ANBOB™

最近操作系统的问题有点多，上周有套Oracle数据库RAC部分节点的日志在频繁输出“ignoring: The maximum number of pending replies per connection has been reached” 信息，找我协助分析一下。

现象

Feb 16 15:32:39 anbob1 su: (to grid) root on none
Feb 16 15:32:39 anbob1 systemd-logind: Failed to start user slice user-1002.slice, ignoring: The maximum number of pending replies per connection has been reached (org.freedesktop.DBus.Error.LimitsExceeded)
Feb 16 15:32:39 anbob1 systemd-logind: Failed to start session scope session-c20950510.scope: The maximum number of pending replies per connection has been reached
Feb 16 15:32:40 anbob1 su: (to grid) root on none
Feb 16 15:32:40 anbob1 systemd-logind: Failed to start user slice user-1002.slice, ignoring: The maximum number of pending replies per connection has been reached (org.freedesktop.DBus.Error.LimitsExceeded)
Feb 16 15:32:40 anbob1 systemd-logind: Failed to start session scope session-c20950511.scope: The maximum number of pending replies per connection has been reached
Feb 16 15:32:40 anbob1 systemd-logind: Failed to start session scope session-1664728.scope: The maximum number of pending replies per connection has been reached
Feb 16 15:32:59 anbob1 su: (to grid) root on none
Feb 16 15:32:59 anbob1 systemd-logind: Failed to start user slice user-1002.slice, ignoring: The maximum number of pending replies per connection has been reached (org.freedesktop.DBus.Error.LimitsExceeded)
Feb 16 15:32:59 anbob1 systemd-logind: Failed to start session scope session-c20950512.scope: The maximum number of pending replies per connection has been reached
Feb 16 15:33:05 anbob1 su: (to grid) root on none
Feb 16 15:33:05 anbob1 systemd-logind: Failed to start user slice user-1002.slice, ignoring: The maximum number of pending replies per connection has been reached (org.freedesktop.DBus.Error.LimitsExceeded)
Feb 16 15:33:05 anbob1 systemd-logind: Failed to start session scope session-c20950513.scope: The maximum number of pending replies per connection has been reached

诊断

$ man dbus-daemon 
 "max_connections_per_user"   : max number of completed connections from the same user
 "max_pending_service_starts" : max number of service launches in   progress at the same time
 "max_names_per_connection"   : max number of names a single  connection can own
 "max_match_rules_per_connection": max number of match rules for a single  connection
 "max_replies_per_connection" : max number of pending method replies per connection  (number of calls-in-progress)
  
[root@db1 system.d]# cat /usr/share/dbus-1/system.conf|grep repli
 <!-- <limit name="max_replies_per_connection">128</limit> -->

$ grep connect /etc/dbus-1/*

$ systemd-cgls
-- or --

[root@db1 system.d]# systemctl status systemd-logind.service
● systemd-logind.service - Login Service
Loaded: loaded (/usr/lib/systemd/system/systemd-logind.service; static; vendor preset: disabled)
Active: active (running) since Fri 2022-12-16 19:48:12 CST; 2 months 0 days ago
Docs: man:systemd-logind.service(8)
man:logind.conf(5)
https://www.freedesktop.org/wiki/Software/systemd/logind
https://www.freedesktop.org/wiki/Software/systemd/multiseat
Main PID: 1281 (systemd-logind)
Status: "Processing requests..."
Tasks: 1 (limit: 203184)
Memory: 2.1M
CGroup: /system.slice/systemd-logind.service <<<<<<< system
└─1281 /usr/lib/systemd/systemd-logind

The default in the file system.conf is:
<!-- <limit name="max_replies_per_connection">128</limit> -->


-- 查看是否有僵尸连接
# loginctl  list-sessions
# loginctl show-session  xx
# busctl --list
$ busctl --list|awk '/login|NAME/'
NAME                                              PID PROCESS         USER             CONNECTION    UNIT                      SESSION    DESCRIPTION
:1.109868                                      718981 systemd-logind  root             :1.109868     systemd-logind.service    -          -
org.freedesktop.login1                         718981 systemd-logind  root             :1.109868     systemd-logind.service    -          -

# systemctl | grep 'of user'| grep 'abandoned'
# systemctl status user-xxx.slice

默认的d-bus限制可能并不适合每个用户。或者，可能存在某些进程行为不正确并耗尽了d-bus资源

建议方案
1， 如果是有僵尸 “abandoned”连接,清理文件

--check run disk usage
df 

ls -l /run/systemd/sessions/*
ls -l /run/systemd/system/*

rm -rf /run/systemd/sessions/sessionscope
rm -rf /run/systemd/system/sessionscope
systemctl daemon-reexec  
-- or 
kill 1

2， 如果确实是默认的128 限制不够，可以增加该限制，但见有OS bug, 在system和session都调到10万后才不提示
创建/etc/dbus-1/system-local.conf 文件（for user session limits it would be session-local.conf）
增加以下

<!DOCTYPE busconfig PUBLIC "-//freedesktop//DTD D-Bus Bus Configuration 1.0//EN"
"http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
<busconfig>
<limit name="max_replies_per_connection">1200</limit>
</busconfig>

note: 不需要重启，通常d-bus daemon监控这些文件的改变。

3，重启d-bus和login服务
因为d-bus或login服务的一些bug异常，可尝试重启两个服务。

# systemctl stop systemd-logind
# systemctl stop dbus.service
# systemctl start dbus.service
# systemctl start systemd-logind

打赏

微信扫一扫，打赏作者吧～