ANBOB™

前段时间分析的一个问题，节点2驱逐后无法再加入集群，日志显示是网络通信问题，查看开始时驱逐的原因也是VD CRS-1615:No I/O has completed 和 Network communication missing， 同时DISK HB和Network HB同时失败，并且存储和private network是双链路，用的也不是同一交换机。什么会导致同时出问题呢？简单记录一下

# node2 GI alert log

2019-09-17 02:10:06.619: 
[cssd(20050)]CRS-1612:Network communication with node node1 (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.870 seconds
2019-09-17 02:10:06.940: 
[cssd(20050)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /dev/asm-diskj will be considered not functional in 7990 milliseconds
2019-09-17 02:10:06.940: 
[cssd(20050)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /dev/asm-diskk will be considered not functional in 8110 milliseconds
2019-09-17 02:10:08.445: 
[cssd(20050)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file /dev/asm-diskj will be considered not functional in 6480 milliseconds
2019-09-17 02:10:08.445: 
[cssd(20050)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file /dev/asm-diskk will be considered not functional in 6600 milliseconds
2019-09-17 02:10:14.623: 
[cssd(20050)]CRS-1662:Member kill requested by node node1 for member number 1, group DBORCL
2019-09-17 02:11:41.644: 
[cssd(20050)]CRS-1611:Network communication with node node1 (1) missing for 75% of timeout interval.  Removal of this node from cluster in 6.620 seconds
2019-09-17 02:11:45.645: 
[cssd(20050)]CRS-1610:Network communication with node node1 (1) missing for 90% of timeout interval.  Removal of this node from cluster in 2.620 seconds
2019-09-17 02:11:48.268: 
[cssd(20050)]CRS-1609:This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00008:) in /u01/app/11.2.0/grid/log/node2/cssd/ocssd.log.
2019-09-17 02:11:48.268: 
[cssd(20050)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0/grid/log/node2/cssd/ocssd.log
2019-09-17 02:11:48.315: 

2019-09-17 02:13:57.761: 
[cssd(1248047)]CRS-1713:CSSD daemon is started in clustered mode
2019-09-17 02:14:13.489: 
[cssd(1248047)]CRS-1707:Lease acquisition for node node2 number 2 completed
2019-09-17 02:14:14.761: 
[cssd(1248047)]CRS-1605:CSSD voting file is online: /dev/asm-diski; details in /u01/app/11.2.0/grid/log/node2/cssd/ocssd.log.
2019-09-17 02:14:14.770: 
[cssd(1248047)]CRS-1605:CSSD voting file is online: /dev/asm-diskj; details in /u01/app/11.2.0/grid/log/node2/cssd/ocssd.log.
2019-09-17 02:14:14.779: 
[cssd(1248047)]CRS-1605:CSSD voting file is online: /dev/asm-diskk; details in /u01/app/11.2.0/grid/log/node2/cssd/ocssd.log.
2019-09-17 02:14:33.309: 
[cssd(1248047)]CRS-1612:Network communication with node node1 (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.040 seconds
2019-09-17 02:14:40.310: 
[cssd(1248047)]CRS-1611:Network communication with node node1 (1) missing for 75% of timeout interval.  Removal of this node from cluster in 7.040 seconds
2019-09-17 02:14:45.312: 
[cssd(1248047)]CRS-1610:Network communication with node node1 (1) missing for 90% of timeout interval.  Removal of this node from cluster in 2.040 seconds
2019-09-17 02:14:47.355: 
[cssd(1248047)]CRS-1609:This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00008:) in /u01/app/11.2.0/grid/log/node2/cssd/ocssd.log.
2019-09-17 02:14:47.355: 
[cssd(1248047)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0/grid/log/node2/cssd/ocssd.log
2019-09-17 02:14:47.387: 
[cssd(1248047)]CRS-1603:CSSD on node node2 shutdown by user.
2019-09-17 02:14:47.589: 
[cssd(1248047)]CRS-1660:The CSS daemon shutdown has completed

诊断方法
# on node1
ping node2-priv-ip
traceroute node2-priv-ip

# on node2
ping node1-priv-ip
traceroute node1-priv-ip

— 测试网络也是没有问题的。

cat /etc/*release*|head -n 3
— RHEL 6.6

又看到了这个多病的操作系统版本，想起了以前的老问题 https://www.anbob.com/archives/2851.html

ping 11.11.11.11 -s 8192
— 果然没反映，即使没有使用大帧也应该拆包发送呀

netstat -s |grep reass
-sleep 5
netstat -s |grep reass

— 值在增加，原因就可能在这里了

echo 16777216 > /proc/sys/net/ipv4/ipfrag_low_thresh
echo 15728640 > /proc/sys/net/ipv4/ipfrag_high_thresh
echo 60 > /proc/sys/net/ipv4/ipfrag_time

一切恢复正常

如果你解决不了， 可以联系www.anbob.com 首页的联系方式。

打赏

微信扫一扫，打赏作者吧～