Warning ! RAC Listener offline& VIP failover when dynamically adding disks using udev after executing
增加新存储到现有ASM,使用udev配置绑定的存储,当增加了udev rule 配置文件中新增的lun, 执行了start_udev后,执行该命令RAC当前节点资源 的VIP fail 到另一个实例,并且listener offline, 这是俩月前的一个案例,简单的记录一下。 数据库版本11.2.0.3 2node RAC on RHEL 6.4 (vmware私有云)。
Udev 配置的ASM , RHEL6 ,修改了udev rule文件,操作系统执行start_udev后,会导致Listener offline?
[grid@anbob02 ~]$ crsctl stat res ora.LISTENER.lsnr -p ... STOP_DEPENDENCIES=hard(intermediate:type:ora.cluster_vip_net1.type) STOP_TIMEOUT=0 .... [grid@anbob02 ~]$ crsctl stat res ora.anbob02.vip -p ... CHECK_INTERVAL=1 <--------################### STOP_DEPENDENCIES=hard(ora.net1.network) STOP_TIMEOUT=0 ... [grid@anbob02 ~]$ crsctl stat res ora.net1.network -p ... CHECK_INTERVAL=1 STOP_DEPENDENCIES= STOP_TIMEOUT=0 USR_ORA_IF=eth0 ... [grid@anbob02 network-scripts]$ vi ifcfg-eth0 DEVICE="eth0" NM_CONTROLLED="yes" ONBOOT=yes HWADDR=00:50:56:9E:38:F3 TYPE=Ethernet BOOTPROTO=none IPADDR=136.142.30.140 PREFIX=25 GATEWAY=136.142.30.254 DEFROUTE=yes IPV4_FAILURE_FATAL=yes IPV6INIT=no NAME="System eth0" UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 [oracle@anbob02 rules.d]$cd /etc/udev/rules.d [oracle@anbob02 rules.d]$ ls *net* 70-persistent-net.rules [oracle@anbob02 rules.d]$ cat 70-persistent-net.rules # This file was automatically generated by the /lib/udev/write_net_rules # program, run by the persistent-net-generator.rules rules file. # PCI device 0x15ad:0x07b0 (vmxnet3) SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9e:38:f3", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0" # PCI device 0x15ad:0x07b0 (vmxnet3) SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9e:38:f5", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
TIP:
listener 依赖 vip, vip 依赖 network eth0 ,而且 network \vip 的检查是每秒(CHECK_INTERVAL=1), 所以网络闪断也很可能发现, 该eth0 网络设备是 使用的udev 绑定, 所以start_udev 后导致一系统变动。 记的以前在RHEL 5上 不存在问题的,后来在MOS上找到一篇note
Network interface going down when dynamically adding disks to storage using udev in RHEL 6 (文档 ID 1569028.1)
CAUSE
During start_udev, udev has deleted the public network interface and this caused the listener to crash, and clusterware moved all services, scan listeners and the VIP on node 1 to node 2.
Solution
As per RHEL you are advised to set HOTPLUG=”no” for the network configuration scripts.
Add HOTPLUG=”no” to the ifcfg-eth0 (public), ifcfg-eth1 (private) and ifcfg-eth2 (backup) network config files in /etc/sysconfig/network-scripts directory.
其它方法也可以试试
Since RHEL6, kudzu is no longer available since the kernel and udev take care of hardware changes.(6.3 ? not sure)
Remove udev rules:
# rm /etc/udev/rules.d/70-persistent-net.rules
# reboot
Verify network devices ( use graphical tool if needed for changes )
Or
Once your UDEV rule is set-up you can trigger them without a reboot
# udevadm control –reload-rules
# udevadm trigger
— 2015-05-21 update 感谢traveldba提供下面的方法
–加载磁盘map
udevadm trigger subsystem-match=block
–加载权限
udevadm trigger subsystem-nomatch=net
对不起,这篇文章暂时关闭评论。