首页 » ORACLE 9i-23ai » Troubleshooting oracle CRS start fail after OS reboot ASM device(udev) unable to open
Troubleshooting oracle CRS start fail after OS reboot ASM device(udev) unable to open
最近一个客户的Oracle 11g RAC 一个节点操作系统（Linux7）重启后，CRS无法启动，另一个节点(NODE2)未重启暂时正常。查看GI日志是Vote disk无法读取，sd设备dd正常，kfed read asmdisk失败，使用udev映射，说明问题出在udev, 简单记录该问题。
验证设备
[grid@anbob1 ~]$ dd if=/dev/asmdiskc bs=1024 count=3
dd: failed to open '/dev/asmdiskc': No such device or address
[grid@anbob1 ~]$ dd if=/dev/sdc bs=1024 count=3
dd: failed to open '/dev/sdc': Permission denied
[grid@anbob1 ~]$ exit
logout
[root@anbob1 dev]# dd if=/dev/sdc bs=1024 count=3
▒▒▒_▒▒ORCLDISK
OCR_VOTING_0001OCR_VOTINGOCR_VOTING_0001▒▒▒0▒:'▒▒8e▒▒(
▒▒▒▒: ;3+0 records in
3+0 records out
3072 bytes (3.1 kB) copied, 0.000535419 s, 5.7 MB/s
[root@anbob1 dev]# /usr/lib/udev/scsi_id -g -u -d /dev/sdc
3614eb08100e1c60a1b7a502f0000000c
[root@anbob1 dev]# grep 3614eb08100e1c60a1b7a502f0000000c /etc/udev/rules.d/*oracle*
/etc/udev/rules.d/97-oracle-asmdevices.rules:KERNEL=="sd*",ENV{DEVTYPE}=="disk",SUBSYSTEM=="block",PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode",RESULT=="3614eb08100e1c60a1b7a502f0000000c", RUN+="/bin/sh -c 'mknod /dev/asmdiskc b $major $minor; chown grid:asmadmin /dev/asmdiskc; chmod 0660 /dev/asmdiskc'"
[grid@anbob1 ~]$ kfed read /dev/asmdiskc
KFED-00303: unable to open file '/dev/asmdiskc'
[grid@anbob1 ~]$ dd if=/dev/asmdiskc bs=1024 count=3
dd: failed to open '/dev/asmdiskc': No such device or address
Note:
注意OS层的sd设备是可以read, 但ASM设备无法找不到设备. ASM使用的mknod根据主备设备号创建。
另一节正常节点
[root@anbob2 ~]# su - grid
Last login: Mon Dec 25 09:16:10 CST 2023
[grid@anbob2 ~]$ dd if=/dev/sdaw bs=1024 count=2
dd: failed to open '/dev/sdaw': Permission denied
[grid@anbob2 ~]$ exit
logout
[root@anbob2 ~]# dd if=/dev/sdaw bs=1024 count=2
▒▒▒_▒▒ORCLDISK
OCR_VOTING_0000OCR_VOTINGOCR_VOTING_0000▒▒▒0▒:'▒▒8e▒▒(
▒▒▒▒: ;2+0 records in
2+0 records out
2048 bytes (2.0 kB) copied, 0.000578212 s, 3.5 MB/s
NODE2 验证UDEV
[root@anbob2 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdaw
3614eb08100e1c60a1b7a502f0000000b
[root@anbob2 ~]# grep 3614eb08100e1c60a1b7a502f0000000b /etc/udev/rules.d/97-oracle-asmdevices.rules
KERNEL=="sd*",ENV{DEVTYPE}=="disk",SUBSYSTEM=="block",PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode",RESULT=="3614eb08100e1c60a1b7a502f0000000b", RUN+="/bin/sh -c 'mknod /dev/asmdiskb b $major $minor; chown grid:asmadmin /dev/asmdiskb; chmod 0660 /dev/asmdiskb'"
[grid@anbob2 ~]$ kfed read /dev/asmdiskb
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 2147483648 ; 0x008: disk=0
对比ASM与BLOCK device
[root@anbob1 dev]# ls -l /dev/asm* brw-rw---- 1 grid asmadmin 68, 240 Dec 25 07:08 /dev/asmdiskc brw-rw---- 1 grid asmadmin 69, 16 Dec 25 07:08 /dev/asmdiskd brw-rw---- 1 grid asmadmin 69, 48 Dec 25 07:08 /dev/asmdiske brw-rw---- 1 grid asmadmin 71, 80 Dec 25 07:08 /dev/asmdiskf brw-rw---- 1 grid asmadmin 69, 96 Dec 25 07:08 /dev/asmdiskg brw-rw---- 1 grid asmadmin 69, 128 Dec 25 07:08 /dev/asmdiskh brw-rw---- 1 grid asmadmin 69, 160 Dec 25 07:08 /dev/asmdiski brw-rw---- 1 grid asmadmin 69, 176 Dec 25 07:08 /dev/asmdiskj brw-rw---- 1 grid asmadmin 69, 208 Dec 25 07:08 /dev/asmdiskk brw-rw---- 1 grid asmadmin 66, 32 Dec 25 07:08 /dev/asmdiskl brw-rw---- 1 grid asmadmin 70, 32 Dec 25 07:08 /dev/asmdiskm brw-rw---- 1 grid asmadmin 70, 64 Dec 25 07:08 /dev/asmdiskn brw-rw---- 1 grid asmadmin 70, 96 Dec 25 07:08 /dev/asmdisko brw-rw---- 1 grid asmadmin 70, 128 Dec 25 07:08 /dev/asmdiskp brw-rw---- 1 grid asmadmin 70, 160 Dec 25 07:08 /dev/asmdiskq [root@anbob1 dev]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 1.3T 0 disk |-sda1 8:1 0 199.5M 0 part /boot/efi |-sda2 8:2 0 1G 0 part /boot `-sda3 8:3 0 1.3T 0 part |-centos-root 253:0 0 1.3T 0 lvm / `-centos-swap 253:1 0 4G 0 lvm [SWAP] sdb 8:16 0 10G 0 disk sdc 8:32 0 10G 0 disk sdd 8:48 0 10G 0 disk sde 8:64 0 1T 0 disk sdf 8:80 0 1T 0 disk sdg 8:96 0 2.9T 0 disk sdh 8:112 0 2T 0 disk sdi 8:128 0 2T 0 disk sdj 8:144 0 2T 0 disk sdk 8:160 0 2T 0 disk sdl 8:176 0 2T 0 disk sdm 8:192 0 2T 0 disk sdn 8:208 0 2T 0 disk sdo 8:224 0 2T 0 disk sdp 8:240 0 2T 0 disk sdq 65:0 0 2T 0 disk
Note:
注意重启后的ASM设备主设备号基本全是8，而重启后设备号发生改变导致无法读取。 使用设备号的UDEV挂载方式，是不建议使用的，就是担心设备号的改变，同样也不建议基于device name, 对于UDEV的挂载的建议可参考《How to create ASM devices with UDEV？》。
重建 ASM UDEV设备
[root@anbob1 dev]# ls -l /dev/asm* brw-rw---- 1 grid asmadmin 8, 16 Dec 25 10:16 /dev/asmdiskb brw-rw---- 1 grid asmadmin 68, 240 Dec 25 07:08 /dev/asmdiskc brw-rw---- 1 grid asmadmin 69, 16 Dec 25 07:08 /dev/asmdiskd brw-rw---- 1 grid asmadmin 69, 48 Dec 25 07:08 /dev/asmdiske ... [root@anbob1 dev]# rm /dev/asmdisk* rm: remove block special file '/dev/asmdiskb'? n rm: remove block special file '/dev/asmdiskc'? y rm: remove block special file '/dev/asmdiskd'? y rm: remove block special file '/dev/asmdiske'? y ... [root@anbob1 dev]# /sbin/udevadm control --reload-rules [root@anbob1 dev]# /sbin/udevadm trigger --type=devices --action=change [root@anbob1 dev]# systemctl stop systemd-udevd Warning: Stopping systemd-udevd.service, but it can still be activated by: systemd-udevd-kernel.socket systemd-udevd-control.socket [root@anbob1 dev]# systemctl start systemd-udevd [root@anbob1 dev]# ls -l /dev/asm* brw-rw---- 1 grid asmadmin 8, 16 Dec 25 10:16 /dev/asmdiskb brw-rw---- 1 grid asmadmin 8, 32 Dec 25 10:26 /dev/asmdiskc brw-rw---- 1 grid asmadmin 8, 48 Dec 25 10:26 /dev/asmdiskd brw-rw---- 1 grid asmadmin 8, 64 Dec 25 10:26 /dev/asmdiske ... [root@anbob1 dev]# su - grid Last login: Mon Dec 25 10:19:48 CST 2023 on pts/1 [grid@anbob1 ~]$ kfed read /dev/asmdiskq |head kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 2147483657 ; 0x008: disk=9 kfbh.check: 3281947134 ; 0x00c: 0xc39e89fe kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000
Note:
重建 ASM设备后恢复正常。
分析OS为什么重启？
# gi alert log 2023-12-25 06:57:48.553: [cssd(156463)]CRS-1612:Network communication with node anbob1 (1) missing for 50% of timeout interval. Removal of this node from cluster in 14.880 seconds 2023-12-25 06:57:56.555: [cssd(156463)]CRS-1611:Network communication with node anbob1 (1) missing for 75% of timeout interval. Removal of this node from cluster in 6.870 seconds 2023-12-25 06:58:00.556: [cssd(156463)]CRS-1610:Network communication with node anbob1 (1) missing for 90% of timeout interval. Removal of this node from cluster in 2.870 seconds 2023-12-25 06:58:03.429: [cssd(156463)]CRS-1609:This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00008:) in /oracle/app/11.2.0/grid/log/anbob2/cssd/ocssd.log. # OS message Dec 25 06:57:41 anbob1 kernel: mce: Uncorrected hardware memory error in user-access at 779dfd1200 Dec 25 06:57:42 anbob1 kernel: Memory failure: 0x779dfd1: already hardware poisoned Dec 25 06:57:42 anbob1 kernel: reserve_ram_pages_type failed [mem 0x779dfd1000-0x779dfd1fff], track 0x2, req 0x2 Dec 25 06:57:43 anbob1 kernel: Could not invalidate pfn=0x779dfd1 from 1:1 map Dec 25 06:57:43 anbob1 kernel: mce: Uncorrected hardware memory error in user-access at 779dfd1200 Dec 25 06:57:44 anbob1 kernel: Memory failure: 0x779dfd1: already hardware poisoned Dec 25 06:57:44 anbob1 kernel: reserve_ram_pages_type failed [mem 0x779dfd1000-0x779dfd1fff], track 0x2, req 0x2 Dec 25 06:57:44 anbob1 kernel: Could not invalidate pfn=0x779dfd1 from 1:1 map Dec 25 06:57:45 anbob1 kernel: mce: Uncorrected hardware memory error in user-access at 779dfd1200 Dec 25 06:57:45 anbob1 kernel: Memory failure: 0x779dfd1: already hardware poisoned Dec 25 06:57:46 anbob1 kernel: reserve_ram_pages_type failed [mem 0x779dfd1000-0x779dfd1fff], track 0x2, req 0x2 Dec 25 06:57:47 anbob1 kernel: Could not invalidate pfn=0x779dfd1 from 1:1 map Dec 25 06:57:47 anbob1 kernel: mce: Uncorrected hardware memory error in user-access at 779dfd1200 Dec 25 06:57:48 anbob1 kernel: Memory failure: 0x779dfd1: already hardware poisoned Dec 25 06:57:48 anbob1 kernel: reserve_ram_pages_type failed [mem 0x779dfd1000-0x779dfd1fff], track 0x2, req 0x2 Dec 25 06:57:48 anbob1 kernel: Could not invalidate pfn=0x779dfd1 from 1:1 map Dec 25 06:57:48 anbob1 kernel: mce: Uncorrected hardware memory error in user-access at 779dfd1200 Dec 25 06:57:48 anbob1 kernel: Memory failure: 0x779dfd1: already hardware poisoned Dec 25 06:57:49 anbob1 kernel: reserve_ram_pages_type failed [mem 0x779dfd1000-0x779dfd1fff], track 0x2, req 0x2
Note:
注意好像是因为OS层发现内存相关的访问错误，导致OS panic， 建议分析硬件分析。
https://access.redhat.com/solutions/67599 以下可能的根本原因并不详尽，但可能涵盖大多数情况：
内存 DIMM 故障。
内存控制器故障（通常是板载的）。
主板上的内存线有故障。
BIOS 有故障。
过热系统。
RAM 潜在结点故障（用户静电放电）。
电源问题或短路。
https://access.redhat.com/solutions/6412581
System restarts when memory uncorrectable error injection is performed in memory MCA mode In the Operation System: Perform cmd：
# mcelog --daemon # modprobe einj param_extension=1 # mount -t debugfs none /sys/kernel/debug # git clone https://github.com/andikleen/mce-inject.git # cd mce-inject; make # ./mce-inject test/uncorrected
SCAN IP访问问题
发现使用scan ip访问DB服务器访问正常，但远程访问提示没有正常的服务名.注意检查 local_listener和remote_listener不为空，并配置默认如下
SQL> show parameter listener NAME TYPE VALUE ------------------------------------ --------------------------------- ------------------------------ listener_networks string local_listener string (ADDRESS=(PROTOCOL=TCP)(HOST=192.xxVIPxx)(PORT=1521)) remote_listener string anbob-scan:1521
另外发现有可能存在主机重启后crs 自动启动时使用crs中配置的spfile， 覆盖本地$ORACLE_HOME/dbs/initXXX.ora 中的内容，如果之前是手动配置的spfile，与CRS不是相同的spfile，可能会导致使用错误的SPFILE启动DB 实例。
— over —
微信扫一扫，打赏作者吧～
对不起，这篇文章暂时关闭评论。