首页 » ORACLE 9i-23ai » Troubleshooting Oracle ASM diskgroup dismount with ORA-15335 ORA-15066 ORA-15196 when delete instance use DBCA
Troubleshooting Oracle ASM diskgroup dismount with ORA-15335 ORA-15066 ORA-15196 when delete instance use DBCA
前几日我们遇到了一个问题，环境为Oracle 11.2.0.4 2节点RAC的情况下，同事在使用DBCA删除一个已经损坏的数据库实例时，意外地导致了当前唯一存活的数据库实例崩溃。进一步的检查发现，ASM磁盘组不可用，而ASM警报日志显示了ASM磁盘文件头损坏、ASM元数据损坏以及ORA-15196: 无效的ASM块头的错误。为什么删除数据库实例会导致ASM磁盘组不可用，并且发现ASM元数据损坏呢？
ASM alert LOG
WARNING: cache read a corrupt block: group=3(DATA) dsk=3 blk=3 disk=3 (DATA_0003) incarn=3916228183 au=0 blk=3 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_84719.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [3] [0 != 1] NOTE: a corrupted block from group DATA was dumped to /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_84719.trc WARNING: cache read (retry) a corrupt block: group=3(DATA) dsk=3 blk=3 disk=3 (DATA_0003) incarn=3916228183 au=0 blk=1 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_84719.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [3] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [3] [0 != 1] ERROR: cache failed to read group=3(DATA) dsk=0 blk=1 from disk(s): 0(DATA_0000) ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [3] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [3] [0 != 1] NOTE: cache initiating offline of disk 3 group DATA NOTE: process _user84719_+asm2 (84719) initiating offline of disk 3.3916228183 (DATA_0000) with mask 0x7e in group 3 NOTE: initiating PST update: grp = 3, dsk = 0/0xe96ce657, mask = 0x6a, op = clear Thu Apr 11 18:10:30 2024 GMON updating disk modes for group 3 at 17 for pid 34, osid 84719 ERROR: Disk 3 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 3) Thu Apr 11 18:10:30 2024 NOTE: cache dismounting (not clean) group 3/0xAD3C16E3 (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 110697, image: oracle@xryhxyw-db02 (B000) WARNING: Offline for disk DATA_0003 in mode 0x7f failed. Thu Apr 11 18:10:30 2024 NOTE: halting all I/Os to diskgroup 3 (DATA) Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_84719.trc (incident=217233): ORA-15335: ASM metadata corruption detected in disk group 'DATA' ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "DATA_0003" in group "DATA" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [3] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [3] [0 != 1] Incident details in: /u01/app/grid/diag/asm/+asm/+ASM2/incident/incdir_217233/+ASM2_ora_84719_i217233.trc
Note:
这里显示DATA磁盘组DISK 3的盘头的metadata 元数据损坏,au 0 block 3通常是AT(Allocation Table)， 因为是外部冗余无法只offline磁盘而是offline diskgroup.
验证磁盘
#kfod disks=all status=true -------------------------------------------------------------------------------- Disk Size Path User Group ================================================================================ 1: 1048576 Mb /dev/mapper/arch01 grid asmadmin 2: 1048576 Mb /dev/mapper/data01 grid asmadmin 3: 1048576 Mb /dev/mapper/data02 grid asmadmin 4: 1048576 Mb /dev/mapper/data03 grid asmadmin 5: 1048576 Mb /dev/mapper/data04 grid asmadmin 6: 1048576 Mb /dev/mapper/data05 grid asmadmin 7: 1048576 Mb /dev/mapper/data06 grid asmadmin 8: 1048576 Mb /dev/mapper/data07 grid asmadmin 9: 20480 Mb /dev/mapper/ocr01 grid asmadmin 10: 20480 Mb /dev/mapper/ocr02 grid asmadmin 11: 20480 Mb /dev/mapper/ocr03 grid asmadmin -------------------------------------------------------------------------------- ORACLE_SID ORACLE_HOME ================================================================================ grid@db002:/home/grid #kfed read /dev/mapper/data02 kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 0 ; 0x001: 0x00 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 0 ; 0x008: file=0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 7FBC09B75400 00000000 00000000 00000000 00000000 [.............. grid@db002:/home/grid #kfed read /dev/mapper/data03 kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 0 ; 0x001: 0x00 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 0 ; 0x008: file=0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 grid@db002:/home/grid #kfed read /dev/mapper/data04 kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 0 ; 0x001: 0x00 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 0 ; 0x008: file=0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 grid@db002:/home/grid #kfed read /dev/mapper/data06 kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 2147483649 ; 0x008: disk=1 kfbh.check: 402677388 ; 0x00c: 0x18005e8c kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8 kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000 kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000 kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000
Note:
可见并不是所有硬盘头都损坏。
验证更多数据
grid@db002:/home/grid #dd if=/dev/mapper/data03 bs=1024 count=1024 skip=4 |hexdump 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 1024+0 records in 1024+0 records out 1048576 bytes (1.0 MB) copied, 0.00940994 s, 111 MB/s 00ff000 0000 0000 0000 0000 0000 0000 0000 0000 00ff010 0000 0000 0000 0000 0000 0000 0000 0000 * 0100000
前1M都是0，也可以用dd和hexdump读更多，默认hexdump会分组相同用*显示。下面简单说明hexdump
hexdump演示
root@19c1:/root #dd if=/dev/zero of=/root/test.dd count=5 bs=1M 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.00279103 s, 1.9 GB/s root@19c1:/root #hexdump /root/test.dd 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 0500000 root@19c1:/root #hexdump /root/test.dd -n 64 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 0000040 root@19c1:/root #hexdump -v /root/test.dd -n 64 0000000 0000 0000 0000 0000 0000 0000 0000 0000 0000010 0000 0000 0000 0000 0000 0000 0000 0000 0000020 0000 0000 0000 0000 0000 0000 0000 0000 0000030 0000 0000 0000 0000 0000 0000 0000 0000 0000040
恢复disk header
grid@db002:/home/grid #kfed repair /dev/mapper/data01 grid@db002:/home/grid #kfed repair /dev/mapper/data02 grid@db002:/home/grid #kfed repair /dev/mapper/data03 grid@db002:/home/grid #kfed repair /dev/mapper/data04
验证修复
grid@db002:/home/grid #kfed read /dev/mapper/data01
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 2147483648 ; 0x008: disk=0
kfbh.check: 1497308808 ; 0x00c: 0x593f1e88
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 0 ; 0x024: 0x0000
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATA_0000 ; 0x028: length=9
kfdhdb.grpname: DATA ; 0x048: length=4
kfdhdb.fgname: DATA_0000 ; 0x068: length=9
kfdhdb.capname: ; 0x088: length=0
grid@db002:/home/grid #dd if=/dev/mapper/data03 bs=$(expr 1024 \* 1024) count=1|hexdump
0000000 8201 0101 0000 0000 0002 8000 7bb6 c8a5
0000010 0000 0000 0000 0000 0000 0000 0000 0000
0000020 524f 4c43 4944 4b53 0000 0000 0000 0000
0000030 0000 0000 0000 0000 0000 0000 0000 0000
0000040 0000 0b20 0002 0301 4144 4154 305f 3030
0000050 0032 0000 0000 0000 0000 0000 0000 0000
0000060 0000 0000 0000 0000 4144 4154 0000 0000
0000070 0000 0000 0000 0000 0000 0000 0000 0000
0000080 0000 0000 0000 0000 4144 4154 305f 3030
0000090 0032 0000 0000 0000 0000 0000 0000 0000
00000a0 0000 0000 0000 0000 0000 0000 0000 0000
*
00000c0 0000 0000 0000 0000 e716 01f8 bc00 0c3c
00000d0 1170 01fa 7000 1c38 0200 1000 0000 0010
00000e0 bc80 0001 0000 0010 000b 0000 0001 0000
00000f0 0002 0000 0000 0000 0000 0000 0000 0000
MOUNT data diskgroup
SQL> ALTER DISKGROUP DATA MOUNT;
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 1048576 0 0 0 0 0 N ARCH/
MOUNTED EXTERN N 512 4096 1048576 0 0 0 0 0 N BAK/
MOUNTED EXTERN N 512 4096 1048576 4194304 3004518 0 3004518 0 N DATA/
MOUNTED NORMAL N 512 4096 1048576 0 0 0 0 0 Y OCR/
ASMCMD>
Note:
其它DISKGROUP如arch,bak 虽然MOUNTED, 但total_mb为0，这是不正常的也有风险, 也是因为ASM元数据损坏，如果有ASM 文件类操作可能会触发磁盘组随时因metadata损坏而dismount。
为什么DBCA删DB实例会触发呢？
NS-00505: Operation timed out
nt secondary err code: 110
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.10.99.165)(PORT=49824))
opidcl aborting process unknown ospid (103505) as a result of ORA-28
Thu Apr 11 15:32:03 2024
ALTER DATABASE DISABLE THREAD 1
Completed: ALTER DATABASE DISABLE THREAD 1
ALTER SYSTEM ARCHIVE LOG
ALTER DATABASE DROP LOGFILE GROUP 1
Thu Apr 11 15:32:03 2024
SUCCESS: diskgroup ARCH was dismounted
WARNING: Cannot delete Oracle managed file +DATA/anbob/onlinelog/group_1.257.1019860099
Errors in file /u01/app/oracle/diag/rdbms/anbob/anbob2/trace/anbob2_ora_55172.trc:
ORA-01265: Unable to delete LOG +DATA/anbob/onlinelog/group_1.257.1019860099
ORA-15130: diskgroup "DATA" is being dismounted
ORA-15335: ASM metadata corruption detected in disk group 'DATA'
ORA-15130: diskgroup "DATA" is being dismounted
ORA-15066: offlining disk "DATA_0003" in group "DATA" may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483651] [3] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483651] [3] [0 != 1]
Thu Apr 11 15:32:04 2024
Errors in file /u01/app/oracle/diag/rdbms/anbob/anbob2/trace/anbob2_lgwr_39665.trc:
ORA-00345: redo log write error block 149399 count 2
ORA-00312: online log 6 thread 2: '+DATA/anbob/onlinelog/group_6.268.1019860427'
ORA-15079: ASM file is closed
ORA-15079: ASM file is closed
Errors in file /u01/app/oracle/diag/rdbms/anbob/anbob2/trace/anbob2_ora_55172.trc:
ORA-00202: control file: '+DATA/anbob/controlfile/current.256.1019860099'
ORA-15079: ASM file is closed
SUCCESS: diskgroup DATA was dismounted
SUCCESS: diskgroup DATA was dismounted
Thu Apr 11 15:32:04 2024
KCF: read, write or open error, block=0x3307b4 online=1
Note:
dbca 删除db实例时会递归删除DATA组中的thread 对应的online redo logfile，属于元数据类操作，所以失败。 上面提示ARCH也diskmount了，数据库归档也无法完成，实际也确实在data mount后启库正常，但几分钟后因归档问题再次crash，db instance实例运行在switch logfile会再次down，不过arch dest 可以先改为filesytem临时规避问题。
归档磁盘组分析
Thu Apr 11 18:10:30 2024 WARNING: cache read a corrupt block: group=1(ARCH) dsk=0 blk=1 disk=0 (ARCH_0000) incarn=3916228183 au=0 blk=1 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_84719.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [1] [0 != 1] NOTE: a corrupted block from group ARCH was dumped to /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_84719.trc WARNING: cache read (retry) a corrupt block: group=1(ARCH) dsk=0 blk=1 disk=0 (ARCH_0000) incarn=3916228183 au=0 blk=1 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_84719.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [1] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [1] [0 != 1] ERROR: cache failed to read group=1(ARCH) dsk=0 blk=1 from disk(s): 0(ARCH_0000) ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [1] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [1] [0 != 1] NOTE: cache initiating offline of disk 0 group ARCH NOTE: process _user84719_+asm2 (84719) initiating offline of disk 0.3916228183 (ARCH_0000) with mask 0x7e in group 1 NOTE: initiating PST update: grp = 1, dsk = 0/0xe96ce657, mask = 0x6a, op = clear Thu Apr 11 18:10:30 2024 GMON updating disk modes for group 1 at 35 for pid 34, osid 84719 ERROR: Disk 0 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 1) Thu Apr 11 18:10:30 2024 NOTE: cache dismounting (not clean) group 1/0xAD3C16E3 (ARCH) NOTE: messaging CKPT to quiesce pins Unix process pid: 110697, image: oracle@db002 (B000) WARNING: Offline for disk ARCH_0000 in mode 0x7f failed. Thu Apr 11 18:10:30 2024 NOTE: halting all I/Os to diskgroup 1 (ARCH) Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_84719.trc (incident=217233): ORA-15335: ASM metadata corruption detected in disk group 'ARCH' ORA-15130: diskgroup "ARCH" is being dismounted grid@db002:/u01/app/grid/diag/asm/+asm/+ASM2/trace #dd if=/dev/mapper/arch01 bs=4096 count=1000|hexdump 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 00001c0 0001 feee ffff 0001 0000 ffff 7fff 0000 00001d0 0000 0000 0000 0000 0000 0000 0000 0000 * 00001f0 0000 0000 0000 0000 0000 0000 0000 aa55 0000200 4645 2049 4150 5452 0000 0001 005c 0000 grid@db002:/u01/app/grid/diag/asm/+asm/+ASM2/trace #kfed read /dev/mapper/arch01 kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 0 ; 0x001: 0x00 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 0 ; 0x008: file=0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 0 ; 0x010: 0x00000000 #kfed repair /dev/mapper/arch01 #dd if=/dev/mapper/arch01 bs=4096 count=100|hexdump 0000000 8201 0101 0000 0000 0000 8000 6d8d 5037 0000010 0000 0000 0000 0000 0000 0000 0000 0000 0000020 524f 4c43 4944 4b53 0000 0000 0000 0000 0000030 0000 0000 0000 0000 0000 0000 0000 0000 0000040 0000 0b20 0000 0301 5241 4843 305f 3030 0000050 0030 0000 0000 0000 0000 0000 0000 0000 0000060 0000 0000 0000 0000 5241 4843 0000 0000 0000070 0000 0000 0000 0000 0000 0000 0000 0000 0000080 0000 0000 0000 0000 5241 4843 305f 3030 0000090 0030 0000 0000 0000 0000 0000 0000 0000 00000a0 0000 0000 0000 0000 0000 0000 0000 0000 * 00000c0 0000 0000 0000 0000 e716 01f8 cc00 09a3 00000d0 104c 01fa b800 8d92 0200 1000 0000 0010 SQL> alter diskgroup arch mount; Diskgroup altered. grid@db002:/u01/app/grid/diag/asm/+asm/+ASM2/trace #asmcmd lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED EXTERN N 512 4096 1048576 1048576 599078 0 599078 0 N ARCH/ MOUNTED EXTERN N 512 4096 1048576 0 0 0 0 0 N BAK/ MOUNTED EXTERN N 512 4096 1048576 4194304 3004518 0 3004518 0 N DATA/ MOUNTED NORMAL N 512 4096 1048576 0 0 0 0 0 Y OCR/
Summary:
在这个环境中，我们无法确定是什么操作导致了部分ASM磁盘的前1M盘头被清理。然而，当进行ASM磁盘组文件操作时，可能会导致ASM发现元数据损坏，从而引发数据库实例崩溃。如果没有进行文件操作，这个环境的问题可能不会出现。操作时间段应该是在ASM重启时间、最后一次归档切换和最后一次文件操作之间。在另一个客户那里也曾遇到了ASM元数据损坏的问题，其中au 0 block 1受损，当时是在ASM实例重启或挂载diskgroup时才会被发现。因此，后续的安全巡检应该加入对ASM磁盘头的检查，以避免类似的问题再次发生。
微信扫一扫，打赏作者吧～
对不起，这篇文章暂时关闭评论。