首页 » ORACLE 9i-23ai » Instance Crash in 11.2.0.3 RAC and ORA-600 [kcbo_switch_cq_1]
Instance Crash in 11.2.0.3 RAC and ORA-600 [kcbo_switch_cq_1]
最近有套新割接的环境DB 11203 2NODE RAC与原环境唯一的区别是新环境未修改任何DB参数，结果数据库两次crash，同时alert日志中显示ORA-600 [kcbo_switch_cq_1] ， 该错误与UNDO 相关，还不确认是否是_undo_autotune=FALSE在以前的环境中规避了该问题，可以明确的应该是一bug。
# db alert log
Sat Nov 05 01:31:04 2016 Thread 1 advanced to log sequence 3738 (LGWR switch) Current log# 2 seq# 3738 mem# 0: +DATADG/anbob/onlinelog/group_2.258.923576167 Sat Nov 05 01:31:31 2016 Thread 1 advanced to log sequence 3739 (LGWR switch) Current log# 5 seq# 3739 mem# 0: +DATADG/anbob/onlinelog/group_5.268.923581181 Sat Nov 05 01:31:44 2016 Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc (incident=242865): ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_242865/anbob1_ora_29818922_i242865.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_242865/anbob1_ora_29818922_i242865.trc" Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached Writing to the above trace file is disabled for now on... Sat Nov 05 01:31:52 2016 Dumping diagnostic data in directory=[cdmp_20161105013152], requested by (instance=1, osid=29818922), summary=[incident=242865]. Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc (incident=242866): ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_242866/anbob1_ora_29818922_i242866.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Sat Nov 05 01:31:53 2016 Sweep [inc][242865]: completed Sweep [inc2][242865]: completed Sat Nov 05 01:31:53 2016 Sweep [inc][242866]: completed Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_242866/anbob1_ora_29818922_i242866.trc" Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached Writing to the above trace file is disabled for now on... Some DDE async actions failed or were cancelled Dumping diagnostic data in directory=[cdmp_20161105013154], requested by (instance=1, osid=29818922), summary=[incident=242866]. Sat Nov 05 01:31:54 2016 Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc: ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] Sat Nov 05 01:31:54 2016 Sweep [inc2][242866]: completed Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc" Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached Writing to the above trace file is disabled for now on... Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc: ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] Dumping diagnostic data in directory=[cdmp_20161105013202], requested by (instance=1, osid=29818922), summary=[abnormal process termination]. Sat Nov 05 01:32:03 2016 Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc (incident=252001): ORA-00603: ORACLE server session terminated by fatal error ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_252001/anbob1_ora_29818922_i252001.trc Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_252001/anbob1_ora_29818922_i252001.trc: ORA-00603: ORACLE server session terminated by fatal error ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] Sat Nov 05 01:32:03 2016 Dumping diagnostic data in directory=[cdmp_20161105013203], requested by (instance=1, osid=29818922), summary=[abnormal process termination]. Sat Nov 05 01:32:06 2016 ERROR: Unable to normalize symbol name for the following short stack (at offset 457): dbgexProcessError()+180<-dbgeExecuteForError()+72<-dbgePostErrorKGE()+2048<-dbkePostKGE_kgsf()+68<-kgeadse()+380<-kgerinv_internal()+48<-kgerinv()+48<-kserin()+76<-kcbo_switch_cq1()+748<-kcbso1()+5252<-kcbpsod()+288<-kcbsod()+92
# trace file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc
----------------------------------------------------------
LEVEL PAYLOAD
----- ------------------------------------------------------------------------------------------------------------------------------------------------
Trace file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/app/oracle/product/11.2.0.3/dbhome_1
System name: AIX
Node name: qdanbob1
Release: 1
Version: 6
Machine: 00F739554C00
Instance name: anbob1
Redo thread mounted by this instance: 1
Oracle process number: 358
Unix process pid: 29818922, image: oracle@qdanbob1 (TNS V1-V3)
*** 2016-11-05 01:31:44.460
*** SESSION ID:(5540.25401) 2016-11-05 01:31:44.460
*** CLIENT ID:() 2016-11-05 01:31:44.460
*** SERVICE NAME:(SYS$USERS) 2016-11-05 01:31:44.460
*** MODULE NAME:(SQL*Plus) 2016-11-05 01:31:44.460
*** ACTION NAME:() 2016-11-05 01:31:44.460
1> ***** Incident 242865 created, dump file: *****
/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_242865/anbob1_ora_29818922_i242865.trc
1< ***** incident_file ***** 1> ***** Error Stack *****
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
1< ***** Error Stack ***** *** 2016-11-05 01:31:52.830 1> ***** Incident 242866 created, dump file: *****
/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_242866/anbob1_ora_29818922_i242866.trc
1< ***** incident_file ***** 1> ***** Error Stack *****
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
1< ***** Error Stack ***** *** 2016-11-05 01:31:54.449 dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=3, mask=0x0) 1> ***** Error Stack Dump *****
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], []
1< ***** Error Stack ***** 1> ***** Current SQL Statement for this session (sql_id=28gq1caugn5dc) *****
insert into sm_tab
select REGION,TELNUM,TEMPLATE_NO,PART,CONTENT,SFROM,FORMNUM,RELATEDFORMNUM,STATUS,MSGID,INTIME,SENDDATE,REALDATE,SHOULDTIMES,LEFTTIMES,TRYTIME
S,NOTIFYTYPE,RETCODE,RETMSG,YWFORMNUM from sm_tab@lnkd2
where REGION=:Region
and ( INTIME>=:StartTime and INTIME ***** PL/SQL Stack *****
----- PL/SQL Call Stack -----
object line object
handle number name
700000bb4975e10 190 package body TBCS.PKG_REPLX_DAILY
700000bc41cd358 1 anonymous block
2< ***** PL/SQL Stack *****
1< ***** current_sql_statement ***** 1> ***** Call Stack Trace *****
calling call entry
location type point
-------------------- -------- --------------------
skdstdst()+40 bl 107ca55c0
ksedst1()+112 call skdstdst()
ksedst()+40 call ksedst1()
dbkedDefDump()+1516 call ksedst()
ksedmp()+72 call dbkedDefDump()
ksupop()+4012 call ksedmp()
opiodr()+4048 call ksupop()
rpidrus()+216 call opiodr()
skgmstack()+120 call rpidrus()
rpidru()+132 call skgmstack()
rpiswu2()+480 call rpidru()
rpidrv()+1060 call rpiswu2()
psddr0()+260 call rpidrv()
psdnal()+472 call psddr0()
pevm_EXIM()+240 call psdnal()
pfrinstr_EXIM()+60 call pevm_EXIM()
pfrrun_no_tool()+96 call pfrinstr_EXIM()
pfrrun()+1032 call pfrrun_no_tool()
plsql_run()+616 call pfrrun()
peicnt()+320 call plsql_run()
kkxexe()+508 call peicnt()
opiexe()+17224 call kkxexe()
kpoal8()+4616 call opiexe()
opiodr()+720 call kpoal8()
ttcpip()+1028 call opiodr()
opitsk()+1508 call ttcpip()
opiino()+940 call opitsk()
opiodr()+720 call opiino()
opidrv()+1132 call opiodr()
sou2o()+136 call opidrv()
opimai_real()+608 call sou2o()
ssthrdmain()+268 call opimai_real()
main()+204 call ssthrdmain()
__start()+112 call main()
1< ***** call_stack_dump *****
# trace file anbob1_ora_29818922_i242865.trc
Dump continued from file: /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] ========= Dump for incident 242865 (ORA 600 [kcbo_switch_cq_1]) ======== ----- Beginning of Customized Incident Dump(s) ----- BH (0x70000084fafdb20) file#: 176 rdba: 0x2c1626fc (176/1451772) class: 1 ba: 0x7000008485c0000 set: 199 pool: 3 bsz: 16384 bsi: 0 sflg: 1 pwc: 0,0 dbwrid: 6 obj: 35154 objn: 26924 tsn: 19 afn: 176 hint: f hash: [0x700000c24aa51e0,0x700000847b0eb88] lru: [0x700000847b029e0,0x70000083fb0e8d0] obj-flags: object_write_list ckptq: [0x700000c254d0728,0x700000847b04db0] fileq: [0x700000c254d1d28,0x700000c254d1d28] objq: [0x70000084fafba68,0x700000847b02cf8] objaq: [0x700000847b090e8,0x70000084faff0d0] use: [0x700000bfc0dd180,0x700000bfc0dd180] wait: [NULL] st: XCURRENT md: EXCL fpin: 'kdiwh22: kdifind' tch: 1 atm: 1236014467,1236494533 flags: mod_started block_written_once redo_since_read affinity_lock change state: NOT_ACTIVE_YET change count: 1 LRBA: [0xe9b.bbb6d.0] LSCN: [0x0.0] HSCN: [0xffff.ffffffff] HSUB: [4] Dump of buffer cache at level 10 for tsn=19 rdba=739649276 BH (0x70000084fafdb20) file#: 176 rdba: 0x2c1626fc (176/1451772) class: 1 ba: 0x7000008485c0000 set: 199 pool: 3 bsz: 16384 bsi: 0 sflg: 1 pwc: 0,0 dbwrid: 6 obj: 35154 objn: 26924 tsn: 19 afn: 176 hint: f hash: [0x700000c24aa51e0,0x700000847b0eb88] lru: [0x700000847b029e0,0x70000083fb0e8d0] obj-flags: object_write_list ckptq: [0x700000c254d0728,0x700000847b04db0] fileq: [0x700000c254d1d28,0x700000c254d1d28] objq: [0x70000084fafba68,0x700000847b02cf8] objaq: [0x700000847b090e8,0x70000084faff0d0] use: [0x700000bfc0dd180,0x700000bfc0dd180] wait: [NULL] st: XCURRENT md: EXCL fpin: 'kdiwh22: kdifind' tch: 1 atm: 1236014467,1236499550 flags: mod_started block_written_once redo_since_read affinity_lock change state: NOT_ACTIVE_YET change count: 1 LRBA: [0xe9b.bbb6d.0] LSCN: [0x0.0] HSCN: [0xffff.ffffffff] HSUB: [4] Using State Objects ... ... ... Main Stack: kcbo_switch_cq1
# trace file /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_252001/anbob1_ora_29818922_i252001.trc
*** 2016-11-05 01:32:03.618 Dump continued from file: /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_29818922.trc ORA-00603: ORACLE server session terminated by fatal error ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], ========= Dump for incident 252001 (ORA 603) ======== *** 2016-11-05 01:32:03.636 dbkedDefDump(): Starting incident default dumps (flags=0x0, level=3, mask=0x0) ----- Error Stack Dump ----- ORA-00603: ORACLE server session terminated by fatal error ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kcbo_switch_cq_1], [], [], [], [], [], [], [], [], [], [], [] ----- SQL Statement (None) ----- Current SQL information unavailable - no SGA. ----- Call Stack Trace ----- calling call entry location type point -------------------- -------- -------------------- skdstdst()+40 bl 107ca55c0 ksedst1()+112 call skdstdst() ksedst()+40 call ksedst1() dbkedDefDump()+1516 call ksedst() ksedmp()+72 call dbkedDefDump() ksfdmp()+100 call ksedmp() dbgexPhaseII()+1940 call ksfdmp() dbgexProcessError() call dbgexPhaseII() +932 dbgeExecuteForError call dbgexProcessError() ()+72 dbgePostErrorKGE()+ call dbgeExecuteForError 2048 () dbkePostKGE_kgsf()+ call dbgePostErrorKGE() 68 kgeade()+364 call dbkePostKGE_kgsf() ### kgeade kgefec()+204 call kgeade() kgefic()+104 call kgefec() ksefic()+200 call kgefic() ksupop()+4176 call ksefic() opiodr()+4048 call ksupop() rpidrus()+216 call opiodr() skgmstack()+120 call rpidrus() rpidru()+132 call skgmstack() rpiswu2()+480 call rpidru() rpidrv()+1060 call rpiswu2() psddr0()+260 call rpidrv() psdnal()+472 call psddr0() pevm_EXIM()+240 call psdnal() pfrinstr_EXIM()+60 call pevm_EXIM() pfrrun_no_tool()+96 call pfrinstr_EXIM() pfrrun()+1032 call pfrrun_no_tool() plsql_run()+616 call pfrrun() peicnt()+320 call plsql_run() kkxexe()+508 call peicnt() opiexe()+17224 call kkxexe() kpoal8()+4616 call opiexe() opiodr()+720 call kpoal8() ttcpip()+1028 call opiodr() opitsk()+1508 call ttcpip() opiino()+940 call opitsk() opiodr()+720 call opiino() opidrv()+1132 call opiodr() sou2o()+136 call opidrv() opimai_real()+608 call sou2o() ssthrdmain()+268 call opimai_real() main()+204 call ssthrdmain() __start()+112 call main()
在MOS中可以匹配上Bug 15883525
解决方法：
1， 安装对应版本中的ONE-OFF PATCH 15883525
2, 升级到11.2.0.4, 12.1.0.2 and later, and 12.2
3, Disable undo autotune(Maybe) set _undo_autotune=FALSE –I’m not sure
微信扫一扫，打赏作者吧～
对不起，这篇文章暂时关闭评论。