首页 » Cloud, ORACLE 9i-23ai » Troubleshooting ASM allocation is failed due to ORA-4030 though OS has enough free memories.
Troubleshooting ASM allocation is failed due to ORA-4030 though OS has enough free memories.
某客户一套Oracle 11.2.0.4 4-node RAC ON RHEL 7.6 环境 ，ASM High冗余Diskgroup 有600TB存储(没错是个超级大库)， 其中有1个1TB的ACFS DG. 一日突然节点1个节点ASM和DB实例crash, 重启后正常, 分析当时的日志是ASM 实例的VDBG后台进程出现的ora-4030错误，目前需要分析一下原因。 简单记录。
DB alert log
Thu Aug 04 19:45:05 2022
WARNING: ASM communication error: op 18 state 0x50 (3113)
ERROR: slave communication error with ASM
NOTE: Deferred communication with ASM instance
Errors in file /u01/app/oracle/diag/rdbms/newanbob/anbob4/trace/anbob4_ora_47083.trc:
ORA-03113: end-of-file on communication channel
...
NOTE: Deferred communication with ASM instance
Thu Aug 04 19:45:05 2022
NOTE: ASMB terminating
Thu Aug 04 19:45:05 2022
WARNING: ASM communication error: op 17 state 0x50 (3113)NOTE: deferred map free for map id 7572857
Errors in file /u01/app/oracle/diag/rdbms/newanbob/anbob4/trace/anbob4_ora_65011.trc:
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 2380 Serial number: 31
ERROR: slave communication error with ASM
...
Session ID: 1404 Serial number: 149
ASMB (ospid: 56359): terminating the instance due to error 15064
NOTE: Deferred communication with ASM instance
Errors in file /u01/app/oracle/diag/rdbms/newanbob/anbob4/trace/anbob4_ora_65017.trc:
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-03113: end-of-file on communication channel
Session ID: 2014 Serial number: 2539
NOTE: deferred map free for map id 7572856
Errors in file /u01/app/oracle/diag/rdbms/newanbob/anbob4/trace/anbob4_ora_47096.trc (incident=198037):
ORA-00600: internal error code, arguments: [2618], [8], [], [], [], [], [], [], [], [], [], []
NOTE: Deferred communication with ASM instance
Errors in file /u01/app/oracle/diag/rdbms/newanbob/anbob4/trace/anbob4_ora_65011.trc:
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-15077: could not locate ASM instance serving a required diskgroup
NOTE: deferred map free for map id 7572859
Note:
db instance crash是因为ASM intance请求失败.
ASM alert log
Thu Aug 04 19:44:52 2022 Errors in file /u01/app/grid/diag/asm/+asm/+ASM4/trace/+ASM4_vdbg_9668.trc (incident=2638776): ORA-04030: out of process memory when trying to allocate 118584 bytes (pga heap,kfvsuIOStatOp) Incident details in: /u01/app/grid/diag/asm/+asm/+ASM4/incident/incdir_2638776/+ASM4_vdbg_9668_i2638776.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Thu Aug 04 19:45:03 2022 Dumping diagnostic data in directory=[cdmp_20220804194503], requested by (instance=4, osid=9668 (VDBG)), summary=[incident=2638776]. Thu Aug 04 19:45:05 2022 Errors in file /u01/app/grid/diag/asm/+asm/+ASM4/trace/+ASM4_vdbg_9668.trc: ORA-04030: out of process memory when trying to allocate 118584 bytes (pga heap,kfvsuIOStatOp) System state dump requested by (instance=4, osid=9668 (VDBG)), summary=[abnormal instance termination]. VDBG (ospid: 9668): terminating the instance due to error 4030 System State dumped to trace file /u01/app/grid/diag/asm/+asm/+ASM4/trace/+ASM4_diag_3689_20220804194505.trc Thu Aug 04 19:45:06 2022 ORA-1092 : opitsk aborting process Thu Aug 04 19:45:06 2022 ORA-1092 : opitsk aborting process Thu Aug 04 19:45:06 2022 License high water mark = 45
vdbg trace file
Dump file /u01/app/grid/diag/asm/+asm/+ASM4/trace/+asm4_vdbg_9668.trc mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4 mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4 .. mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4 mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4 DDE: Flood control is not active ========= Dump for critical error (no incident) (DIA 48002) ======== Starting a Diag Context default dump (level=3) DIA-48002: thread terminated by fatal error DIA-48003: out of process memory when trying to allocate 56 bytes (diag pga,dbgr entry) ----- Call Stack Trace ----- calling call entry argument values in hex location type point (? means dubious value) -------------------- -------- -------------------- ---------------------------- mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4 mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4 mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4 .. .. mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4 kgdsdstsg()+53 call kgdsdst() 000000000 ? 000000000 ? 7FFC14E0A000 ? 7FFC14E0A0D8 ? 7FFC14E0EB80 ? 000000003 ? 00000000077CEDDD call kgdsdstsg() 7F067C909878 ? 7FFC14E0EE80 ? 000000003 ? 7FFC14E0A0D8 ? 7FFC14E0EB80 ? 000000003 ? 000000000770FF80 call 00000000077CEDDD 7F067C909878 ? 000000003 ? 000000003 ? 7FFC14E0A0D8 ? 7FFC14E0EB80 ? 000000003 ? dbgexProcessError() call 000000000770FF80 7F067C8FE728 ? 7F067C938E58 ? +2680 7FFC14E12FF8 ? 7FFC14E0A0D8 ? 7FFC14E0EB80 ? 000000003 ? 00000000076CF596 call dbgexProcessError() 7F067C8FE728 ? 7F067C938E58 ? 000000001 ? 000000000 ? 7FFC14E0EB80 ? 000000003 ? dbgePostErrorKGE()+ call 00000000076CF596 7F067C8FE728 ? 7F067C938E58 ? 2136 000000001 ? 000000001 ? 000000000 ? 000000003 ? 00000000077CEF95 call dbgePostErrorKGE() 7F067C909878 ? 7F067C92CA98 ? 00000BB82 ? 000000001 ? 000000000 ? 000000003 ? 0000000002A28E05 call 00000000077CEF95 7F067C909878 ? 7F067C92CA98 ? 00000BB82 ? 000000001 ? 000000000 ? 000000003 ? kgefec()+176 call 0000000002A28E05 7F067C909878 ? 7F067C909A38 ? 7F067C92CA98 ? 00000BB82 ? 000000000 ? 000000003 ? 0000000002A19192 call kgefec() 7F067C909878 ? 7F067C92CA98 ? 000000000 ? 000000001 ? 000000000 ? 000000003 ? kghalf()+1139 call 0000000002A19192 7F067C909878 ? 7F067C8FE7F8 ? 000000038 ? 000000001 ? 000000000 ? 000000003 ? dbgric_init_cache() call kghalf() 7F067C909878 ? 7F067C8FE7F8 ? +527 000000020 ? 000000001 ? 000000000 ? 0098F08C8 ? dbgriap_init_adr_pg call dbgric_init_cache() 7F067C8FE728 ? 7FFC14E144F0 ? a()+198 000000020 ? 000000001 ? 000000000 ? 0098F08C8 ? dbgc_init_all()+848 call dbgriap_init_adr_pg 7F067C8FE728 ? 000000003 ? a() 000000020 ? 000000001 ? 000000000 ? 0098F08C8 ? 0000000002D5FFCE call dbgc_init_all() 000000003 ? 7F067C8FE728 ? 000000020 ? 000000001 ? 000000000 ? 0098F08C8 ? 0000000002D5FB37 call 0000000002D5FFCE 7F067C8FE728 ? 7F067C8FE728 ? 000000020 ? 000000001 ? 000000000 ? 0098F08C8 ? dbgc_rls_diagctx()+ call 0000000002D5FB37 000000002 ? 00AC3BA20 ? 58 000000020 ? 000000001 ? 000000000 ? 0098F08C8 ? 0000000001F9A23F call dbgc_rls_diagctx() 000000002 ? 00AC3BA20 ? 000000020 ? 000000001 ? 000000000 ? 0098F08C8 ? dbkc_sga_unmap()+75 call 0000000001F9A23F 000000002 ? 00AC3BA20 ? 000000020 ? 000000001 ? 000000000 ? 0098F08C8 ? ksmsdes()+111 call dbkc_sga_unmap() 000000000 ? 00AC3BA20 ? 000000020 ? 000000001 ? 000000000 ? 0098F08C8 ? ksuitm()+2165 call ksmsdes() 7FFC14E14AF0 ? 00AC3BA20 ? 000000020 ? 000000001 ? 000000000 ? 0098F08C8 ? ksbrdp()+3507 call ksuitm() 000000000 ? 000000FBE ? 154484DD0 ? 000000001 ? 000000000 ? 0098F08C8 ? opirip()+623 call ksbrdp() 000000000 ? 000000FBE ? 154484DD0 ? 000000001 ? 000000000 ? 0098F08C8 ? opidrv()+603 call opirip() 000000032 ? 000000004 ? 7FFC14E15FA8 ? 000000001 ? 000000000 ? 0098F08C8 ? sou2o()+103 call opidrv() 000000032 ? 000000004 ? 7FFC14E15FA8 ? 000000001 ? 000000000 ? 0098F08C8 ? mmap(offset=210198528, len=4096) failed with errno=12 for the file asm_vdbg_+ASM4 opimai_real()+250 call sou2o() 7FFC14E15F80 ? 000000032 ? 000000004 ? 7FFC14E15FA8 ? 000000000 ? 0098F08C8 ? ssthrdmain()+265 call opimai_real() 000000000 ? 7FFC14E16170 ? 000000004 ? 7FFC14E15FA8 ? 000000000 ? 0098F08C8 ? main()+201 call ssthrdmain() 000000003 ? 7FFC14E16170 ? 000000001 ? 000000000 ? 000000000 ? 0098F08C8 ? __libc_start_main() call main() 000000003 ? 7FFC14E16310 ? +245 000000001 ? 000000000 ? 000000000 ? 0098F08C8 ? 00000000008489F9 call __libc_start_main() 000848AC0 ? 000000001 ? 7FFC14E16308 ? 000000000 ? 000000000 ? 0098F08C8 ? ... ----- Process Map Dump ----- ******************* Dumping process map **************** 00400000-0a8ac000 r-xp 00000000 fd:03 67451950 /u01/app/11.2.0/grid/bin/oracle 0aaac000-0aaad000 r--p 0a4ac000 fd:03 67451950 /u01/app/11.2.0/grid/bin/oracle 0aaad000-0ac15000 rw-p 0a4ad000 fd:03 67451950 /u01/app/11.2.0/grid/bin/oracle 0ac15000-0ac5e000 rw-p 00000000 00:00 0 0bd44000-0bdea000 rw-p 00000000 00:00 0 [heap] 60000000-60001000 r--s 00000000 00:13 450040 /dev/shm/ora_+ASM4_131075_0 60001000-61000000 rw-s 00001000 00:13 450040 /dev/shm/ora_+ASM4_131075_0 61000000-62000000 rw-s 00000000 00:13 450041 /dev/shm/ora_+ASM4_131075_1 ... 7f04bd20c000-7f04bd21c000 rw-p 00000000 00:05 1030 /dev/zero 7f04bd21c000-7f04bd23c000 rw-p 00000000 00:05 1030 /dev/zero 7f04bd23c000-7f04bd25c000 rw-p 00000000 00:05 1030 /dev/zero 7f04bd25c000-7f04bd27c000 rw-p 00000000 00:05 1030 /dev/zero 7f04bd27c000-7f04bd29c000 rw-p 00000000 00:05 1030 /dev/zero 7f04bd29c000-7f04bd2ac000 rw-p 00000000 00:05 1030 /dev/zero 7f04bd2ac000-7f04bd2cc000 rw-p 00000000 00:05 1030 /dev/zero 7f04bd2cc000-7f04bd2ec000 rw-p 00000000 00:05 1030 /dev/zero ... 7ffc14f8e000-7ffc14f90000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] ####### ---- total number of 65532 lines ******************* End of process map dump ************ ----- Process Resource Limits ----- ***************** Dumping Resource Limits(s/h) ***************** core file size UNLIMITED/UNLIMITED data seg size UNLIMITED/UNLIMITED file size UNLIMITED/UNLIMITED pending signals 2010 KB/2010 KB max locked memory UNLIMITED/UNLIMITED max memory size UNLIMITED/UNLIMITED open files 64 KB/64 KB POSIX message queues 800 KB/800 KB stack size 32 MB/UNLIMITED cpu time UNLIMITED/UNLIMITED max user processes 16 KB/16 KB virtual memory UNLIMITED/UNLIMITED file locks UNLIMITED/UNLIMITED ***************** End of Resource Limits Dump ******************
Note:
call stack中有mmap(offset=210198528, len=4096) failed with errno=12报错， 当时的data和stack 在当前用户已经是unlimited, MAX memory size和virtual memory 也是unlimited . 因为mmap失败导致asm intance的VDBG进程出现的ora-4030错误 。
也可以手动收集maps
$ps -ef|grep vdbg $ pmap -p <vdbg进程号>
mmap failed with errno=12
[root@oel7db1 ~]# perror 12 OS error code 12: Cannot allocate memory
Note:
that 12 is exactly ENOMEM error code. Investigation for mmap() syscall shows that ENOMEM stands for “No memory is available, or the process’s maximum number of mappings would have been exceeded.”
VDBG Process
Volume Driver Process Forwards Oracle ASM requests to perform various volume-related tasks VDBG handles requests to lock or unlock an extent for rebalancing, volume resize, disk offline, add or drop a disk, force and dismount disk group to the Dynamic Volume Manager driver.
Note:
判断与ASM ACFS相关。
OSW vmstat
zzz ***Thu Aug 4 19:41:05 CST 2022 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 25 2 436116 118542424 81556 42647252 0 0 2605 406 0 0 10 3 85 1 0 19 4 436116 118563664 81556 42648388 0 0 684500 2338 478437 189366 17 7 74 2 0 21 4 436116 118491496 81556 42648276 0 0 570272 2741 459318 203020 13 6 79 2 0 zzz ***Thu Aug 4 19:41:36 CST 2022 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 24 7 436116 118174656 81556 42654560 0 0 2605 406 0 0 10 3 85 1 0 17 7 436116 118107056 81556 42654244 0 0 960114 4220 488187 295850 16 7 73 3 0 23 5 436116 118005568 81556 42654516 0 0 1095528 9780 476740 248006 15 7 75 4 0 zzz ***Thu Aug 4 19:42:07 CST 2022 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 36 1 436116 118660784 81556 42661968 0 0 2605 406 0 0 10 3 85 1 0 29 2 436116 118633664 81556 42662728 0 0 621297 8233 565649 258656 17 9 73 1 0 29 3 436116 118634256 81556 42663212 0 0 707392 2908 590466 315953 15 9 75 1 0 zzz ***Thu Aug 4 19:42:38 CST 2022 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 25 4 436116 118511632 81556 42665980 0 0 2605 406 0 0 10 3 85 1 0 21 3 436116 118320824 81556 42666804 0 0 758968 11114 420537 277405 16 5 75 4 0 19 3 436116 118262856 81556 42667052 0 0 774818 13257 439119 296825 13 7 77 3 0 zzz ***Thu Aug 4 19:43:09 CST 2022 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 23 0 436116 118024848 81556 42677192 0 0 2605 406 0 0 10 3 85 1 0 23 1 436116 117908224 81556 42678480 0 0 828708 7031 528760 157467 15 8 76 1 0 21 0 436116 117911840 81556 42678396 0 0 829886 3924 544309 201965 16 7 76 1 0 zzz ***Thu Aug 4 19:43:40 CST 2022 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 24 6 436116 118217664 81556 42685468 0 0 2605 406 0 0 10 3 85 1 0 16 6 436116 118186256 81556 42685844 0 0 1247490 1470 472355 210836 14 7 76 3 0 20 4 436116 118188568 81556 42686296 0 0 1252882 9776 467208 207419 13 6 78 3 0 zzz ***Thu Aug 4 19:44:11 CST 2022 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 23 3 436116 117897696 81556 42694212 0 0 2605 406 0 0 10 3 85 1 0 26 0 436116 117777872 81552 42695332 0 0 665390 11447 539412 167680 17 10 72 1 0 25 1 436116 117754168 81500 42695716 0 0 715547 4452 549530 185636 17 8 73 2 0 zzz ***Thu Aug 4 19:44:43 CST 2022 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 27 4 436116 117627000 81556 42701572 0 0 2605 406 0 0 10 3 85 1 0 21 7 436116 118095872 81556 42702520 0 0 1182373 2582 493360 193122 19 8 69 4 0 23 5 436116 118133888 81556 42702364 0 0 1095942 2632 473514 196193 16 6 74 4 0 zzz ***Thu Aug 4 19:45:14 CST 2022 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 6 0 436116 125675744 44588 45139048 0 0 2605 406 0 0 10 3 85 1 0 8 0 436116 125555808 44588 45296892 0 0 2873 772219 344671 316581 2 6 93 0 0 4 0 436116 125464688 44588 45400196 0 0 3830 1466162 162413 196481 1 3 96 0 0
Note:
可见swap和free memory 在操作系统层还有空间。
OSW PS
ps RSS kb为单位 USER PID PPID PRI %CPU %MEM VSZ RSS WCHAN S STARTED TIME COMMAND zzz ***Thu Aug 4 19:00:11 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7308376 SYSC_s S Jul 09 1-22:52:09 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:00:42 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7308496 SYSC_s S Jul 09 1-22:52:09 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:01:13 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7308496 SYSC_s S Jul 09 1-22:52:09 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:01:44 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7308648 SYSC_s S Jul 09 1-22:52:09 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:02:15 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7308648 SYSC_s S Jul 09 1-22:52:10 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:02:46 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7308648 SYSC_s S Jul 09 1-22:52:10 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:03:17 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7308648 SYSC_s S Jul 09 1-22:52:10 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:03:48 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7308912 SYSC_s S Jul 09 1-22:52:11 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:04:20 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7308912 SYSC_s S Jul 09 1-22:52:11 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:04:51 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7308912 SYSC_s S Jul 09 1-22:52:11 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:05:22 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7308912 SYSC_s S Jul 09 1-22:52:12 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:05:53 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309032 SYSC_s S Jul 09 1-22:52:12 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:06:24 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309032 SYSC_s S Jul 09 1-22:52:12 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:06:55 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309160 SYSC_s S Jul 09 1-22:52:13 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:07:26 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309280 SYSC_s S Jul 09 1-22:52:13 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:07:57 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309280 SYSC_s S Jul 09 1-22:52:13 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:08:28 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309280 SYSC_s S Jul 09 1-22:52:14 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:08:59 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309420 SYSC_s S Jul 09 1-22:52:14 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:09:30 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309420 SYSC_s S Jul 09 1-22:52:15 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:10:01 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309540 SYSC_s S Jul 09 1-22:52:15 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:10:32 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309540 SYSC_s S Jul 09 1-22:52:15 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:11:03 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309540 SYSC_s S Jul 09 1-22:52:16 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:11:34 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309540 SYSC_s S Jul 09 1-22:52:16 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:12:05 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309724 SYSC_s S Jul 09 1-22:52:17 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:12:36 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309724 SYSC_s S Jul 09 1-22:52:17 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:13:08 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309848 - S Jul 09 1-22:52:17 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:13:39 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309848 SYSC_s S Jul 09 1-22:52:18 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:14:10 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309992 SYSC_s S Jul 09 1-22:52:18 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:14:41 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7309992 SYSC_s S Jul 09 1-22:52:18 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:15:12 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310112 SYSC_s S Jul 09 1-22:52:19 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:15:43 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310112 SYSC_s S Jul 09 1-22:52:19 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:16:14 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310248 SYSC_s S Jul 09 1-22:52:20 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:16:45 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310248 SYSC_s S Jul 09 1-22:52:20 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:17:16 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310368 SYSC_s S Jul 09 1-22:52:20 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:17:47 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310368 SYSC_s S Jul 09 1-22:52:21 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:18:18 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310368 SYSC_s S Jul 09 1-22:52:21 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:18:49 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310368 SYSC_s S Jul 09 1-22:52:21 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:19:20 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310368 SYSC_s S Jul 09 1-22:52:22 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:19:51 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310556 SYSC_s S Jul 09 1-22:52:22 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:20:22 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310556 SYSC_s S Jul 09 1-22:52:22 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:20:54 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310692 SYSC_s S Jul 09 1-22:52:22 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:21:25 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310824 SYSC_s S Jul 09 1-22:52:23 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:21:56 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310944 SYSC_s S Jul 09 1-22:52:23 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:22:27 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310944 SYSC_s S Jul 09 1-22:52:23 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:22:58 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310944 SYSC_s S Jul 09 1-22:52:23 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:23:29 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7310944 SYSC_s S Jul 09 1-22:52:24 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:24:00 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311128 SYSC_s S Jul 09 1-22:52:24 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:24:31 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311128 SYSC_s S Jul 09 1-22:52:24 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:25:02 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311248 SYSC_s S Jul 09 1-22:52:24 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:25:33 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311248 SYSC_s S Jul 09 1-22:52:25 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:26:04 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311248 SYSC_s S Jul 09 1-22:52:25 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:26:35 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311248 SYSC_s S Jul 09 1-22:52:26 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:27:06 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311404 SYSC_s S Jul 09 1-22:52:26 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:27:37 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311404 SYSC_s S Jul 09 1-22:52:27 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:28:08 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311404 SYSC_s S Jul 09 1-22:52:27 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:28:39 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311600 SYSC_s S Jul 09 1-22:52:28 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:29:10 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311720 SYSC_s S Jul 09 1-22:52:28 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:29:41 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311720 SYSC_s S Jul 09 1-22:52:29 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:30:12 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311720 SYSC_s S Jul 09 1-22:52:29 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:30:43 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311720 SYSC_s S Jul 09 1-22:52:29 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:31:14 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311920 SYSC_s S Jul 09 1-22:52:30 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:31:45 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7311920 SYSC_s S Jul 09 1-22:52:30 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:32:17 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312040 SYSC_s S Jul 09 1-22:52:31 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:32:48 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312040 SYSC_s S Jul 09 1-22:52:31 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:33:19 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312040 SYSC_s S Jul 09 1-22:52:32 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:33:50 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312216 SYSC_s S Jul 09 1-22:52:32 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:34:21 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312216 SYSC_s S Jul 09 1-22:52:32 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:34:52 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312216 SYSC_s S Jul 09 1-22:52:33 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:35:23 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312360 SYSC_s S Jul 09 1-22:52:33 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:35:54 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312360 SYSC_s S Jul 09 1-22:52:34 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:36:25 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312488 SYSC_s S Jul 09 1-22:52:34 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:36:56 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312488 SYSC_s S Jul 09 1-22:52:34 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:37:27 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312488 SYSC_s S Jul 09 1-22:52:35 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:37:58 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312616 SYSC_s S Jul 09 1-22:52:35 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:38:29 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312616 SYSC_s S Jul 09 1-22:52:35 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:39:00 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312736 SYSC_s S Jul 09 1-22:52:36 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:39:32 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312736 SYSC_s S Jul 09 1-22:52:36 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:40:03 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312864 - S Jul 09 1-22:52:36 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:40:34 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312864 SYSC_s S Jul 09 1-22:52:36 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:41:05 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7312864 SYSC_s S Jul 09 1-22:52:37 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:41:37 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7313048 SYSC_s S Jul 09 1-22:52:37 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:42:08 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7313048 SYSC_s S Jul 09 1-22:52:37 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:42:39 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7313048 SYSC_s S Jul 09 1-22:52:37 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:43:10 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7313180 SYSC_s S Jul 09 1-22:52:38 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:43:41 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7313180 SYSC_s S Jul 09 1-22:52:38 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:44:12 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7313320 SYSC_s S Jul 09 1-22:52:38 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:44:43 CST 2022 grid 9668 1 19 0.4 1.3 11730252 7313320 SYSC_s S Jul 09 1-22:52:38 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:45:14 CST 2022 grid 9668 1 19 0.4 1.3 11730372 7339608 hrtime S Jul 09 1-22:52:50 asm_vdbg_+ASM4 zzz ***Thu Aug 4 19:45:45 CST 2022 zzz ***Thu Aug 4 19:46:16 CST 2022
Note:
持续关注进程是否有内存泄露是否过大或持续增长，当前如果正在做asm Operater有操作，我认为vdbg增长也是正常现象。
TFA OS report
# more /proc/sys/vm/max_map_count 65530
Note:
如果有tfactl 收集日志在os report中有OS 内核参数信息。
max_map_count
max_map_count 文件允许限制特定进程可以拥有的 VMA（虚拟内存区域）的数量。虚拟内存区是虚拟地址空间的连续区域。当程序尝试对文件进行内存映射、链接到共享内存段或分配堆空间时，会在进程的生命周期中创建这些区域。调整此值会限制进程可以拥有的这些 VMA 的数量。限制进程可以拥有的 VMA 数量可能会导致有问题的应用程序行为，因为当进程达到其 VMA 限制时系统将返回内存不足错误，但可以释放低内存以供其他内核使用。如果您的系统在 NORMAL 区域内存不足，则降低此值将有助于释放内存供内核使用. mmap,OS error code 12: Cannot allocate memory 这些错误通常是由于从操作系统中耗尽了map entries而出现的。 每个进程默认有64k map entries(max_map_count),默认的realfree allocator pagesize 是64 kB (65536), 所以64K entries可以分配4GB.加大map entries可以分配更多的内存，如 256kB (262144) pages, 限制增加到16GB.
查看realfree pages
select i.ksppinm name , v.ksppstvl cur_val,
v.ksppstdf default_val, v.ksppstvf
from x$ksppi i, x$ksppcv v
where i.indx = v.indx
and i.ksppinm in ('_realfree_heap_pagesize_hint', '_use_realfree_heap')
Diag event
ora-4030 发生时没有生成PGA heapdump信息，可以开启error TRACE
ALTER SYSTEM SET EVENTS '4030 trace name heapdump level 536870917;name errorstack level 3';
解决方案建议
1,增加map entries记录数或增加realfree pagesize
Change the page count at the OS level: $ more /proc/sys/vm/max_map_count $ sysctl -w vm.max_map_count=262144 -- OR -- Adjust the realfree heap pagesize within the database by setting the following parameters in the init/spfile and restart the database. ALTER SYSTEM SET "_use_realfree_heap"=TRUE SCOPE=SPFILE; ALTER SYSTEM SET "_realfree_heap_pagesize_hint"=262144 SCOPE=SPFILE; NOTE: In Oracle 12c, the parameter _realfree_heap_pagesize_hint is changed to _realfree_heap_pagesize. In 12C, _realfree_heap_pagesize = 262144
2, 增加MEMROY_TARGET
当前系统OS free 内存较多，ASM DG较大，建议也增加一些MEMORY_TARGET(SGA+PGA AUTO). ASM 默认使用AMM, 在ASM实例中SGA的shared_pool占用较大， 所以我们的memory_target要给shared_pool预留外建议再增加2-4GB空间。
# FOR SHARED POOL
SELECT SUM(bytes)/(1024*1024*1024) FROM V$DATAFILE; SELECT SUM(bytes)/(1024*1024*1024) FROM V$LOGFILE a, V$LOG b WHERE a.group#=b.group#; SELECT SUM(bytes)/(1024*1024*1024) FROM V$TEMPFILE WHERE status='ONLINE'; o For disk groups using external redundancy, every 100 GB of space needs 1 MB of extra shared pool plus 2 MB. o For disk groups using normal redundancy, every 50 GB of space needs 1 MB of extra shared pool plus 4 MB. o For disk groups using high redundancy, every 33 GB of space needs 1 MB of extra shared pool plus 6 MB.
微信扫一扫，打赏作者吧～
对不起，这篇文章暂时关闭评论。