ANBOB™

oracle 11.2.0.4 RAC 安装在OEL7没想到还能出这么多问题，几年前做过总结《RHEL7（Linux7）安装Oracle 11g R2(11.2.0.4) RAC 问题小结》，这次使用的是某软件的自动化安装部署，遇到了新的问题，root.sh时oc4j无法启动，虽然大部分情况该资源并不影响数据库的使用，甚至应对某些安全扫描的风险端口还需要停掉该服务，但这里是希望找到报错原因，使用crsctl stat res -t查看资源状态ora.oc4j ONLINE OFFLINE STARTING 状态

root.sh 失败

...
FirstNode configuration failed at /u01/app/11.2.0.4/grid/crs/install/crsconfig_lib.pm line 9379.
 /u01/app/11.2.0.4/grid/perl/bin/perl -I/u01/app/11.2.0.4/grid/perl/lib -I/u01/app/11.2.0.4/grid/crs/install /u01/app/11.2.0.4/grid/crs/install/rootcrs.pl execution failed

GI alert log

...
Resource 'ora,oc4j' failed to start automatically.
...

oc4j error log ($GRID_HOME/oc4j/j2ee/home/log)

$ vi oc4j_xxxxxx.err
INFO: Updating JtaAdmin account
2024-06-05 14:01:45.783 ERROR ServerFile[PersistentMap,/opt/oracle/app/12.1.0.2/grid/oc4j/j2ee/home/persistence/jms.state]: persistence file could not be locked.  
Resolve the sharing violation by ending the process currently holding the file lock or by using a different persistence directory for one or both processes.
2024-06-05 14:01:45 *** (SEVERE) ServerFile[PersistentMap,
/opt/oracle/app/12.1.0.2/grid/oc4j/j2ee/home/persistence/jms.state]: persistence file could not be locked.  
Resolve the sharing violation by ending the process currently holding the file lock or by using a different persistence directory for one or both processes.
2024-06-05 14:01:45.786 ERROR Failed to set the internal configuration of the OC4J JMS Server with: XMLJMSServerConfig[file:$GRID_HOME/oc4j/j2ee/home/OC4J_DBWLM_config/jms.xml]
2024-06-05 14:01:45 *** (SEVERE) Failed to set the internal configuration of the OC4J JMS Server with: XMLJMSServerConfig[file:$GRID_HOME/oc4j/j2ee/home/OC4J_DBWLM_config/jms.xml]
2024-06-05 14:01:45.788 ERROR J2EE OJR-00011 Exception starting JMS server: Failed to set the internal configuration of 
the OC4J JMS Server with: XMLJMSServerConfig[file:$GRID_HOME/oc4j/j2ee/home/OC4J_DBWLM_config/jms.xml]
2024-06-05 14:01:45.814 ERROR J2EE RMI-00002 Error starting ORMI server  port 23,792: Address already in use
2024-06-05 14:01:45.815 ERROR java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.socketBind(PlainSocketImpl.java:521)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:414)
at java.net.ServerSocket.bind(ServerSocket.java:326)
at java.net.ServerSocket.(ServerSocket.java:192)
at com.evermind.server.rmi.RMIServer.createServerSocket(RMIServer.java:459)
at com.evermind.server.rmi.RMIServer.createNonSecureSocket(RMIServer.java:373)
at com.evermind.server.rmi.RMIServer.setConfig(RMIServer.java:236)
at com.evermind.server.ApplicationServer.initializeRMI(ApplicationServer.java:2464)
at com.evermind.server.ApplicationServer.setConfig(ApplicationServer.java:1020)
at com.evermind.server.ApplicationServerLauncher.run(ApplicationServerLauncher.java:131)
at java.lang.Thread.run(Thread.java:682)

2024-06-05 14:01:45 Error starting ORMI server  port 23,792: Address already in use 

who  use 23792 port?

anbob:~ # nestat -tanelp|grep 23792
tcp6        0      0 :::23792                :::*                    LISTEN      92780/java        

anbob:~ # ps -ef|grep 92780|grep -v grep
grid     92780     1  0 19:14 ?        00:02:54$GRID_HOME/jdk/bin/java -server -Xms128M -Xmx384M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Dstdstream.filesize=100 -Dstdstream.filenumber=10 -DTRACING.ENABLED=false -Doracle.wlm.dbwlmlogger.logging.level=INFO -jar $GRID_HOME/oc4j/j2ee/home/oc4j.jar -config $GRID_HOME/oc4j/j2ee/home/OC4J_DBWLM_config/server.xml -properties -ports default-web-site:http:8888,rmi:23792 -out $GRID_HOME/oc4j/j2ee/home/log/oc4j.out -err $GRID_HOME/oc4j/j2ee/home/log/oc4j.err

anbob:~ # lsof $GRID_HOME/oc4j/j2ee/home/persistence/jms.state

是因为这个环境重装了几次，原因是由于/tmp下的孤儿11g文件夹没有清理， 自己程序占用了自己的端口:

# ls -ld /tmp/hs*

解决方法

1. delete those old hsperfdata_xxx folders in /tmp
2. kill oc4j java process
3. Deconfig
4. run root.sh again.

How to deconfig

On all cluster nodes except the last, run the following command as the “root” user.

# perl $GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force

On the last cluster node, run the following command as the “root” user.

# perl $GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode

This final command will blank the OCR configuration and voting disk.

12C以后用rootcrs.sh替换为rootcrs.sh

Deconfig fail

# /u01/app/11.2.0.4/grid_home/crs/install/rootcrs.pl -verbose -deconfig -force
Can't locate Env.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /u01/app/11.2.0.4/grid_home/crs/install) at /u01/app/11.2.0.4/grid_home/crs/install/crsconfig_lib.pm line 703.
BEGIN failed--compilation aborted at /u01/app/11.2.0.4/grid_home/crs/install/crsconfig_lib.pm line 703.
Compilation failed in require at /u01/app/11.2.0.4/grid_home/crs/install/rootcrs.pl line 305.
BEGIN failed--compilation aborted at /u01/app/11.2.0.4/grid_home/crs/install/rootcrs.pl line 305.

此报错是由bug 18650453导致的，已经在在12.1.0.2中被修复

解决方法

补文件
[root@anbob2 ~]#ls /u01/app/11.2.0.4/crs/perl/lib/
5.10.0/    site_perl/
[root@anbob2 ~]# cp -p /u01/app/11.2.0.4/crs/perl/lib/5.10.0/Env.pm /usr/lib64/perl5/vendor_perl/
-- or --
[root@anbob2 ~]# cp -p /u01/app/11.2.0.4/crs/perl/lib/5.10.0/Env.pm  /usr/share/perl5/vendor_perl/

或
安装 补丁程序18650453: OL7: CLUSTERWARE ROOT SCRIPTS FAIL DUE TO: CAN'T LOCATE ENV.PM IN @INC

或 
使用ORACLE perl
[root@anbob2 ~]# /u01/app/grid/11.2.0.4/perl/bin/perl   /u01/app/grid/11.2.0.4/crs/install/roothas.pl -deconfig -force

重跑root.sh 正常。

Reference
root.sh failed with “CLSRSC-1003: Failed to start resource OC4J” during 12c grid install. (Doc ID 1996831.1)

— 希望对您有帮助。

打赏

微信扫一扫，打赏作者吧～