Downgrade Grid Infrastructure 12.1.0.2 to 11.2.0.4降级后crs无法启动 No voting files found
Last month, after a set of Exadata test environment of our customer environment was downgraded,Downgrade Grid Infrastructure 12.1.0.2 to 11.2.0.4, the downgrade operation was successful, but when starting the CRS, it prompted that the VD could not be found, and the VD checked that it exists. I found out that he had the same case from the website. Reprint and record it here.
# /opt/app/12.1.0/grid2/crs/install/rootcrs.sh -downgrade ... CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector. Successfully downgraded Oracle Clusterware stack on this node
since this is a two node RAC the downgrade command is run with lastnode option (OCR-node). This will remove the GI management repository, downgrade the OCR
# /opt/app/12.1.0/grid2/crs/install/rootcrs.sh -downgrade -lastnode
CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
Successfully downgraded Oracle Clusterware stack on this node
Run '/opt/app/11.2.0/grid4/bin/crsctl start crs' on all nodes to complete downgrade
Before starting the cluster with 11.2 update the inventory by setting crs=true for 11.2 GI home. At this point 12.1 GI home will have crs=true.
$ cd /opt/app/12.1.0/grid2/oui/bin/
$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/opt/app/12.1.0/grid2
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 4095 MB Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/opt/app/11.2.0/grid4
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 4095 MB Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.
Also make sure the contents in /etc/init.d/ohasd and /etc/init.d/init.ohasd refers the 11.2 as the ORA_CRS_HOME and there are no references to 12.1. Few times the last downgrade node (rhel6m1) did contain references to 12.1 even though downgrade command had successfully completed. However these files on the other node had the correct references to 11.2.
# cat /etc/init.d/ohasd | grep ORA_CRS_HOME ORA_CRS_HOME=/opt/app/11.2.0/grid4 export ORA_CRS_HOME # cat /etc/init.d/init.ohasd | grep ORA_CRS_HOME ORA_CRS_HOME=/opt/app/11.2.0/grid4 export ORA_CRS_HOME PERL="/opt/app/11.2.0/grid4/perl/bin/perl -I${ORA_CRS_HOME}/perl/lib" crsctl start crs crsctl query crs activeversion -f Oracle Clusterware active version on the cluster is [11.2.0.4.0]. The cluster upgrade state is [NORMAL]. crsctl query crs softwareversion Oracle Clusterware version on node [rhel6m1] is [11.2.0.4.0] # ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 3356 Available space (kbytes) : 258764 ID : 2072206343 Device/File Name : +CLUSTER_DG This concludes the steps to successfully downgrading GI from 12.1.0.2 to 11.2.0.4.
There could be occasions the downgrade command successfully completes but start of crs fails. Symptoms in this case include unable to discover any voting disks crsctl query css votedisk doesn’t return any vote disk infomration and ocssd.log will have entries similar to
2015-03-31 14:00:09.618: [ CSSD][898090752]clssnmvDiskVerify: Successful discovery of 0 disks 2015-03-31 14:00:09.618: [ CSSD][898090752]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery 2015-03-31 14:00:09.618: [ CSSD][898090752]clssnmvFindInitialConfigs: No voting files found
Other times includes corrupted OCR with ocssd.log having entries similar to
2015-03-26 11:33:05.633: [ CRSMAIN][3817551648] Initialing cluclu context...
[ OCRMAS][3776734976]th_calc_av:8': Failed in vsnupr. Incorrect SV stored in OCR. Key [SYSTEM.version.hostnames.] Value []
2015-03-26 11:33:06.618: [ OCRSRV][3776734976]th_upgrade:9 Shutdown CacheMaster. prev AV [186647552] new calc av [186647552] my sv [186647552]
No root cause was found of these cases and could only assume this may be due to some of the earlier mentioned reasons such as having backups of OCR from previous upgrades i.e. ocr11.2.0.3.0 (But it must be said a successful downgrade was archived while ocr11.2.0.3.0 was in the cdata directory), wrong binaries referred during downgrade due to environment variable settings. The only option to recover from such a situation is to restore the OCR taken while cluster was 11.2.
crsctl stop crs -f # run on all nodes crsctl start crs -excl -nocrs # run only on one node ocrconfig -restore /opt/app/11.2.0/grid4/cdata/rhel6m-cluster/backup_20150331_170634.ocr crsctl replace votedisk +cluster_dg
对不起,这篇文章暂时关闭评论。