And the third and latest issue we got today with RAC installation was the following error while running rootdelete.sh which is used to disable clusterware applications :

[code]

[root@node1 install]# srvctl
/u01/oracle/product/10.2.0/crs_1/jdk/jre/bin/java: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory
[root@node1 install]# ./rootdelete.sh
/u01/oracle/product/10.2.0/crs_1/jdk/jre/bin/java: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory
[/code]

Then we got the same error while using srvctl utility to manage configuration information.

After a little investigation, we’ve found that we need to edit two executables 1. srvctl and vipca (as it’s called when running rootdelete.sh) and comment the line where LD_ASSUME_KERNEL parameter is set

[code]

       then
            LD_ASSUME_KERNEL=2.4.19
          #  export LD_ASSUME_KERNEL
[/code]

After commenting the line, we were able to bypass the above mentioned error

You can refer the following metalink note for more information:

Executing Applypreferences.sh Fails With “Error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory” [ID 729952.1]

[code]

[root@node1 install]# ./rootdelete.sh
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources.
Error while stopping resources. Possible cause: CRSD is down.
Stopping CSSD.
Unable to communicate with the CSS daemon.
Shutdown has begun. The daemons should exit soon.
Checking to see if Oracle CRS stack is down…
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script for Oracle Cluster Ready services
Updating ocr file for downgrade
Cleaning up SCR settings in ‘/etc/oracle/scls_scr’
[root@node1 install]#

[/code]

I would like to talk about the second problem we got during RAC installation. Again, after clusterware installation while running root.sh on the first node, we got some delay and the script were going to keeping running. While checking css.log file under $ORA_CRS_HOME/log/<HOSTNAME>/client/css.log file we got the following error:

[code]

[ CSSCLNT][9553600]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying

[ CSSCLNT][9553600]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying

[/code]

However, we were able to run the same shell script (root.sh) on the second node successfully.

After a little investigation in the metalink, we’ve foud that it’s a bug – Bug 6353645: ROOT.SH FAILED TO STARTUP CLUSTERWARE

Although there were no any workaround, we’ve found a suggestion in the end of the note that after running root.sh on the second node, the message in the first node should disappear

It was true

[code]

[root@FIRSTNODE client]# crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[/code]

Today, while configuring 2 node RAC on VMware ESX server, I got the following error afer clusterware installation:

[code]

Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
Now formatting voting device: /dev/raw/raw3
Failure writing offset 1766400 in voting device (raw3)
Failed to initialize Oracle Cluster Registry for cluster
[/code]

After some investigation, we’ve found that the problem is caused by dividing a raw device into two different partitions and the failure occurs after formatting the device. So we deleted those partitions and combined them into one partition and the error fixed.