As you’ve already known, you should use odd number of disks for voting disk. A node must be able to strictly access more than half of the voting disks at any time. Let me show you how it works. I have installed and configured two node 11gR3 RAC on VirtualBox and use the following case to show how it works:

– Create a diskgroup with 3 failure groups and 3 different disks

– Move voting disk to the new diskgroup. Shutdown the second node and deattach one of the disks. In this case, cluster should start as it can access more than half of the voting disks (2 from 3)

– Start the second node. The cluster should be up. Shut the second node again and deattach the second voting disk. And start it. The cluster will not start. Check the ocssd.log file

– Shut down all node, attach the previous disks and start it again. Cluster will be up

Here’re the detailed steps:

– Create a diskgroup :

– Mount the diskgroup at the second node:

SQL> ALTER DISKGROUP vdisk MOUNT;

 

– Replace voting disk, move it to the new diskgroup and query the voting disk:

– 

 

 

 

 

 

 

 

 

– Shutdown the second instance and reattach one of the disks of VDISK diskgroup :

 

–

 

 

 

 

 

 

 

 

 

– Star the second node, query the Voting disk and check if the clusterware is up:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

– Shutdown the second node again, remove the second disk from the Voting diskgroup and start the node:

 

 

 

 

 

 

– Check the log file at $GRID_HOME/log/node2/cssd/ocssd.log :

2014-05-29 01:51:23.055: [ CSSD][2946955008]clssnmvVerifyCommittedConfigVFs: Insufficient voting files found, found 1 of 3 configured, needed 2 voting files
2014-05-29 01:51:23.055: [ CSSD][2946955008](:CSSNM00020:)clssnmvVerifyCommittedConfigVFs: voting file 0, id 279c162c-1b964f88-bfb1d622-aecc9e4e not found
2014-05-29 01:51:23.055: [ CSSD][2946955008](:CSSNM00020:)clssnmvVerifyCommittedConfigVFs: voting file 1, id 7e282f3f-5e514f42-bfb79396-c69fda76 not found
2014-05-29 01:51:23.055: [ CSSD][2946955008](:CSSNM00021:)clssnmCompleteVFDiscovery: Found 1 voting files, but 2 are required. Terminating due to insufficient configured voting files

– As you see, cluster is down. Now, shutdown both nodes, add disks to the second node and check the status of the clusterware:

 

 

 

 

 

 

 

 

 

It took me some time to investigate why CRSCTL REPLACE VOTEDISK command is not working.
[oracle@node1 ~]$ crsctl replace votedisk VDISK
CRS-4264: The operation could not be validated
CRS-4000: Command Replace failed, or completed with errors.
When you get an error during VOTEDISK replacement, make sure you check the following items:

– Make sure the disk group you’re moving the voting disk is mounted on all nodes.

– Make sure the compatibility parameter is set to the version of Grid software you’re using. You can change it using the following command:

alter diskgroup VDISK set attribute ‘compatible.asm’=’11.2’;

Query V$ASM_DISKGROUP view to make sure it’s the same with the rest disk groups and with the version of the Grid Software:

select group_number, name, compatibility, database_compatibility from v$asm_diskgroup;

– Check alert.log file of an ASM instance, any available trace file of the ASM instance. Check /var/log/messages file and trace the replace command usint strace file. See if you can catch any error from the log file:

[grid@node5 ~]strace crsctl replace votedisk VDISK
– Make sure you’ve an odd number of votedisk

– Make sure there’s enough space in the diskgroup

– Make sure disk permissions is correct

– Make sure you’re running the command using Grid Software owner

Today, all above mentioned checks are failed :). In my case, the problem was using incorrect “crsctl” command. After upgrading the RAC environment from 11.2.0 to 11.2.3 I was still using old crsctl (by accident, forgot to set environment variables correctly). But no need to worries, it was a test database.

Let me know if you have any additional check to investigate voting disk replace failure

Cheers