Kamran Agayev's Oracle Blog

Oracle Certified Master

Archive for the 'RAC issues' Category

Real Application Clusters

How to troubleshoot CRSCTL REPLACE VOTEDISK error?

Posted by Kamran Agayev A. on 27th May 2014

It took me some time to investigate why CRSCTL REPLACE VOTEDISK command is not working.
[oracle@node1 ~]$ crsctl replace votedisk VDISK
CRS-4264: The operation could not be validated
CRS-4000: Command Replace failed, or completed with errors.
When you get an error during VOTEDISK replacement, make sure you check the following items:

– Make sure the disk group you’re moving the voting disk is mounted on all nodes.

– Make sure the compatibility parameter is set to the version of Grid software you’re using. You can change it using the following command:

alter diskgroup VDISK set attribute ‘compatible.asm’=’11.2’;

Query V$ASM_DISKGROUP view to make sure it’s the same with the rest disk groups and with the version of the Grid Software:

select group_number, name, compatibility, database_compatibility from v$asm_diskgroup;

– Check alert.log file of an ASM instance, any available trace file of the ASM instance. Check /var/log/messages file and trace the replace command usint strace file. See if you can catch any error from the log file:

[grid@node5 ~]strace crsctl replace votedisk VDISK
– Make sure you’ve an odd number of votedisk

– Make sure there’s enough space in the diskgroup

– Make sure disk permissions is correct

– Make sure you’re running the command using Grid Software owner

Today, all above mentioned checks are failed :). In my case, the problem was using incorrect “crsctl” command. After upgrading the RAC environment from 11.2.0 to 11.2.3 I was still using old crsctl (by accident, forgot to set environment variables correctly). But no need to worries, it was a test database.

Let me know if you have any additional check to investigate voting disk replace failure

Cheers

Posted in RAC issues | 4 Comments »

Cluster won’t start if diagnostic_dest folder is missing

Posted by Kamran Agayev A. on 3rd March 2014

One of the reason of why cluster won’t start is DIAGNOSTIC_DEST folder is missing. Here it is what I got today in of the nodes of the RAC environment:

db-bash: crs_stat -t

HA Resource Target State
———– —— —–
error connecting to CRSD at [(ADDRESS=(PROTOCOL=IPC)(KEY=ora_crsqs))] clsccon 184

 

While checking alert log file of the clusterware ($GRID_HOME/log/node1/alertnode1.log

[/home/oracle/11.2.0/grid_1124/bin/oraagent.bin(4745)]CRS-5011:Check of resource “+ASM” failed: details at “(:CLSN00006:)” in “/home/oracle/11.2.0/grid_1124/log/node01/agent/ohasd/oraagent_oracle/oraagent_oracle.log”

 

ASM instance failed to start. I connected to ASM instance and tried to start it manually:

db-bash-$ asm

SQL*Plus: Release 11.2.0.4.0 Production on Mon Mar 3 10:32:24 2014

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

ASM> startup

ORA-48108: invalid value given for the diagnostic_dest init.ora parameter

ORA-48140: the specified ADR Base directory does not exist [/home/oracle/11.2.0/dbhome]

ORA-48187: specified directory does not exist

HPUX-ia64 Error: 2: No such file or directory

Additional information: 1

ASM>

 

ADR Base Directory is missing. After creating it, I successfully started the CRS and got the happiest message :) :

db-bash-$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
db-bash-$

Posted in Administration, RAC issues | 2 Comments »

V$ASM_DISKGROUP displays information from the header of ASM disks

Posted by Kamran Agayev A. on 17th January 2014

While playing with OCR recovery, suddenly I realized that V$ASM_DISKGROUP view gets information from the headers of the ASM disk files that are specified at *.ASM_DISKSTRING parameter. Here’s the description of V$ASM_DISKGROUP view from documentation:

V$ASM_DISKGROUP displays one row for every ASM disk group discovered by the ASM instance on the node.

http://docs.oracle.com/cd/E18283_01/server.112/e17110/dynviews_1027.htm

 

I got explain plan of V$ASM_DISKGROUP to know which X$ table stand behind it and got – X$KFGRP

SQL> set autotrace on

SQL> select count(1) from v$asm_diskgroup;

COUNT(1)

———-

3

 

pic1

 

 

 

 

 

 

 

SQL> select name_kfgrp from x$kfgrp;

NAME_KFGRP

——————————

 

DATA

FLASH

OCR

 

SQL> select GRPNUM_KFDSK, NUMBER_KFDSK, STATE_KFDSK, ASMNAME_KFDSK, PATH_KFDSK from x$KFDSK;

pic2

 

 

 

 

 

 

 

 

Then I queried ASM_DISKGROUPS parameter :

 

SQL> show parameter disk

 

NAME                                 TYPE        VALUE

———————————— ———– ——————————

asm_diskgroups                       string      OCR, DATA, FLASH

asm_diskstring                       string      /dev/oracleasm/disks

 

No I will create a new tablespace under FLASH diskgroup, create a new table, change owner of the disk of FLASH diskgroup and make it #*disappear* from V$ASM_DISKGROUP view, and then return everything back 

SQL> create tablespace new_tbs datafile ‘+FLASH’;

Tablespace created.

SQL> create table new_table (id number) tablespace new_tbs;

Table created.

SQL> insert into new_table values(1);

1 row created.

SQL> commit;

Commit complete.

SQL> select * from new_table;

ID
———-
1

 

I create a parameter file from spfile, change ASM_DISKGROUPS parameter to OCR,DATA (remove FLASH) and mount the ASM instance again using a parameter file with

ASM_DISKGROUPS=’OCR’,’DATA’ specified:

 

pic7

 

So DISK6 is member and as the disk is discovered by ASM instance, FLASH diskgroup is dismounted, but still there.

Let’s change the owner of the disk and check it again. But before checking the owner, let’s read it’s header by KFED:

[root@node1 disks]# kfed read DISK6

pic4

 

Now let’s start the instance and check V$ASM_DISKGROUP view:
pic5

 

Query X$KFGRP view:


 
SQL> select NAME_KFGRP from X$KFGRP;

NAME_KFGRP

——————————

 

DATA

OCR

SQL>

 

Switch to the database and check if you can query the table:

SQL> select * from new_table;

select * from new_table

*

ERROR at line 1:

ORA-01157: cannot identify/lock data file 8 – see DBWR trace file

ORA-01110: data file 8: ‘+FLASH/rac/datafile/new_tbs.257.837080939’

SQL>

 

 

Now shutdown the ASM instance, return the owner back and check V$ASM_DISKGROUP again:

[root@node1 disks]# chown -R oracle:dba DISK6
pic6

 

FLASH diskgroup appeared, however it’s not specified at ASM_DISKGROUPS parameter. Now mount the diskgroup and query the table again:

SQL> alter diskgroup flash mount;

Diskgroup altered.

SQL>

SQL> select * from new_table;

ID
———-
1

SQL>

 

This means that if you want to move the ASM instance to another host, it’s enough to specify ASM_DISKSTRING parameter, V$ASM_DISKGROUP will discover all diskgroups

Posted in Administration, RAC issues | 2 Comments »

Step by Step Installing Oracle 12c RAC on VirtualBox – Video Tutorial

Posted by Kamran Agayev A. on 22nd September 2013

It’s 3.50 AM at San Francisco and I’m finalizing a new video tutorial that I have been creating for the users who will attend RAC Attack at Oracle Openworld 2013 (Ninja Operation). Tomorrow is the first day of OOW13 and I’m very happy that I was able to finish it in time. The first minutes of this tutorial was created in the plane when I was flying to SFO this week :)

For the step by step instruction, I’ve used the following article

http://en.wikibooks.org/wiki/RAC_Attack_-_Oracle_Cluster_Database_at_Home/RAC_Attack_12c

 

You can watch the video from the following link:
12c_rac_installation

You can download the video from this link:

http://www.kamranagayev.com/Video_Tutorials/Step_by_Step_installing_Oracle_12c_RAC.mp4

 

Hope you’ll enjoy the video!

Posted in Administration, RAC issues | 7 Comments »

I’m RACAttack Ninja!

Posted by Kamran Agayev A. on 19th September 2013

I’m proud to announce that I’m one of the 10 RAC Attack Ninjas this year! If you’re attending OOW13, don’t forget to come and say “hi” to Ninjas. We’ll be at OTN Lounge in the lobby of Moscone South on 24th and 25th September between 10am-2pm

Using documentation created by Ninjas, you’ll be able to install and run Oracle 12c RAC on your laptop!

For the step by step introduction, check the following link:
http://racattack.org/12c

To get more information about RacAttack event, check the following link:
http://www.pythian.com/blog/rac-attack-at-oracle-openworld-2013-operation-ninja/

You can check our facebook page:
https://www.facebook.com/racattackoow13

And don’t forget to use #RACAttack hashtag.

I’m working on “Step by Step Installing Oracle 12c RAC Video tutorial” and hope it will be ready for the RACAttack event. If you don’t have time to install it during OOW days, come and watch the video tutorial and learn how to install Oracle 12c RAC

See you all soon!

131982_634077923291835_2111683842_o

Posted in Administration, RAC issues | 1 Comment »

User equivalence check failed for user “oracle”.

Posted by Kamran Agayev A. on 14th August 2013

If you’ve installed and configured RAC environment, you should face “User equivalenece check” error during pre-requisite check

There’re three common reasons for getting this error:

– You didn’t configure ssh between nodes. Make sure you can ssh to all nodes which are used at runcluvfy.sh script

– UID are different on any of node

– runcluvfy.sh script can’t find necessary executable to perform the connectivity or equivalence check

 

In my case, I configured ssh on both nodes and all parameters of an oracle user were same. So in this case, I decided to debug runcluvfy.sh script. To debug this script, you need to set some environment variables:

export CV_HOME=/home/oracle/cv   (Export Cluster Verify debugging home page)

export SRVM_TRACE=true

./runcluvfy.sh stage -post hwos -n node1,node2 -verbose

 

Then I checked the log file under /home/oracle/cv/pid/cv/log/log_file.log and got the following output:

[Worker 0] [8:57:44:666] [UnixSystem.checkRemoteExecutionSetup:1833] checkRemoteExecutionSetup:: Error checking user equivalence us
ing Secured Shell ‘/usr/local/bin/ssh’; Wed Aug 14 08:57:44 GMT+04:00 2013

[Thread-7] [8:57:44:850] [StreamReader.run:65] ERROR>remshd: Login incorrect.; Wed Aug 14 08:57:44 GMT+04:00 2013

[Thread-5] [8:58:15:773] [StreamReader.run:65] ERROR>rcmd_af: connect: node1: Connection refused; Wed Aug 14 08:5
8:15 GMT+04:00 2013

 

Hey, wait. I don’t have /usr/local/bin/ssh file! I do have ssh executable but it’s under the different directory. And why the script didn’t use “which ssh” command and get the exact path of the ssh executable?

Anyway, I created a symbolic link to the original ssh file and run the script again. It worked!

Posted in Administration, RAC issues | No Comments »

Waiting at “Performing remote operations” during RAC installation

Posted by Kamran Agayev A. on 30th January 2013

Today, while installing RAC, the installation began to wait at “Performing remote operations” step. I’ve tried to check log files, but there was no information written

Then I tried to trace the currently running process, but didn’t get enough information about the wait. At last, I checked both servers and found firewall running. After stopping firewall, (stopping, deinstalling and starting the installation again) the setup was able to pass that step and copied the Grid home files to the remote server

 

Posted in Administration, RAC issues | No Comments »

Testing Load balancing in Oracle RAC

Posted by Kamran Agayev A. on 14th January 2013

If you’ve installed Oracle RAC (Real Application Clusters) and want to test how Load Balancing works, you can run the following shell script and check GV$SESSION view:

#!/bin/bash
. /home/oracle/.bash_profile
for ((i=1; i <= 50 ; i++))
do
nohup sqlplus -S system/oracle@racdb<<eof &
begin
dbms_lock.sleep(10);
end;
/

eof
done

This will open 50 sessions in the background. Check GV$SESSION view before and after  running this query:

SQL> select inst_id,count(*) from gv$session where username is not null group by inst_id;

INST_ID COUNT(*)
———- ———-
1           10
2          9

Run the following command from the different session:

[oracle@node1 ~] ./check_load_balancing.sh

SQL> /

INST_ID COUNT(*)
———- ———-
1           33
2           36

Wait for 10 seconds (as we’ve defined “10” seconds at DBMS_LOCK.SLEEP procedure) and run the query again :

SQL> /

INST_ID COUNT(*)
———- ———-
1           10
2           9

SQL>

From the output you can see that all new sessions were distributed between two nodes appropriately.

Posted in RAC issues | 12 Comments »

Solving the bug – ORA-07445 [KFKPROCESSREQ()+429] in ASM instance

Posted by Kamran Agayev A. on 10th July 2012

Today, one of the instances of 2 node RAC went down with ORA-07445 error. After checking metalink, I found out that this is a bug:

Bug 10422354: ORA-07445 [KFKPROCESSREQ()+429] [SIGSEGV] ON ASM INSTANCE

And it happens after the upgrade of RAC to 10.2.5 patchset. Unfortunately there was no workaround suggested by support. After checking ASM log file again, I find the following error:

ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]

The ASM instance was not mounting. The instance wasn’t able to find the shared storage, so I changed asm_diskstring parameter as follows:

[php]asm_diskstring=’/dev/mapper/*'[/php]

It was a test RAC environment, both ASM instances were using different initialization parameter files (however, both database instances were using the same shared initialization parameter file)

I’ve created new server parameter file from the existing one, put it in the shared folder, changed parameter file of each instance (pfile=’/shared_folder/spfile+ASM.ora’) and started the failed ASM instance. Fortunately, it has started. Then I restarted the second ASM instance to make it use the shared parameter file and both instances were up

 

Posted in Administration, RAC issues | 2 Comments »

The installer has detected the existence of Oracle Clusterware

Posted by Kamran Agayev A. on 12th June 2012

After removing the RAC installation on the test server, I got the following error during the installation:

“The installer has detected the existence of 10g Oracle Clusterware on the remote node, —-, however, 10g Oracle Clusterware does not exist on the local node …

 

Although I run $CRS_HOME/install/rootdelete.sh, removed the $CRS_HOME and oraInventory folders and killed all running processes, it was throwing above mentioned error. After making some checks, I saw that /etc/oracle/ocr.loc file is still there. When removing that file, the error message disappeared. I was interested whether installation checks the file’s availability or its entry also, so created an empty file with the same name “ocr.loc” and tried the installation. Guess what? I got the error again. This means that Oracle looks only for the name of the file in the mentioned folder

In case you want to clear the failed RAC installation, check the following article:

http://blog.flimatech.com/2008/10/02/10g-rac-how-to-clean-up-after-a-failed-crs-install/

 

Posted in Administration, RAC issues | No Comments »