Just came across a wonderful blog where you can find dozens of different Oracle related video tutorials

www.OracleVideoTutorials.com

One of the reason of why cluster won’t start is DIAGNOSTIC_DEST folder is missing. Here it is what I got today in of the nodes of the RAC environment:

db-bash: crs_stat -t

HA Resource Target State
———– —— —–
error connecting to CRSD at [(ADDRESS=(PROTOCOL=IPC)(KEY=ora_crsqs))] clsccon 184

 

While checking alert log file of the clusterware ($GRID_HOME/log/node1/alertnode1.log

[/home/oracle/11.2.0/grid_1124/bin/oraagent.bin(4745)]CRS-5011:Check of resource “+ASM” failed: details at “(:CLSN00006:)” in “/home/oracle/11.2.0/grid_1124/log/node01/agent/ohasd/oraagent_oracle/oraagent_oracle.log”

 

ASM instance failed to start. I connected to ASM instance and tried to start it manually:

db-bash-$ asm

SQL*Plus: Release 11.2.0.4.0 Production on Mon Mar 3 10:32:24 2014

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

ASM> startup

ORA-48108: invalid value given for the diagnostic_dest init.ora parameter

ORA-48140: the specified ADR Base directory does not exist [/home/oracle/11.2.0/dbhome]

ORA-48187: specified directory does not exist

HPUX-ia64 Error: 2: No such file or directory

Additional information: 1

ASM>

 

ADR Base Directory is missing. After creating it, I successfully started the CRS and got the happiest message :

db-bash-$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
db-bash-$

After migrating 10gR2 single instance to 11gR2 RAC db, the next step should be configuring Enterprise Manager (if you don’t use centralized Grid Control)

 

It can be done by a simple command. But as usual (at least for me) it didn’t go smooth and I got some errors :). In this article I will show you solutions for some of them.

The command should be run is as follows:

emca -config dbcontrol db -repos create –cluster

 

It will ask you to provide the following information:

db-bash-$ emca -config dbcontrol db -repos create -cluster

 

STARTED EMCA at Mar 2, 2014 3:03:19 PM

EM Configuration Assistant, Version 11.2.0.3.0 Production

Copyright (c) 2003, 2011, Oracle.  All rights reserved.

 

Enter the following information:

Database unique name:

Service name:

Listener port number:

Listener ORACLE_HOME

Password for SYS user:

Password for DBSNMP user:

Password for SYSMAN user:

Cluster name:

Email address for notifications (optional):

Outgoing Mail (SMTP) server for notifications (optional):

ASM ORACLE_HOME :

ASM port :

ASM username [ ASMSNMP ]:

ASM user password:

 

–          Get Database unique name from db_unique_name parameter from the database

SQL> show parameter db_unique_name

–          To provide a service name, get the SCAN listener name from the host (ps –ef | grep tns), get the status of SCAN listener by running lsnrctl status LISTENER_SCAN1 command from $GRID_HOME and get the service name from the output

Service “RACDB.MYCOMP.COM” has 3 instance(s).
     Instance “RACDB1”, status READY, has 1 handler(s) for this service…
     Instance “RACDB2”, status READY, has 1 handler(s) for this service…
     Instance “RACDB3”, status READY, has 1 handler(s) for this service…

 

–          Make sure all password files at all nodes – orapwNODE1[2-3] – are same. If not, create a password file at each node and provide the same password

–          As a Cluster Name, provide the name of the cluster. To get the cluster name, use the following command:

cd $GRID_HOME/bin

./cemutlo –n

–          Make sure you’ve ASMSNMP user created at ASM instance and SYSDBA privilege is granted. Do it in the first node, then copy ora+ASM1 to the different nodes under $GRID_HOME/dbs/+ASM[1-2]

 

Hope above mentioned notes will help you to create EM db control successfully for your RAC environment.

 

To get more information from metalink, use the following MOS:

How to manage DB Control 11.x for RAC Database with emca (Doc ID 578011.1)

 

While playing with OCR recovery, suddenly I realized that V$ASM_DISKGROUP view gets information from the headers of the ASM disk files that are specified at *.ASM_DISKSTRING parameter. Here’s the description of V$ASM_DISKGROUP view from documentation:

V$ASM_DISKGROUP displays one row for every ASM disk group discovered by the ASM instance on the node.

http://docs.oracle.com/cd/E18283_01/server.112/e17110/dynviews_1027.htm

 

I got explain plan of V$ASM_DISKGROUP to know which X$ table stand behind it and got – X$KFGRP

SQL> set autotrace on

SQL> select count(1) from v$asm_diskgroup;

COUNT(1)

———-

3

 

 

 

 

 

 

 

 

SQL> select name_kfgrp from x$kfgrp;

NAME_KFGRP

——————————

 

DATA

FLASH

OCR

 

SQL> select GRPNUM_KFDSK, NUMBER_KFDSK, STATE_KFDSK, ASMNAME_KFDSK, PATH_KFDSK from x$KFDSK;

 

 

 

 

 

 

 

 

Then I queried ASM_DISKGROUPS parameter :

 

SQL> show parameter disk

 

NAME                                 TYPE        VALUE

———————————— ———– ——————————

asm_diskgroups                       string      OCR, DATA, FLASH

asm_diskstring                       string      /dev/oracleasm/disks

 

No I will create a new tablespace under FLASH diskgroup, create a new table, change owner of the disk of FLASH diskgroup and make it #*disappear* from V$ASM_DISKGROUP view, and then return everything back 

SQL> create tablespace new_tbs datafile ‘+FLASH’;

Tablespace created.

SQL> create table new_table (id number) tablespace new_tbs;

Table created.

SQL> insert into new_table values(1);

1 row created.

SQL> commit;

Commit complete.

SQL> select * from new_table;

ID
———-
1

 

I create a parameter file from spfile, change ASM_DISKGROUPS parameter to OCR,DATA (remove FLASH) and mount the ASM instance again using a parameter file with

ASM_DISKGROUPS=’OCR’,’DATA’ specified:

 

 

So DISK6 is member and as the disk is discovered by ASM instance, FLASH diskgroup is dismounted, but still there.

Let’s change the owner of the disk and check it again. But before checking the owner, let’s read it’s header by KFED:

[root@node1 disks]# kfed read DISK6

 

Now let’s start the instance and check V$ASM_DISKGROUP view:

 

Query X$KFGRP view:

 SQL> select NAME_KFGRP from X$KFGRP;

NAME_KFGRP

——————————

 

DATA

OCR

SQL>

 

Switch to the database and check if you can query the table:

SQL> select * from new_table;

select * from new_table

*

ERROR at line 1:

ORA-01157: cannot identify/lock data file 8 – see DBWR trace file

ORA-01110: data file 8: ‘+FLASH/rac/datafile/new_tbs.257.837080939’

SQL>

 

 

Now shutdown the ASM instance, return the owner back and check V$ASM_DISKGROUP again:

[root@node1 disks]# chown -R oracle:dba DISK6

 

FLASH diskgroup appeared, however it’s not specified at ASM_DISKGROUPS parameter. Now mount the diskgroup and query the table again:

SQL> alter diskgroup flash mount;

Diskgroup altered.

SQL>

SQL> select * from new_table;

ID
———-
1

SQL>

 

This means that if you want to move the ASM instance to another host, it’s enough to specify ASM_DISKSTRING parameter, V$ASM_DISKGROUP will discover all diskgroups

It’s 3.50 AM at San Francisco and I’m finalizing a new video tutorial that I have been creating for the users who will attend RAC Attack at Oracle Openworld 2013 (Ninja Operation). Tomorrow is the first day of OOW13 and I’m very happy that I was able to finish it in time. The first minutes of this tutorial was created in the plane when I was flying to SFO this week

For the step by step instruction, I’ve used the following article

http://en.wikibooks.org/wiki/RAC_Attack_-_Oracle_Cluster_Database_at_Home/RAC_Attack_12c

 

You can watch the video from the following link:

You can download the video from this link:

http://www.kamranagayev.com/Video_Tutorials/Step_by_Step_installing_Oracle_12c_RAC.mp4

 

Hope you’ll enjoy the video!

I’m proud to announce that I’m one of the 10 RAC Attack Ninjas this year! If you’re attending OOW13, don’t forget to come and say “hi” to Ninjas. We’ll be at OTN Lounge in the lobby of Moscone South on 24th and 25th September between 10am-2pm

Using documentation created by Ninjas, you’ll be able to install and run Oracle 12c RAC on your laptop!

For the step by step introduction, check the following link:
http://racattack.org/12c

To get more information about RacAttack event, check the following link:
http://www.pythian.com/blog/rac-attack-at-oracle-openworld-2013-operation-ninja/

You can check our facebook page:
https://www.facebook.com/racattackoow13

And don’t forget to use #RACAttack hashtag.

I’m working on “Step by Step Installing Oracle 12c RAC Video tutorial” and hope it will be ready for the RACAttack event. If you don’t have time to install it during OOW days, come and watch the video tutorial and learn how to install Oracle 12c RAC

See you all soon!

My book is out!

I would like to announce to all my fellow DBA friends that my book is published and is available at Amazon.

http://www.amazon.com/Oracle-Backup-Recovery-secrets-In-Focus/dp/0984428232/

It took me more than one year to write it. 3 years ago I got an email from Rampant Techpress where they asked me if I can write a book about RMAN Backup and Recovery. I wasn’t ready at that time, but decided to accept that challenge and agreed. The first man came in my mind to help me at that time was my friend Aman Sharma. He also agreed and we started the journey together.

After writing the first chapter I realized that we really need to have technical reviewers, so I asked Syed Sabdar and Hemant K. Chitale if they can help us. Fortunately they agreed.  I would like to thank them both as they did a great job and made a lot of corrections

All chapters are based on practical scenarios and deep research and I hope it will help you to understand RMAN in depth, to create your own RMAN backup strategy and to recover from any type of failures you can face in your production database. There’re a lot of scenarios under each chapter with an explanation and step by step guide to help you to reproduce them on your test environment

This is it! Finally I’ve successfully passed OCM exam. It was almost more than 1 year that I’ve been preparing for this.

It’s almost impossible to find additional information about the exam online, so I decided to make my own preparation plan as I’ve done it for OCP exams before.  And I would like to share my preparation tips with you and hope it will motivate you to start thinking about OCM.

“It is literally true that you can succeed best and quickest by helping others to succeed.” – Napoleon Hill

Please note that this document has already been checked and approved by Oracle Certification Team.  

 

My preparation

–          In parallel to working as a production DBA at a leading mobile operator company in Azerbaijan (Azercell Telecom LLC), I decided to make a preparation plan for 9 months. I printed exam topics from this link and put it on my desktop. I planned to spend a month on each section for preparation (there are total 8 sections)

–          For each section and topic I went through the documentation, checked blog posts, studied metalink notes and created my own practices. I installed VirtualBox on my laptop and used it for all type of practices including RAC and Data Guard. I created different practices for each topic of each session, assigned a time frame for each practice and tried to solve them in a very limited time. I tried to not use GUI in all kind of scenarios. I was able to install Oracle Software and create a database both in silent mode using almost all parameters in a few minutes (with closed eyes ) . I learned all commands by heart; I was able to create a Data Guard using command line without looking to the documentation. I created RAC for several times, added and deleted nodes, went through all configuration and management issues. If you’re preparing for OCM exam, you have to forget GUI at all!

EM is a very important tool which has to be used on your production system and which helps you to administer all your databases from one window. You have to know every single link and page on EM, however you shouldn’t rely only on EM, because there may be some tasks which should be performed without EM. And don’t forget to master EM, as the second Section is all about Grid Control

–          You have to be prepared for any type of corruption at any time during the exam. You have to test all kind of failure scenarios and recover from any type of error. If you are not able to recover your database during the exam, you will have to return home earlier. I created all kind of corruption and failure scenarios, and prepared and tested all “How to recover if …” cases.

–          Each time I studied everything related to any topic and solved all practices, I removed that line from the paper. My aim was to have a paper with everything removed out like this 

–          I sensed that I would have to fight against the time during the exam, so I decided to improve my typing speed. For the last 7 years I’ve been typing without looking at the keyboard, but it wasn’t enough for me. I decided to improve my typing speed and accuracy, thus registered at www.typeracer.com and competed every day. As a result my typing accuracy improved further.

–          After 8 months I finished my research almost for every section and topic of an OCM exam. For the 3 weeks of the last month I performed all practices that I had made for myself (all practices per week!). For the last week I did all practices per DAY! Every time it took me a few hours to finish all tasks. In the last week I realized that my fingers type faster than my brain process It was very hard to devote some hours per day for all practices, so I was preparing over the lunch breaks, after work, and mostly on weekends.

 

Finally, registration day arrived. Actually, I had planned that date before the preparation and started the registration 2 months before the exam. When you register for the exam try to check the nearest country, book the nearest hotel to the exam center, get a document to have a VISA from the embassy and book flight tickets. Don’t forget to contact OCM exam center 1-2 weeks before an exam to make sure that the exam will proceed and make sure you’ve received a confirmation letter from Oracle.

Before taking an OCM exam, you have to earn OCP certification and take 2 Oracle University (OU) courses. You can check this link for more information. If you haven’t taken the required courses before, it would be better to take the topic that you don’t feel at ease. After taking the required courses, you should submit them at Course Submission Form using course Enrollment ID

Also don’t forget to print addresses of the hotel and the exam center. On the first day I could not explain where to go unless I showed the printed address to the taxi driver. Don’t forget to take your ID to the exam center. In my case, I forgot it, came back to the hotel 1 hour before the exam started, was stuck in a traffic jam and finally got to the exam center 10 minutes ago

Although I had not a chance to relax (slept only 3 hours before the exam), but I would strongly advise you to have a rest a day before the exam.

 

About the exam 

Exam starts at 9.00 and finishes at 18.00. You should arrive 1 hour before the exam. Don’t forget to bring your Oracle Testing ID with you

You will not get any bug during the exam. The proctor is responsible for the discipline, provides information in the beginning of each section and doesn’t answer to any technical question. You can use nonsearchable documentation.

You need to find syntax of any command in a second, you have to know which book, which chapter and which section you need to look for specific topic or question. During preparation days, don’t use documentation search and google AT ALL! In some cases, you don’t have the second chance to correct the mistake you’ve made. You don’t have any chance to learn something during the exam. If you don’t know any topic or haven’t done it or practiced it before, or even don’t know where to find the information in the documentation just go to the next question. I’m sure you’ve already done the same at OCP exam

As I’ve signed NDA, I can’t provide more information about the exam. In a nutshell, you’ll struggle against the time during the whole exam. I want to repeat again – If you come across a question that you haven’t prepared for and seen any time before, that can be a big problem. Theoretical knowledge is not enough; you have to have a practical experience.

In order to avoid feeling nervous during the exam, you are supposed to have gone through the same experience during your DBA life. If you’ve faced such critical issues and solved them in time, then the exam will be less excited.

 

In conclusion …  

It doesn’t matter how experienced you are, it is still very challenging to use only command line interface to perform any requirement, to solve data corruption and recover the database from any data loss, create and manage RAC and Data Guard in a very limited timeline, tune the database and perform different kind of administrative tasks one by one without any mistake.

I hope my preparation method will help you start the preparation for the OCM exam. If you have any question, please do not hesitate to post your comment.

First and foremost I would like to thank The Alimighty God. He has given me the power to believe in my passion and pursue my dreams.  I could never have done this without the faith I have. Sincere thanks to my family, especially to my father who motivated me every day. I would also like to thank to my collegues, my friends and all my students for supporting me. Thank you for being genuinely happy for my success and good fortune, as if it were your own

And last but not least, I would like to share some motivation quotes which can help you to be motivated during the preparation and also in your entire life

• If my mind can conceive it, and my heart can believe it – then I can achieve it. – Muhammad Ali

• “Most people have the will to win, few have the will to prepare to win.” -Bobby Knight

• You’re never a loser until you quit trying. -Mike Ditka”

• Many of life’s failures are people who did not realize how close they were to success when they gave up

• He who is not courageous enough to take risks will accomplish nothing in life.

• Accept challenges in life, so that you may feel the exhilaration of victory.

• “The greatest danger for most of us is not that we aim too high and we miss it, but we aim too low and reach it.” – Michelangelo

• Edison failed 10,000 times before he made the electric light. Do not be discouraged if you fail a few times. – Napoleon Hil

 

 

If you’ve installed and configured RAC environment, you should face “User equivalenece check” error during pre-requisite check

There’re three common reasons for getting this error:

– You didn’t configure ssh between nodes. Make sure you can ssh to all nodes which are used at runcluvfy.sh script

– UID are different on any of node

– runcluvfy.sh script can’t find necessary executable to perform the connectivity or equivalence check

 

In my case, I configured ssh on both nodes and all parameters of an oracle user were same. So in this case, I decided to debug runcluvfy.sh script. To debug this script, you need to set some environment variables:

export CV_HOME=/home/oracle/cv   (Export Cluster Verify debugging home page)

export SRVM_TRACE=true

./runcluvfy.sh stage -post hwos -n node1,node2 -verbose

 

Then I checked the log file under /home/oracle/cv/pid/cv/log/log_file.log and got the following output:

[Worker 0] [8:57:44:666] [UnixSystem.checkRemoteExecutionSetup:1833] checkRemoteExecutionSetup:: Error checking user equivalence us
ing Secured Shell ‘/usr/local/bin/ssh’; Wed Aug 14 08:57:44 GMT+04:00 2013

[Thread-7] [8:57:44:850] [StreamReader.run:65] ERROR>remshd: Login incorrect.; Wed Aug 14 08:57:44 GMT+04:00 2013

[Thread-5] [8:58:15:773] [StreamReader.run:65] ERROR>rcmd_af: connect: node1: Connection refused; Wed Aug 14 08:5
8:15 GMT+04:00 2013

 

Hey, wait. I don’t have /usr/local/bin/ssh file! I do have ssh executable but it’s under the different directory. And why the script didn’t use “which ssh” command and get the exact path of the ssh executable?

Anyway, I created a symbolic link to the original ssh file and run the script again. It worked!

Some days ago I was requested to deploy 9i database to 10g Grid Control. According to compatibility matrix (check metalink note – Oracle Enterprise Manager Grid Control Certification Checker (Doc ID 412431.1)) you can install 10g agent on 9i database and deploy it

So I downloaded 10.2.0.2 agent from this link and installed it on 9i machine

Before getting the installation screen, I got the following error :

Unable to convert from “UTF-8” to “ISO8859-1” for NLS!

After struggling a little bit, I set the DISPLAY environment variable and was able to get the installation screen. Installation went smooth and agent deployed successfully. However, when trying to reach to that database over OEM Grid Control, I got “peer not authenticated” error:

 

 

I checked OEM and agent logs and didn’t get much information about this error. The output from OEM log was as follows:

2013-08-13 11:19:13,950 [EMUI_11_19_13_/console/database/config] ERROR perf.sitemap logStackTrace.2401 – oracle.sysman.emSDK.emd.comm.CommException: peer not authenticated
at oracle.sysman.emSDK.emd.comm.EMDClient.getResponseForRequest(EMDClient.java:1680)
at oracle.sysman.emSDK.emd.comm.EMDClient.getTarget(EMDClient.java:1214)
at oracle.sysman.db.conf.DBConfigObject.getTargetInstance(DBConfigObject.java:463)
at oracle.sysman.db.conf.DBConfigObject.initPropertiesFromRepository(DBConfigObject.java:634)
at oracle.sysman.db.conf.DBConfigObject.initialize(DBConfigObject.java:434)
at oracle.sysman.db.conf.DBConfigUI.initialize(DBConfigUI.java:271)

 

Then I tried to compare targets.xml file on both 9i and on 10g machine where agent was working properly and found that the main two lines were missing from 9i targets.xml file:

<Property NAME=”UserName” VALUE=”emuser”/>
<Property NAME=”password” VALUE=”bae2342769a2fac163dfdf256813e716″ ENCRYPTED=”TRUE”/>

 

I added both lines to the 9i targets.xml file, created “emuser” and changed VALUE parameter of the password field and set ENCRYPTED to “FALSE”

Then I performed the following steps in order to delete all files that should be uploaded and start the agent again in non-secure mode :

 

$AGENT_HOME/bin/emctl stop agent

 

rm -rf $AGENT_HOME/sysman/emd/recv

rm -rf $AGENT_HOME/sysman/emd/collection

rm -rf $AGENT_HOME/sysman/emd/state

rm -rf $AGENT_HOME/sysman/emd/upload

rm -rf  $AGENT_HOME/sysman/emd/lastupld.xml

$AGENT_HOME/bin/emctl clearstaete agent

$AGENT_HOME/bin/emctl start agent

 

After some minutes I was able to manage the database from OEM and the error disappeared.