Kamran Agayev's Oracle Blog

Oracle Certified Master

Archive for the 'Administration' Category

Step by Step Installing Oracle 12c RAC on VirtualBox – Video Tutorial

Posted by Kamran Agayev A. on 22nd September 2013

It’s 3.50 AM at San Francisco and I’m finalizing a new video tutorial that I have been creating for the users who will attend RAC Attack at Oracle Openworld 2013 (Ninja Operation). Tomorrow is the first day of OOW13 and I’m very happy that I was able to finish it in time. The first minutes of this tutorial was created in the plane when I was flying to SFO this week :)

For the step by step instruction, I’ve used the following article



You can watch the video from the following link:

You can download the video from this link:



Hope you’ll enjoy the video!

Posted in Administration, RAC issues | 7 Comments »

I’m RACAttack Ninja!

Posted by Kamran Agayev A. on 19th September 2013

I’m proud to announce that I’m one of the 10 RAC Attack Ninjas this year! If you’re attending OOW13, don’t forget to come and say “hi” to Ninjas. We’ll be at OTN Lounge in the lobby of Moscone South on 24th and 25th September between 10am-2pm

Using documentation created by Ninjas, you’ll be able to install and run Oracle 12c RAC on your laptop!

For the step by step introduction, check the following link:

To get more information about RacAttack event, check the following link:

You can check our facebook page:

And don’t forget to use #RACAttack hashtag.

I’m working on “Step by Step Installing Oracle 12c RAC Video tutorial” and hope it will be ready for the RACAttack event. If you don’t have time to install it during OOW days, come and watch the video tutorial and learn how to install Oracle 12c RAC

See you all soon!


Posted in Administration, RAC issues | 1 Comment »

My book is published!

Posted by Kamran Agayev A. on 13th September 2013

My book is out!

I would like to announce to all my fellow DBA friends that my book is published and is available at Amazon.


It took me more than one year to write it. 3 years ago I got an email from Rampant Techpress where they asked me if I can write a book about RMAN Backup and Recovery. I wasn’t ready at that time, but decided to accept that challenge and agreed. The first man came in my mind to help me at that time was my friend Aman Sharma. He also agreed and we started the journey together.

After writing the first chapter I realized that we really need to have technical reviewers, so I asked Syed Sabdar and Hemant K. Chitale if they can help us. Fortunately they agreed.  I would like to thank them both as they did a great job and made a lot of corrections

All chapters are based on practical scenarios and deep research and I hope it will help you to understand RMAN in depth, to create your own RMAN backup strategy and to recover from any type of failures you can face in your production database. There’re a lot of scenarios under each chapter with an explanation and step by step guide to help you to reproduce them on your test environment


Posted in Administration, RMAN Backup and Recovery | 14 Comments »

How to become an Oracle Certified Master (My OCM journey)

Posted by Kamran Agayev A. on 16th August 2013

This is it! Finally I’ve successfully passed OCM exam. It was almost more than 1 year that I’ve been preparing for this.

It’s almost impossible to find additional information about the exam online, so I decided to make my own preparation plan as I’ve done it for OCP exams before.  And I would like to share my preparation tips with you and hope it will motivate you to start thinking about OCM.

It is literally true that you can succeed best and quickest by helping others to succeed.” – Napoleon Hill

Please note that this document has already been checked and approved by Oracle Certification Team.  


My preparation

–          In parallel to working as a production DBA at a leading mobile operator company in Azerbaijan (Azercell Telecom LLC), I decided to make a preparation plan for 9 months. I printed exam topics from this link and put it on my desktop. I planned to spend a month on each section for preparation (there are total 8 sections)


–          For each section and topic I went through the documentation, checked blog posts, studied metalink notes and created my own practices. I installed VirtualBox on my laptop and used it for all type of practices including RAC and Data Guard. I created different practices for each topic of each session, assigned a time frame for each practice and tried to solve them in a very limited time. I tried to not use GUI in all kind of scenarios. I was able to install Oracle Software and create a database both in silent mode using almost all parameters in a few minutes (with closed eyes :) ) . I learned all commands by heart; I was able to create a Data Guard using command line without looking to the documentation. I created RAC for several times, added and deleted nodes, went through all configuration and management issues. If you’re preparing for OCM exam, you have to forget GUI at all!

EM is a very important tool which has to be used on your production system and which helps you to administer all your databases from one window. You have to know every single link and page on EM, however you shouldn’t rely only on EM, because there may be some tasks which should be performed without EM. And don’t forget to master EM, as the second Section is all about Grid Control :)

–          You have to be prepared for any type of corruption at any time during the exam. You have to test all kind of failure scenarios and recover from any type of error. If you are not able to recover your database during the exam, you will have to return home earlier. I created all kind of corruption and failure scenarios, and prepared and tested all “How to recover if …” cases.

–          Each time I studied everything related to any topic and solved all practices, I removed that line from the paper. My aim was to have a paper with everything removed out like this :)


–          I sensed that I would have to fight against the time during the exam, so I decided to improve my typing speed. For the last 7 years I’ve been typing without looking at the keyboard, but it wasn’t enough for me. I decided to improve my typing speed and accuracy, thus registered at www.typeracer.com and competed every day. As a result my typing accuracy improved further.

–          After 8 months I finished my research almost for every section and topic of an OCM exam. For the 3 weeks of the last month I performed all practices that I had made for myself (all practices per week!). For the last week I did all practices per DAY! Every time it took me a few hours to finish all tasks. In the last week I realized that my fingers type faster than my brain process :) It was very hard to devote some hours per day for all practices, so I was preparing over the lunch breaks, after work, and mostly on weekends.


Finally, registration day arrived. Actually, I had planned that date before the preparation and started the registration 2 months before the exam. When you register for the exam try to check the nearest country, book the nearest hotel to the exam center, get a document to have a VISA from the embassy and book flight tickets. Don’t forget to contact OCM exam center 1-2 weeks before an exam to make sure that the exam will proceed and make sure you’ve received a confirmation letter from Oracle.

Before taking an OCM exam, you have to earn OCP certification and take 2 Oracle University (OU) courses. You can check this link for more information. If you haven’t taken the required courses before, it would be better to take the topic that you don’t feel at ease. After taking the required courses, you should submit them at Course Submission Form using course Enrollment ID

Also don’t forget to print addresses of the hotel and the exam center. On the first day I could not explain where to go unless I showed the printed address to the taxi driver. Don’t forget to take your ID to the exam center. In my case, I forgot it, came back to the hotel 1 hour before the exam started, was stuck in a traffic jam and finally got to the exam center 10 minutes ago :)

Although I had not a chance to relax (slept only 3 hours before the exam), but I would strongly advise you to have a rest a day before the exam.


About the exam 

Exam starts at 9.00 and finishes at 18.00. You should arrive 1 hour before the exam. Don’t forget to bring your Oracle Testing ID with you

You will not get any bug during the exam. The proctor is responsible for the discipline, provides information in the beginning of each section and doesn’t answer to any technical question. You can use nonsearchable documentation.

You need to find syntax of any command in a second, you have to know which book, which chapter and which section you need to look for specific topic or question. During preparation days, don’t use documentation search and google AT ALL! In some cases, you don’t have the second chance to correct the mistake you’ve made. You don’t have any chance to learn something during the exam. If you don’t know any topic or haven’t done it or practiced it before, or even don’t know where to find the information in the documentation just go to the next question. I’m sure you’ve already done the same at OCP exam

As I’ve signed NDA, I can’t provide more information about the exam. In a nutshell, you’ll struggle against the time during the whole exam. I want to repeat again – If you come across a question that you haven’t prepared for and seen any time before, that can be a big problem. Theoretical knowledge is not enough; you have to have a practical experience.

In order to avoid feeling nervous during the exam, you are supposed to have gone through the same experience during your DBA life. If you’ve faced such critical issues and solved them in time, then the exam will be less excited.


In conclusion …  

It doesn’t matter how experienced you are, it is still very challenging to use only command line interface to perform any requirement, to solve data corruption and recover the database from any data loss, create and manage RAC and Data Guard in a very limited timeline, tune the database and perform different kind of administrative tasks one by one without any mistake.

I hope my preparation method will help you start the preparation for the OCM exam. If you have any question, please do not hesitate to post your comment.

First and foremost I would like to thank The Alimighty God. He has given me the power to believe in my passion and pursue my dreams.  I could never have done this without the faith I have. Sincere thanks to my family, especially to my father who motivated me every day. I would also like to thank to my collegues, my friends and all my students for supporting me. Thank you for being genuinely happy for my success and good fortune, as if it were your own

And last but not least, I would like to share some motivation quotes which can help you to be motivated during the preparation and also in your entire life

  • If my mind can conceive it, and my heart can believe it – then I can achieve it. – Muhammad Ali
  • “Most people have the will to win, few have the will to prepare to win.” -Bobby Knight
  • You’re never a loser until you quit trying. -Mike Ditka”
  • Many of life’s failures are people who did not realize how close they were to success when they gave up
  • He who is not courageous enough to take risks will accomplish nothing in life.
  • Accept challenges in life, so that you may feel the exhilaration of victory.
  • “The greatest danger for most of us is not that we aim too high and we miss it, but we aim too low and reach it.” – Michelangelo
  • Edison failed 10,000 times before he made the electric light. Do not be discouraged if you fail a few times. – Napoleon Hil



Posted in Administration | 87 Comments »

User equivalence check failed for user “oracle”.

Posted by Kamran Agayev A. on 14th August 2013

If you’ve installed and configured RAC environment, you should face “User equivalenece check” error during pre-requisite check

There’re three common reasons for getting this error:

– You didn’t configure ssh between nodes. Make sure you can ssh to all nodes which are used at runcluvfy.sh script

– UID are different on any of node

– runcluvfy.sh script can’t find necessary executable to perform the connectivity or equivalence check


In my case, I configured ssh on both nodes and all parameters of an oracle user were same. So in this case, I decided to debug runcluvfy.sh script. To debug this script, you need to set some environment variables:

export CV_HOME=/home/oracle/cv   (Export Cluster Verify debugging home page)

export SRVM_TRACE=true

./runcluvfy.sh stage -post hwos -n node1,node2 -verbose


Then I checked the log file under /home/oracle/cv/pid/cv/log/log_file.log and got the following output:

[Worker 0] [8:57:44:666] [UnixSystem.checkRemoteExecutionSetup:1833] checkRemoteExecutionSetup:: Error checking user equivalence us
ing Secured Shell ‘/usr/local/bin/ssh’; Wed Aug 14 08:57:44 GMT+04:00 2013

[Thread-7] [8:57:44:850] [StreamReader.run:65] ERROR>remshd: Login incorrect.; Wed Aug 14 08:57:44 GMT+04:00 2013

[Thread-5] [8:58:15:773] [StreamReader.run:65] ERROR>rcmd_af: connect: node1: Connection refused; Wed Aug 14 08:5
8:15 GMT+04:00 2013


Hey, wait. I don’t have /usr/local/bin/ssh file! I do have ssh executable but it’s under the different directory. And why the script didn’t use “which ssh” command and get the exact path of the ssh executable?

Anyway, I created a symbolic link to the original ssh file and run the script again. It worked!

Posted in Administration, RAC issues | No Comments »

Peer not authenticated – OEM error

Posted by Kamran Agayev A. on 14th August 2013

Some days ago I was requested to deploy 9i database to 10g Grid Control. According to compatibility matrix (check metalink note – Oracle Enterprise Manager Grid Control Certification Checker (Doc ID 412431.1)) you can install 10g agent on 9i database and deploy it

So I downloaded agent from this link and installed it on 9i machine

Before getting the installation screen, I got the following error :

Unable to convert from “UTF-8” to “ISO8859-1” for NLS!

After struggling a little bit, I set the DISPLAY environment variable and was able to get the installation screen. Installation went smooth and agent deployed successfully. However, when trying to reach to that database over OEM Grid Control, I got “peer not authenticated” error:




I checked OEM and agent logs and didn’t get much information about this error. The output from OEM log was as follows:

2013-08-13 11:19:13,950 [EMUI_11_19_13_/console/database/config] ERROR perf.sitemap logStackTrace.2401 – oracle.sysman.emSDK.emd.comm.CommException: peer not authenticated
at oracle.sysman.emSDK.emd.comm.EMDClient.getResponseForRequest(EMDClient.java:1680)
at oracle.sysman.emSDK.emd.comm.EMDClient.getTarget(EMDClient.java:1214)
at oracle.sysman.db.conf.DBConfigObject.getTargetInstance(DBConfigObject.java:463)
at oracle.sysman.db.conf.DBConfigObject.initPropertiesFromRepository(DBConfigObject.java:634)
at oracle.sysman.db.conf.DBConfigObject.initialize(DBConfigObject.java:434)
at oracle.sysman.db.conf.DBConfigUI.initialize(DBConfigUI.java:271)


Then I tried to compare targets.xml file on both 9i and on 10g machine where agent was working properly and found that the main two lines were missing from 9i targets.xml file:

<Property NAME=”UserName” VALUE=”emuser”/>
<Property NAME=”password” VALUE=”bae2342769a2fac163dfdf256813e716″ ENCRYPTED=”TRUE”/>


I added both lines to the 9i targets.xml file, created “emuser” and changed VALUE parameter of the password field and set ENCRYPTED to “FALSE”

Then I performed the following steps in order to delete all files that should be uploaded and start the agent again in non-secure mode :


$AGENT_HOME/bin/emctl stop agent


rm -rf $AGENT_HOME/sysman/emd/recv

rm -rf $AGENT_HOME/sysman/emd/collection

rm -rf $AGENT_HOME/sysman/emd/state

rm -rf $AGENT_HOME/sysman/emd/upload

rm -rf  $AGENT_HOME/sysman/emd/lastupld.xml

$AGENT_HOME/bin/emctl clearstaete agent

$AGENT_HOME/bin/emctl start agent


After some minutes I was able to manage the database from OEM and the error disappeared.

Posted in Administration | No Comments »

Step by step video tutorial of installing Oracle Database 12c on OEL 6

Posted by Kamran Agayev A. on 2nd July 2013

In this video tutorial you’ll learn how to install Oracle Database 12c on Oracle Enterprise Linux 6



You can download .mp4 format of the video from this link

You can watch it online from this link  (Don’t forget to watch it in HD format)

You an download Installation configuration file from this link

Posted in Administration, Oracle on Linux, Video Tutorials | 6 Comments »

Moving RAC redolog files to the new ASM diskgroup

Posted by Kamran Agayev A. on 17th May 2013

Here’re the steps you need to perform in order to move redo log files to the different ASM diskgroup in a RAC environment

First of all, make sure you’ve been provided shared disks by system administrators. If you’re using asmlib, create disks using oracleasm as follows:

[oracle@node1 ~] sudo /etc/init.d/oracleasm createdisk RLOGS /dev/mapper/mpath6

Then run scan and query new disks from other nodes:

[oracle@node2 ~]$ sudo /etc/init.d/oracleasm scandisks
[oracle@node2 ~]$ sudo /etc/init.d/oracleasm listdisks

After having your disk added to the list, query it from /dev/oracleasm/disks folder:
[oracle@node1 ~]$ ls -ltr /dev/oracleasm/disks/

If you got the disk, then everything is ok. Now you can connect to the ASM instance and create a diskgroup. For this, get the name of the ASM instance from OS and connect to it:
ps -ef | grep ASM
sqlplus / as sysdba

SQL> create diskgroup RLOGS external redundancy
2 DISK ‘/dev/oracleasm/disks/RLOGS’;

Diskgroup created.


If it’s not mounted on the rest nodes, connect to those nodes and mount it manually:
SQL>alter diskgroup RLOGS mount;

And also make sure the name of the disk group is added to asm_diskstrings parameter at ASM instance

After having your disk group created, create new redo log group per thread, and remove on redo log group from the same thread which has INACTIVE status
You can check the status of any group based on thread using V$LOG view:

SELECT group#, thread#, status
FROM v$log
WHERE thread# = 2

———- ———- —————-

Now add new group to the same thread (thread#=2) and check v$logfile view:


SELECT lf.group#, lf.MEMBER
FROM v$logfile lf, v$log l
WHERE lf.group# = l.group# AND l.thread# = 2

———- ——————————————————-
20 +OLD_LOG/PROD/onlinelog/group_20.259.815656575
21 +OLD_LOG/PROD/onlinelog/group_21.260.815656475
22 +RLOGS/PROD/onlinelog/group_22.260.815656475

Now you can safely delete INACTIVE redo log group from the same thread:


Perform above mentioned steps one by one for each thread.

Each time switch log file and make sure the new redo log member is used. Moreover, try to query both sql commands that are written above after each time you add and drop redo log group to see if it’s added/deleted successfully.

Posted in Administration | No Comments »

Waiting at “Performing remote operations” during RAC installation

Posted by Kamran Agayev A. on 30th January 2013

Today, while installing RAC, the installation began to wait at “Performing remote operations” step. I’ve tried to check log files, but there was no information written

Then I tried to trace the currently running process, but didn’t get enough information about the wait. At last, I checked both servers and found firewall running. After stopping firewall, (stopping, deinstalling and starting the installation again) the setup was able to pass that step and copied the Grid home files to the remote server


Posted in Administration, RAC issues | No Comments »

Some processes are missing from V$SESSION view

Posted by Kamran Agayev A. on 22nd January 2013

Today, while checking both V$PROCESS and V$SESSION view, I realized that there’re some processes that are missing from V$SESSION view:

SQL> select count(*) from v$process where addr not in (select paddr from v$session);




How can a process be created without having any corresponding entry in V$SESSION view?

I started to check those processes on OS level. I got SPID of one of those processes and firstly tried to debug it using ORADEBUG from sqlplus:

sqlplus / as sysdba


By getting the trace file name, I’ve checked the file but didn’t find any helpful message inside.

Then I decided to debug it on OS level. As the OS was HP-UX, I used truss executable to trace the unix system calls:

/usr/local/bin/truss -o truss.out -vall -laefdD -p 3422

And checked the output again:

tail -f truss.out
( Attached to process 18997 (“oraclePRODDB (LOCAL=NO)”) [64-bit] )
18997/9934051: 18446744073.7095 read(16, 0x600000000021a616, 2064) [sleeping]

This output wasn’t helpful (however, Tanel uses pfiles executable to get all opened files in the process by getting the first parameter of the read function – which is 16) , so I decided to use the next executable – lsof – which lists the opened files by the process:

/usr/local/bin/lsof -p 3422

oracle 20472 oracle cwd DIR 64,0×4 2048 3660 /home/oracle/product/10g/dbs
oracle 20472 oracle txt REG 64,0×4 284477360 101589 /home/oracle/product/10g/bin/oracle
oracle 20472 oracle mem REG 64,0×4 5337208 25311 /home/oracle/product/10g/lib/libnnz10.so
oracle 20472 oracle mem REG 64,0×7 21281 27339 /usr/lib/tztab
oracle 20472 oracle mem REG 64,0×5 312440 54074 /opt/star-ncf-prod/ep_patch/usr/lib/hpux64/libxti.so.1
oracle 20472 oracle mem REG 64,0×7 4371800 14392 /usr/lib/hpux64/libc.so.1
oracle 20472 oracle mem REG 64,0×7 1649152 91060 /usr/lib/hpux64/libnsl.so.1
oracle 20472 oracle mem REG 64,0×7 67664 91056 /usr/lib/hpux64/libnss_compat.so.1
oracle 20472 oracle mem REG 64,0×4 20041776 8569 /home/oracle/product/10g/lib/libjox10.so
oracle 20472 oracle mem REG 64,0×7 1046480 11645 /usr/lib/hpux64/libpthread.so.1
oracle 20472 oracle mem REG 64,0×7 55088 491 /usr/lib/hpux64/libuca.so.1
oracle 20472 oracle mem REG 64,0×7 706832 91078 /usr/lib/hpux64/libunwind.so.1
oracle 20472 oracle mem REG 64,0×7 6444800 91068 /usr/lib/hpux64/libm.so.1
oracle 20472 oracle mem REG 64,0×7 78168 90961 /usr/lib/hpux64/libdl.so.1
oracle 20472 oracle mem REG 64,0×7 89208 91053 /usr/lib/hpux64/libnss_dns.so.1
oracle 20472 oracle mem REG 64,0×7 44832 3881 /usr/lib/hpux64/librt.so.1
oracle 20472 oracle mem REG 64,0×4 199296 8277 /home/oracle/product/10g/lib/libdbcfg10.so
oracle 20472 oracle mem REG 64,0×4 134552 13143 /home/oracle/product/10g/lib/libocrutl10.so
oracle 20472 oracle mem REG 64,0×4 533792 13142 /home/oracle/product/10g/lib/libocrb10.so
oracle 20472 oracle mem REG 64,0×4 731352 25330 /home/oracle/product/10g/lib/libocr10.so
oracle 20472 oracle mem REG 64,0×4 1296848 25478 /home/oracle/product/10g/lib/libhasgen10.so
oracle 20472 oracle mem REG 64,0×4 315008 13264 /home/oracle/product/10g/lib/libskgxp10.so
oracle 20472 oracle mem REG 64,0×7 1163152 90960 /usr/lib/hpux64/dld.so
oracle 20472 oracle mem REG 64,0×7 184592 90962 /usr/lib/hpux64/uld.so
oracle 20472 oracle 0u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 1u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 2u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 3u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 4u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 5u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 6u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 7u IPv4 0xe0000001cbfe0900 0t0 TCP *:* (IDLE)
oracle 20472 oracle 8u REG 64,0×4 849408 13244 /home/oracle/product/10g/rdbms/mesg/oraus.msb
oracle 20472 oracle 14u REG 64,0×4 268644352 79279 /home/oracle/product/10g/cdata/localhost/local.ocr
oracle 20472 oracle 27u IPv4 0xe0000001c34f0740 0t3603 TCP ***.***.***.***:1521->name.ofthehost.com:4934 (ESTABLISHED)
oracle 20472 oracle 28w FIFO 0xe00000017b3ba748 0t0 184090157
oracle 20472 oracle 29r FIFO 0xe0000001f99d3e48 0t0 184090158


At the end of the output, I saw an IP address and the host name where the process were connecting. After checking that server, I saw that it was opening a connection to the database and created a process on OS. Then it wasn’t able to create a session and didn’t detect the dead process and kill it. So it remained in the V$PROCESS view and on the OS as a process and was missing in the V$SESSION view.

I killed all missing sessions from OS level and rebooted the server which was opening connections to the database. It seems it has memory or network issues, so I asked sysadmins to check it


You can also check the following metalink note and learn how you can enable Dead Connection Detection (DCD)

A discussion of Dead Connection Detection, Resource Limits, V$SESSION, V$PROCESS and OS processes [ID 601605.1]


Posted in Administration | 2 Comments »