Kamran Agayev's Oracle Blog

Oracle Certified Master

User equivalence check failed for user “oracle”.

Posted by Kamran Agayev A. on August 14th, 2013

If you’ve installed and configured RAC environment, you should face “User equivalenece check” error during pre-requisite check

There’re three common reasons for getting this error:

- You didn’t configure ssh between nodes. Make sure you can ssh to all nodes which are used at runcluvfy.sh script

- UID are different on any of node

- runcluvfy.sh script can’t find necessary executable to perform the connectivity or equivalence check


In my case, I configured ssh on both nodes and all parameters of an oracle user were same. So in this case, I decided to debug runcluvfy.sh script. To debug this script, you need to set some environment variables:

export CV_HOME=/home/oracle/cv   (Export Cluster Verify debugging home page)

export SRVM_TRACE=true

./runcluvfy.sh stage -post hwos -n node1,node2 -verbose


Then I checked the log file under /home/oracle/cv/pid/cv/log/log_file.log and got the following output:

[Worker 0] [8:57:44:666] [UnixSystem.checkRemoteExecutionSetup:1833] checkRemoteExecutionSetup:: Error checking user equivalence us
ing Secured Shell ‘/usr/local/bin/ssh’; Wed Aug 14 08:57:44 GMT+04:00 2013

[Thread-7] [8:57:44:850] [StreamReader.run:65] ERROR>remshd: Login incorrect.; Wed Aug 14 08:57:44 GMT+04:00 2013

[Thread-5] [8:58:15:773] [StreamReader.run:65] ERROR>rcmd_af: connect: node1: Connection refused; Wed Aug 14 08:5
8:15 GMT+04:00 2013


Hey, wait. I don’t have /usr/local/bin/ssh file! I do have ssh executable but it’s under the different directory. And why the script didn’t use “which ssh” command and get the exact path of the ssh executable?

Anyway, I created a symbolic link to the original ssh file and run the script again. It worked!

Posted in Administration, RAC issues | No Comments »

Peer not authenticated – OEM error

Posted by Kamran Agayev A. on August 14th, 2013

Some days ago I was requested to deploy 9i database to 10g Grid Control. According to compatibility matrix (check metalink note - Oracle Enterprise Manager Grid Control Certification Checker (Doc ID 412431.1)) you can install 10g agent on 9i database and deploy it

So I downloaded agent from this link and installed it on 9i machine

Before getting the installation screen, I got the following error :

Unable to convert from “UTF-8″ to “ISO8859-1″ for NLS!

After struggling a little bit, I set the DISPLAY environment variable and was able to get the installation screen. Installation went smooth and agent deployed successfully. However, when trying to reach to that database over OEM Grid Control, I got “peer not authenticated” error:




I checked OEM and agent logs and didn’t get much information about this error. The output from OEM log was as follows:

2013-08-13 11:19:13,950 [EMUI_11_19_13_/console/database/config] ERROR perf.sitemap logStackTrace.2401 – oracle.sysman.emSDK.emd.comm.CommException: peer not authenticated
at oracle.sysman.emSDK.emd.comm.EMDClient.getResponseForRequest(EMDClient.java:1680)
at oracle.sysman.emSDK.emd.comm.EMDClient.getTarget(EMDClient.java:1214)
at oracle.sysman.db.conf.DBConfigObject.getTargetInstance(DBConfigObject.java:463)
at oracle.sysman.db.conf.DBConfigObject.initPropertiesFromRepository(DBConfigObject.java:634)
at oracle.sysman.db.conf.DBConfigObject.initialize(DBConfigObject.java:434)
at oracle.sysman.db.conf.DBConfigUI.initialize(DBConfigUI.java:271)


Then I tried to compare targets.xml file on both 9i and on 10g machine where agent was working properly and found that the main two lines were missing from 9i targets.xml file:

<Property NAME=”UserName” VALUE=”emuser”/>
<Property NAME=”password” VALUE=”bae2342769a2fac163dfdf256813e716″ ENCRYPTED=”TRUE”/>


I added both lines to the 9i targets.xml file, created “emuser” and changed VALUE parameter of the password field and set ENCRYPTED to “FALSE”

Then I performed the following steps in order to delete all files that should be uploaded and start the agent again in non-secure mode :


$AGENT_HOME/bin/emctl stop agent


rm -rf $AGENT_HOME/sysman/emd/recv

rm -rf $AGENT_HOME/sysman/emd/collection

rm -rf $AGENT_HOME/sysman/emd/state

rm -rf $AGENT_HOME/sysman/emd/upload

rm -rf  $AGENT_HOME/sysman/emd/lastupld.xml

$AGENT_HOME/bin/emctl clearstaete agent

$AGENT_HOME/bin/emctl start agent


After some minutes I was able to manage the database from OEM and the error disappeared.

Posted in Administration | No Comments »

I’m Oracle Certified Master now!

Posted by Kamran Agayev A. on July 10th, 2013

Last week I got an email from Oracle Certification Team that I’ve successfully passed OCM exam. I can proudly say that I’m Oracle Certified Master now!

Although I’m working as a production DBA, I prepared for this exam for more than 6 months. I made a plan for each month, even for each week, prepared own scenarios and practiced for hours.

The exam was tough, you need to be prepared for any case, you have to make decisions very quickly, and even you have to type very fast.

I will prepare blog post on how I prepared for OCM exam and hope it will help and inspire you to take it


Posted in Uncategorized | 39 Comments »

Step by step video tutorial of installing Oracle Database 12c on OEL 6

Posted by Kamran Agayev A. on July 2nd, 2013

In this video tutorial you’ll learn how to install Oracle Database 12c on Oracle Enterprise Linux 6



You can download .mp4 format of the video from this link

You can watch it online from this link  (Don’t forget to watch it in HD format)

You an download Installation configuration file from this link

Posted in Administration, Oracle on Linux, Video Tutorials | 5 Comments »

Moving RAC redolog files to the new ASM diskgroup

Posted by Kamran Agayev A. on May 17th, 2013

Here’re the steps you need to perform in order to move redo log files to the different ASM diskgroup in a RAC environment

First of all, make sure you’ve been provided shared disks by system administrators. If you’re using asmlib, create disks using oracleasm as follows:

[oracle@node1 ~] sudo /etc/init.d/oracleasm createdisk RLOGS /dev/mapper/mpath6

Then run scan and query new disks from other nodes:

[oracle@node2 ~]$ sudo /etc/init.d/oracleasm scandisks
[oracle@node2 ~]$ sudo /etc/init.d/oracleasm listdisks

After having your disk added to the list, query it from /dev/oracleasm/disks folder:
[oracle@node1 ~]$ ls -ltr /dev/oracleasm/disks/

If you got the disk, then everything is ok. Now you can connect to the ASM instance and create a diskgroup. For this, get the name of the ASM instance from OS and connect to it:
ps -ef | grep ASM
sqlplus / as sysdba

SQL> create diskgroup RLOGS external redundancy
2 DISK ‘/dev/oracleasm/disks/RLOGS’;

Diskgroup created.


If it’s not mounted on the rest nodes, connect to those nodes and mount it manually:
SQL>alter diskgroup RLOGS mount;

And also make sure the name of the disk group is added to asm_diskstrings parameter at ASM instance

After having your disk group created, create new redo log group per thread, and remove on redo log group from the same thread which has INACTIVE status
You can check the status of any group based on thread using V$LOG view:

SELECT group#, thread#, status
FROM v$log
WHERE thread# = 2

———- ———- —————-

Now add new group to the same thread (thread#=2) and check v$logfile view:


SELECT lf.group#, lf.MEMBER
FROM v$logfile lf, v$log l
WHERE lf.group# = l.group# AND l.thread# = 2

———- ——————————————————-
20 +OLD_LOG/PROD/onlinelog/group_20.259.815656575
21 +OLD_LOG/PROD/onlinelog/group_21.260.815656475
22 +RLOGS/PROD/onlinelog/group_22.260.815656475

Now you can safely delete INACTIVE redo log group from the same thread:


Perform above mentioned steps one by one for each thread.

Each time switch log file and make sure the new redo log member is used. Moreover, try to query both sql commands that are written above after each time you add and drop redo log group to see if it’s added/deleted successfully.

Posted in Administration | No Comments »

TT0802: Database permanent space exhausted (TimesTen db)

Posted by Kamran Agayev A. on April 18th, 2013

During TimesTen stress test, we’ve encountered the following error while trying to perform bulk insert :

TT0802: Database permanent space exhausted — file “blk.c”, lineno 3477, procedure “sbBlkAlloc”"

The reason is because of the PERM_ALLOCATED_SIZE parameter defined in sys.odbc.ini file. The default value is 64MB which is small for performing bulk-insert and stress-test. You can get this value either from sys.odbc.ini file, or using dssize command from ttIsql

Command> dssize


Command> exit


In order to increase this paramter, switch to $TIMESTEN_HOME/info folder and change PERM_ALLOCATED_SIZE parameter inside the required cache group:



Here, we changed the parameter to 10GB.  Now restart the TimesTen deamon and check the value again:

[oracle@timesten info]$ ttDaemonAdmin -stop
TimesTen Daemon stopped.
[oracle@timesten info]$ ttDaemonAdmin -start
TimesTen Daemon startup OK.


Command> dssize




Now, run the bulk insert again and you’ll not get the error, instead you’ll get PERM_IN_USE_SIZE value increased


Posted in TimesTen | No Comments »

Step by Step Oracle 10g RAC installation ebook

Posted by Kamran Agayev A. on March 15th, 2013

From the following link you can download my Step by Step Oracle 10g RAC installation on VMware ebook

Downloaded 1527 times

Posted in Uncategorized | 8 Comments »

Debugging Data Pump session

Posted by Kamran Agayev A. on March 11th, 2013

While importing a dump file, sometimes it takes too long and seems to be “hanging” and processing something unknown in the background, or queuing for something. As a DBA, you HAVE NOT wait “until it finishes”. You have to try to find out the solution.

Yesterday, while importing a metadata of a big database, the session hold and took hours to finish.

This is the command I run to import the dump file into the new database:

impdp system/oracle directory=mydir dumpfile=dump01.dmp exclude=tablespace, statistics parfile=param.ini METRICS=y 

The session was hold in the following section:


     Completed 1 CLUSTER objects in 0 seconds


     Completed 1 INDEX objects in 0 seconds


Note: I suggest that we use undocumented METRICS=Y parameter to get information about the number of objects and the time it takes to process them into the log file

I waited shortly and was curious about the activities in the background, so decided to debug the Data Pump session and run the following query to get the SID and SERIAL number of the running data pump sessions. These values will be used to trace the session in the next step:

col username format a10
set linesize 150
col job_name format a20
col program format a25
WHERE p.addr = s.paddr AND s.saddr = d.saddr;

DATE                PROGRAM                          SID STATUS   USERNAME   JOB_NAME             SPID            SERIAL#        PID

——————- ————————- ———- ——– ———- ——————– ———— ———- ———-

2013-03-07 14:52:04 udi@TestDB02 (TNS V1-V3)         152 ACTIVE   SYSTEM     SYS_IMPORT_FULL_01   12143                36         17

2013-03-07 14:52:04 oracle@TestDB02 (DM00)           148 ACTIVE   SYSTEM     SYS_IMPORT_FULL_01   12147                 3         19

2013-03-07 14:52:04 oracle@TestDB02 (DW01)           142 ACTIVE   SYSTEM     SYS_IMPORT_FULL_01   12149                15         20





Then I queried V$SESSION_WAIT view to get the waiting event:

SELECT   w.sid, w.event, w.seconds_in_wait


 WHERE   s.saddr = d.saddr AND s.sid = w.sid;

     SID EVENT                                                            SECONDS_IN_WAIT

———- —————————————————————- —————

       142 control file sequential read                                                                  1

       148 wait for unread message on broadcast channel                                  43

       152 wait for unread message on broadcast channel                                 306

If you run the same query a number of times, you’ll see how the wait event changes:

To get detailed information I decided to debug the first session (SID = 142) and provided values of SID and SERIAL# columns to DBMS_SYSTEM.SET_EV procedure and started the trace:

SQL>  EXECUTE SYS.DBMS_SYSTEM.SET_EV (142,15,10046,8,”);

Then I switched to udump folder and checked the file labeled “dbname_ora_PID” – “testdb_ora_12149”

-bash-4.2$ tail -f testdb_ora_12149.trc

WAIT #4: nam=’Data file init write’ ela= 3 count=4294967295 intr=32 timeout=2147483647 obj#=51780 tim=13410366772411

WAIT #4: nam=’control file sequential read’ ela= 21 file#=0 block#=1 blocks=1 obj#=51780 tim=13410366772623

WAIT #4: nam=’control file sequential read’ ela= 15 file#=1 block#=1 blocks=1 obj#=51780 tim=13410366772660

WAIT #4: nam=’control file sequential read’ ela= 15 file#=2 block#=1 blocks=1 obj#=51780 tim=13410366772695

WAIT #4: nam=’control file sequential read’ ela= 13 file#=0 block#=16 blocks=1 obj#=51780 tim=13410366772728

WAIT #4: nam=’control file sequential read’ ela= 13 file#=0 block#=18 blocks=1 obj#=51780 tim=13410366772760

WAIT #4: nam=’control file sequential read’ ela= 13 file#=0 block#=24 blocks=1 obj#=51780 tim=13410366772802

WAIT #4: nam=’db file sequential read’ ela= 12 file#=4 block#=1 blocks=1 obj#=51780 tim=13410366772834

WAIT #4: nam=’db file single write’ ela= 11313 file#=4 block#=1 blocks=1 obj#=51780 tim=13410366784170

WAIT #4: nam=’control file parallel write’ ela= 38910 files=3 block#=17 requests=3 obj#=51780 tim=13410366823104

WAIT #4: nam=’control file parallel write’ ela= 46532 files=3 block#=15 requests=3 obj#=51780 tim=13410366869661

WAIT #4: nam=’control file parallel write’ ela= 40258 files=3 block#=1 requests=3 obj#=51780 tim=13410366909955

WAIT #4: nam=’control file sequential read’ ela= 12 file#=0 block#=1 blocks=1 obj#=51780 tim=13410366909986

WAIT #4: nam=’rdbms ipc reply’ ela= 7016 from_process=5 timeout=21474836 p3=0 obj#=51780 tim=13410366917049


PARSING IN CURSOR #15 len=296 dep=3 uid=0 oct=6 lid=0 tim=13410366917452 hv=2379717279 ad=’de799c58′

update seg$ set type#=:4,blocks=:5,extents=:6,minexts=:7,maxexts=:8,extsize=:9,extpct=:10,user#=:11,iniexts=:12,lists=decode(:13, 65535, NULL, :13),groups=decode(:14, 65535, NULL, :14), cachehint=:15, hwmincr=:16, spare1=DECODE(:17,0,NULL,:17),scanhint=:18 where ts#=:1 and file#=:2 and block#=:3


PARSE #15:c=0,e=37,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=3,tim=13410366917447

EXEC #15:c=0,e=337,p=0,cr=5,cu=1,mis=0,r=1,dep=3,og=3,tim=13410366918000

STAT #15 id=1 cnt=0 pid=0 pos=1 obj=0 op=’UPDATE  SEG$ (cr=5 pr=0 pw=0 time=277 us)’

STAT #15 id=2 cnt=1 pid=1 pos=1 obj=14 op=’TABLE ACCESS CLUSTER SEG$ (cr=5 pr=0 pw=0 time=166 us)’

STAT #15 id=3 cnt=1 pid=2 pos=1 obj=9 op=’INDEX UNIQUE SCAN I_FILE#_BLOCK# (cr=2 pr=0 pw=0 time=31 us)’

By using tkprof I formatted content of the trace file into a readable output:

tkprof testdb_ora_12149.trc impdp_output.dat

And read the file:

insert into seg$ (file#,block#,type#,ts#,blocks,extents,minexts,maxexts,


spare1, scanhint)




call     count       cpu    elapsed       disk      query    current        rows

——- ——  ——– ———- ———- ———- ———-  ———-

Parse      905      0.04       0.01          0          0          0           0

Execute    905      0.26       0.32          0       2756       6420         905

Fetch        0      0.00       0.00          0          0          0           0

——- ——  ——– ———- ———- ———- ———-  ———-

total     1810      0.30       0.34          0       2756       6420         905

Everything is fine. It’s importing the metadata into the database by inserting data to the seg$ (segments) table. I used tail command to get the output of the trace file and run the first query.

After a delay, I got “statement suspended, wait error to be cleared” message, checked alert.log file and found that the USERS tablespace is FULL

SQL> /

SID EVENT                                                            SECONDS_IN_WAIT

———- —————————————————————- —————

142 statement suspended, wait error to be cleared                       20

148 wait for unread message on broadcast channel                                 905

152 wait for unread message on broadcast channel                                 246


tail –f Alert.log file

statement in resumable session ‘SYSTEM.SYS_IMPORT_FULL_01.1′ was suspended due to

    ORA-01658: unable to create INITIAL extent for segment in tablespace USERS

After adding some space to the USERS tablespace the message disappeared.

Then suddenly I begin receiving “data file init write” wait event. That was the main reasons why the import took much time.

The reason was that the AUTOEXTEND value of the datafile was set very low and Oracle was making too many calls to OS in order to add new extents

The output from the trace file was as follows:

WAIT #4: nam=’Data file init write’ ela= 3 count=4294967295 intr=32 timeout=2147483647 obj#=51780 tim=13410366772411

In order to define the tablespace which is the cause of this event, I got the object id from the trace file (obj#=51780) and checked obj$ table.

SELECT   name


 WHERE   obj# = 51780

Then using DBMS_METADATA.GET_DLL function I got the tablespace name of the object that couldn’t be imported


As the ‘Data file init write’ event directly relates to AUTOEXTEND and occurs while adding extents to the datafile, I checked AUTOEXTEND value of the datafiles. Their values were too small. I increased the AUTOEXTEND values and the event disappeared from the trace log and import job finished successfully.

Reference: Export/Import DataPump Parameter TRACE – How to Diagnose Oracle Data Pump [ID 286496.1]

Posted in Uncategorized | 5 Comments »

Waiting at “Performing remote operations” during RAC installation

Posted by Kamran Agayev A. on January 30th, 2013

Today, while installing RAC, the installation began to wait at “Performing remote operations” step. I’ve tried to check log files, but there was no information written

Then I tried to trace the currently running process, but didn’t get enough information about the wait. At last, I checked both servers and found firewall running. After stopping firewall, (stopping, deinstalling and starting the installation again) the setup was able to pass that step and copied the Grid home files to the remote server


Posted in Administration, RAC issues | No Comments »

Some processes are missing from V$SESSION view

Posted by Kamran Agayev A. on January 22nd, 2013

Today, while checking both V$PROCESS and V$SESSION view, I realized that there’re some processes that are missing from V$SESSION view:

SQL> select count(*) from v$process where addr not in (select paddr from v$session);




How can a process be created without having any corresponding entry in V$SESSION view?

I started to check those processes on OS level. I got SPID of one of those processes and firstly tried to debug it using ORADEBUG from sqlplus:

sqlplus / as sysdba


By getting the trace file name, I’ve checked the file but didn’t find any helpful message inside.

Then I decided to debug it on OS level. As the OS was HP-UX, I used truss executable to trace the unix system calls:

/usr/local/bin/truss -o truss.out -vall -laefdD -p 3422

And checked the output again:

tail -f truss.out
( Attached to process 18997 (“oraclePRODDB (LOCAL=NO)”) [64-bit] )
18997/9934051: 18446744073.7095 read(16, 0x600000000021a616, 2064) [sleeping]

This output wasn’t helpful (however, Tanel uses pfiles executable to get all opened files in the process by getting the first parameter of the read function – which is 16) , so I decided to use the next executable – lsof – which lists the opened files by the process:

/usr/local/bin/lsof -p 3422

oracle 20472 oracle cwd DIR 64,0×4 2048 3660 /home/oracle/product/10g/dbs
oracle 20472 oracle txt REG 64,0×4 284477360 101589 /home/oracle/product/10g/bin/oracle
oracle 20472 oracle mem REG 64,0×4 5337208 25311 /home/oracle/product/10g/lib/libnnz10.so
oracle 20472 oracle mem REG 64,0×7 21281 27339 /usr/lib/tztab
oracle 20472 oracle mem REG 64,0×5 312440 54074 /opt/star-ncf-prod/ep_patch/usr/lib/hpux64/libxti.so.1
oracle 20472 oracle mem REG 64,0×7 4371800 14392 /usr/lib/hpux64/libc.so.1
oracle 20472 oracle mem REG 64,0×7 1649152 91060 /usr/lib/hpux64/libnsl.so.1
oracle 20472 oracle mem REG 64,0×7 67664 91056 /usr/lib/hpux64/libnss_compat.so.1
oracle 20472 oracle mem REG 64,0×4 20041776 8569 /home/oracle/product/10g/lib/libjox10.so
oracle 20472 oracle mem REG 64,0×7 1046480 11645 /usr/lib/hpux64/libpthread.so.1
oracle 20472 oracle mem REG 64,0×7 55088 491 /usr/lib/hpux64/libuca.so.1
oracle 20472 oracle mem REG 64,0×7 706832 91078 /usr/lib/hpux64/libunwind.so.1
oracle 20472 oracle mem REG 64,0×7 6444800 91068 /usr/lib/hpux64/libm.so.1
oracle 20472 oracle mem REG 64,0×7 78168 90961 /usr/lib/hpux64/libdl.so.1
oracle 20472 oracle mem REG 64,0×7 89208 91053 /usr/lib/hpux64/libnss_dns.so.1
oracle 20472 oracle mem REG 64,0×7 44832 3881 /usr/lib/hpux64/librt.so.1
oracle 20472 oracle mem REG 64,0×4 199296 8277 /home/oracle/product/10g/lib/libdbcfg10.so
oracle 20472 oracle mem REG 64,0×4 134552 13143 /home/oracle/product/10g/lib/libocrutl10.so
oracle 20472 oracle mem REG 64,0×4 533792 13142 /home/oracle/product/10g/lib/libocrb10.so
oracle 20472 oracle mem REG 64,0×4 731352 25330 /home/oracle/product/10g/lib/libocr10.so
oracle 20472 oracle mem REG 64,0×4 1296848 25478 /home/oracle/product/10g/lib/libhasgen10.so
oracle 20472 oracle mem REG 64,0×4 315008 13264 /home/oracle/product/10g/lib/libskgxp10.so
oracle 20472 oracle mem REG 64,0×7 1163152 90960 /usr/lib/hpux64/dld.so
oracle 20472 oracle mem REG 64,0×7 184592 90962 /usr/lib/hpux64/uld.so
oracle 20472 oracle 0u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 1u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 2u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 3u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 4u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 5u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 6u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 7u IPv4 0xe0000001cbfe0900 0t0 TCP *:* (IDLE)
oracle 20472 oracle 8u REG 64,0×4 849408 13244 /home/oracle/product/10g/rdbms/mesg/oraus.msb
oracle 20472 oracle 14u REG 64,0×4 268644352 79279 /home/oracle/product/10g/cdata/localhost/local.ocr
oracle 20472 oracle 27u IPv4 0xe0000001c34f0740 0t3603 TCP ***.***.***.***:1521->name.ofthehost.com:4934 (ESTABLISHED)
oracle 20472 oracle 28w FIFO 0xe00000017b3ba748 0t0 184090157
oracle 20472 oracle 29r FIFO 0xe0000001f99d3e48 0t0 184090158


At the end of the output, I saw an IP address and the host name where the process were connecting. After checking that server, I saw that it was opening a connection to the database and created a process on OS. Then it wasn’t able to create a session and didn’t detect the dead process and kill it. So it remained in the V$PROCESS view and on the OS as a process and was missing in the V$SESSION view.

I killed all missing sessions from OS level and rebooted the server which was opening connections to the database. It seems it has memory or network issues, so I asked sysadmins to check it


You can also check the following metalink note and learn how you can enable Dead Connection Detection (DCD)

A discussion of Dead Connection Detection, Resource Limits, V$SESSION, V$PROCESS and OS processes [ID 601605.1]


Posted in Administration | 2 Comments »