In this video tutorial you’ll learn how to install Oracle Database 12c on Oracle Enterprise Linux 6

You can download .mp4 format of the video from this link

You can watch it online from this link  (Don’t forget to watch it in HD format)

You an download Installation configuration file from this link

Here’re the steps you need to perform in order to move redo log files to the different ASM diskgroup in a RAC environment

First of all, make sure you’ve been provided shared disks by system administrators. If you’re using asmlib, create disks using oracleasm as follows:

[oracle@node1 ~] sudo /etc/init.d/oracleasm createdisk RLOGS /dev/mapper/mpath6

Then run scan and query new disks from other nodes:

[oracle@node2 ~]$ sudo /etc/init.d/oracleasm scandisks
[oracle@node2 ~]$ sudo /etc/init.d/oracleasm listdisks

After having your disk added to the list, query it from /dev/oracleasm/disks folder:
[oracle@node1 ~]$ ls -ltr /dev/oracleasm/disks/

If you got the disk, then everything is ok. Now you can connect to the ASM instance and create a diskgroup. For this, get the name of the ASM instance from OS and connect to it:
ps -ef | grep ASM
export ORACLE_SID=+ASM1
sqlplus / as sysdba

SQL> create diskgroup RLOGS external redundancy
2 DISK ‘/dev/oracleasm/disks/RLOGS’;

Diskgroup created.

SQL>

If it’s not mounted on the rest nodes, connect to those nodes and mount it manually:
SQL>alter diskgroup RLOGS mount;

And also make sure the name of the disk group is added to asm_diskstrings parameter at ASM instance

After having your disk group created, create new redo log group per thread, and remove on redo log group from the same thread which has INACTIVE status
You can check the status of any group based on thread using V$LOG view:

SELECT group#, thread#, status
FROM v$log
WHERE thread# = 2
ORDER BY 2, 1

GROUP# THREAD# STATUS
———- ———- —————-
20 2 CURRENT
21 2 INACTIVE

Now add new group to the same thread (thread#=2) and check v$logfile view:

SQL> ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 22 (‘+RLOGS’) SIZE 50M

SELECT lf.group#, lf.MEMBER
FROM v$logfile lf, v$log l
WHERE lf.group# = l.group# AND l.thread# = 2

GROUP# MEMBER
———- ——————————————————-
20 +OLD_LOG/PROD/onlinelog/group_20.259.815656575
21 +OLD_LOG/PROD/onlinelog/group_21.260.815656475
22 +RLOGS/PROD/onlinelog/group_22.260.815656475

Now you can safely delete INACTIVE redo log group from the same thread:

SQL> ALTER DATABASE DROP LOGFILE GROUP 21;

Perform above mentioned steps one by one for each thread.

Each time switch log file and make sure the new redo log member is used. Moreover, try to query both sql commands that are written above after each time you add and drop redo log group to see if it’s added/deleted successfully.

Today, while installing RAC, the installation began to wait at “Performing remote operations” step. I’ve tried to check log files, but there was no information written

Then I tried to trace the currently running process, but didn’t get enough information about the wait. At last, I checked both servers and found firewall running. After stopping firewall, (stopping, deinstalling and starting the installation again) the setup was able to pass that step and copied the Grid home files to the remote server

Today, while checking both V$PROCESS and V$SESSION view, I realized that there’re some processes that are missing from V$SESSION view:

SQL> select count(*) from v$process where addr not in (select paddr from v$session);

 COUNT(*)

———-
120

How can a process be created without having any corresponding entry in V$SESSION view?

I started to check those processes on OS level. I got SPID of one of those processes and firstly tried to debug it using ORADEBUG from sqlplus:

sqlplus / as sysdba

ORADEBUG SETOSPID 3422
ORADEBUG TRACEFILE_NAME;

By getting the trace file name, I’ve checked the file but didn’t find any helpful message inside.

Then I decided to debug it on OS level. As the OS was HP-UX, I used truss executable to trace the unix system calls:

/usr/local/bin/truss -o truss.out -vall -laefdD -p 3422

And checked the output again:

tail -f truss.out
( Attached to process 18997 (“oraclePRODDB (LOCAL=NO)”) [64-bit] )
18997/9934051: 18446744073.7095 read(16, 0x600000000021a616, 2064) [sleeping]

This output wasn’t helpful (however, Tanel uses pfiles executable to get all opened files in the process by getting the first parameter of the read function – which is 16) , so I decided to use the next executable – lsof – which lists the opened files by the process:

/usr/local/bin/lsof -p 3422

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
oracle 20472 oracle cwd DIR 64,0×4 2048 3660 /home/oracle/product/10g/dbs
oracle 20472 oracle txt REG 64,0×4 284477360 101589 /home/oracle/product/10g/bin/oracle
oracle 20472 oracle mem REG 64,0×4 5337208 25311 /home/oracle/product/10g/lib/libnnz10.so
oracle 20472 oracle mem REG 64,0×7 21281 27339 /usr/lib/tztab
oracle 20472 oracle mem REG 64,0×5 312440 54074 /opt/star-ncf-prod/ep_patch/usr/lib/hpux64/libxti.so.1
oracle 20472 oracle mem REG 64,0×7 4371800 14392 /usr/lib/hpux64/libc.so.1
oracle 20472 oracle mem REG 64,0×7 1649152 91060 /usr/lib/hpux64/libnsl.so.1
oracle 20472 oracle mem REG 64,0×7 67664 91056 /usr/lib/hpux64/libnss_compat.so.1
oracle 20472 oracle mem REG 64,0×4 20041776 8569 /home/oracle/product/10g/lib/libjox10.so
oracle 20472 oracle mem REG 64,0×7 1046480 11645 /usr/lib/hpux64/libpthread.so.1
oracle 20472 oracle mem REG 64,0×7 55088 491 /usr/lib/hpux64/libuca.so.1
oracle 20472 oracle mem REG 64,0×7 706832 91078 /usr/lib/hpux64/libunwind.so.1
oracle 20472 oracle mem REG 64,0×7 6444800 91068 /usr/lib/hpux64/libm.so.1
oracle 20472 oracle mem REG 64,0×7 78168 90961 /usr/lib/hpux64/libdl.so.1
oracle 20472 oracle mem REG 64,0×7 89208 91053 /usr/lib/hpux64/libnss_dns.so.1
oracle 20472 oracle mem REG 64,0×7 44832 3881 /usr/lib/hpux64/librt.so.1
oracle 20472 oracle mem REG 64,0×4 199296 8277 /home/oracle/product/10g/lib/libdbcfg10.so
oracle 20472 oracle mem REG 64,0×4 134552 13143 /home/oracle/product/10g/lib/libocrutl10.so
oracle 20472 oracle mem REG 64,0×4 533792 13142 /home/oracle/product/10g/lib/libocrb10.so
oracle 20472 oracle mem REG 64,0×4 731352 25330 /home/oracle/product/10g/lib/libocr10.so
oracle 20472 oracle mem REG 64,0×4 1296848 25478 /home/oracle/product/10g/lib/libhasgen10.so
oracle 20472 oracle mem REG 64,0×4 315008 13264 /home/oracle/product/10g/lib/libskgxp10.so
oracle 20472 oracle mem REG 64,0×7 1163152 90960 /usr/lib/hpux64/dld.so
oracle 20472 oracle mem REG 64,0×7 184592 90962 /usr/lib/hpux64/uld.so
oracle 20472 oracle 0u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 1u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 2u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 3u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 4u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 5u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 6u CHR 3,0×2 0t0 74 /dev/null
oracle 20472 oracle 7u IPv4 0xe0000001cbfe0900 0t0 TCP *:* (IDLE)
oracle 20472 oracle 8u REG 64,0×4 849408 13244 /home/oracle/product/10g/rdbms/mesg/oraus.msb
oracle 20472 oracle 14u REG 64,0×4 268644352 79279 /home/oracle/product/10g/cdata/localhost/local.ocr
oracle 20472 oracle 27u IPv4 0xe0000001c34f0740 0t3603 TCP ***.***.***.***:1521->name.ofthehost.com:4934 (ESTABLISHED)
oracle 20472 oracle 28w FIFO 0xe00000017b3ba748 0t0 184090157
oracle 20472 oracle 29r FIFO 0xe0000001f99d3e48 0t0 184090158

At the end of the output, I saw an IP address and the host name where the process were connecting. After checking that server, I saw that it was opening a connection to the database and created a process on OS. Then it wasn’t able to create a session and didn’t detect the dead process and kill it. So it remained in the V$PROCESS view and on the OS as a process and was missing in the V$SESSION view.

I killed all missing sessions from OS level and rebooted the server which was opening connections to the database. It seems it has memory or network issues, so I asked sysadmins to check it

You can also check the following metalink note and learn how you can enable Dead Connection Detection (DCD)

A discussion of Dead Connection Detection, Resource Limits, V$SESSION, V$PROCESS and OS processes [ID 601605.1]

My friend and former ACE Director Porus Homi Havewala’s new book on Oracle Enterprise Manager 12c Cloud Control, is now available, this is the First published EM 12c Cloud Control book in the world.

If you are interested in learning about the capabilities of Enterprise Manager 12c, please have a look. Electronic copies are available too.

http://www.packtpub.com/oracle-enterprise-manager-12c-cloud-control/book

Oracle Enterprise Manager 12c Cloud Control: Managing Data Center Chaos

Today, when I was using BULK COLLECT to fetch some rows to the collection, I got the following error:

ORA-06502: PL/SQL: numeric or value error: Bulk Bind: Truncated Bind

I checked my code and didn’t find anything special that can cause this error. After investigating a while, I found that the row which has length more than 2000 bytes causes that error. I checked my collection and saw that this row was defined as – dbms_sql.varchar2_table

What do you think, how it was defined in the package level (DBMS_SQL)? I was thinking it was defined as “VARCHAR2(4000)“, but it was “VARCHAR2(2000)“.

type Varchar2_Table is table of varchar2(2000) index by binary_integer;

So I declared new variable, changed the declaration of my collection as follows and it worked:

type Varchar4000_Table is table of varchar2(4000) index by binary_integer;

TYPE trec  IS RECORD (

…..

…..

column Varchar4000_Table;

)

Do you know you get database failure each time when you create a new database with DBCA? 

Today, while testing troubleshooting Data Recovery Advisor (DRA) I saw an interesting issue. When I queried GV$IR_FAILURE view on the newly created database, I got some rows … This view is used and updated by DRA and Oracle background processes. When Oracle encounter an error such as data bock corruption, missing datafile or controlfile and etc. , it automatically updates the mentioned view. But how it’s possible to get an error on newly created database when no operation is performed.

Here’s the output from this view:

SQL> set linesize 150

SQL> col description format a35

SQL> col impacts format a30

SQL> select failure_id, to_char(time_detected,’dd.mm.yyyy hh24:mi:ss’) time_detected, modified, description, impacts, priority, status from GV$IR_FAILURE;

FAILURE_ID TIME_DETECTED       MODIFIED  DESCRIPTION                         IMPACTS                        PRIORITY STATUS

———- ——————- ——— ———————————– —————————— ——– ————

2 28.08.2012 14:44:57           Control file needs media recovery   Database cannot be opened      CRITICAL OPEN

And in one case I got fou 4 failures (one was child failure)

Just want to mention that if you want to know the source of the view “GV$IR_FAILURE”, query v$fixed_view_definition view:

SQL> col view_definition format a70

SQL>  select * from v$fixed_view_definition

where view_name=’GV$IR_FAILURE’;

VIEW_NAME                      VIEW_DEFINITION

—————————— ———————————————————————-

GV$IR_FAILURE                  select inst_id,       id,       pid,       cid_count,       clsname,

cast(ctime as date),       cast(mtime as date),       fdg_msg,

damage_msg,       decode(priority, 0, ‘CRITICAL’,

1, ‘HIGH’,                         2, ‘LOW’,

‘UNKNOWN’),       decode(status, 0, ‘OPEN’,

1, ‘CLOSED’,                       2, ‘UNDER-REPAIR’,

‘UNKNOWN’) from x$dbkfdg    where type = 1 and clsnameid = 2

and bitand(FLAGS,3) = 0

So the original table is – x$dbkfdg

Now let’s go back to our issue. After getting this error on GV$IR_FAILURE view, I logged in to RMAN and run LIST FAILURE command to get the list of failures. Although the failure is logged as “CRITICAL” and the status is “OPEN”, the command returned no result:

RMAN> list backup;

using target database control file instead of recovery catalog

specification does not match any backup in the repository

RMAN>

Then I checked detected time of the failure from the view – GV$IR_FAILURE:

SQL> select to_char(time_detected,’dd.mm.yyyy hh24:mi:ss’) time_detected from GV$IR_FAILURE;

TIME_DETECTED

——————-

28.08.2012 14:44:57

So this means that at the mentioned time we got a critical problem with control file. Then I checked the log file of the database creation for the same time period:

cd /u01/home/oracle/cfgtoollogs/dbca/testdb

vi trace.log

[Thread-55] [ 2012-08-28 14:44:10.243 AZST ] [CloneDBCreationStep.executeImpl:448]  createCTLSql=Create controlfile reuse set database “mydb”

[Thread-55] [ 2012-08-28 14:44:20.299 AZST ] [CloneDBCreationStep.executeImpl:470]  Shutdown database

[Thread-55] [ 2012-08-28 14:44:20.307 AZST ] [CloneDBCreationStep.executeImpl:492]  Startup ……nomount……

[Thread-55] [ 2012-08-28 14:44:49.702 AZST ] [CloneDBCreationStep.executeImpl:511]  Enabling restricted session.

[Thread-55] [ 2012-08-28 14:44:52.062 AZST ] [CloneDBCreationStep.executeImpl:513]  alter database “mydb” open resetlogs;

[Thread-55] [ 2012-08-28 14:45:07.405 AZST ] [CloneDBCreationStep.executeImpl:521]  Removing existing services from sourcedb seeddata

[Thread-55] [ 2012-08-28 14:45:07.733 AZST ] [CloneDBCreationStep.executeImpl:526]  Renaming globale_name

So the error came from “ALTER DATABASE “MYDB” OPEN RESETLOGS” command.

After running ADVISE FAILURE command and the querying the view again, you’ll see that the failure is “solved”, status is CLOSED and MODIFIED columns is updated to the current time:

SQL> select failure_id, to_char(time_detected,’dd.mm.yyyy hh24:mi:ss’) time_detected, to_char(modified,’dd.mm.yyyy hh24:mi:ss’) modified, description, impacts, priority, status from GV$IR_FAILURE;

FAILURE_ID TIME_DETECTED       MODIFIED            DESCRIPTION                         IMPACTS                        PRIORITY STATUS

———- ——————- ——————- ———————————– —————————— ——– ————

         2 28.08.2012 14:44:57 28.08.2012 16:27:06 Control file needs media recovery   Database cannot be opened      CRITICAL CLOSED

Today, after installing Critical Security Patches, the listeners didn’t start up and I got the following error:

-bash-3.2$ lsnrctl status

LSNRCTL for HPUX: Version 10.2.0.4.0 – Production on 31-JUL-2012 14:24:51

Copyright (c) 1991, 2007, Oracle. All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
HPUX Error: 239: Connection refused

When I tried to start it, I got the following error:

-bash-3.2$ lsnrctl start LISTENER

LSNRCTL for HPUX: Version 10.2.0.4.0 – Production on 31-JUL-2012 14:25:01

Copyright (c) 1991, 2007, Oracle. All rights reserved.

Starting /home/oracle/product/10gR2/bin/tnslsnr: please wait…

TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
HPUX Error: 32: Broken pipe

Actually I hate these type of unknown and unexplainable errors. After checking metalink notes, I didn’t get any useful information.

As I got the error after patch installation, I tried to relink tns again:

-bash-3.2$ cd $ORACLE_HOME/network/lib

-bash-3.2$ make -f ins_net_server.mk.32 install

Then I started the listener and succeeded

I strongly believe if you’ve deployed agents a coupled of times, you’ve encountered the following error after the installation:

[oracle@dg2 bin]$ ./emctl upload agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet..

The first thing that should be done is checking the log file. Switch to the following directory and tail the log file:

cd /u01/oracle/product/10.2.0/agent10g/sysman/log
tail -f emagent.trc

Here’s the output from the log file:

2012-07-26 03:46:33 Thread-51022736 ERROR pingManager: nmepm_pingReposURL: Did not receive a response header from repository
2012-07-26 03:47:08 Thread-51022736 ERROR pingManager: nmepm_pingReposURL: Did not receive a response header from repository
2012-07-26 03:47:08 Thread-51022736 ERROR pingManager: nmepm_pingReposURL: Did not receive a response header from repository

I checked the status of the agent and saw that it’s running :

[oracle@dg2 bin]$ ./emctl status agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.3.0
OMS Version : 10.2.0.3.0
Protocol Version : 10.2.0.2.0
Agent Home : /u01/oracle/product/10.2.0/agent10g
Agent binaries : /u01/oracle/product/10.2.0/agent10g
Agent Process ID : 27772
Parent Process ID : 27755
Agent URL : http://dg2:3872/emd/main/
Repository URL : http://dg1:4889/em/upload/
Started at : 2012-07-26 03:42:04
Started by user : oracle
Last Reload : 2012-07-26 03:42:04
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 21
Size of XML files pending upload(MB) : 3.25
Available disk space on upload filesystem : 58.86%
Last attempted heartbeat to OMS : 2012-07-26 03:45:22
Last successful heartbeat to OMS : unknown
—————————————————————
Agent is Running and Ready
[oracle@dg2 bin]$
[oracle@dg2 bin]$

So I decided to clear unnecessary files that can’t be uploaded to the Grid Control under the following directories:

$AGENT_HOME/sysman/emd/collection

$AGENT_HOME/sysman/emd/state

$AGENT_HOME/sysman/emd/recv

$AGENT_HOME/sysman/emd/upload

$AGENT_HOME/sysman/emd/lastupld.xml

Then stopped the agent, secured it, cleared unnecessary files again with clearstate parameter, started the agent and uploaded.

[oracle@dg2 bin]$ ./emctl stop agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Stopping agent …. stopped.

 
[oracle@dg2 bin]$ ./emctl secure agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Enter Agent Registration password :
Agent is already stopped… Done.
Securing agent… Started.
Requesting an HTTPS Upload URL from the OMS… Done.
Requesting an Oracle Wallet and Agent Key from the OMS…
Done.
Check if HTTPS Upload URL is accessible from the agent… Done.
Configuring Agent for HTTPS in CENTRAL_AGENT mode… Done.
EMD_URL set in /u01/oracle/product/10.2.0/agent10g/sysman/config/emd.properties
Securing agent… Successful.
[oracle@dg2 bin]$

 
[oracle@dg2 bin]$ ./emctl clearstate agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
EMD clearstate completed successfully
[oracle@dg2 bin]$ ./emctl start agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Starting agent ….. started.

 
[oracle@dg2 bin]$ ./emctl upload agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
EMD upload completed successfully

 
[oracle@dg2 bin]$ ./emctl upload agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
EMD upload completed successfully
[oracle@dg2 bin]$

Then I checked Grid Control and made sure that the agent is working

Today, while installing agent I failed again. The reason was that I was trying to install 10gR2 agent on OEL5.5 which is not supported. During the installation I got the following error:

./agentDownload.linux_x64 -b /u01/oracle/product/10.2.0/ -m dg1 -r 4889
Performing check for CertifiedVersions
Checking operating system requirements …
Expected result: One of redhat-3,redhat-4,SuSE-8,SuSE-9,SuSE-10
Actual Result: redhat-5.5
Check complete. The overall result of this check is: Failed <<<<
Check complete: Failed <<<<
Problem: Oracle EnterPrise Manager 10gR2 is not certified on the current operating system.
Recommendation: Make sure you are installing the software on the correct platform.

PrereqChecks complete

Completed with Status=255

[oracle@dg2 tmp]

Although I’ve changed /etc/redhat-release file to “redhat-4” I got the above error. So I decided to edit the agentDownload script and make it to bypass the PreRequisite checks

I opened the file and find that the runInstaller is called ExecAgentInstall() procedure. I get down to the line 269 and edit it as follows:

Before: ignoreSysPrereqs=${silentPatchCheck:+”-ignoreSysPrereqs”}

After: ignoreSysPrereqs=”-ignoreSysPrereqs use_prereq_checker=false “

Here’s the output from the documentation related with this parameter:

Then I saved the file and run the first command again. Fortunately it bypassed the PreRequisite check and installation finished successfully

Installation in progress (Thu Jul 26 03:34:46 AZST 2012)
……………………………………………………… 39% Done.
……………………………………………………… 78% Done.
Install successful