Today, after installing Critical Security Patches, the listeners didn’t start up and I got the following error:

-bash-3.2$ lsnrctl status

LSNRCTL for HPUX: Version 10.2.0.4.0 – Production on 31-JUL-2012 14:24:51

Copyright (c) 1991, 2007, Oracle. All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
HPUX Error: 239: Connection refused

When I tried to start it, I got the following error:

-bash-3.2$ lsnrctl start LISTENER

LSNRCTL for HPUX: Version 10.2.0.4.0 – Production on 31-JUL-2012 14:25:01

Copyright (c) 1991, 2007, Oracle. All rights reserved.

Starting /home/oracle/product/10gR2/bin/tnslsnr: please wait…

TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
HPUX Error: 32: Broken pipe

Actually I hate these type of unknown and unexplainable errors. After checking metalink notes, I didn’t get any useful information.

As I got the error after patch installation, I tried to relink tns again:

-bash-3.2$ cd $ORACLE_HOME/network/lib

-bash-3.2$ make -f ins_net_server.mk.32 install

Then I started the listener and succeeded

I strongly believe if you’ve deployed agents a coupled of times, you’ve encountered the following error after the installation:

[oracle@dg2 bin]$ ./emctl upload agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet..

The first thing that should be done is checking the log file. Switch to the following directory and tail the log file:

cd /u01/oracle/product/10.2.0/agent10g/sysman/log
tail -f emagent.trc

Here’s the output from the log file:

2012-07-26 03:46:33 Thread-51022736 ERROR pingManager: nmepm_pingReposURL: Did not receive a response header from repository
2012-07-26 03:47:08 Thread-51022736 ERROR pingManager: nmepm_pingReposURL: Did not receive a response header from repository
2012-07-26 03:47:08 Thread-51022736 ERROR pingManager: nmepm_pingReposURL: Did not receive a response header from repository

I checked the status of the agent and saw that it’s running :

[oracle@dg2 bin]$ ./emctl status agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.3.0
OMS Version : 10.2.0.3.0
Protocol Version : 10.2.0.2.0
Agent Home : /u01/oracle/product/10.2.0/agent10g
Agent binaries : /u01/oracle/product/10.2.0/agent10g
Agent Process ID : 27772
Parent Process ID : 27755
Agent URL : http://dg2:3872/emd/main/
Repository URL : http://dg1:4889/em/upload/
Started at : 2012-07-26 03:42:04
Started by user : oracle
Last Reload : 2012-07-26 03:42:04
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 21
Size of XML files pending upload(MB) : 3.25
Available disk space on upload filesystem : 58.86%
Last attempted heartbeat to OMS : 2012-07-26 03:45:22
Last successful heartbeat to OMS : unknown
—————————————————————
Agent is Running and Ready
[oracle@dg2 bin]$
[oracle@dg2 bin]$

So I decided to clear unnecessary files that can’t be uploaded to the Grid Control under the following directories:

$AGENT_HOME/sysman/emd/collection

$AGENT_HOME/sysman/emd/state

$AGENT_HOME/sysman/emd/recv

$AGENT_HOME/sysman/emd/upload

$AGENT_HOME/sysman/emd/lastupld.xml

Then stopped the agent, secured it, cleared unnecessary files again with clearstate parameter, started the agent and uploaded.

[oracle@dg2 bin]$ ./emctl stop agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Stopping agent …. stopped.

 
[oracle@dg2 bin]$ ./emctl secure agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Enter Agent Registration password :
Agent is already stopped… Done.
Securing agent… Started.
Requesting an HTTPS Upload URL from the OMS… Done.
Requesting an Oracle Wallet and Agent Key from the OMS…
Done.
Check if HTTPS Upload URL is accessible from the agent… Done.
Configuring Agent for HTTPS in CENTRAL_AGENT mode… Done.
EMD_URL set in /u01/oracle/product/10.2.0/agent10g/sysman/config/emd.properties
Securing agent… Successful.
[oracle@dg2 bin]$

 
[oracle@dg2 bin]$ ./emctl clearstate agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
EMD clearstate completed successfully
[oracle@dg2 bin]$ ./emctl start agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Starting agent ….. started.

 
[oracle@dg2 bin]$ ./emctl upload agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
EMD upload completed successfully

 
[oracle@dg2 bin]$ ./emctl upload agent
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
EMD upload completed successfully
[oracle@dg2 bin]$

Then I checked Grid Control and made sure that the agent is working

Today, while installing agent I failed again. The reason was that I was trying to install 10gR2 agent on OEL5.5 which is not supported. During the installation I got the following error:

./agentDownload.linux_x64 -b /u01/oracle/product/10.2.0/ -m dg1 -r 4889
Performing check for CertifiedVersions
Checking operating system requirements …
Expected result: One of redhat-3,redhat-4,SuSE-8,SuSE-9,SuSE-10
Actual Result: redhat-5.5
Check complete. The overall result of this check is: Failed <<<<
Check complete: Failed <<<<
Problem: Oracle EnterPrise Manager 10gR2 is not certified on the current operating system.
Recommendation: Make sure you are installing the software on the correct platform.

PrereqChecks complete

Completed with Status=255

[oracle@dg2 tmp]

Although I’ve changed /etc/redhat-release file to “redhat-4” I got the above error. So I decided to edit the agentDownload script and make it to bypass the PreRequisite checks

I opened the file and find that the runInstaller is called ExecAgentInstall() procedure. I get down to the line 269 and edit it as follows:

Before: ignoreSysPrereqs=${silentPatchCheck:+”-ignoreSysPrereqs”}

After: ignoreSysPrereqs=”-ignoreSysPrereqs use_prereq_checker=false “

Here’s the output from the documentation related with this parameter:

Then I saved the file and run the first command again. Fortunately it bypassed the PreRequisite check and installation finished successfully

Installation in progress (Thu Jul 26 03:34:46 AZST 2012)
……………………………………………………… 39% Done.
……………………………………………………… 78% Done.
Install successful

I’ve wrongly installed 64 bit Oracle 10g Grid Control installation on 32 bit Linux. The installation went smooth, but while trying to deploy an agent on the different host, I got the following error:

Resolving dg1… 192.168.153.128

Connecting to dg1|192.168.153.128|:4889… connected.
HTTP request sent, awaiting response… 200 OK
Length: 19533 (19K)

[text language=”/plain”][/text]

Saving to: `agent_download.rsp’

100%[======================================>] 19,533 –.-K/s in 0s

2012-07-25 23:43:32 (91.2 MB/s) – `agent_download.rsp’ saved [19533/19533]

Finished Downloading with Status=0
Downloaded response with status=0
Downloading Oracle Installer …
–2012-07-25 23:43:32– http://dg1:4889/agent_download/10.2.0.3.0/linux/oui/oui_linux.jar
Resolving dg1… 192.168.153.128
Connecting to dg1|192.168.153.128|:4889… connected.
HTTP request sent, awaiting response… 404 Not Found
2012-07-25 23:43:32 ERROR 404: Not Found.

After checking the output, I saw the folder – “…/10.2.0.3.0/linux/oui…..”

I checked the folder at the Grid installation site and and as it was 64 bit installation, the folder was names “linux_x64”

$ORACLE_HOME/sysman/agent_download/10.2.0.3.0/linux_x64

I renamed the folder and run the installation command again. But I got the same error. When checking the output, I saw that it’s looking for oui_linux.jar file which is probably “linux_x64.jar” at the Grid site. I opened agentDownload.linux_x64 file which I copied from Grid site and checked for linux_x64 keyword. I changed line 362:

“Linux.x86_64”) os =linux_x64;;

to

“Linux.x86_64”) os =linux;;

And renamed oui_linux_x64.jar file to oui_linux.jar

And the installation succeeded

I would strongly reccomend you to install correct version of the Grid Control and Oracle Software. If you’re running 32 bit OS, then install 32 bit Software and do the same for 64 bit installation

Today I got my second internal error while starting the database.

[php]

Starting up ORACLE RDBMS Version: 10.2.0.1.0.
Errors in file /opt/oracle/10.2/admin/ORCL/udump/ORCL_ora_535.trc:
ORA-00600: internal error code, arguments: [keltnfy-ldmInit], [46], [1], [], [], [], [], []
USER: terminating instance due to error 600
Instance terminated by USER, pid = 535

[/php]

I checked all parameters at spfile, but everything seems to be ok. After checking metalink note, I find out that this is because Oracle can’t get host information.

Startup Database Produces ORA-00600: [Keltnfy-Ldminit] [ID 336447.1]

As I already changed the hostname, it wasn’t changed at /etc/hosts file. After changing /etc/hosts file and providing correct hostname, the database opened

Today, while configuring a standby database, I got the following internal error “during spfile creation”

[php]ORA-00600: internal error code, arguments: [kspsetpao1], [1753], [1700], [*], [user_dump_dest], [33], [], [][/php]

Actually it’s the first time I get an “internal error” while creating spfile. Not a message, not a warning, but internal error

After checking spfile I saw that I forgot to close the quote mark

[php]*.log_archive_dest_2=’service=standby[/php]

I added the quote mark and the problem solved.

Last week I’ve done a migration from 32 bit to 64 bit on Linux server. Just would like to share the experience and steps that should be performed during the migration

First of all, I would advise you to check Oracle Support for existence document. I’ve found the following document very useful:

How To Migrate a Database From Linux x86 ( 32-bit ) To Linux Itanium 64-bit (IA64) [ID 553868.1]

Before starting the migration (or any critical operation) take backup of the database and make sure you’ve tested the migration on the test environment (It’s very easy. Just create a new database on the 32bit OS, move it to 64 bit OS and practice the migration)

The first step is to get the list of all datafiles (dba_data_files), redo log files (v$logfile) and control files (v$controlfile).

Then, take backup of parameter file and extract creation script of the control file:

[php]

SQL> create pfile=’/tmp/pfile_mydb.ora’ from spfile;

SQL> alter database backup controlfile to trace as ‘/tmp/controlfile_script.dat’;

[/php]

The shutdown the database, copy all datafiles, redo log files, new parameter file and control file creation script to the new server. Make necessary changes to the parameter file (show the new destination for the controlfiles) and to the control file creation script (change the file names to the new folder)

If everything ok, start the database in NOMOUNT mode, and create the controlfiles by running the script:

[php]

SQL> startup nomount pfile=’/tmp/pfile_mydb.ora’;

SQL> create spfile from pfile=’/tmp/pfile_mydb.ora’;

SQL> startup nomount force;

[/php]

After the controlfile created successfully open the database. If you’ll try to open the database in normal mode, you’ll get some internal errors which is very scare to get at midnight on the production database

I got the following internal error after trying to open the database in normal mode as metalink note suggested (however, I was wondering and thinking to NOT open it in normal mode)

[php]

ORA-00604: error occurred at recursive SQL level 1
ORA-06553: PLS-801: internal error [56319]

Error 0 in kwqmnpartition(), aborting txn
Error 0 in kwqmnpartition(), aborting txn
Error 0 in kwqmnpartition(), aborting txn
Error 0 in kwqmnpartition(), aborting txn
Shutting down instance: further logons disabled
Wed Jul 4 02:42:30 2012
Stopping background process QMNC
Wed Jul 4 02:42:30 2012
Stopping background process CJQ0
Wed Jul 4 02:42:32 2012
Stopping background process MMNL
Wed Jul 4 02:42:33 2012
Stopping background process MMON
Wed Jul 4 02:42:34 2012
Errors in file /home/oracle/admin/MYDB/udump/mydb_ora_3133.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-06553: PLS-801: internal error [56319]

[/php]

And instance closed automatically …

I didn’t payed attention the that error (however panicked a little )

I’ve tried to open it again with UPGRADE option which succeeded

[php]

SQL> startup upgrade;

[/php]

There’s a metalink note on the above mentioned error:

ORA-00600 [17093] and ORA-00600 [kkmendsel-pin] Errors Continuously [ID 756592.1]

Actually I didn’t checked it when I got the error, just bypassed it and tried to open the database

So I suggest you to open with with RESETLOGS and UPGRADE option at once using the following command :

[php]

SQL> alter database open resetlogs upgrade;

[/php]

After the database opened successfully, you need to run the following sql files:

[php]

SQL> @$ORACLE_HOME/rdbms/admin/utlirp.sql;

SQL> @$ORACLE_HOME/rdbms/admin/utlrp.sql ;

[/php]

Both scripts are different. The first one will INVALIDATE all objects at the database, and the second one VALIDATEs.

After both scripts completed, shutdown the instance and start it in normal mode. Then check if you have any INVALIDE objects

If you’re using OLAP components, (or not using but just installed), you’ll get some internal errors and get some objects INVALIDE. In this case either remove the OLAP objects (if you don’t use them) and install it again, or check the following metalink note :

Migrating OLAP From 32 To 64 Bits [ID 352306.1]

After finishing all, make sure you’ve moved listener.ora and tnsnames.ora file from the old server to the new one and start the listener

I strongly believe you already know it, but would like to share if you’ve forgot and can’t understand why you’re not able to connect to the instance which is already running

[php]-bash-3.2$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.4.0 – Production on Wed Jul 11 09:24:18 2012

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Connected to an idle instance.

SQL> exit

-bash-3.2$ ps -ef | grep smon
oracle 21202 1 0 Nov 8 ? 118:15 ora_smon_MYDB

[/php]

The database is running, but I can’t login to the instance. Let’s check ORACLE_HOME environment variable:

[php]

/home/oracle/product/10g/

[/php]

Do you see the slash in the end of the line? That’s the reason! Let’s set it again without the slash at the end and try again

[php]

-bash-3.2$ export ORACLE_HOME=/home/oracle/product/10g
-bash-3.2$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.4.0 – Production on Wed Jul 11 09:33:39 2012

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 – 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL>

[/php]

Here it is …

Today, one of the instances of 2 node RAC went down with ORA-07445 error. After checking metalink, I found out that this is a bug:

Bug 10422354: ORA-07445 [KFKPROCESSREQ()+429] [SIGSEGV] ON ASM INSTANCE

And it happens after the upgrade of RAC to 10.2.5 patchset. Unfortunately there was no workaround suggested by support. After checking ASM log file again, I find the following error:

ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]

The ASM instance was not mounting. The instance wasn’t able to find the shared storage, so I changed asm_diskstring parameter as follows:

[php]asm_diskstring=’/dev/mapper/*'[/php]

It was a test RAC environment, both ASM instances were using different initialization parameter files (however, both database instances were using the same shared initialization parameter file)

I’ve created new server parameter file from the existing one, put it in the shared folder, changed parameter file of each instance (pfile=’/shared_folder/spfile+ASM.ora’) and started the failed ASM instance. Fortunately, it has started. Then I restarted the second ASM instance to make it use the shared parameter file and both instances were up