STANDBY DATABASE + Error is 16191.

Last night I was demonstrating to my friends about standby database creation and its administration using Oracle10g, Release2 (10.2.0.1) on Windows XP Operating System and at the end of standby database creation (creating on the same host) found out that log information is not shipping from the primary database to the standby location despite setting all the required parameters correctly on primary and standby.

After wasting an hour time, finally had a look at primary database alert log and found the following error messages:

====
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191
It may be necessary to define the DB_ALLOWED_LOGON_VERSION
initialization parameter to the value "10". Check the
manual for information on this initialization parameter.
------------------------------------------------------------
Thu Jun 07 01:40:44 2007
Errors in file d:\oradata\ocm\bdump\ocm_arc0_876.trc:
ORA-16191: Primary log shipping client not logged on standby

PING[ARC0]: Heartbeat failed to connect to standby 'SDB'. Error is 16191.

Errors in file d:\oradata\ocm\bdump\ocm_arc0_876.trc:
ORA-01031: insufficient privileges
====

When searching in the metalink about 'DB_ALLOWED_LOGON_VERSION' found the following

Note: Bug 2981553, which is implemented in 10.1.0.2, removes the parameter db_allowed_logon_version. This is replaced by the sqlnet.ora parameter called sqlnet_allowed_logon_version.

As per note, setting SQLNET_ALLOWED_LOGON_VERSION doesn't solve the issues.

Looking more close of the error message, I then, realized that I gave the different password to standby database password file than the primary password file.

After recreating the password file of standby database with the similar password of primary database password file, automatically log shipping started transmitting from primary to standby.

Well, I don't realized that just having different passwords to primary and standby parameter files will leads to this problem.

The other change I noted in the 10g Standby versus 9i is that, in 9i, when you say simply say startup Oracle throws an error saying the the controlfile is standby, where as in 10g, upon using startup, it automatically opens the standby database in READ ONLY state. Umm...

Happy reading,

Jaffar

--

Mystry resolved (RMAN/VERITAS +ORA-19511)

Since last month, all of sudden our data warehouse database(size 1.7tb) RMAN full database backup started failing with the following error:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch09 channel at 06/01/2007 13:15:15ORA-19502: write error on file "OFDMP_Dly_BACKUP_DBFFri010607krij6c1h_1_1", blockno 10731521 (blocksize=1024)ORA-27030: skgfwrt: sbtwrite2 returned errorORA-19511: Error received from media manager layer, error text:VxBSASendData: Failed with error:Server Status: Communication with the server has not been iniatated or the server status has not been retrieved fromthe server.

Looking into the above error message it is clearly understood that the problem is definitely relates to Media Manager as the communication between the channels from the client side to the netbackup server side loosing the communication after sometime.

We reported this problem with Veritas (vendor) and their support staff were here to run the back in trace mode and took those trace file for further investigations.

Meanwhile, I have also opened a p1 TAR with Oracle Support, they simply said that the problem is moreover related to Veritas and we should contact Veritas to resolve the issue.
It was like not having backups for more than 1 month (luckily we have DRC in sync with the primary).

Surprisingly, backing up a single, set of datafiles or tablespace was working fine, the problem is only when backing up entire database. Once it finishes 1 hr of time, backup was failing with the above stated error message.

Upon Veritas recommendation we gave look at NET_BUFFER_SZ value in the Netbackup Server and the Client side(the database server configured as netbackup client, since there is no server edition available on HP Superdom, we were using Netbackup client) we found out that the client and the server has different values in the NET_BUFFER_SIZE file.

We changed the value of this file in the client side to match with the server side file and backup started working fine.

As per general recommendation, NET_BUFFER_SZ file value on the client should be <= to the value on the server side.

Now the backup is running without any issues. No one has clue how this file got changed the values.

Happy Reading,

Jaffar

How to get a stack trace from a CORE file

We might have of come across of many situation where Oracle create CORE file, usually when the process has attempted perform something which Operating System does not like.

 

Since the CORE file is in binary format, it is difficult to read the contents of it and upon contacting Oracle Support, they might ask you to get the stack trace from a CORE file which will be helpful for their investigation.

 

I will shows you couple of steps using which you can produce the stack trace for a CORE file, this example is only for UNIX platforms.

 

By default, 'core' file will be generated under the directory mentioned with CORE_DUMP_DEST init parameter, or  in $ORACLE_HOME/dbs or in the directory from where you run the applicationf.

 

First step is to know which executable (program) cause core file genration.

                

               file core (make sure you are in the core file directory and execute the command).

 

Second step is to get the stack trace from the core file using one of the following command which supports to your OS:

 

                  dbx, xdb, gdb, dde, sdb, adb, debug, gdb

 

For me on HPUX Superdom, 'adb' command works.

 

Example:

                        script mystack

                        adb $ORACLE_HOME/bin/program core

                             $c

                             $q

                          exit

 

When the above commands are executed, stack trace of the core file will be in 'mystack' file and you can upload this file to the Oracle Support for their investigation.

 

 

Referecnes:

 

Note : 1812.1 Getting a Stack Trace from CORE file

Turned to 34 years today

Ummm... today is my birthday and I have turned to 34 years. Early thirties!!! myself feeling bit older now.

I usually don't celebrate my birthday, I treat this day as a normal day. Birthday has a funny meaning in Urdu. In Urdu it is called as 'Saal Ghira', which means, a year fallen.

 

I have said my kids to celebrate my birthday as their birthday by distributing sweets and chocolates in their respective classes. They were born in July and we usually go to India during July on annual vacation and celebrate their birthday as a small family function. This time we are going to India in September not in July as we use to. Since they know that we are not going to India, they agreed to celebrate their birthday today. We brought new clothes and gifts for them to feel happy.

 

My wife and kids planned a surprise party at home for me and a cake has been ordered from french corner shop, my favorite biriyani being prepared for lunch. Well, they don't that I know their surprise!. But, I kept it as a secrete that I don't about their surprise!

 

Thanks for reading,

 

Jaffar

RAC - finally into production

It was like a long cherished dream become true when we successfully moved one of our highly OLTP database from the single node to 2 node RAC on AIX platofrm.

We initially thought of implementing extended RAC between head office and DRC sites. At the last moment we had to drop the plan due to some non-technical reasons. Rather, we simulated the extend RAC in the head office using two different storages.

They were couple problems faced during the implementation, 1) when we changed the default port from 1521 to another, there was an issue with Enterprise Manager Console, as it was still looking for the default port (1521). We couldn't successes in the first attempt of running 'emca -reconfig dbconsloe db' command. However, running the same command second solve the issue. 2) AMS Listener entry missing. After some research we found out the the LOCAL LISTENER entry missing from the ASM initialization parameter. Setting the LOCAL LISTENER parameter in the ASM init file solve the issue. These were related with Enterprise Manager (console), however, the database was functioning without any issues.

Although, we have 2 node RAC, we thought not to distribute the load equally on the two nodes, rather, 90% on one node and 10% on another node, (of course missing failover connection concept here). This was simply to monitor the behavior of the RAC. If we finish the week successfully without issues, we will then implement the load distribution equally to the both nodes.

Once we get the stable line between the head office and DRC site (which a kilometer away from the head office), we will another third node.

I should say that the RAC installation doesn't fancy me was I initially thought. The major challenges lays of maintaining RAC environment and survival commands to quickly come from problematic issues. I am sure that experience will teach us.

If everything goes will, Mr. Murali Vallath will be presenting 4 day RAC in-house training at our premises in coming months. Bit excited about this opportunity.

Thursday and Friday are off (week-end) here in ME (Saudi Arabia). The database will be in live mode this Saturday.

I will keep posting about my experience of RAC.

Happy reading,

Jaffar