How to startup RAC database services automatically

As of Oracle 10gR2 (10.2.0.4 patch set) when RAC database is started with 'srvctl start database -d DBNAME', unfortunately, the associated database services do not startup automatially. Therefore,   services must be started manually after the database startup. It could be painfull in some situations, like, when you have many databases running on RAC with daily or weekly cold backup schedule. When services are not up, clients unable to connect to the respective databsases, in case they use the service name to connect to the database.

A possible workaround is to write  FAN server side callouts. You may download perl scripts, 'Start Services on Instance Up' from http://www.oracle.com/technology/sample_code/products/rac/index.html. Its a smaple perl script which can be used to as a FAN Server Side callout to start services when an instanes up event is received on the node. You need to put the scripts under $CRS_HOME/racg/usrco. The PDF contains the procedure how to deploy the scripts, setting permission and other stuff.

In 11g,  When services are already started on some remote nodes, then the startup of the instance on the local node will autostart the services on it.

References:

ML 416178.1: After Srvctl Start Database, Database Services Will Not Start Up Automatically

http://www.oracle.com/technology/sample_code/products/rac/index.html

http://www.oracle.com/technology/products/database/clustering/pdf/twpracwkldmgmt.pdf

http://download-west.oracle.com/docs/cd/B19306_01/rac.102/b14197/hafeats.htm#sthref374

Happy reading,

Jaffar

99% memory consumption on an Idle RAC nodes

We have been observed that the memory consumption on all RAC nodes(8 node setup) on HPUX Superdom (Itaninum II)  servers are consuming almost 99% memory round the clock.  There are no databases running, just Oracle clusterware was installed on these nodes. We would be soon using this setup for our production, thats our plan.

We initially thought there could be a memory leak which is causing the high memory consumption. Upon opening a TAR for this issue, Oracle support suggested that the memory consumption by the clusterware is pretty normal and there is no high memory consumption by any of Oracle CW processes. Shutting down the cluster in order to test the memory consumption doesn't help either as the memory consumption still was the same.

Then, we thought of investigating from OS point of view and we  found out that the OS kernel parameter 'dbc_max_pct' was set to the default value, i.e. 50%.  The general recommendation on most of the environments is 10%. Thank god, its a dyanmic parameter which can be modified without the shutdown. Upon resetting the value to 10%, the memory consumption drastically down from 99% to 35% on all the nodes.

I have also found a very good Metalink Note (68105.1 : Commonly Misconfigured HP-UX Kernel Parameters) which explains the most commonly misconfigured HP-UX kernal parameters. Dears, you are running any databases on HPUX, I strongly recommend you to have a look at this note and make changes accordingly.

Happy Reading,

Jaffar

Tough time ahead (going to have fun with RAC)

It is going to be a tough this week-end for me, and at the same time very challeging too. Well, this week-end, in simple terms, I can say that I am going to have fun with RAC.

We currently have 4 node develpment and 8 node Production RAC environment at our premises. The development currently running on Oracle 10g R2 with patchset 10.2.0.3, which we are planning to upgrade to 10.2.0.4. My god, there are over 20 databases running across these 4 nodes. Uff.. upgrading cluster, asm and all those databases to 10.2.0.4 patchset.. tough time!

Along with upgrade activity, we also plan the following at the same time:

Changing the IPs of public and interconnect on production.

Moving a RAC database from development to production.

Apart from the above, there are couple of small changes to be done.

I know one time such a big change is not recommended and a very tough task too. Thankfully, we are not really  into production yet,  we have ample of downtime to carry out/complete all these tasks.

Wish me best of luck and I will blog about any issues that I faced during this activity.

Happy reading,

Jaffar

Database migration across platforms and conversion of single instance to RAC

Working in a huge RAC environment, each passing day is a learning day for me. After setting up 4 node RAC on HP UX Itanium 2 Severs, the subsequent task/challenge is to migrate single instance databases (around 16) from AIX OS to HP OS and plus converting single instance to RAC.

I have started this task on a low confidence note, I would be lying if I say I was very much confident. However, as thing progress, my confidence level went high and at the end I have successfully migrated a database (single instance) from AIX OS to HP OS and then converted the single instance database to RAC database.

To give you a rough idea what I have done:

Converted database from AIX to HP using RMAN.
On HP, then, converted this databases from single-instance to RAC database. There are few ways using which you can achieve this. However, I have opted the DBCA option to execute the task.

Next step is to migrate and convert the rest of the databases using the same method.

Happy reading,

Jaffar

16 node RAC with 12 databases

Umm.. its been really a quite long time that I did update my blog. becauseI have left my previous company (bank) after over 4 fruitful years. I have joined a new upcoming bank where I have got an opportunity to setup a 16 node RAC on HPUX (development) with 12 databases plus grid computing setup.

Though the soft lunch of the bank would be in coming few months , we are too busy setting up the RAC and Grid environment. In process, we have just started with a setup of 8 node RAC for the development environment this evening and will be ending up adding 8 more nodes on HP-UX Superdom (Itanimum 2) servers using EMC storage.

It is going to be the RAC setup all the way, i.e., starting with development, testing, pre-production and finally to production environments. I am not sure whether any coompany has such sort setup here in Saudi Arabia and this is going to be very challenging to our DBA's team as well to the OS administrators.

The next challenging which lays after the RAC setup is moving/migrating stand alone databases from AIX OS to the RAC setup on HPUX os. Which required database conversion and then adding to RAC.

I am sure that it gonna be a great fun handling such setup. There is plan to fully utilize the Computing technology, most probably looking at the workload management, TAF, FAN and etc. Its is not yet clear that all the databases (probably 12) will have scalability across all the nodes or something else.

I will keep writing in my blog about the developments, issues we faced and other technical issues during this juroney.

Till then, have good time.

Jaffar