RMAN best practices for RAC presentation

As part of BrainSurface's VirtaThon Road tour, I made a 30 minutes presentation (webinar) 'RMAN best practices for RAC' for Massachusetts Institute of Technology (MIT).

Since the manageability of RMAN backup, restore and recovery operations are nearly identical in non-clustered and clustered databases, the objective of the presentation was to summarize the options using which an RMAN can be best utilized for a RAC database to speed up the RMAN operations. The presentation also explains in-detail about instance,crash recovery operations for RAC database in contract to a stand-alone database instance recovery.

If you have missed the opportunity to attend or listen the presentation, I am uploading the slides here at my blog for your reference.

RMAN best practices for RAC

View more presentations from Syed Hussain.

Node slave process (pz99) on 10g RAC

We have observed a strange slave process, 'pz99' on our RAC databases for every connection to the database and we were wondering what type of processes are they. Thank god our doubt was cleared without much hassles. According to ML Doc :734139.1 'From Oracle 10g onwards you will find on every node slave processes like pz99. This are the processes that query the gv$ views'

Happy reading

Jaffar

Oracle DB & RAC 10gR2 on IBM AIX : Tips and Considerations by Erik Salander

While browsing this afternoon, I have come across of 'Oracle DB & RAC 10gR2 on IBM AIX: Tips and Considerations'

paper by Erik Salander. Well, if you have any planns to setup RAC on AIX, I recommend you to have a look at this paper.

You can download the PDF@ http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101089

If you wanna need the abstract of this paper before you want to download, here it is:

This paper consolidates the information that needs to be considered when implementing and Oracle Database 10gR2 or Oracle RAC (Real Application Clusters) 10gR2 on AIX 

This paper is written to a level of detail that assumes readers have an in-depth knowledge of AIX , Oracle Database 10g, Oracle RAC 10g and the related products.

Happy Reading,

Jaffar

How to startup RAC database services automatically

As of Oracle 10gR2 (10.2.0.4 patch set) when RAC database is started with 'srvctl start database -d DBNAME', unfortunately, the associated database services do not startup automatially. Therefore,   services must be started manually after the database startup. It could be painfull in some situations, like, when you have many databases running on RAC with daily or weekly cold backup schedule. When services are not up, clients unable to connect to the respective databsases, in case they use the service name to connect to the database.

A possible workaround is to write  FAN server side callouts. You may download perl scripts, 'Start Services on Instance Up' from http://www.oracle.com/technology/sample_code/products/rac/index.html. Its a smaple perl script which can be used to as a FAN Server Side callout to start services when an instanes up event is received on the node. You need to put the scripts under $CRS_HOME/racg/usrco. The PDF contains the procedure how to deploy the scripts, setting permission and other stuff.

In 11g,  When services are already started on some remote nodes, then the startup of the instance on the local node will autostart the services on it.

References:

ML 416178.1: After Srvctl Start Database, Database Services Will Not Start Up Automatically

http://www.oracle.com/technology/sample_code/products/rac/index.html

http://www.oracle.com/technology/products/database/clustering/pdf/twpracwkldmgmt.pdf

http://download-west.oracle.com/docs/cd/B19306_01/rac.102/b14197/hafeats.htm#sthref374

Happy reading,

Jaffar

Tough time ahead (going to have fun with RAC)

It is going to be a tough this week-end for me, and at the same time very challeging too. Well, this week-end, in simple terms, I can say that I am going to have fun with RAC.

We currently have 4 node develpment and 8 node Production RAC environment at our premises. The development currently running on Oracle 10g R2 with patchset 10.2.0.3, which we are planning to upgrade to 10.2.0.4. My god, there are over 20 databases running across these 4 nodes. Uff.. upgrading cluster, asm and all those databases to 10.2.0.4 patchset.. tough time!

Along with upgrade activity, we also plan the following at the same time:

Changing the IPs of public and interconnect on production.

Moving a RAC database from development to production.

Apart from the above, there are couple of small changes to be done.

I know one time such a big change is not recommended and a very tough task too. Thankfully, we are not really  into production yet,  we have ample of downtime to carry out/complete all these tasks.

Wish me best of luck and I will blog about any issues that I faced during this activity.

Happy reading,

Jaffar

My experience of converting a cross platform (AIX to HP) single instance database to RAC using RCONFIG tool

Its been really more than a month that I didn't write anything on my blog due to my busy schedule.  Well, I am here now to share my experience of converting a cross platform (AIX to HP) single instance database to 2 node RAC using the RCONFIG tool.

If you were following my blog, you might aware that we have a plan of implementing 16 node production RAC on HP UX Superdom Itanium servers using EMC DMX4 SAN storage. 

As part of production implementation, last week, I have configured initial 8 node RAC successfully and the immediate plans were to migrate the databases across operating systems (AIX to HP ux) and convert a single instance database to RAC. I have done this exercise on the development setup sucessfully, the only change this time is that I have used RCONFIG utility to convert single instance database to RAC. Earlier I have used DBCA method to do the same.

Oracle provides following methods to convert a single instance database to RAC:

Grid Control

DBCA

Manual

RCONFIG(from 10gR2)

I must say, each DBA may choose a different method according to his/her convineance to achieve the result. The following illustrates how I have migrated a database from AIX to HP UX and from a single instance to RAC (on 2 node).

Database and OS details

Source:

Database 10gR2 (10.2.0.4)

OS : AIX

non-ASM

Target:

Databae 10gR2 (10.2.0.4)

OS : HP UX Superdom Itaninum

ASM

RCONFIG brief

RCONFIG tool is introuced in Oracle 10gR2 and the main functionality of this tool is to convert single instance database to RAC. 

The tool(rconfig) can be found under $ORACLE_HOME/bin directry.

The xml input file (ConvertToRAC.xml) resides under $ORACLE_HOME/assistants/rconfig/sampleXMLs (it is recommended to copy the file before using it).

Log files(rconfig.log and etc) can be found under $ORACLE_HOME/cfgtoollogs/rconfig

The Convert verify option in the ConvertToRAC.xml file has three options:

• Convert verify="YES": rconfig performs checks to ensure that the prerequisites for single-instance to RAC conversion have been met before it starts conversion
• Convert verify="NO": rconfig does not perform prerequisite checks, and starts conversion
• Convert verify="ONLY" : rconfig only performs prerequisite checks; it does not start conversion after completing prerequisite checks 

Before you actually starts the convert process, you can run the rconfig with convert verify=ONLY option which basically check all the prerequisites. At the end, the result code either would be 0 or 1. Where 0 stands for operation success and 1 stand for operation failures and in case of failure, it will also gives you the reason of failure.

Following RMAN command converts the database from AIX to HP UX platform.

RMAN> convert database new database 'ORCL'

 transport script '/tmp_backup/tempdb/transport.sql'

 to platform 'HP-UX IA (64-bit)'

 db_file_name_convert

'+DG_ORCL/ORCL/datafile','/tmp_backup/tempdb';

Move all the converted datafiles to target machine and follow the ML NOTE: 414878.1  for further details on how to start the on target machine.

Following illustrate how to convert single instance database to RAC using the RCONFIG tool:

cp $ORACLE_HOME/assistants/rconfig/sampleXMLs/ConvertToRAC.xml convertdb.xml

modify the convertdb.xml file according to your environment. Following is the example:

==

convert_t24uat.xml 46 lines, 2964 characters 

 ?xml version=1.0 encoding=UTF-8?

 RConfig xmlns:n=http://www.oracle.com/rconfig

           xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance

           xsi:schemaLocation=http://www.oracle.com/rconfig

    ConvertToRAC

  Verify does a precheck to ensure all pre-requisites are met, before the conversion is attempted. Allowable values are: YES|NO|O

NLY

        Convert verify=NO

 Specify current OracleHome of non-rac database for SourceDBHome

              SourceDBHome /u00/app/oracle/product/10.2.0/db_1 SourceDBHome

 Specify OracleHome where the rac database should be configured. It can be same as SourceDBHome

              TargetDBHome/u00/app/oracle/product/10.2.0/db_1 TargetDBHome

 Specify SID of non-rac database and credential. User with sysdba role is required to perform conversion

              SourceDBInfo SID=ORCL

                Credentials

                  User sys User

                  Password password Password

                  RolesysdbaRole

                Credentials

              SourceDBInfo

 ASMInfo element is required only if the current non-rac database uses ASM Storage

              ASMInfo SID=+ASM1

                Credentials

                  User sys User

                  Password passwordasm Password

                  RolesysdbaRole

                Credentials

              ASMInfo

 Specify the list of nodes that should have rac instances running. LocalNode should be the first node in this nodelist.

              NodeList

                Node name=rac01

                Node name=rac02

              NodeList

 Specify prefix for rac instances. It can be same as the instance name for non-rac database or different. The instance number will be attached to this prefix.

              InstancePrefix ORCL InstancePrefix

 !--Specify port for the listener to be configured for rac database.If port=, alistener existing on localhost will be used for rac

 database.The listener will be extended to all nodes in the nodelist --

              Listener port=""

 Specify the type of storage to be used by rac database. Allowable values are CFS|ASM. The non-rac database should have same storage type.

              SharedStorage type=ASM

 Specify Database Area Location to be configured for rac database.If this field is left empty, current storage will be used for rac database. For CFS, this field will have directory path.

                TargetDatabaseArea+DG_ORCL TargetDatabaseArea

 Specify Flash Recovery Area to be configured for rac database. If this field is left empty, current recovery area of non-rac dat

abase will be configured for rac database. If current database is not using recovery Area, the resulting rac database will not have

a recovery area.

         TargetFlashRecoveryArea +DG_ORCL Target FlashRecoveryArea

        SharedStorage

        Convert

     ConvertToRAC

 RConfig

===

Once you modify the convert.xml file according to your environment, use the following command to run the tool:

rconfig convertdb.xml

Oracle does the following during conversion:

oracle@usogp01: /u00/app/oracle/product/10.2.0/db_1/bin # ./rconfig convertdb.xml

Converting Database ORCL. to Cluster Database. Target Oracle Home : /u00/app/oracle/product/10.2.0/db_1.

Setting Data Files and Control Files

Adding Database Instances

Adding Redo Logs

Enabling threads for all Database Instances

Setting TEMP tablespace

Adding UNDO tablespaces

Adding Trace files

Setting Flash Recovery Area

Updating Oratab

Creating Password file(s)

Configuring Listeners

Configuring related CRS resources

Adding NetService entries

Starting Cluster Database

Starting Listeners

    

      

        

          Operation Succeeded

        

      

      

         /u00/app/oracle/product/10.2.0/db_1

      

        

          ORCL1<\SID>

          ORCL2<\SID>

        <\SIDList>    

    

  

oracle@usogp01: /u00/app/oracle/product/10.2.0/db_1/bin #

==

As you can see the exit code is 0, which indicates the success of the conversion process.

Uphills faced during the conversion process

The only error which I got during the conversion was missing listener.ora under ORACLE_HOME/network/admin directory as we had created listener under the ASM home (we have separate RDBMS and ASM homes). Problem has been resolved by creating the link under the RDBMS network admin directory of ASM listener.ora file.

We have contacted one of the developer of rconfig utiity to clarify the two changes (following) to make sure it will work:

non-ASM to ASM 

single instance to RAC.

He said that the main purpose of this utility is to convert single instance database to RAC and optinally few non-ASM of target database to ASM while converting to RAC. But, he is not sure about moving the entire non-ASM to ASM while conversion. As they don't have any test case. However, asked us to go ahed and try. Well, we were amazed that it worked for us and we delightly informed the developer that two changes are worked for us and you can have our reference.

Our next migration/conversion is our EBusiness suit and of course, I am going to share my expereince here.

References

http://download-west.oracle.com/docs/cd/B19306_01/install.102/b14201/cvrt2rac.htm#sthref1273

Important Meta Link Notes:

ML Note:387046.1 RCONFIG : Frequently Asked Questions

ML Note:371519.1 RCONFIG Error When Converting From Single Instance To RAC

ML Note:375472.1 RCONFIG FAILS with an null pointer EXCEPTION

ML Note:391297.1 RCONFIG fails when using ASM as Shared Storage

ML Note:388577.1 Using Oracle 10g Release 2 Real Application Clusters and

 Automatic Storage Management with Oracle E-Business Suite 

 Release 12

Happy Reading,

Jaffar

Database migration across platforms and conversion of single instance to RAC

Working in a huge RAC environment, each passing day is a learning day for me. After setting up 4 node RAC on HP UX Itanium 2 Severs, the subsequent task/challenge is to migrate single instance databases (around 16) from AIX OS to HP OS and plus converting single instance to RAC.

I have started this task on a low confidence note, I would be lying if I say I was very much confident. However, as thing progress, my confidence level went high and at the end I have successfully migrated a database (single instance) from AIX OS to HP OS and then converted the single instance database to RAC database.

To give you a rough idea what I have done:

Converted database from AIX to HP using RMAN.
On HP, then, converted this databases from single-instance to RAC database. There are few ways using which you can achieve this. However, I have opted the DBCA option to execute the task.

Next step is to migrate and convert the rest of the databases using the same method.

Happy reading,

Jaffar

Oracle doesn't allow replacing ocr disk or adding the mirror copy if the new disk size is less than the existing ocr disk size

At least my self now decided not to believe in the ML Notes (not always though), its better testing what they have said. I was trying to replace an existing ocr location and also trying to add mirror copy of ocr. During RAC setup, we have used 1gb disks for OCR and voting disk (with single copy of each).
In order to multiplex the OCR and Voting disk we have requsted our UNXI team to gives us the four raw disks with 500mb each (oracle recommends 300MB for OCR/Voting Disk, may be less) and whil trying to replace/mirror the ocr to the new raw disks, I have got the following errors:

for OCR Mirror copy: PROT-22:Storage is too small
for replace OCR : PROT-16: Internal Error

As usaul, we search in the ML and found couple of notes for the same errors (317628.1 & 444757.1). Well, the ML Note : 317628.1 said the following:

" fails with "PROT-22: Storage too small" error. The problem is due to an Oracle bug where this operation requires the OCR mirror partition/file tobe larger than the original by up to 128MB. The bug has been fixed in the 10.2.0.2 patchset."

The note says that this bug has been fixed in 10.2.0.2 patch, we are on 10.2.0.3 patch and the bug still persists.

Happy reading,

Jaffar

ocrconfig 11g improvement

Starting in 11g, the ocrconfig command now supports manual backup option which enables you to perform ocr file backup at any time, rather than waiting for the default automatic backup (i.e. every 4 or more hrs) interval.

Using the ocrconfig -manualbackup option, you can perform the manual (binary) backup on demand.

The -showbackup option now includes auto and manual flags that you can optionally specify to view the auto/manual existing backup information.

http://download.oracle.com/docs/cd/B28359_01/rac.111/b28255/votocr.htm#i1012456


Happy Reading,

Jaffar

RAC - finally into production

It was like a long cherished dream become true when we successfully moved one of our highly OLTP database from the single node to 2 node RAC on AIX platofrm.

We initially thought of implementing extended RAC between head office and DRC sites. At the last moment we had to drop the plan due to some non-technical reasons. Rather, we simulated the extend RAC in the head office using two different storages.

They were couple problems faced during the implementation, 1) when we changed the default port from 1521 to another, there was an issue with Enterprise Manager Console, as it was still looking for the default port (1521). We couldn't successes in the first attempt of running 'emca -reconfig dbconsloe db' command. However, running the same command second solve the issue. 2) AMS Listener entry missing. After some research we found out the the LOCAL LISTENER entry missing from the ASM initialization parameter. Setting the LOCAL LISTENER parameter in the ASM init file solve the issue. These were related with Enterprise Manager (console), however, the database was functioning without any issues.

Although, we have 2 node RAC, we thought not to distribute the load equally on the two nodes, rather, 90% on one node and 10% on another node, (of course missing failover connection concept here). This was simply to monitor the behavior of the RAC. If we finish the week successfully without issues, we will then implement the load distribution equally to the both nodes.

Once we get the stable line between the head office and DRC site (which a kilometer away from the head office), we will another third node.

I should say that the RAC installation doesn't fancy me was I initially thought. The major challenges lays of maintaining RAC environment and survival commands to quickly come from problematic issues. I am sure that experience will teach us.

If everything goes will, Mr. Murali Vallath will be presenting 4 day RAC in-house training at our premises in coming months. Bit excited about this opportunity.

Thursday and Friday are off (week-end) here in ME (Saudi Arabia). The database will be in live mode this Saturday.

I will keep posting about my experience of RAC.

Happy reading,

Jaffar

RAC - Reading, Implement & Test

Finally we have got the go signal from the mgmt to test and implement RAC on extended (stretched) clusters for one of the highly OLTP database, this was being discussed for a quite long time. Since, the database is so important, we are carefully doing/testing measuring our scenarios on the RAC.
As an initial step, we have successfully installed Oracle 10gR2 RAC, able to maintain clusters without any issues on AIX, although it took considerable time, finally, we have done it. Now we are thinking of deploying the application to have a test.

I don’t have much experience with RAC nor do I have much knowledge about the through functionality of clusters, RAC and etc.

Considering the above, I though of start learning the concepts of clusters and Oracle RAC functionality. I have ordered following three books from Amazon site:

Oracle 10g RAC Grid, Services & Clustering by Murali Vallath
Oracle Database 10g Real Application Clusters Handbook (Osborne Oracle Press) by K Gopalakrishnan
Pro Oracle Database 10g RAC on Linux: Installation, Administration, and Performance (Expert's Voice in Oracle) by Julian Dyke today!

Yesterday, I got Mr. Gopla’s book and waiting for other two books, expecting end of this week.

Last night I read one chapter about ASM Instance of Gpola’s book. It was a good read; as the author explained very detailed information in a simple format which can be easily digest by anyone who is reading.

I thought of just giving the review of this book once I finish my reading(s).

I am sure, I will be doing it, at least for the sake of writing review, I will read the book thoroughly again and again, until the concepts get cleared to me.

Happy Reading,

Jaffar

Desktop RAC - Virtually installed and configured.

I have successfully implemented a Desktop RAC scenario which is provided by Oracle at http://www.oracle.com/technology/tech/linux/vmware/index.html.

Initially I have downloaded all the four zip files required to test the Desktop RAC for Red Hot Linux, there are two zip files for Novell SUSE as well.

Note: you are required to register with VMware and Red Hot Linux in order to get the trail version of that software and a unique number.

Following are the two issues I have come across while implementing Desktop RAC.

1. One of the zip file download contains VMware workstation 5.3 download from the VMware site. Unfortunately you see a message at VMware site that VMware workstation 5.3 is no more available for download and it suggest you to download VMware workstation 5.5 version.
This is required because you need to get the unique key to validate the VMware installation.

I have first installed VMware workstation 5.5 and skipped the VMware installation part while running Oracle installation.

2. First two zips contain Oracle RAC software for Linux and each zip contains Disk1 folders respectively. The problem was that while unzipping zip files which has same folder names respectively, needed to rename and this cause problem while setup as the step looking for all the required files under the Disk1 folder. What I did was to cross compare the folder in the Disk1 folders respectively and copy the missing in to one Disk1 folder.

After couple of unsuccessful attempts, finally I have done with Desktop RAC. I can see that Red Hot Linux 4.0 OS installed, Oracle RAC (10gR2) with ASM configured.
Also, it gives you bunch of examples step-by-step in documentation to test the scenarios.
The implementation gives you two instances and a database named as RACDB with ASM configured.

This is good for the beginners to have RAC installed on a PC which doesn’t require state-of-the-art hardware.

I guess, this is not good for an experienced DBA. You can’t install CRS, create RAC database and configuration and etc.

For me, it’s a good chance to start and make myself familiar with RAC commands and do some testing before I go for any training or work with production.

If I come across any good issues, I will update on my blog.

Happy reading,

Want to learn RAC????

I have come across of many request on serveral Oracle database forums and here and there asking about to have RAC env. on a PC for testing and learning purpose.
After some browsing, I have got two possibilities for the people who wants the same.

Basic Requirements:
Have a PC with good configuration, at least 1GB of RAM and good amount of DISK space.

Possibility 1:
Mr. Howard J Roger at his web size DIZWELL INFORMATICS has written an excellent articel on 'Building a 10gR2 Laptop RAC on Windows'' in which he greately explained about installation and maintenance of RAC on a PC. Following link will take you the article.

http://www.dizwell.com/prod/node/25

Possibility 2:

If you feel installing Oracle RAC, VMWARE and other stuff would be little bit hectice, at least, for the biginner, then, don't worry, Oracle has a ready made solution for you.

Desktop Data Centers: Oracle-on-Linux VMware Tool Kits

Oracle, VMware, Red Hat and Novell have teamed up to provide everything you need to run a full Oracle Database 10g on Linux environment on your Windows desktop—including the operational infrastructure and step-by-step instructions on how to use Oracle Database 10g and Oracle Real Application Clusters.

Note: These kits are for evaluation and development purposes only. Oracle will not support any deployments of these packages.

Currently the test cases are available on RED HAT and NOVELL only.

http://www.oracle.com/technology/tech/linux/vmware/index.html

Good books (in my view) are:

Pro Oracle Database 10g RAC on Linux Hardcover by Julian Dyke
Oracle 10g RAC Grid, Services... Paperback by Murali Vallath
Oracle Database 10g Real Application... Paperback by K Gopalakrishnan

Blog which discuss about RAC stuff:

Mr. Kevin Closson maintain's a good blog discussing about RAC stuff. Wroth to add to favourites or book mark his blog.

http://kevinclosson.wordpress.com/

Few good articles on RAC.

Mr. Julian Dyke wrote couple of good articles on RAC. It can be downloaded from his site.

http://www.juliandyke.com/

HAVE FUN WITH RAC!!!!!