1. set AIXTHREAD to S on /etc/environment.
2. " ioo -o maxpgahead=256 " to set maxpgahead parameter
Initial settings were : Min/Maxpgahead 2 16
3. " vmo -o minfree=360 -o maxfree=1128 " to set minfree and maxfree...
Initial settings were : Min/Maxfree 240 256
My experience of converting a cross platform (AIX to HP) single instance database to RAC using RCONFIG tool
The Convert verify option in the ConvertToRAC.xml file has three options:
- Convert verify="YES": rconfig performs checks to ensure that the prerequisites for single-instance to RAC conversion have been met before it starts conversion
- Convert verify="NO": rconfig does not perform prerequisite checks, and starts conversion
- Convert verify="ONLY" : rconfig only performs prerequisite checks; it does not start conversion after completing prerequisite checks
RMAN> convert database new database 'ORCL'
transport script '/tmp_backup/tempdb/transport.sql'
to platform 'HP-UX IA (64-bit)'
Today, we have just completed day 1 class in which all the basic stuff about SAN storage (specially EMC) has been discussed and I must say it is really worthy attending this training. It did really help me to clear some of the basic concepts which I was thinking about SAN Storage.
Now, I can talk about, what is a SAN Block Box (what does it consists of), what is a VLAN/VSAN, Fiber Channel, type of ports on a switch, how volume manager works, what is a LUN, port zoaning. prot channels, what is multipathing & it works, and etc. stuff. I am not sure how long I am going to keep this terminology in my memory, but, surely, this course going to give me bit of confidence to configure our RAC enviornment optimally.
I will be posting about the interesting stuff that learned in the class in coming days.
A final word about our storage. Our bank has brought EMC DMX4 SAN storage.
I have started this task on a low confidence note, I would be lying if I say I was very much confident. However, as thing progress, my confidence level went high and at the end I have successfully migrated a database (single instance) from AIX OS to HP OS and then converted the single instance database to RAC database.
To give you a rough idea what I have done:
Converted database from AIX to HP using RMAN.
On HP, then, converted this databases from single-instance to RAC database. There are few ways using which you can achieve this. However, I have opted the DBCA option to execute the task.
Next step is to migrate and convert the rest of the databases using the same method.
Oracle doesn't allow replacing ocr disk or adding the mirror copy if the new disk size is less than the existing ocr disk size
In order to multiplex the OCR and Voting disk we have requsted our UNXI team to gives us the four raw disks with 500mb each (oracle recommends 300MB for OCR/Voting Disk, may be less) and whil trying to replace/mirror the ocr to the new raw disks, I have got the following errors:
for OCR Mirror copy: PROT-22:Storage is too small
for replace OCR : PROT-16: Internal Error
As usaul, we search in the ML and found couple of notes for the same errors (317628.1 & 444757.1). Well, the ML Note : 317628.1 said the following:
" fails with "PROT-22: Storage too small" error. The problem is due to an Oracle bug where this operation requires the OCR mirror partition/file tobe larger than the original by up to 128MB. The bug has been fixed in the 10.2.0.2 patchset."
The note says that this bug has been fixed in 10.2.0.2 patch, we are on 10.2.0.3 patch and the bug still persists.
The error message which was coming 'Unable to copy the '/etc/oratab' to '/tmp/oratab.
We certainly knew that the issue is either with the permission or copying the said file and our immediate step was checking the permisson and ownership on the oratab file. Everything was just fine and then we try to create a file in the /tmp directory using touch command from the Oracle user and it worked without any hassles.
We were able to create the RAC database successfully on a single node and when tried to add instance on the other node, the same issue encountered again and again.
We finally thought of running the dbca in tracing mode and when the same problem arises and when we had a look at the trace file, we couldn't stop wondering about the silly/stupid thing which Oracle was complaining.
The trace file reveals that it is looking for 'scp' executable in '/usr/local/bin' location, whereas, the 'scp' excutable was in '/usr/bin' directory. We simply created a link for 'scp' in the '/usr/local/bin' and dbca run smoothly and we were able to complete the RAC database creation.
Past three months, its been really a great working expeirence on RAC and I must say, we have faced hell a lot of issues while trying to implement RAC on HP Virtual Machines. Definately, its a great learning phase.
Guess what's next? MOVING 8 DEVELOPMENT DATABASE TO RAC from AIX to HP. Umm.. a good and taugh task ahead.
I will keep posting my experience and the issues that have faced while setting up our 16 node RAC with 8 databases.
I am not one of those good people who do set some goals every year to achieve. I am bit lazy doing so. However, this time I do set couple of goals, like, achieving the OCM status and a very careful planning about financial savings(which I never think seriously before).
It was really a very fruitful last year where I did achieve something good in my life, like, given my ever first international Oracle seminars in Sweden & Denmark, spending more time working on RAC environments and brought a good house.
Unlike last year, no French Corner Cacke this time. No plans of celebration. Its like a normal day for me.
Thanks for passing by my blog.
Though the soft lunch of the bank would be in coming few months , we are too busy setting up the RAC and Grid environment. In process, we have just started with a setup of 8 node RAC for the development environment this evening and will be ending up adding 8 more nodes on HP-UX Superdom (Itanimum 2) servers using EMC storage.
It is going to be the RAC setup all the way, i.e., starting with development, testing, pre-production and finally to production environments. I am not sure whether any coompany has such sort setup here in Saudi Arabia and this is going to be very challenging to our DBA's team as well to the OS administrators.
The next challenging which lays after the RAC setup is moving/migrating stand alone databases from AIX OS to the RAC setup on HPUX os. Which required database conversion and then adding to RAC.
I am sure that it gonna be a great fun handling such setup. There is plan to fully utilize the Computing technology, most probably looking at the workload management, TAF, FAN and etc. Its is not yet clear that all the databases (probably 12) will have scalability across all the nodes or something else.
I will keep writing in my blog about the developments, issues we faced and other technical issues during this juroney.
Till then, have good time.
As per the press news, Dr. Majid Altuwaijri, CIO, National Guard Health Affairs will be the president of Saudi Oracle User Group.
Following is the excerpt from the press release:
The autonomous group comprises 14 most distinguished CIO's from the various industry verticals in the Kingdom.
Apart from addressing member problems, the group will provide networking and collaboration opportunities to Oracle users in the Kingdom, enabling them to have a better understanding on specialist areas of Oracle software. The Saudi Oracle User Group will enable Oracle customers from around the Kingdom to share their firsthand experiences in using Oracle technology and solutions, and how to best leverage their IT investments.
'Though independent, the Saudi Oracle User Group will work closely with Oracle to offer constructive feedback and assist different customers with their crisis,' said Dr. Majid Altuwaijri, CIO, National Guard Health Affairs, and President of the Saudi Oracle User Group. 'The feedback will help Oracle determine long term customer requirements, improvise on product and solution offerings to help customers in the Kingdom have a global presence with groups around the world.'
Moving forward, the group will meet members to discuss the Charter to establish a set of milestones to be achieved in the short and long term. Once the Charter is defined, members will meet to define online objectives. The website will be crucial in sharing information with member and will boost memberships. It will also act as a medium of managing and publishing the group's presence. The group will also attend meetings at international and local Oracle User Groups, participate in IT exhibitions and third party forums.
Over the next 12 months, the group will meet on a quarterly basis to form special interest groups across industries. This initiative will allow groups to be more focused and interactive as it speaks the same language for a set of customers.
I am very curious about joining this group and would really like to involve actively.
Now-a-day I am busy delivering Oracle University 5 days courses at various places (oracle onsite and customer site). Last week I have delivered a 5 day course on 'Oracle 10g: programming with PL/SQL' and starting with today, I am teaching 'Oracle 10g: Admin I' here at Oracle faclities in Riyadh, Saudi Arabia. I always felt that teaching is a tough job, however, I am loving it.
There were 12 atendees on the day 1 for this class, all of them have ample of programming experience in VB and .NET technologies. Though, not many serious questions have been asked on day 1, but, I loved teaching the Programming concepts.
Today day was the first day and still 4 days to go.
Today, I was shocked and surprised to see couple of my articles (which were written on blog) appeared on this site. I don't remember that they have ever taken my permissions to publish at their site.
Following is the link of few articles published on their web site:
Simulating 11g Snapshot Standby Database feature on Oracle 10g?
Change behavior of GROUP BY clause in 10g
I don't mind publishing my articles on their web site if they would have taken my permission.
Following are my specialized ares for consultancy work:
Data Guard/Physical Standby Setup (Disaster Recovery)
RMAN Policies and Setup
Following are my expertise for Oracle Technology Training:
Advanced RMAN concepts - 1 day
Automatic Storage Management (ASM) - 1 day
Data Guard - 2 days
Feel free to contact me on email@example.com for further enquirers.
Using the ocrconfig -manualbackup option, you can perform the manual (binary) backup on demand.
The -showbackup option now includes auto and manual flags that you can optionally specify to view the auto/manual existing backup information.
I don't know, how many of you aware that the changing of email id on OTN forums is possible now. I knew that many of you are least bothered about this, however, if anybody wants to change the email id, now you can do that easily and get rid off your legay email id.
How to change the profile information on OTN forums:
Once you log in on OTN forums, click 'Your Control Panel' option on right top corner which gives the facility to change the profile information, including your email id.
Metalink Note : Note:429825.1. (latest revision on 05-MAR-2008)
SELECT * FROM V$VERSION;
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Prod
PL/SQL Release 10.2.0.1.0 - Production
CORE 10.2.0.1.0 Production
TNS for 32-bit Windows: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 - Production
SQL> select instance_name from v$instance;
SQL> show parameter control_files
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_files string D:\ORACLE\PRODUCT\10.2.0\DB_1\DATABASE
Probably we all know that ASM consist of an Instance, doesn't have database, i.e. physical structure. Therefore, the above definitely misleads.
According to the ASM book, the control_files parameters point to a dummy (nonexistent) file. This is the default internal value for this parameter and it is a meaningless value in the ASM instance.
The book started with the ‘Foreword’ page consists of ‘History of ASM’ headline which is written by Bill Bridge (an architect in Oracle over 18 yrs and the creator of ASM).
He revealed few interesting points about ASM history, like when did he get an Idea about ASM, how it can be useful with Oracle technology and the total time which took to build ASM technology.
Chapter 1 – Storage Stack Overview
This chapter starts explaining an overview about Disk drive technology, disk layouts, Disk Drive performance, different class of storage system architectures (such as SAN, NAS, DAS and SCSI) .
RAID levels also been discussed, and storage arrays concepts too.
It might look interesting, but, I have skipped this chapter in order to read more about ASM concepts.
Chapter 2 – ASM Instances
This chapter starts on a note about ASM introduction, managing ASM (startup/shutdown) instances, few important ASM Instance parameters (such as instance_type, asm_diskstring & asm_diskgroups), ASM instance new background processes, V$ ASM views, ASM installation, upgrade and patching concepts.
It also talk about the significance of two ASM files, the ‘ab_
These files are used by the RDBMS instances for env. Information and health check monitoring.
Note : Beware, these files should not be deleted.
Nothing new has been offered in this chapter. This information is already available in the Oracle documents.
Chapter 3 – ASM disk
This chapter starts explaining about logical term LUN and explained about creating LUNs, LUN optimal sizes and what is the minimum disk size (4mb) and maximum (2 power 32 MB). Also, explained disks discovery(asm_disktring init parameter) and explained the meaning of disk header status (Former, Candidate, Member & Provisioned).
It also talks about the disk preparation/configuration on different OS, such as, Solaris, AIX and Windows and how to use the asmtool & asmtoolg utilities.
Nothing new offered in this chapter as well.
Chapter 4 – Diskgroups and Failure Groups.
This chapter started explaining about ASM diskgroup concepts and why ASM diskgroups are differ from typical LVM (logical volume manager). Also discussed ASM diskgroups redundancy types (normal, high and external) and their practice.
In the next segment of this chapter, it talk about Creating Diskgroups, default ASM Disk Names & how to change default disk naming convention and Diskgroup Numbers. An interesting point was made, i.e., “you are not restricted to having two failure groups for normal redundancy and three for high redundancy”.
Couple of new feature of v11g ASM, ASM Preferred Read, Diskgroups Attributes and Fast Disk Resync have been discussed.
In the last segment of this chapter, it explained I/O concepts and I/O failures. The chapter finished explaining about the Space Management Views and important columns that need to consult for space management.
This chapter gave some good inputs about I/O errors, 11g new features, ASM extent management , understanding of redundancy level and etc.
Chapter 5 – ASM Space Allocation and Rebalance
I particularly liked this chapter, it presented deeper insights about ASM Space allocation, operations, such as ASM AU (Allocation Units), ASM Extents distribution evenly among existing disks, how ASM Strips works, and difference between coarse and fine-grained striping types and which type of striping is good for what type of database files.
It also discussed when to change the default values of _asm_ausize (deault 1m) & _asm_stripesize (128k/1M) and how to set/change values for VLDs databases.
I also like the way author explained about ASM Rebalance operation concepts and the impact of setting various values to the ASM_POWER_LIMIT init parameters.
I really like the point “It is commonly believed that adding and dropping a disk against the same diskgroup required two separate rebalance activities. In fact, disks can be dropped and then added and a single rebalance command can be evenly distribute the I/O”. It shows how to avoid two rebalancing operations on this occasion.
A 11g new feature Fast Rebalancing which overhead of ASM-to-ASM and ASM-toRDBMS messaging also been discussed.
I really enjoyed reading this chapter which gave me great insight of ASM extent mechanism, rebalancing and striping internals. It also provides few very useful scripts.
Chapter 6 – ASMLIB Concepts and Overview:
Since I have not worked much on Linux, I have quickly reviewed this chapter about ASMLIB API which is currently available on Linux OS.
This chapter explains the benefits of ASMLIB API, and the following functionality of ASMLIB:
Device Discovery I/O processing Performance and reliability
Explained ASMLIB configuring, Managing disk in ASMLIB, Upgrading ASMLIB, Migrating to ASMLIB and troubleshooting ASMLIB.
Chapter 7 – ASM Files, Alias and Templates
This chapter explains about the default ASM Filenames, creating tablespaces, ASM Alias and Templates.
Nothing new offer in this chapter.
Chapter 8 – ASM Command-Line
This chapter explained about the ASMCMD (unix based) utility and its usage. It starts explaining existing means to manage and access ASM and its components.
I really liked the following 11g new features:
Bad Block Remapping in ASMCMD. Starting with 11g, when ASM redundancy is used, it is possible to recover bad sot by restoring the data from their mirror copy using the ‘remap’ command.
Note : ‘remap’ don’t fix logical errors or corruptions detected by the RDBMS.
Starting with 11g, moving datafiles in and out of ASM made easy with “copy” command introduced in ASM.
11g couple of new features about ASMCMD utility worth to know.
Chapter 9 – Managing Databases in ASM
This chapter started on a note on how to use ASM while creating databases and covered some good points about the internal functionality of CSS (Cluster Synchronize Services) daemon/process, how RDBMS interacts with ASM, important background process (role of RBAL, ASMB & 00nn b/g), few vital parameters that influence ASM performance (like, processes + add 16, large_pool add additional 60k and shared_pool size additional memory to store extent maps and concluded with good examples of converting ASM no non-ASM and vice-versa.
Explained well about CSS internal behavior, how RDBMS communicates with ASM instance, converting instances from ASM-to-Non ASM and vice versa.
Chapter 10 – ASM Persistent Data Structures
Stars explaining about MEATADATA and where does ASM stores the metadata, and different types of metadata. ASM has two main classes of metadata, physical and virtual metadata.
Physical metadata which located at fixed locations on disk contains the following:
Disk Header Allocation Table(AT) Free Space Table (FST) Partnership Status Table (PST)
Virtual Metadata stored in ASM files and contains the following:
File Directory Disk Directory Active Change Directory (ACD) Continuing Operations Directory (COD) Template Directory Alias Directory Attribute Directory Staleness Directory Staleness Registry
Chapter 11 – ASM Operations
This chapter is one of those chapters I liked in this book. It really explains how ASM works internally on the following condition:
First time when RDBMS instance tries to access an ASM file. Operations on ASM Files File creation File open File input/output (I/O) File close File delete
It also talk about the variable sized extents and Striping types (such as Coarse and fine-grind), Resync, Rebalancing and ASM Instance Recovery & Crash Recovery concepts.
This is one of interesting chapters present in this book which offers an internal mechanism of ASM functionality on the above mentioned operations.
Appendix A: explained about ASM and Storage Array Configurations.
Appendix B : Troubleshooting – In page, it discussed about ASM startup issues, Diskgroup mounting issues, Space related Issues and a walkthrough of an ASM alert log.
Appendix C: Scripts and Tips - Automatically Starting up and Shutting down ASM instance in Non-RAC env.
Migrating Individual Non-ASM datafiles to ASM.
This book is recommended for someone who serious wants to study about ASM technology and its in & outs.
Having a copy of this book is better than scanning 100th of Oracle documents.
This book is summarized and explained all ASM contents to a great extent.
For me, I liked 4 chapters in this books and might need another read.
While reading the note, following bug fixes really caught my attention:
- SYSTEM RESET of an spfile parameter can corrupt the SPFILE
- /*append parallel*/ can corrupt an index
-deadlock hang on "cursor: pin S wait on X" (typically from DBMS_STATS)
I have come across of many DBAs complaing about this behavior. Few links:
I am really interested in the following bugs fix:
+ Higher CPU / Higher "cache buffer chains" latch gets / Higher "consistent gets" after truncate/Rebuild.
Jonathan Lewis and others have talked about this bug at their respective blogs
Few other bugs fixed in this patch set:
+ Wrong results with bind variables/CURSOR_SHARING
5079978 High US enqueue contention in RAC
5387030 Automatic tuning of undo_retention causes unusual extra space allocation
5439554 "buffer busy wait" timeouts with automatic undo management and in memory undo
5442919 Expired extents not being reused (ORA-30036)
5512921 Instance crash caused by SMON OERI[kcblus_1] / dump
It been really a long list of bugs fixed in this patch set. From my personal experience prospective, I would say, first study the bugs fixed in the patch set before applying on a production database. If all resolving majority of your issues.
Let me reveal about my preparations before my mail box gets filled by hundreds of mails asking me to provide documents and etc stuff. For me, Oracle RAC documentation, Oracle university (RAC course) material and of course the extensive production practice knowledge of RAC certainly helped me a lot to prepare and pass this exam. I am sorry, I can't offer anything more than this.
After the exam, a small thought comes into my mind. Since this exam comes under EXPERT category, I thought, it would have been more better doing a day practical lab exam than simply doing the multiple choice questioning exam.
Have nice time,