3.01.2008

Oracle Automatic Storage Management - Book Review

I have borrowed this book from my colleague to read during the week-end and at the same time, thought of writing a summary/review. When I told my idea, I knew that my family disliked this, specially during week-end.

The book started with the ‘Foreword’ page consists of ‘History of ASM’ headline which is written by Bill Bridge (an architect in Oracle over 18 yrs and the creator of ASM).

He revealed few interesting points about ASM history, like when did he get an Idea about ASM, how it can be useful with Oracle technology and the total time which took to build ASM technology.

Chapter 1 – Storage Stack Overview

This chapter starts explaining an overview about Disk drive technology, disk layouts, Disk Drive performance, different class of storage system architectures (such as SAN, NAS, DAS and SCSI) .

RAID levels also been discussed, and storage arrays concepts too.

Conclusion:

It might look interesting, but, I have skipped this chapter in order to read more about ASM concepts.


Chapter 2 – ASM Instances

This chapter starts on a note about ASM introduction, managing ASM (startup/shutdown) instances, few important ASM Instance parameters (such as instance_type, asm_diskstring & asm_diskgroups), ASM instance new background processes, V$ ASM views, ASM installation, upgrade and patching concepts.

It also talk about the significance of two ASM files, the ‘ab_.dat ‘ and the ‘hc_.dat’ that are present under $ORACLE_HOME/dbs directory.

These files are used by the RDBMS instances for env. Information and health check monitoring.

Note : Beware, these files should not be deleted.

Conclusion:

Nothing new has been offered in this chapter. This information is already available in the Oracle documents.

Chapter 3 – ASM disk

This chapter starts explaining about logical term LUN and explained about creating LUNs, LUN optimal sizes and what is the minimum disk size (4mb) and maximum (2 power 32 MB). Also, explained disks discovery(asm_disktring init parameter) and explained the meaning of disk header status (Former, Candidate, Member & Provisioned).

It also talks about the disk preparation/configuration on different OS, such as, Solaris, AIX and Windows and how to use the asmtool & asmtoolg utilities.

Conclusion:

Nothing new offered in this chapter as well.

Chapter 4 – Diskgroups and Failure Groups.

This chapter started explaining about ASM diskgroup concepts and why ASM diskgroups are differ from typical LVM (logical volume manager). Also discussed ASM diskgroups redundancy types (normal, high and external) and their practice.

In the next segment of this chapter, it talk about Creating Diskgroups, default ASM Disk Names & how to change default disk naming convention and Diskgroup Numbers. An interesting point was made, i.e., “you are not restricted to having two failure groups for normal redundancy and three for high redundancy”.

Couple of new feature of v11g ASM, ASM Preferred Read, Diskgroups Attributes and Fast Disk Resync have been discussed.

In the last segment of this chapter, it explained I/O concepts and I/O failures. The chapter finished explaining about the Space Management Views and important columns that need to consult for space management.

Conclusion:

This chapter gave some good inputs about I/O errors, 11g new features, ASM extent management , understanding of redundancy level and etc.

Chapter 5 – ASM Space Allocation and Rebalance

I particularly liked this chapter, it presented deeper insights about ASM Space allocation, operations, such as ASM AU (Allocation Units), ASM Extents distribution evenly among existing disks, how ASM Strips works, and difference between coarse and fine-grained striping types and which type of striping is good for what type of database files.

It also discussed when to change the default values of _asm_ausize (deault 1m) & _asm_stripesize (128k/1M) and how to set/change values for VLDs databases.

I also like the way author explained about ASM Rebalance operation concepts and the impact of setting various values to the ASM_POWER_LIMIT init parameters.

I really like the point “It is commonly believed that adding and dropping a disk against the same diskgroup required two separate rebalance activities. In fact, disks can be dropped and then added and a single rebalance command can be evenly distribute the I/O”. It shows how to avoid two rebalancing operations on this occasion.

A 11g new feature Fast Rebalancing which overhead of ASM-to-ASM and ASM-toRDBMS messaging also been discussed.

Conclusion:

I really enjoyed reading this chapter which gave me great insight of ASM extent mechanism, rebalancing and striping internals. It also provides few very useful scripts.

Chapter 6 – ASMLIB Concepts and Overview:

Since I have not worked much on Linux, I have quickly reviewed this chapter about ASMLIB API which is currently available on Linux OS.

This chapter explains the benefits of ASMLIB API, and the following functionality of ASMLIB:

  • Device Discovery
  • I/O processing
  • Performance and reliability

Explained ASMLIB configuring, Managing disk in ASMLIB, Upgrading ASMLIB, Migrating to ASMLIB and troubleshooting ASMLIB.


Chapter 7 – ASM Files, Alias and Templates

This chapter explains about the default ASM Filenames, creating tablespaces, ASM Alias and Templates.

Conclusion:

Nothing new offer in this chapter.

Chapter 8 – ASM Command-Line

This chapter explained about the ASMCMD (unix based) utility and its usage. It starts explaining existing means to manage and access ASM and its components.

I really liked the following 11g new features:

Bad Block Remapping in ASMCMD. Starting with 11g, when ASM redundancy is used, it is possible to recover bad sot by restoring the data from their mirror copy using the ‘remap’ command.

Note : ‘remap’ don’t fix logical errors or corruptions detected by the RDBMS.

ASMCMD Copy:

Starting with 11g, moving datafiles in and out of ASM made easy with “copy” command introduced in ASM.

Conclusion:

11g couple of new features about ASMCMD utility worth to know.


Chapter 9 – Managing Databases in ASM

This chapter started on a note on how to use ASM while creating databases and covered some good points about the internal functionality of CSS (Cluster Synchronize Services) daemon/process, how RDBMS interacts with ASM, important background process (role of RBAL, ASMB & 00nn b/g), few vital parameters that influence ASM performance (like, processes + add 16, large_pool add additional 60k and shared_pool size additional memory to store extent maps and concluded with good examples of converting ASM no non-ASM and vice-versa.

Conclusion:

Explained well about CSS internal behavior, how RDBMS communicates with ASM instance, converting instances from ASM-to-Non ASM and vice versa.


Chapter 10 – ASM Persistent Data Structures

Stars explaining about MEATADATA and where does ASM stores the metadata, and different types of metadata. ASM has two main classes of metadata, physical and virtual metadata.

Physical metadata which located at fixed locations on disk contains the following:

  • Disk Header
  • Allocation Table(AT)
  • Free Space Table (FST)
  • Partnership Status Table (PST)

Virtual Metadata stored in ASM files and contains the following:

  • File Directory
  • Disk Directory
  • Active Change Directory (ACD)
  • Continuing Operations Directory (COD)
  • Template Directory
  • Alias Directory
  • Attribute Directory
  • Staleness Directory
  • Staleness Registry


Chapter 11 – ASM Operations

This chapter is one of those chapters I liked in this book. It really explains how ASM works internally on the following condition:

  • First time when RDBMS instance tries to access an ASM file.
  • Operations on ASM Files
  • File creation
  • File open
  • File input/output (I/O)
  • File close
  • File delete

It also talk about the variable sized extents and Striping types (such as Coarse and fine-grind), Resync, Rebalancing and ASM Instance Recovery & Crash Recovery concepts.

Conclusion:

This is one of interesting chapters present in this book which offers an internal mechanism of ASM functionality on the above mentioned operations.


On APPENDIX

Appendix A: explained about ASM and Storage Array Configurations.

Appendix B : Troubleshooting – In page, it discussed about ASM startup issues, Diskgroup mounting issues, Space related Issues and a walkthrough of an ASM alert log.

Appendix C: Scripts and Tips - Automatically Starting up and Shutting down ASM instance in Non-RAC env.

Migrating Individual Non-ASM datafiles to ASM.


Overall thoughts:

This book is recommended for someone who serious wants to study about ASM technology and its in & outs.

Having a copy of this book is better than scanning 100th of Oracle documents.

This book is summarized and explained all ASM contents to a great extent.

For me, I liked 4 chapters in this books and might need another read.

Happy Reading


Jaffar

2.25.2008

Why should we apply 10.2.0.4 patch set?

At least for us, it is not really necessary to apply every latest Oracle patch set on our databases unless we really need them. After knowing about the release of 10.2.0.4 patch set from Laurent and Mr.Doug's blogs, I thought of reading a metalink note that describes the list of bugs/issues fixed by this patch set. (ML Note : 401436.1).

While reading the note, following bug fixes really caught my attention:

- SYSTEM RESET of an spfile parameter can corrupt the SPFILE
- /*append parallel*/ can corrupt an index

-deadlock hang on "cursor: pin S wait on X" (typically from DBMS_STATS)

I have come across of many DBAs complaing about this behavior. Few links:

http://momendba.blogspot.com/2007/06/cursor-pin-s-wait-on-x-wait-event-in.html
http://www.dbaportal.eu/?q=node/49/print

I am really interested in the following bugs fix:

+ Higher CPU / Higher "cache buffer chains" latch gets / Higher "consistent gets" after truncate/Rebuild.

Jonathan Lewis and others have talked about this bug at their respective blogs

http://jonathanlewis.wordpress.com/2008/02/09/index-rebuild-10g/


Few other bugs fixed in this patch set:
+ Wrong results with bind variables/CURSOR_SHARING
5079978 High US enqueue contention in RAC
5387030 Automatic tuning of undo_retention causes unusual extra space allocation
5439554 "buffer busy wait" timeouts with automatic undo management and in memory undo
5442919 Expired extents not being reused (ORA-30036)
5512921 Instance crash caused by SMON OERI[kcblus_1] / dump

It been really a long list of bugs fixed in this patch set. From my personal experience prospective, I would say, first study the bugs fixed in the patch set before applying on a production database. If all resolving majority of your issues.

Happy reading,

Jaffar

1.24.2008

1Z0-048 Oracle Database 10g: Real Applications Clusters Administrator Certified Expert - CLEARED

Well, this morning I have passed the 1ZO-048 RAC expert exam.

Let me reveal about my preparations before my mail box gets filled by hundreds of mails asking me to provide documents and etc stuff. For me, Oracle RAC documentation, Oracle university (RAC course) material and of course the extensive production practice knowledge of RAC certainly helped me a lot to prepare and pass this exam. I am sorry, I can't offer anything more than this.

After the exam, a small thought comes into my mind. Since this exam comes under EXPERT category, I thought, it would have been more better doing a day practical lab exam than simply doing the multiple choice questioning exam.

Have nice time,

Jaffar

12.20.2007

Odd issues while preparing Physical Standby

After an uphill task of upgrade/migration of 2.5TB sized data warehouse Oracle database from 9i to 10g and then from one OS to another (cross platform), the subsequent challenge is building a DR (Standby Database) for this database.

The DR (Disaster Recovery) site is around 3 km away from the HO. Since the size of the database is huge, we thought of moving tapes (physically) or duplicating the backup tapes from HO to the DR site, but, none of the ideas was materialized due to few technical/ un-technical difficulties. Fortunately, the speed of the lease line (leaser link) between the two sites is good enough and we decided trigger the duplicate database (standby) command from HO.

Ufff.. It took 3 days to finish the restore. However, the subsequent recovery on the standby database failed by the following problem:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 12/19/2007 05:16:32
RMAN-03015: error occurred in stored script Memory Script
RMAN-06136: ORACLE error from auxiliary database: ORA-19625: error identifying file /datafile1.dbf
ORA-27041: unable to open file
IBM AIX RISC System/6000 Error: 22: Invalid argument
Additional information: 2

Surprisingly, the datafile, datafile1.dbf (filename is changed here) which Oracle complaints was exists on the mount point. Then, thought of starting the MRP (recover managed standby database disconnect from session), unfortunately, the MRP terminated by the following errors:

Errors in file /dbdata/oradba/admin/OFDMP/bdump/ofdmp_dbw0_1077358.trc:
ORA-01157: cannot identify/lock data file 1479 - see DBWR trace file
ORA-01110: data file 1479: '/mountpoint/
datafile2.dbf'

Again, the complained file exists on the mount point, however, this time the problem was that the file has ‘carriage return’ before the name, (this may happen when you try to create/add a datafile from OEM). The workaround was following:

mv /mountpoint/ datafile2.dbf’ /mountpoint/datafile2.dbf

and then, change file name in the controlfile with the following command:

(make sure the standby_file_management is set to MANUAL on the standby).

alter database rename file ‘/mountpoint/ datafile2.dbf’ /mountpoint/datafile2.df’;

After the above workaround, the MRP (Media Recovery Process) was started successfully and looking for the archive gap.

When I tried to restore required archived log on the standby database from the backups (archived backups), the restored command failed with the following:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 12/20/2007 13:44:20
RMAN-20242: specification does not match any archive log in the recovery catalog

I thought of just testing the restore on production, the restore (tried t restore single archive log) was successfully.

I lately realized that the missing archives are generated after creating the standby controlfile and obviously the standby controlfile was not having the information about these archived logs. (Backups were performed with nocatalog option).

Following is the workaround:

1. Create a catalog and register production database.

2. On standby, using the catalog (rman target / catalog). Now, standby can get the info about the backups through the catalog.

3. Restoring required archived logs.

Earlier, I have done dozens of DG configuration facing different challenges. Well, everything it’s a new challenge and learning experience.

Happy reading,

Jaffar

12.10.2007

Simulating 11g Snapshot Standby Database feature on Oracle 10g?

As we all knew that the Oracle 11g improved the capabilities of standby database immensely, where a physical standby database can easily open in read-write mode, which can be ideally suitable for test and development environments. At the same time, it maintains protection by continuing to receive data from the production database, archiving it for later use.

What if you want to achieve the same on Oracle 10g? Well, I absolutely don’t have any clue about others, but, we have come across of such situation couple of days ago when our DR (Disaster Recovery Solution) team came to us with a request to test our standby database. They want the standby database in read write mode to do some real scenario tests and once the testing is done, they want the database to be back to standby mode.

We initially said, we can open the database in read only mode for their testing, but, the requirement demands the database to be in read write mode. We thought, we can break the standby database for their testing and once the testing is done, we can rebuild the standby database again. We know that this is very well possible with Oracle 11g but not with Oracle 10g. My colleague, Mr. Asif Momen, did some R&D come up with a solution where a Oracle 10g standby database can open in read write mode and can also be reverted back to standby mode.

The procedure as follows:

1. Set the following parameters on the standby database:

db_recovery_file_dest_size & db_recovery_file_dest

- Make sure the values are reflected.

2. Stop the media recovery process, if active.

3. When the standby is in MOUNT mode, Create a guaranteed restore point:

CREATE restore point before_rw guarantee flashback database;

3. Stop the log shipping on the primary database. (for safer side)

alter system archive log current;

alter system set log_archive_dest_state_2=DEFER;

4. Failover the standby database using the following command:

ALTER DATABASE ACTIVATE STANDBY DATABASE;

-Make sure the media recovery process is turned off

-Minimize the protection mode to MAXIMUM PERFORMANCE, if the mode is set other than the MAXIMUM PERFORMANCE.

5. Open the database (read write mode).


AT THIS POINT, YOU CAN USE THIS DATABASE AS NORMAL READ WRITE DATABASE.

Reverting the database back to standby mode:

  • Shutdown the database
  • Startup database in mount mode
  • Flash back database to restore point using the following:
FLASH BACK DATABASE TO RESTORE POINT before_rw;
  • Convert the database back to standby mode using the following:
ALTER DATABASE CONVERT TO PHYSICAL STANDBY;
  • Shutdown the standby database and remove the previously set parameters.
  • Start the standby database in mount state and drop the restored point.
  • Enable the Media Recovery on the Standby database.
  • Activate log shipping on the primary using the following:
ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENALE;

Since, we have done very little changes in the database, after converting to read write mode, the time which took to revert the database back to standby mode took few minutes only. Well, it is definitely need to be seen the time that take during the conversion to standby mode after huge changes in the read write database.


It worked well with us and Mr. Asif definitely deserved an appreciation

The tests have been carried out on AIX 64 bit with Oracle 10.2.0.3 release.

In the Part II, I will be posting my testing.

Happy Reading,


Jaffar