Bug 18223021 : DISK FILE OPERATIONS I/O ON HP-UX.

We have been dealing with a strange ASM  behavior over the past few months across all ASM instances in our multiple Oracle 11gR2 (11.2.0.2) Cluster envs on HPUX 11.31 OS. Even a simple and typical ASM task like, adding new disk to ASM disk group, disk group mount/unmount and querying for CANDIDATE asm disks was taking min of 20 minutes, and sometimes infinity time. This behavior caused a lot of performance degradation across all database instance in the cluster, and most of the databases suffered with 'Disk file operations i/o' wait with other consequences.

I must say, Oracle support indeed made a few unsuccessful attempts by suggesting increasing the ASM instance SGA, enabling async i/o on OS, reducing the number of disks etc to address the issue. Finally, they have logged a BUG '18223021' for our issue and we are yet to receive the fix.

Below are a few consequences of this behavior we are confronting:

• Oracle 10g database instances running on the node where a new ASM disk being added to the existing disk group had suffered with control file locking and database hung issues
• We ensure the disk group to which a new disk is being added is not mounted on non relative ASM instances (in other nodes)
• When adding disk takes infinite time on one of the ASM instances, the only way was to shutdown the instance with dismount the ASM disk group subsequently to speed-up and complete the procedure
• Most databases are suffering with 'Disk File operations I/O'

If you have MOS access, you may refer the BUG for more details.

Stay tuned for the fix and solution...

ASM instance in INTERMEDIATE state + CHECK TIMED OUT state_details

Yet another clash with ASM 11gR2 BUG during a new database creation under a 6 node development RAC environment. Got a requirement to create a new database, thereby, created a new ASM diskgroup manually and began creating the database with DBCA tool. However,  ran over the following error when the newly created ASM diskgroup was selected to host the data files of the database:

After getting the error, I thought I would better ensure once again that the diskgroup is properly mounted on the local ASM instance, despite the fact that the diskgroup in very well mount state, the issue persists. I therefore check over the resource (asm and dg) status in the cluster issuing the 'crsctl stat res -t' command. The output displays that the diskgroup in the question has OFFLINE STATE and scrolling further down it was a bit strange and scary moment. The ora.asm resource was in INTERMEDIATE state with CHECK TIMED OUT state_details.

My curiosity let me do some research over the MOS and quickly come across the following document:

oraagent.bin Exits After Check Timed Out [ID 1323679.1]

The issue was nearly identical to what has been explained in the document and the behavior is due to bug 11807012 that apparently fixed in 11.2.0.3, 12.1 and above. However, the document also did mention that the Interim patch 11807012 exists for certain platform/version and advised to engage Oracle Support to request if it does not exist for your platform/version. Unfortunately, we couldn't find the interim patch for our OS (HPUX). We then opened a SR and the engineer indeed confirmed it is due to the bug in the context. As a workaround, recommend the following action plan followed ASM instance restart:

crsctl modify resource "ora.asm" -attr "CHECK_TIMEOUT=132"

The above change was applied and bounced the ASM instance to reflect the change.(We seek the downtime to deploy the change, since it was a dev. environment, we easily get it). The ora.asm resource then cam back to normal condition and the database was successfully created subsequently..

Out of six ASM instances, we found three of them were in the same state. However, on a positive side, it will not cause any troubles with the ongoing operations unless you wanna do something what we needed. The following image outlines the status of the ASM resources:

Happy reading,

Jaffar

Unable to mount a diskgroup with ORA-15032,ORA-15202,ORA-15041 errors

Post 8 node Oracle 10g cluster upgrade to 11g R2 with nearly 60 databases on HPUX OS, we have observed a significant increase in the server resources consumption, in particular for CPU and Memory. Due to the heavy  resource consumption (or lack of sufficient resources time to time), a couple of nodes with nearly 10 databases running across each node started evicting more frequently. However, a couple of days back, one of the node went down for the same reason, and the subsequent node start up, the databases on the node refuse to start up automatically and we have noticed the memory and virtually memory consumption was just shooting up in no time. We identified the symptoms of the behavior and repaired the issue after spending nearly a half day time (it was a very silly mistake, I will blog about it later on). However, one particular RAC database was not able to start as its diskgroup was not mounted. When we tried mount the diskgroup in question manauly, we come across the following ORA error:

ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15202: cannot create additional ASM internal change segment
ORA-15041: diskgroup "DG_XXXX" space exhausted

It was a bit of surprise to receive the ORA-15041 error while mounting the ASM diskgroup as I was in a impression that the error would come when the diskgroup is mounted and when it runs out of adequate space to cope up with the space reqruiements from the database. After all R&D and trying all the options in our capacity, we have logged a SR with the Oracle support and the issue was opened for 4 days... yes, the database was down for 4 days, luckily it was a development database plus a week-end too.

The following has been recorded in the ASM alert.log :

Thu May 19 16:04:47 2011
GMON dismounting group 57 at 143 for pid 32, osid 23046
NOTE: Disk in mode 0x8 marked for de-assignment
ERROR: diskgroup DG_XXXX was not mounted
ORA-15032: not all alterations performed
ORA-15202: cannot create additional ASM internal change segment
ORA-15041: diskgroup "DG_XXXX" space exhausted
ERROR: alter diskgroup DG_XXXX mount
Thu May 19 16:05:10 2011
SQL> alter diskgroup DG_XXXX mount
NOTE: cache registered group DG_XXXX number=57 incarn=0xdc516e6a
NOTE: cache began mount (not first) of group DG_XXXX number=57 incarn=0xdc516e6a
NOTE: Assigning number (57,0) to disk (/dev/rdsk/oracle/data/ln1/xxxxxxx)
Thu May 19 16:05:14 2011
GMON querying group 57 at 145 for pid 31, osid 23739
NOTE: cache opening disk 0 of grp 57: DG_XXXX_0000 path:/dev/rdsk/oracle/data/ln1/xxxxxx
NOTE: F1X0 found on disk 0 au 2 fcn 0.0
NOTE: cache mounting (not first) external redundancy group 57/0xDC516E6A (DG_XXXX)
Thu May 19 16:05:14 2011
kjbdomatt send to inst 1
kjbdomatt send to inst 3
kjbdomatt send to inst 4
kjbdomatt send to inst 5
kjbdomatt send to inst 6
Thu May 19 16:05:14 2011
NOTE: attached to recovery domain 57
NOTE: redo buffer size is 256 blocks (1053184 bytes)
Thu May 19 16:05:14 2011
NOTE: LGWR attempting to mount thread 4 for diskgroup 57 (DG_XXXX)
NOTE: ACD expansion required for disk group 57
Thu May 19 16:05:33 2011
WARNING: unable to grow ACD, probably out of space
ERROR: ORA-15041 signalled during mount of diskgroup DG_XXXX
NOTE: cache dismounting (clean) group 57/0xDC516E6A (DG_XXXX)
NOTE: lgwr not being msg'd to dismount

The above messages indicates that ASM is trying to update or recover some metadata information and no space has been left in the diskgroup.

Nearly four different Oracle Engineers handled the issue and neither a workaround nor a solution was offered by any of them. Upon escalating the issue further, the support engineer asked me to use the AMDU utility and asked us to execute the following script:

./amdu -diskstring 'ASM disks location' -dump 'diskgroup_name'

When the above script was executed, the following was appeared on the screen:

AMDU-00204: Disk N0109 is in currently mounted diskgroup DG_XXXXX
AMDU-00201: Disk N0109: 'ASM disks location'

The first message raised my eyebrows, and started to think where the hell the diskgroup is mounted as I had tried on multiple instances to mount the diskgroup and ended up with the error. I then decided to go through the rest of the 7 ASM instances to check where the diskgroup is mounted. I found that the diskgroup was mounted on the first node and I was bit disappointed and sort of cursing myself why I didn't check all the ASM instances before. Upon adding a new asm disk to the diskgroup from the node1 ASM instance, I was able to mount the diskgroup successfully on the other nodes in the context.

The good thing about this investigation was coming across of the AMDU utility which is ported with Oracle 11g Grid Control software. I have also learned from MOS ID 553639.1 that the utility also can be configured on Oracle 10g (read the note for further instructions).

A directory with a couple of files will be created in the current location after executing the script. You can read the report.txt file, however, you can't read the other informative files, i.e., .map, as Map files are ASCII files that describe the data in the image files for a particular disk group.
 
Here is the extract of the note:

On Oracle 10g the content of a diskgroup can be reviewed if the diskgroup is mounted and there are internal views that display the specific allocation of the files (ASM files and Database files) inside of the disks.

When the diskgroup is not mounted, this information is not available, which makes difficult to diagnose the errors avoiding the diskgroup to be mounted.   This problem has been resolved with AMDU.

AMDU is a tool introduced in 11g where it is posible to extract all the available metadata from one or more ASM disks, generate formatted block printouts from the dump output, extract one or more files from a diskgroup (mounted/unmounted) and write them to the OS file system.

This tool is very important when dealing with internal errors related to the ASM metadata. Although this tool was released with 11g, it can be used with ASM 10g.

You can get the help about the utility with amdu -help

References:

https://twiki.cern.ch/twiki/bin/view/PDBService/ASM_utilities
Placeholder for AMDU binaries and using with ASM 10g [ID 553639.1]


Happy reading,

Jaffar

Does ASM has controlfile?

SELECT * FROM V$VERSION;
BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Prod
PL/SQL Release 10.2.0.1.0 - Production
CORE    10.2.0.1.0      Production
TNS for 32-bit Windows: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 - Production

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
+asm

SQL> show parameter control_files

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
control_files                        string      D:\ORACLE\PRODUCT\10.2.0\DB_1\DATABASE
                                                                     \CTL1+ASM.ORA

Probably we all know that ASM consist of an Instance, doesn't have database, i.e. physical structure. Therefore, the above definitely misleads.

According to the ASM book, the control_files parameters point to a dummy (nonexistent) file. This is the default internal value for this parameter and it is a meaningless value in the ASM instance.


Happy Reading,

Jaffar