Bug 14026888 - Datapump import gets ORA-600 [kpudpxcs_ctxConvertStream_ref_1] [ID 14026888.8] - ORA-31693,ORA-02354,ORA-00600

When a full database datapump import operations executed this morning to load data from an Oracle 11gR1 db to 11gR2 db, the job got failed with the following ORA errors on over 50 tables that has LOB data type:

ORA-31693: Table data object failed to load/unload & is being skipped due to
ORA-02354: error in exporting/importing data
ORA-00600: internal error code, arguments: [kpudpxcs_ctxConvertStream_ref_1],  

A very quick research about an ORA-600 error in the metalink brought the following facts and workaround to our knolwedge:


Product (Component)Oracle Server (Rdbms)
Range of versions believed to be affected Versions BELOW 12.1
Versions confirmed as being affected
Platforms affectedGeneric (all / most platforms affected)


This issue is fixed in

Since there is no fix available yet, we applied the workaround, used conventional exp/imp, mentioned in the notes. Bravo, the workaround does worked and the issue got resolved..

Note : The BUG appears to be a generic one and applicable to all platforms.


addNode.sh ERROR [PRKC-PRCF-2015] and RESOLUTION

Here is a very quick and short blog update about the issue we had confronted whilst performing an addNode action.

As part our ongoing HPUX servers migration project (Superdom 1 to Superdome 2 ), we agreed on a plan, where we wanted to add same amount of nodes (new configuration) to the exists cluster environment and then delete the old nodes after successfully migrating everything over the new nodes.

Well the addnode procedure was going smoothly until the following error occurrence :
PRCF-2023 : The following contents are not transferred as they are non-readable
1) /u00/app/11.2.0/grid_1/bin/oradaemonagent
2) /u00/app/11.2.0/grid_1/crs/utl/cluvfy
3) /u00/app/11.2.0/grid_1/srvm/admin/logging.properties

Refer to '/u00/app/oracle/oraInventory/logs/addNodeActions2012-06-22_06-27-57AM.log' for details. You may fix the errors on the required remote nodes. Refer to the install guide for error recovery.
96% Done.
The previous experience on this error to perform the following action plan:
  • Copy those files to the target node.
  • Execute orainstRoot.sh on the target node followed by root.sh execution.
  • Create a new local inventory on the new nodes followed by inventory update action across all nodes.

This time around we didn't see the instructions to manually execute the orainstRoot.sh in the error. We also found that the file (orainstRoot.sh) doesn't exists under oraInventory location on the new node.

Although it was a clearly indication of sort of permission issue, we didn't want to take the risk as its a business critical environment, and we opened a TAR with Oracle support to seek their assitance. After going through the painful process (providing logs and other feed back), the engineer asked us to add an additional read permission (chmod o+r to those files) to the files in the context and instructed to re-initate the addNode procedure.

Well, after addiging an additional read previlige to the files, the addNode procedure started over and it went on smoothly.

Small things some time can draw you crazy.  Anyways, its part of life, you need to deal with it.

Happy reading,




It's Alive! OOW2012 - Content Catalog is out now

Hi everyone,

A very quick update about upcoming OOW2012 event.
OOW2012 content catalog is out now (https://blogs.oracle.com/oracleopenworld/entry/it_s_alive). You can now search for your favorite sessions and sessions. Don't wait any longer, start planning your events from now.

See you at OOW2012.



ORA-07445: exception encountered: core dump [evaopn3()+384] [SIGSEGV]

Before I really get started discussing seriously about the subject matter, I would like to shared and echo what Coskan's had said at his blog.

"Every Oracle version, developers at Oracle add or change some of the undocumented parameters to do better optimization. Most of the time these new optimizations works fine but from time to time they have a negative effect for the generated plan which causes post upgrade slowness. When you change OFE (optimizer_features_enable) from to Oracle changes value of 33 parameters !!! Because they are undocumented, unless an Oracle Scientist reveals what they are actually doing or note on MOS explaining the parameter, their effects to the plans are not that clear."  (although a bit older stuff, still worth reading his blog entry about Plan Stability Through Upgrade).

First and foremost, it is indeed a big disappointment (at least for ourselves) the way Oracle 11gR2 CBO behave/manage things with regards to ENCRYPTED columns in contrast to 10gR2 CBO. We really had enough surprises post 11gR2 upgrade with most of our business critical databases where ENCRYPTED column tables involved in a query. The queries in the context had terrible performance degradation and it was totally unacceptable. Since there was no suitable workaround great help from the Oracle Support, we decided to DECRYPT the columns temporary to fall back to 10gR2 behavior.

Couple of days ago, I have been notified by one of the application support guys about the wired situation of some queries, where the queries run OK at first run and fail in another run to produce the results. When I simulated the test case over non-production database, the session constantly terminating with following errors:

ERROR at line 20:
ORA-03113: end-of-file on communication channel
Process ID: 12560
Session ID: 104 Serial number: 21213

Additionally, an ORA- 07445 error also written in the alert.log.

Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x1200000000] [PC:0x4000000006320C80, evaopn3()+384] [flags: 0x0, count: 1]
Errors in file /u00/app/oracle/diag/rdbms/pmsit/DBXX/trace/DBXX_ora_12560.trc  (incident=48362):
ORA-07445: exception encountered: core dump [evaopn3()+384] [SIGSEGV] [ADDR:0x1200000000] [PC:0x4000000006320C80] [Address not mapped to object] []
Incident details in: /u00/app/oracle/diag/rdbms/pmsit/DBXX/incident/incdir_48362/DBXX_ora_12560_i48362.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Jun 04 16:37:55 2012
Dumping diagnostic data in directory=[cdmp_20120604163755], requested by (instance=1, osid=12560), summary=[incident=48362].
Mon Jun 04 16:37:59 2012
Sweep [inc][48362]: completed
Sweep [inc2][48362]: completed

After a very quick initial analysis, found MOS Note "ORA-7445 (evaopn3) [ID 860969.1], that describes the cause as one of the BUG due to ENCRYPTED columns. It was confirmed after looking at the table structure, we indeed have ENCRYPTED columns. The MOS Note explained that the fix would be in version 12 and also have the following suggestion:


I then googled about the parameter in the question and come across several great posts from many people, including Oracle CBO blog. Although the join factorization (JF) appears to be a great improvement for the queries with UNION ALL and with SELF/INNER JOIN conditions, where it could drastically improve the query performance, unfortunately, we had to cut if OFF due to column ENCRYPTION.. The query went well after turning the feature OFF.

I have couple more interesting topics to discuss in my next blog entries.

Till then, happy reading.


Advt: Oracle Enterprise Manager 12c Cloud Control: Managing Data Center Chaos

My friend and former ACE Director Porus Homi Havewala's new book on 
Enterprise Manager 12c Cloud Control, will be available in September 2012 
or earlier, perhaps the First EM 12c Cloud Control book in the world.
If you are interested in learning about the capabilities of Enterprise Manager 12c,
please have a look. The book can be pre-ordered in advance. Electronic copies 
will be available too on publication.
Oracle Enterprise Manager 12c Cloud Control: Managing Data Center Chaos
Happy reading