Exadata migration

Had a wonderful Sangam16 conference in India, and received much applaud for the two presentations delivered,  Oracle 12c multitenancy and Exadata migration best practices.

After a very short trip to India, life started to be business as usual again, and become busy. Was fully occupied with multiple assignments: Oracle EBS database health check assessment at a client for 2 days, GI/RDBMS/PSU deployments on Oracle Sun Super Cluster M7, Exadata configuration preparation and 9 databases migration to Exadata during the week-end.

Over the last week-end, we (me and my colleague) were pretty busy with 9 databases migration to Exadata. There were a few challenges , and learned a few new things too. I would like to discuss couple of scenarios that were interesting:

One of the databases had corrupted blocks, and the expdp was keep failing with ORA-01555: snapshot too old: rollback segment number  with name "" too small. Our initial thoughts were tuning undo_retention, increasing the undo tablespace, setting an event, etc. Unfortunately, none of the workarounds helped in the situation. We then cameacross a MOS note which explains that an ORA-01555 with "", no rollback segment name is probably due to corrupted blocks. After applying the solution explained in the note, we managed to export/import the database successfully. My colleague has blogged about the scenario at his blog: http://bit.ly/2fBOxm7

Another database is running on Windows x86 64-bit, and its full of LOBs, hence, the datapumps (expdp) took significant time, as NFS filesystem used to store the  dump file. We then thought of doing direct RMAN restore from source to target, as the database on Windows x86 64bit and Linux x86 64-bit are the same (Litten) Endian formats. As per one of the MOS notes, we can also do the Data Guard setup, and do RMAN restore. However, RMAN recovery would fail with ORA-600, as cross platform redo conversion won't be possible. We are now thinking of taking a cold backup (consistent) and do a complete restore with reset logs option.

Stayed tuned for more updates on this.

Wanna become an OCM? Go on read out Kamran's new OCM practical guide - MY TAKE

When I first heard about Kamran's new book, OCM practical guide, I said wow, because I was thinking on the same line a few years back, but, dropped the idea due to several factors, including the fact that the time and efforts required on this book.When he approached me to be one of the technical reviewers, I have accepted the deal straightaway without having a second thought. I am really honored to be part of this wonderful book which unfold the knowledge and help OCM aspirants to get what they dream in their lives.

I remember the debates at various places and at several Oracle forums where people discussed about the necessity and advantages of being Oracle certified professional. There were so many discussions and debates whether being certified professional will really add any value to one's career. Anyways, this is not the platform to discuss such things , however, in my own perspective, with real experience and having certification surely boosts the career and gives more chances to advance in the career.

First thing first, Kamran really put his heart out to come-up with such an extraordinary book in the form of practical guide. I thoroughly enjoyed reviewing every bits and bytes of the book, and the amount of practical examples demonstrate in this book is just prodigious. Only one with the real world and tremendous experience in the technology could do that. Take a bow my friend.

Each and every chapter got some great contents, and neatly explained with about 200+ practical examples. Let me walk through the chapters and give you my inputs:

Server Configuration provides a detailed step-by-step guide on Oracle 11g Database Software setup and new database creation through GUI and silent mode likewise. Also, outlines the the procedure to configure network settings, listener, tns names and etc.

Enterprise Manager Grid Control explains step-by-step procedure to install and configure OEM, and how to schedule and manage stuff.  

Managing Database Availability one of the important chapters, not only for OCM preparation perspective, to manage our production databases, deploying optimal backup and recovery strategies to secure the databases.

Data Management chapter provides detailed information about types of materialized view and materialized view log and how Oracle uses precomputed materialized view instead of querying a table with different aggregate functions and provides a quick result.

Data Warehouse Management chapter provides information about main data warehouse topics such as partitioning and managing large objects. Next, we talk about large objects and show how to use various SecureFile LOB features such as compression, deduplication, encryption, caching and logging.

Performance Tuning I particularly enjoyed this reading and reviewing this chapter, Its the heart of the book with so much explanation and practical examples. This one shouldn't be missed out. 

Grid Infrastructure and ASM contains all you wanted to know about GRID infrastructure and ASM technologies. How to install GRID and create disk groups.

Real Application Clusters explores steps to successfully create a RAC database on two nodes. With only few additional steps, you will successfully create a RAC database. Then you see the way how to create and configure ASM with command line interface. Once ASM is configured, silent RAC database creation steps are provided.

Data Guard chapter starts by creating a data guard using command line interface, OEM and data guard broker. It also provides steps on performing switchover and failover using all mentioned tools.


In nutshell, its a practice guide with a perfect recipe and ingredients to become an OCM, which evenly blended with many useful examples and extraordinary explanation. The wait is over, this is the book we all been looking for a long time. Go and place your order and get certified, become an OCM.

You can place the order through Amazon, use the below URL:

https://www.amazon.com/Oracle-Certified-Master-Study-Guide/dp/1536800791/ref=sr_1_fkmr1_1?ie=UTF8&qid=1477409700&sr=8-1-fkmr1&keywords=Kamran+ocmds=Kamran+ocm

Is Fixed Objects Stats needed in PDB?

Doesn't matter if we are working on a new technology or something we are very family with. When we encounter tricky situations, sometimes neither we could find things in the manuals nor could find any useful references.

All what you need is an action by yourself and then become a reference to others .

Mike Dietrich had received an interesting question about whether Gather Fixed Objects required for an indiviudal PDBs or just required on the ROOT Container. He did a small test and explained the answer at oracle.blogs.

https://blogs.oracle.com/UPGRADE/entry/gather_fixed_objects_stats_in

Oracle Critical Patch Update for October 2016

The Critical Patch Update for October 2016 was released on October 18th, 2016. Oracle strongly recommends applying the patches as soon as possible. 

Visit the URL below for more updates

http://www.oracle.com/us/dm/sev100575522-na-ww-ot-de1-ev-3253663.html?elq_mid=59973&sh=0802222319060808261813292520100701&cmid=SPPT160711P00036

reports (cvucheckreport_*.xml) from ora.cvu resource causing excessive disk space

On Exadata dbnodes, we noticed that Clusterware 12.1.0.2 ora.cvu resource generating XML reports every 6 hours and occupied large amounts of space on /u01 filesystem. Perhaps this is the same on non-Exadata systems too, if you are running 12.1.0.2 Clusterware.

Upon looking into MOS ID (reports from ora.cvu resource using excessive disk space (Doc ID 1964482.1) it was learned that this is an expected behavior with 12.1.0.2 where ora.cvu invokes Clusterware Health check (CHC) every 6 hrs and stores the XML reports under $GI_HOME/cvu/baseline or $ORACLE_BASE/crsdata/@global/cvu/baseline' location. However, the CVU didn't clean-up the reports periodically which ultimately causes  disk full.

As per the note, the bug is fixed as follows:

Bug 18143707 is fixed in 12.1.0.2

Bug 19703199 is fixed in 12.1.0.2.4 PSU

Bug 20177779 is fixed in 12.1.0.2.5 PSU

Quick workaround to this problem is to delete the files (xml) manually in order to release the space.

References:

reports from ora.cvu resource using excessive disk space (Doc ID 1964482.1)

ora.cvu Resource Reports not Purged (Doc ID 2054765.1)

12.1.0.2 Grid Infrastructure Patch Set Updates - List of Fixes in each GI PSU (Doc ID 1928853.1)

Learn and Master the concepts of Oracle 12c RAC and EM 13c from Oracle Experts

Learn and master the concepts of Oracle RAC 12c and EM 13c from Experts, online Instructor lead course.

Enroll now and avail 20% discount upfront, exclusively for my network.

http://dbatalent.com/product/oracle-12c-rac/

Coupon, exclusively for my network : SJRAC20

At the end of the course and last 2 days, I will be sharing the following:

Day 1 : RAC best practices in real-world scenarios:

- Installation, Network, Storage, Application Design, Backup & Recovery

RAC internals

Day 2 : Taking all your questions. Exclusively taking your questions and answering them.

Guys, do forward this to your networking and help your friends/dears to master Oracle RAC and EM 13c concepts.

Course Objective:

To master Oracle Real Application Clusters 12c. Course is for Oracle DBAs or Apps DBAs looking to upgrade their skill.

Expectations and Goals

• Install, Configure & Administer 12c RAC
• Install & Configure 13c Cloud Control
• Upgrade Database/RAC from 11g to 12c
• Install, Configure & Administer 12c Grid Infrastructure

Course Agenda:

• RAC Introduction
• RAC Architecture
• Install and configure 12c RAC
• Administer GI & RAC
• Monitoring and Troubleshooting RAC
• Upgrade RAC and Database
• Migrate DB to ASM and RAC
• Install, Configure EM 13c
• Backup, Recovery & Cloning
• Performance Tuning

The New Batch is going to start from Friday, 14th October 2016 from 07:00 PM IST

Explore more about 12c Multitenant with useful MOS Docs

As part of an ongoing Exadata migration assignment, we have been exploring Oracle 12c multitenant capabilities a lot. I would like to share here some of selected useful MOS Doc about 12c multitenant concepts:

Below docs explains most of the 12c multitenant concepts:

• Initialization parameters in a Multitenant database - Facts and additional information (Doc ID 2101596.1)
• Initialization parameters in a Multitenant database - FAQ and Examples (Doc ID 2101638.1)
• Oracle Multitenant Option - 12c : Frequently Asked Questions (Doc ID 1511619.1)
• 12c Multitenant Container Databases (CDB) and Pluggable Databases (PDB) Character set restrictions / ORA-65116/65119: incompatible database/national character set ( Character set mismatch: PDB character set CDB character set ) (Doc ID 1968706.1)
• How to Restore - Dropped Pluggable database (PDB) in Multitenant (Doc ID 2034953.1)
• Script For Getting Complete Basic Information about configured CDB and PDB in Oracle Database Multitenant 12c (Doc ID 2012221.1)
• Difference Between Major Components of Traditional Databases and Multitenant Databases CDB/PDB Introduced in Version 12c (Doc ID 2013529.1)
• How to set a Pluggable Database to have a Different Time Zone to its own CDB (Doc ID 2127835.1)
• Where Manageability Data is Stored in 12c Multi-tenant (CDB) database (Doc ID 1586256.1)
• Master Note for the Oracle Multitenant Option (Doc ID 1519699.1)

Below notes outlines guideline to manage 12c Multitenant Container Database while Oracle Data Guard configuring is in-place:

• Step by Step Examples of Migrating non-CDBs and PDBs Using ASM for File Storage (Doc ID 1576755.1)
• Making Use Deferred PDB Recovery and the STANDBYS=NONE Feature with Oracle Multitenant (Doc ID 1916648.1)
• Data Guard Impact on Oracle Multitenant Environments (Doc ID 2049127.1)
• PDB Failover in a Data Guard environment: Unplugging a Single Failed PDB from a Standby Database and Plugging into a New Container (Doc ID 2088201.1)

New ORAchk & EXAchk Versions 12.2.0.1.1 Released

New ORAchk & EXAchk v12.2.0.1.1 release is out now with so many new features. Its easy to use and upgrade.

Below is the blog entry about the new release and features by RAC Assurance Development team at Oracle:

https://www.linkedin.com/pulse/new-orachk-exachk-versions-122011-released-gareth-chapman?trk=hp-feed-article-title-publish

Upgrade yourself from DBA to DMA

Be it any profession, learning never ends. From my very own personal perspective and exprience, I strongly beleinv in one who upgrade his skills always stays a step ahead from the rest. To stay competitive, it is necessary to upgrade our skills according to the market demand.

For DBA folks who aim and dream to learn and upgrade their knwoledge with Oracle Exadata technologies, I wrote a 5 series article to Toad, below are the links to the articles:

Exploring Exadata - for experienced non-Exadata DBAs (Part I)

Exploring Exadata - for experienced non-Exadata DBAs (Part II)

Exploring Exadata - for experienced non-Exadata DBAs (Part III)

Exploring Exadata - for experienced non-Exadata DBAs (Part IV)

Exploring Exadata - for experienced non-Exadata DBAs (Part V)

Below are few more Exadata related stuff.

Managing & Troubleshooting Exadata - Upgrading & Patching Exadata

Shrink/Grow Exadata diskgroups

Storage difference between 2 identical Exa boxes. How and why?

Pass on the knolwedge to the community.

Happy reading.

E-book: Discover the breakthrough innovations in Oracle Database 12c

Download the eBook and read to learn about 12c (the next generation most popular database) :

• Simplify database consolidation and manage them as cloud services

• Accelerate analytical performance whilst achieving new levels of efficiency, security and availability

• Lower cost and mitigate risk

http://oracle.com.edgesuite.net/ebook/db12c/pdf/Oracle_Database_12c_E-Book_2014.pdf

AIOUG Sangam'16 - India's largest Annual Oracle Confernece - DON'T MISS IT

AIOUG Sangam'16, India's largest annual Oracle conference, to be held in Bangalore this year during 11th and 12th November.

I am glad and super excited to present and attend the conference. I am going to present the following two sessions:

• Exadata Migration Recipes - Secret Sauce for a Tasty Migration

Abstract

The purpose of this presentation is to provide real-world augmented database migration procedures that allowed us to successfully achieve the migrations of 54 business critical databases to Exadata x5-2 of different sizes and Operating Systems. The presentation will layout the different possible migration technologies that meets the business demands such as Data Pumps, RMAN, XTTS with Incremental Backups.  The presentation will also highlight and focus on the challenges that were encountered and their solutions. A very specific scenario will also be presented: Cross platform Oracle EBS migration of 30TB size.

• Cost Optimization and Better Database Management with Oracle 12c Multitenant A 360 Degree Overview

Abstract

Organizations who deploy and manage several numbers of standalone databases in their complex environments typically face the uphill task of better (ease) database management and also Cost optimization. Oracle 12c Multitenant got the ability to consolidate databases, which addresses the major concerns of those Organizations and provide the easier Database management to the DBAs. The presentation will focuses on the key advantages and benefits of 12c Multitenant features, and how one can reduce the IT cost as well as manage their databases efficiently. Outlines the architecture, deployment scenarios, easy management of Multitenant features.

Why should you attend
If you are around in India during those dates, I strongly recommend you to avail the opportunity to meet and learn from many international stars. AIOUG really did a tremeounds job to bring famous Oracle personalities together on board for this year's event.I am really honored and indeed my previlege to present and listen to other fellow speakers.

Visit the website to for more details: enrollment, view list of topics and spearks etc http://www.aioug.org/sangam16/

Look forward to seeng you at the event!

How to view database patching information in Oracle 12.1.0 and higher

We are currently engaged with an assignment to migrate over 50 productions database to Exadata of different sizes , different OS and different business priority. Each passing day, we encounter some technical challenges and of course each situation educates and adds more knowledge.

I would like to share a small post on how to list/view the patching information  in Oracle >= 12.1.0 version:

As of 12.1.0.2, you can extract the database patches information using the following methods:
1. Traditional 'opatch -lsinventory'
2. Review database alert.log
3. Use the dbms_qopatch package

Review database alert.log

We notice an Oracle 12.1.0.2 instance (at least on Exadata x5-2) dumping current patch information in its alert.log. This is pretty good, you no longer need to query lsinventory to know the patches that are applied. Just review your alert.log to know the patching information.

Below is the excerpt of the alert.log:

Wed May 25 17:59:05 2016
===========================================================
Dumping current patch information
===========================================================
Patch Id: 20243804
Patch Description: DATABASE BUNDLE PATCH: 12.1.0.2.5 (20243804)
Patch Apply Time: 2016-04-19 17:01:42 GMT+03:00
Bugs Fixed: 13498243,13640676,14165431,14254610,14643995,15894842,15996428,
16010876,16293223,16356176,16359751,16556115,16619249,16870214,16938780,


Use the dbms_qoptch package:

with a as (select dbms_qopatch.get_opatch_lsinventory patch_output from dual)
select x.*
from a,
xmltable('InventoryInstance/patches/*'
passing a.patch_output
columns
patch_id number path 'patchID',
patch_uid number path 'uniquePatchID',
description varchar2(80) path 'patchDescription',
applied_date varchar2(30) path 'appliedDate',
sql_patch varchar2(8) path 'sqlPatch',
rollbackable varchar2(8) path 'rollbackable'
) x;

Apparently there is a bug wen the query is executed on 12.1.0.1. Read "12.1.0.1 datapatch issue : ORA-27477: "SYS"."LOAD_OPATCH_INVENTORY_1" already exists (Doc ID 1934882.1)"

Learning curve (Oracle 12c Multitenant, Oracle Cloud & Golden Gate)

First thing first. After almost 8 yrs of successful tenure at my previous company, I have moved on to new challenges from 1-Mar-2016. Joined eProseed KSA as Technical Director where my prime responsibility is to involve in pre-sales, technical planning, motivating teams and hands-on technically in critical projects. I must say, this is what I was looking for a very long time and I am sure I gonna enjoy my new role very much.

Over the past couple of weeks, I have been busy exploring the following concepts, though they are not very new to more people:

• Oracle 12c Multitenant
• Oracle Cloud
• Golden Gate
• Enterprise Manager Cloud Control 13c

Also involved in additional task, which I can't reveal due to NDA, but will reveal later on.

Started a new Whatsapp group Trend Oracle Cloud with more than 60 members as of now. 

Hope everyone of you doing great. Stay tuned for more updates.

Storage difference between 2 identical Exa boxes. How and why?

We noticed around 1.6TB storage difference between two Eight (1/8) Exadata boxes while configuring Data Guard. Wondered what went wrong. The Exa box configured for DR was around 1.6TB short compare to the other Exa box. Verified the lun, physical disk and grdidisk status on a cell, which showed active/online status. The tricky part on Exadata is, everything has to be active/online across all cell storage servers. We then figured-out that grid disk status on the 3rd cell storage server was inactive. After making them active on the 3rd cell server, everything become normal, i mean, the missing 1.6TB space appeared.

When you work with Exadata, you need to verify all cell storage servers to confirm the issue, rather than just query things over just one cell server.

Gather system stats on EXADATA X4-2 throws an ORA-20001,ORA-06512

After a recent patch deployment on Exadata X4-2 system, the following error encountered whilst gathering system stats with Exadata mode (dbms_stats.gather_system_stats('EXADATA')):

SQL> exec dbms_stats.gather_system_stats('EXADATA');
BEGIN dbms_stats.gather_system_stats('EXADATA'); END;

*
ERROR at line 1:
ORA-20001: Invalid or inconsistent input values
ORA-06512: at "SYS.DBMS_STATS", line 27155
ORA-06512: at line 1

It appears to be a BUG, and the following workaround should resolve the issue:

(1) As SYS user execute the scripts below:

SQL> @?/rdbms/admin/dbmsstat.sql
SQL> @?/rdbms/admin/prvtstas.plb
SQL> @?/rdbms/admin/prvtstat.plb
SQL> @?/rdbms/admin/prvtstai.plb

(2) Then, re-run your DBMS_STATS call:

exec dbms_stats.gather_system_stats('EXADATA');

Indeed this worked for us and hope this would work for you as well.