11.18.2018

Exadata Cloud Machine - Hardware capacity

Everyone who is aware and utilizes Exadata Database Machine is certainly knew the performance it can deliver. I have involved in many Exadata migration projects, and witnessed how customers gained the database performance and satisfied post migration. I am not talking about the cost, the need etc., as a technical guy, I knew the capabilities of the box and how it can benefit customers to fulfill their need and future demand.

We all knew about Cloud technologies, how every software company and organization trying to race with the trend and need of cloud technologies. In some countries, the cloud adoption is bit slower compare to the other part of the world. But, gradually majority of the companies would be adopting cloud technologies, this is for sure. Certainly, cloud has its own share of advantages and disadvantages. Whoever utilizes it smartly, can gain much flexibility and benefits.

To ensure and meet customers demand to have Exadata availability on cloud, Oracle started Exadata Cloud services offering to facilitate Exadata machine on cloud. Still, some organization couldn't adopt cloud due to industry regulations, corporate policies, security compliance etc. Therefore, Oracle announced Exadata Cloud Machine availability. With this model, customers who want to have cloud on-premises with Exadata hardware, can go for this model.

I would like to highlight the hardware capabilities that Exadata Could Machine (ExaCM) offers.


  • 40Gbps InfiniBand Networking
  • Ultra-fast NVMe Flash storage
  • Up to 257GB/sec Throughput
  • Up to 3.6 Million 8k I/Os per sec
  • 1/4 millisecond response time
  • Fastest Compute 
  • Fastest x86 processors
  • Large Memory Capacity - 720GB per compute node
  • Complete Redundancy
Soon will talk more about Exadata Cloud Machine migrations. Stayed tuned and hunger for knowledge.

10.10.2018

Few Exadata MOS Docs to review

If you have MOS login credentials and managing Exadata database machines, below is the list of few MOS Doc which is worth reading:

  • 888828.1, "Exadata Database Machine and Exadata Storage Server Supported Versions"
  • 1070954.1, "Oracle Exadata Database Machine Exachk or HealthCheck"
  • 1353073.2, "Exadata Diagnostic Collection Guide"
  • 1500257.1, " Exadata Write-Back Flash Cache - FAQ"
  • 1553103.1, "dbnodeupdate.sh and dbserver.patch.zip: Updating Exadata Database Server Software using the DBNodeUpdate Utility and patchmgr"
  • 1589868.1, "Procedure to check for corrupted root file system on Exadata Storage Servers and Linux database servers"

10.08.2018

(EX42) Flash disk failure may lead to ASM metadata corruption when using write-back flash cache

While reviewing the latest Exachk report on X5-2 machine, the following critical alrams were observed:



And details shows below description:


And the MOS Note : 1270094.1 explains the following:


According to MOS Doc: 2356460.1, the said behavior is due to a bug (27372426) which applies on Exa version 12.2.1.1.0 to 12.2.1.1.5 or 18.1.0.0.0 to 18.1.3.0.0.

Impact:

If you are running GI 11.2.0.4 or 12.1 with the above said Exa version, and  with FlashCache configured as Writeback mode, the following ORA error may encounter, during: ASM rebalancing operation, disk group mount, & disk group consistency checks, ASM review asm alert.log:

ORA-00600: internal error code, arguments: [kfdAuDealloc2]

WARNING: cache read a corrupt block: group=1(DATA) fn=381 indblk=27 disk=110 (DATA_CD_04_DM01CEL01)
ORA-15196: invalid ASM block header [kfc.c:26411] [endian_kfbh]

ORA-00600: internal error code, arguments: [kfrValAcd30]

ORA-00600: internal error code, arguments: [kfdAuPivotVec2], [kfCheckDG]

ERROR: file +DATADG1.3341.962251267: F3341 PX38530 => D55 A765853 => F1677
PX1647463: fnum mismatch
ERROR: file +DATADG1.3341.962251267: F3341 PX38531 => D15 A205431 => F3341
PX56068: xnum mismatch



Workaround:
To fix the bug, Following action plan needs to be applied:

1) Update the storage server to >=12.2.1.1.6 or >=18.1.4.0.0
2) Apply patch 27510959 and scan ASM metadata


Note :

The issues doesn't impact on GI 12.2 or whenever you have higher version of Exa software mentioned in this bug;
The bug also doesn't affect if the FlashCache mode is WriteThrough;

References:

Exadata Critical Issues (Doc ID 1270094.1)


9.15.2018

Updating Exadata Software summary

Updating an Exadata software is one of the crucial tasks for any Database Machine Administrator (DMA). Though not necessarily one has to patch the environments whenever there is a new patch released by Oracle, but, it is highly recommended to patch the systems at least twice a year to fix any known &unknown bugs, security vulnerabilities and other issues.

This blog post summarizes the overall overview of software updates on an Exadata Database Machine. The post explains what components are needed the updates, the update order of components, pre-requisites and etc.

Typically, Exadata database machine updates are divided in the following categories:

  • Exadata Infrastructure Software 
  • Grid Infrastructure and Oracle Database Software

Updating the Exadata Software comprises of following components:

  • Storage Servers
  • Database Servers
  • InfiniBand Switches
Software upgrade for Cell and DB nodes typically contains the updates for the following:
  • OLE OS
  • Exadata Software
  • Firmware (Disk, Flash, RAID Controller, HCA, ILOM etc)

Pre-requisites

The following pre-upgrade activities are highly recommended before upgrading the Exadata software in any environment:

  • Review MOS Doc 888828.1 and download the target version software
  • Download observer.patch.zip from MOS Doc 1553103.1
  • Review MOS Doc 1270094.1 for any critical issues
  • Run the latest version of ExaCHK utility. Fix any FAIL and WARNINGS issues reported in the ExaCHK report. Also, review version recommendations in the MAA scoreboard section
  • Ensure you have latest upgrade/patching utilities, such as, patchmgr, opatch etc. (MOS Doc 1070954.1)
  • Perform prerequisites checks
  • Backup the Exadata database servers before the update 
Rolling vs Non-rolling upgrades

Software updates can be performed online or offline (rolling or non-rolling) fashion. For online updates, it is highly recommended ASM high level disk group redundancy to avoid any data or service loss.

As part of best practices, the following is update order is recommended and treated as safe:
  1. GI and Oracle Database home
  2. Database Servers
  3. Storage Servers
  4. IB Switches
patchmgr update utiity

patchmgr update utility is used to patch the Exadata infrastructure components.  Following are the capabilities of patchmgr:
  • Single invocation for Database servers, storage servers and IB Switches
  • updates firmware, OS and Exadata softwares
  • Online update advantage
Conclusion: Though the procedure looks pretty straight forward & simply when reading, with my past experience, patching each environments comes up with surprises and we need to be ready, unless we are very lucky on the particular day to have a smooth patching experience.

In the upcoming posts, I will talk about how to use patchmgr and other update utilizes to update Exadata software, Database, Storage servers and IB Switches.

9.14.2018

Exadata and Capacity on Demand (CoD)

As most of us knew that the Exadata Database Machine comes in different sizes with different resource capacity. Not sure how many of you aware that Capacity on Demand (CoD) option can enable customers to start with limited active cores processors and increase them dynamically on demand. If CoD option is not enabled during the initial EDM configuration, then, all active cores are enabled by default and can't be reduced any further.

With X4-2 or higher, number of active cores can be reduced during the installation and can be increased based on the demand.  For X4-2, cores are increased in two (2) core increment, where as X4-8 increased in eight (8) core factor, see the table below.

Below example demonstrates the procedure to increase the active core processors:

Using DBMCLI utility:

DBMCLI> LIST DBSERVER attributes coreCount

DBMCLI> ALTER DBSERVER pendingCoreCount = new_core_count

DBMCLI> LIST DBSERVER attributes coreCount

Note: Once active cores are enabled (increased), there is no procedure to reduce them again.

Restart the database servers after increasing the core count.

Below table depicts the capacity-on-demand core configuration for various EDM types and releases:














Updates (10-Oct-2018):
Came across of the below blog post where the author described the procedure how to reduce the core count on Exadata.
I haven't tested though, and also not sure whether this is an Oracle approved approach to reduce the core factor on Exadata. However, its good to know the procedure.

https://grepora.com/2018/10/08/reduce-exadata-core-count/