How to manage DB and Cell servers remotely on Exadata with ExaCLI utility - (Part 1)

Whoever is working with Exadata or knew how to administrate Exadata major components like DB and Cell nodes, will certainly knows the use of CellCLI, dcli and DBMCLI utilities. This blog post is focused about managing database and cell servers remotely using the ExCLI and ExadCLI utilities.

Simply put, ExaCLI is a command line tool which comes by default on database & cell nodes that provides the capabilities of remote management for database and cell servers on the Exadata. Unlike CellCLI only runs on cell servers and DBMCLI runs on only DB nodes, the ExaCLI can manage database or cell servers remotely.

There are two key advantages of using the ExaCLI: 1) Its useful when you can't get SSH connectivity and root user credentials to connect DB or Cell nodes. 2) With Exadata at customer or cloud, customers won't get SSH and root user access for CellCLI and DBMCLI. So, the ExaCLI could be handy to access the servers.

ExaCLI works with the non-system (default) users on the DB or cell serers. Therefore, you must create a role based user in order use the utility to connect to a DB or cell server. The utility is found under /user/local/sbin directory.



Creating users for ExaCLI




Note : not all CellCLI commands are compatible on ExaCLI. Below sections givens an overview about the limitations:

Below are some of the examples demonstrate how to establish remote connectivity with DB or cell servers using ExaCLI:




In the next blog post, you will learn how to use the Exadcli utility execute the commands on a set of DB or cell nodes at once.


References:

https://docs.oracle.com/en/engineered-systems/exadata-database-machine/dbmmn/exadcli.html#GUID-1C738F05-2A69-4B75-BB1E-B578C9081487

https://docs.oracle.com/en/engineered-systems/exadata-database-machine/dbmmn/exacli.html#GUID-3D57CDC3-561C-48CE-AB1E-0279E6D24DAE

Exadata Cloud Machine - Hardware capacity

Everyone who is aware and utilizes Exadata Database Machine is certainly knew the performance it can deliver. I have involved in many Exadata migration projects, and witnessed how customers gained the database performance and satisfied post migration. I am not talking about the cost, the need etc., as a technical guy, I knew the capabilities of the box and how it can benefit customers to fulfill their need and future demand.

We all knew about Cloud technologies, how every software company and organization trying to race with the trend and need of cloud technologies. In some countries, the cloud adoption is bit slower compare to the other part of the world. But, gradually majority of the companies would be adopting cloud technologies, this is for sure. Certainly, cloud has its own share of advantages and disadvantages. Whoever utilizes it smartly, can gain much flexibility and benefits.

To ensure and meet customers demand to have Exadata availability on cloud, Oracle started Exadata Cloud services offering to facilitate Exadata machine on cloud. Still, some organization couldn't adopt cloud due to industry regulations, corporate policies, security compliance etc. Therefore, Oracle announced Exadata Cloud Machine availability. With this model, customers who want to have cloud on-premises with Exadata hardware, can go for this model.

I would like to highlight the hardware capabilities that Exadata Could Machine (ExaCM) offers.

• 40Gbps InfiniBand Networking
• Ultra-fast NVMe Flash storage
• Up to 257GB/sec Throughput
• Up to 3.6 Million 8k I/Os per sec
• 1/4 millisecond response time
• Fastest Compute 
• Fastest x86 processors
• Large Memory Capacity - 720GB per compute node
• Complete Redundancy

Soon will talk more about Exadata Cloud Machine migrations. Stayed tuned and hunger for knowledge.

Few Exadata MOS Docs to review

If you have MOS login credentials and managing Exadata database machines, below is the list of few MOS Doc which is worth reading:

• 888828.1, "Exadata Database Machine and Exadata Storage Server Supported Versions"
• 1070954.1, "Oracle Exadata Database Machine Exachk or HealthCheck"
• 1353073.2, "Exadata Diagnostic Collection Guide"
• 1500257.1, " Exadata Write-Back Flash Cache - FAQ"
• 1553103.1, "dbnodeupdate.sh and dbserver.patch.zip: Updating Exadata Database Server Software using the DBNodeUpdate Utility and patchmgr"
• 1589868.1, "Procedure to check for corrupted root file system on Exadata Storage Servers and Linux database servers"

(EX42) Flash disk failure may lead to ASM metadata corruption when using write-back flash cache

While reviewing the latest Exachk report on X5-2 machine, the following critical alrams were observed:

And details shows below description:

And the MOS Note : 1270094.1 explains the following:

According to MOS Doc: 2356460.1, the said behavior is due to a bug (27372426) which applies on Exa version 12.2.1.1.0 to 12.2.1.1.5 or 18.1.0.0.0 to 18.1.3.0.0.

Impact:

If you are running GI 11.2.0.4 or 12.1 with the above said Exa version, and  with FlashCache configured as Writeback mode, the following ORA error may encounter, during: ASM rebalancing operation, disk group mount, & disk group consistency checks, ASM review asm alert.log:

ORA-00600: internal error code, arguments: [kfdAuDealloc2]

WARNING: cache read a corrupt block: group=1(DATA) fn=381 indblk=27 disk=110 (DATA_CD_04_DM01CEL01)
ORA-15196: invalid ASM block header [kfc.c:26411] [endian_kfbh]

ORA-00600: internal error code, arguments: [kfrValAcd30]

ORA-00600: internal error code, arguments: [kfdAuPivotVec2], [kfCheckDG]

ERROR: file +DATADG1.3341.962251267: F3341 PX38530 => D55 A765853 => F1677
PX1647463: fnum mismatch
ERROR: file +DATADG1.3341.962251267: F3341 PX38531 => D15 A205431 => F3341
PX56068: xnum mismatch



Workaround:
To fix the bug, Following action plan needs to be applied:

1) Update the storage server to >=12.2.1.1.6 or >=18.1.4.0.0
2) Apply patch 27510959 and scan ASM metadata


Note :

The issues doesn't impact on GI 12.2 or whenever you have higher version of Exa software mentioned in this bug;
The bug also doesn't affect if the FlashCache mode is WriteThrough;

References:

Exadata Critical Issues (Doc ID 1270094.1)

(EX42) Flash disk failure may lead to ASM metadata corruption when using write-back flash cache (Doc ID 2356460.1)

Updating Exadata Software summary

Updating an Exadata software is one of the crucial tasks for any Database Machine Administrator (DMA). Though not necessarily one has to patch the environments whenever there is a new patch released by Oracle, but, it is highly recommended to patch the systems at least twice a year to fix any known &unknown bugs, security vulnerabilities and other issues.

This blog post summarizes the overall overview of software updates on an Exadata Database Machine. The post explains what components are needed the updates, the update order of components, pre-requisites and etc.

Typically, Exadata database machine updates are divided in the following categories:

• Exadata Infrastructure Software 
• Grid Infrastructure and Oracle Database Software

Updating the Exadata Software comprises of following components:

• Storage Servers
• Database Servers
• InfiniBand Switches

Software upgrade for Cell and DB nodes typically contains the updates for the following:

• OLE OS
• Exadata Software
• Firmware (Disk, Flash, RAID Controller, HCA, ILOM etc)

Pre-requisites

The following pre-upgrade activities are highly recommended before upgrading the Exadata software in any environment:

• Review MOS Doc 888828.1 and download the target version software
• Download observer.patch.zip from MOS Doc 1553103.1
• Review MOS Doc 1270094.1 for any critical issues
• Run the latest version of ExaCHK utility. Fix any FAIL and WARNINGS issues reported in the ExaCHK report. Also, review version recommendations in the MAA scoreboard section
• Ensure you have latest upgrade/patching utilities, such as, patchmgr, opatch etc. (MOS Doc 1070954.1)
• Perform prerequisites checks
• Backup the Exadata database servers before the update 

Rolling vs Non-rolling upgrades

Software updates can be performed online or offline (rolling or non-rolling) fashion. For online updates, it is highly recommended ASM high level disk group redundancy to avoid any data or service loss.

As part of best practices, the following is update order is recommended and treated as safe:

1. GI and Oracle Database home
2. Database Servers
3. Storage Servers
4. IB Switches

patchmgr update utiity

patchmgr update utility is used to patch the Exadata infrastructure components.  Following are the capabilities of patchmgr:

• Single invocation for Database servers, storage servers and IB Switches
• updates firmware, OS and Exadata softwares
• Online update advantage

Conclusion: Though the procedure looks pretty straight forward & simply when reading, with my past experience, patching each environments comes up with surprises and we need to be ready, unless we are very lucky on the particular day to have a smooth patching experience.

In the upcoming posts, I will talk about how to use patchmgr and other update utilizes to update Exadata software, Database, Storage servers and IB Switches.

Exadata and Capacity on Demand (CoD)

As most of us knew that the Exadata Database Machine comes in different sizes with different resource capacity. Not sure how many of you aware that Capacity on Demand (CoD) option can enable customers to start with limited active cores processors and increase them dynamically on demand. If CoD option is not enabled during the initial EDM configuration, then, all active cores are enabled by default and can't be reduced any further.

With X4-2 or higher, number of active cores can be reduced during the installation and can be increased based on the demand.  For X4-2, cores are increased in two (2) core increment, where as X4-8 increased in eight (8) core factor, see the table below.

Below example demonstrates the procedure to increase the active core processors:

Using DBMCLI utility:

DBMCLI> LIST DBSERVER attributes coreCount

DBMCLI> ALTER DBSERVER pendingCoreCount = new_core_count

DBMCLI> LIST DBSERVER attributes coreCount

Note: Once active cores are enabled (increased), there is no procedure to reduce them again.

Restart the database servers after increasing the core count.

Below table depicts the capacity-on-demand core configuration for various EDM types and releases:

Updates (10-Oct-2018):
Came across of the below blog post where the author described the procedure how to reduce the core count on Exadata.
I haven't tested though, and also not sure whether this is an Oracle approved approach to reduce the core factor on Exadata. However, its good to know the procedure.

https://grepora.com/2018/10/08/reduce-exadata-core-count/

Cloud Access Security Broker (CASB) for Oracle Cloud Infrastructure (OCI)

Customer adoption to cloud services (IaaS, PaaS, SaaS)  has been rapidly grown and growing. The most challenging aspect moving to cloud is the ability to secure the application and data that is put on the cloud. Oracle's hetrogenous security solution Cloud Access Security Broker (CASB) helps customers protecting their cloud-based infrastructure, platforms, applications across vendors. CASBs have emerged as  the go-to cloud security solution. CASB has the ability to provide security to entire cloud footprint (SaaS, PaaS, IaaS).

Most essentially, for all Oracle Cloud Infrastructure (OCI) deployments, it provides visibility, threat protection, data security and complaince. Following are a few key advantages of CASB:

• Governance of privileged activities
• Unified incident management
• Proactive remediation
• Continuous security compliance for OCI deployments

As part of complete visibility, it provides holistic view of entire cloud environment, including users and devices.

Threat Detection with User Behavior Analytics (UBA) builds a baseline for typical behavior, down to the user and application. Also, maintain a log when and how a user deviates from the baseline. With the help of predictive analytics, you can easily identify the risky users who performs folder permission change, changing user privileges , or tampering with the configuration settings.

All your cloud compliance configuration settings can be easily maintained. Once the settings are made, CASB monitoring the settings and alerts you whenever there is a change in the setting.

CASB provides three key components to secure your data in cloud:

1. Data visibility
2. Data inspection
3. Data Accessibility

It can easily integrate with the existing cloud security solutions, such as, SWG, NGF, IDaaS, DLP and SIEM.

For more details and information, visit Oracle website:

https://www.oracle.com/cloud/paas/casb-cloud-service.html

All about 'Autonomous Transaction Processing' - Part I

There has been a lot of buzz about 'Self driving & Self tuning database', 'autonomous', 'automation', etc. since Oracle 18c announced. I have decided to do my homework and test/validate some of them. So, this blog post will focus about 'Autonomous Transaction Processing (ATP)', how this is helpful to an organization and what role a DBA can play.

Its nothing but another typical cloud offering from Oracle. To begin with, Oracle ATP is built upon Oracle database and is designed, optimized to deliver scalable transaction performance across all standard business applications. As a service, ATP doesn't require DBA and no DBA intervention for any installation,configuration or management related activities. It handles all the DB related activities, such as, DB creation, backup, patching, upgrade, space management etc.

Its completely elastic service, where you can dynamically increase and decrease the resources (OCPU and storage capacity) without having any service interruption or database downtime. Using the cloud based console, you can easily manage the service, such as, scaling the service and monitoring. Additionally, cloud based notebook application provides easy querying, colobration and  data-visualization capabilities.

Below picture (source : Oracle documentation) describes the ATP architecture:

Below are some of the key features of Oracle Autonomous Transaction Processing:

• Simplified management of : rapid provisioning of new database, dynamic resource management (allocation and de-allocation of cpu and storage), patching & upgrades and backup & recovery
• complete elastic service
• Supports: existing, cloud and on-prime applications
• supports high query performance and concurrent workloads
• Easy data migration
• BI tools support
• Building reports and dashboards with analytics
• All data stores in encrypted formatted to secure the data
• Strong authenticity for connection and data access control

In part II, I will discuss details subscription, creating and users management.

Trying to avail 30 days free account on Oracle could. If I succeed to have the credentials, I will run through practically and post the configuration and management tasks.

Oracle 18c Autonomous Health Framework (AHF) - Part 1

Recently I had to present at eProseed AnualTech conference in Luxembourg and I was requested  to present a topic something about Oracle18c.

Obviously I don't want to talk and repeat the same about Autonomous Database, many experts already said much on this. I then decided to pick a topic which really helps DBAs, Administrators and finally to the organization. I was really fascinated about Oracle 18c autonomous health framework concepts and decided to do a presentation on this topics.

Working in a complex and huge Oracle environment, I knew where most of our energy and time is spend, as a DBA or system administrator. We always focus on avoiding run time availability and performance issues. In a complex and critical environment, every other day, you will face a new challenge and you must be on your toes as DBA during the business hours.

For DBA, most importantly, we need to ensure the database availability, at the same time, ensure its deliver the same performance 24x7. Imagine, if you get stuck with latches, instance crash, node crash, someone changes the binaries permission/ownership, you will spend hours and hours to fix and find the root cause of the issues.

With 18c autonomous health framework, its easy to avoid and auto fix run-time availability and performance issues. There are 8 components that makes this framework. Though some of them are present in 12.2, but, all these are configured automatically upon 18c configuration and run 24x7 in daemon mode. Also, 3 of the components have machine learning capabilities. to fix issue automatically.

I will start discussing about 8 components in next series. Stay tune for more on this topic.

utlrp weird behavior with INVAID objects in an Orace EBS database

In one of the recent database migration & upgrade activities, I have migrated an Oracle EBS database 11.2.0.4 to a new DB host and upgraded to 12.1.0.2. After migration and post upgrade, an utlrp.sql was ran to validate the 100k invalid objects in the database.

Weirdly, during the utlrp execution and when the INVALID objects count goes to 200, the INVALID objects counts started to increase again, and ultimately reaching the 100k number again. The utlrp was in kind of loop and never exited.

As a workaround, I have manually compiled all invalid objects and ran ultrp which ran successfully. I am yet to unlock the theory caused the situation.

Stay tuned for more updates on this post.

Traditional Export/Import Fail With ORA-01031 in DB Vault Environment Starting Release 11.2.0.3

Oracle Database Vault was configured on RAC database v11.2.0.3 and appropriate permissions given to allow the regular duties, such as data pump export/import etc.

However, when classical/traditional import (a dump file from v9 database) was performed, the import on 11.2.0.3 database failed with the following errors:

Import: Release 11.2.0.3.0 - Production on Tue May 29 16:56:23 2018

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

IMP-00058: ORACLE error 1031 encountered

ORA-01031: insufficient privilegesUsername: 

Data pump jobs were successfully executing without any issues. The only problem is with traditional export/import . According to below Oracle notes, it is an expected behavior in 11.2.0.3 with DB Vault in place.

Classic Export/Import Fail With ORA-01031 in Database Vault Environment Starting Release 11.2.0.3 (Doc ID 1419687.1)

Bug 13615338 - EXP/IMP disabled in Database Vault environment (Doc ID 13615338.8)

According to the notes,

Export and Import tools are no longer recommended to be used, especially in Database Vault environments.
When it is not possible to use data pump (for instance when the export has been created from a pre-10g database),
the patch 13615338 must be installed to allow the usage of classic export/import utilities.

The option would be to upgrade the database to higher than 11.2.0.3.


Will apply the patch and update this blog if the issue is fixed or not.