Database hang and Row Cache Lock concurrency troubleshooting

Issue background

This post will help to analyze Oracle database instance slowdown that can happen due to considerable row cache lock (enqueue) wait events. It’s is based on a real case of a database hang that I worked on recently. I must admit this type of situation does not appear often but it’s very dangerous since it can considerably slow down a database instance or even freeze it for a short period of time. In most cases SQL against ASH view and Systemstate dumps can help to nail down the problem unless this is an Oracle bug.

Usually it occurs suddenly and disappears quickly. See an example ASH graph below with brown peak that represents this type of concurrency: row cache lock wait events.

ASH graph - Row Cache Lock concurrency

ASH graph – Row Cache Lock concurrency

What is a Row Cache Enqueue Lock?

The Row Cache or Data Dictionary Cache is a memory area in the shared pool that holds data dictionary information. Row cache holds data as rows instead of buffers. A Row cache enqueue lock is a lock on the data dictionary rows. It is used primarily to serialize changes to the data dictionary and to wait for a lock on a data dictionary cache. The enqueue will be on a specific data dictionary object. This is called the enqueue type and can be found in the v$rowcache view. Waits on this event usually indicate some form of DDL occurring, or possibly recursive operations such as storage management, sequence numbers incrementing frequently, etc. Diagnosing the cause of the contention

Diagnosing the cause of the contention

Read more »

Oracle database housekeeping methods – ADR files purge

Since Oracle Database 11g, all traces, incident dumps and packages, the alert log, Health Monitor reports, core dumps, and more files are stored in the ADR, a file-based Automatic Diagnostic Repository for simplified database diagnostic data management. In spite of that new 11g feature, the housekeeping or purging of those files got not considerably simplified. Unfortunately the new Oracle utility ADRCI can not purge all the files that Oracle database generates in ADR. Thus, you should use the other Unix OS methods and tools to accomplish Oracle 11g database housekeeping. In previous posts I described the Unix log rotation mechanism and SYS auditing files purge in adump directory. This time we will talk exactly about ADR purge functionality. But before, let’s look at some basics.

Location for Diagnostic Traces

Location of Diagnostic Traces of Oracle ADR

Location of Diagnostic Traces in Oracle 11g ADR

The table shown above in the slide describes the different classes of trace data and dumps that reside both in Oracle Database 10g and in Oracle Database 11g.
With Oracle Database 11g, there is no distinction between foreground and background trace files. Both types of files go into the $ADR_HOME/trace directory.
All non-incident traces are stored inside the trace subdirectory. This is the main difference compared with previous releases where critical error information is dumped into the corresponding process trace files instead of incident dumps. Incident dumps are placed in files separated from the normal process trace files starting with Oracle Database 11g. We can also check V$DIAG_INFO view lists all important ADR locations including.

ADR retention policy – automatic purge

A retention policy allows you to specify how long to keep the diagnostic data. ADR incidents are controlled by two different policies:

- LONGP_POLICY:
The incident metadata retention policy controls how long the metadata is kept around. This policy has a default setting of one year (default is 8760 = 365 days = 1 year). Here are the components that currently get purged by LONGP_POLICY:

ALERT — files in the ./alert directory
INCIDENT – files in the ./incident/incdir_ directory.
SWEEP — files in the ./sweep directory
STAGE — files in the ./stage directory
HM — files in the ./hm directory and metadata in the HM schema

- SHORTP_POLICY
The incident files and dumps retention policy controls how long generated dump files are kept around. This policy has Read more »

Oracle database housekeeping methods – adump purge

Since Oracle Database 11g, all traces, incident dumps and packages, the alert log, Health Monitor reports, core dumps, and more files are stored in the ADR, a file-based Automatic Diagnostic Repository for simplified database diagnostic data management. In spite of that new 11g feature, the housekeeping or purging of those files got not considerably simplified. Unfortunately the new Oracle utility ADRCI can not purge all the files that Oracle database generates in ADR. Thus, you should use the other Unix OS methods and tools to accomplish Oracle 11g database housekeeping. I’ve already described the Unix log rotation mechanism. This time we will talk about Oracle System Audit files and how to purge them regularly.

SYS, SYSDBA or SYSOPER connections to the Oracle database are always audited. Sometimes this can lead to the creation of an excessive number of audit files. When this is considered to be a problem it cannot be solved at the database side and it must be investigated why the ‘client’ applications including Oracle Enterprise Manager (OEM) Components and agents are connecting so frequently as SYSDBA/SYSOPER. So this type of auditing mandatory in the Oracle database and can NOT be turned off. However the amount of audited information depends on AUDIT_SYS_OPERATIONS parameter that allows the addition audit of all statements issued by SYS/SYSDBA/SYSOPER in the same OS audit trail file.  In case the parameter AUDIT_SYS_OPERATIONS=TRUE Oracle audits not only SYS/SYSDBA/SYSOPER connection details but also their SQL operations. In any case the audit files with the name like <sid>_ora_<spid >_<instance#>.aud are created in audit_file_dest location on Unix.

sql*plus> show parameter audit
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
audit_file_dest                      string      /opt/app/oracle/admin/orcl/ adump
audit_sys_operations                 boolean     FALSE

I share below one of the methods of purging those audit files on Unix using a simple Unix command based on following: Read more »

Oracle performance tuning with Real World Performance Group

Oracle Real World Performance Tour seminar in Munich

Oracle Real World Performance Tour seminar in Munich

I was a part of a unique Oracle training seminar that happened in Munich, Germany last week.
This is a Day of Real World Performance Tour seminar given by: Andrew Holdsworth, Tom Kyte, Graham Wood. These 3 well known Oracle experts are key members of the Real World Performance Team that consist of about 25 database experts who focus on Oracle database performance tuning, creating live demos and teaching customers how Oracle perform and how they should write the optimal code.

The format of this Oracle event is very unique. The entire seminar day Andrew, Tom and Graham are continuously on stage sharing in the conversational format very unique technical material debating (sometimes in a funny form) with each other and the audience. There are three screens in the seminar room and every instructor drives his own projector. They show the quite interesting live demos loading terabytes of data and manipulating with billions of rows in just several minutes on their own Exadata systems.

Interesting enough that these guys consider performance tuning aspects of the same problem from different angels sometimes contradicting each other but finally giving concrete solutions and best practices to follow. After those tests and lessons learned you definitely change your mind in terms of database and application performance tuning.

This training seminar makes interesting not only the unique technical material but also all three very different personalities of the instructors and their backgrounds. Tom has never been a DBA but he represents mainly Read more »

How CASE expression manipulates with NULL in Oracle SQL

My next post is an answer to the below question about Oracle SQL CASE expression and the way how it manipulates with NULL values compared to DECODE function. This topic in fact is very common in Oracle database developers’ area.

===============
Hi Kirill,

Could you please help me to clarify below mention doubt.

DECODE:-

select decode (null,null,’true’,'false’) as value from dual

OUTPUT:- TRUE

CASE:-

select case null when null then ‘true’ else  ‘false’ end as Case_Test from dual

OUTPUT :- FALSE

Why above mention queries are giving different output  and what is the difference between case and decode?

==============

CASE expression treats NULLs in Oracle database SQL a bit differently compared to DECODE. You have to understand that the Boolean expressions usually result in TRUE or FALSE , but NULLs introduce a third possible result which is : UNKNOWN . At the same time NULL is not the same as UNKNOWN. See an example below:

sum + NULL results in NULL (this is a Scalar expression.)
NULL = NULL results in UNKNOWN (this is a Boolean expression.)
sum < NULL results in UNKNOWN (this is a Boolean expression.)

Ok. Now let’s explore the Oracle SQL CASE expression itself.

Oracle database SQL CASE expression

Oracle database SQL – CASE expression

CASE introduces two ways of conditional expressions: Simple CASE and Searched CASE. Read more »

Next Page »

DBMS Blog Updates : Subscribe RSS RSS: Subscribe to Articles · Subscribe to Comments Subscribe RSS Receive site updates via email