This post will help to analyze Oracle database instance slowdown that can happen due to considerable row cache lock (enqueue) wait events. It’s is based on a real case of a database hang that I worked on recently. I must admit this type of situation does not appear often but it’s very dangerous since it can considerably slow down a database instance or even freeze it for a short period of time. In most cases SQL against ASH view and Systemstate dumps can help to nail down the problem unless this is an Oracle bug.
Usually it occurs suddenly and disappears quickly. See an example ASH graph below with brown peak that represents this type of concurrency: row cache lock wait events.
What is a Row Cache Enqueue Lock?
The Row Cache or Data Dictionary Cache is a memory area in the shared pool that holds data dictionary information. Row cache holds data as rows instead of buffers. A Row cache enqueue lock is a lock on the data dictionary rows. It is used primarily to serialize changes to the data dictionary and to wait for a lock on a data dictionary cache. The enqueue will be on a specific data dictionary object. This is called the enqueue type and can be found in the v$rowcache view. Waits on this event usually indicate some form of DDL occurring, or possibly recursive operations such as storage management, sequence numbers incrementing frequently, etc. Diagnosing the cause of the contention
Diagnosing the cause of the contention
Since Oracle Database 11g, all traces, incident dumps and packages, the alert log, Health Monitor reports, core dumps, and more files are stored in the ADR, a file-based Automatic Diagnostic Repository for simplified database diagnostic data management. In spite of that new 11g feature, the housekeeping or purging of those files got not considerably simplified. Unfortunately the new Oracle utility ADRCI can not purge all the files that Oracle database generates in ADR. Thus, you should use the other Unix OS methods and tools to accomplish Oracle 11g database housekeeping. In previous posts I described the Unix log rotation mechanism and SYS auditing files purge in adump directory. This time we will talk exactly about ADR purge functionality. But before, let’s look at some basics.
Location for Diagnostic Traces
The table shown above in the slide describes the different classes of trace data and dumps that reside both in Oracle Database 10g and in Oracle Database 11g.
With Oracle Database 11g, there is no distinction between foreground and background trace files. Both types of files go into the $ADR_HOME/trace directory.
All non-incident traces are stored inside the trace subdirectory. This is the main difference compared with previous releases where critical error information is dumped into the corresponding process trace files instead of incident dumps. Incident dumps are placed in files separated from the normal process trace files starting with Oracle Database 11g. We can also check V$DIAG_INFO view lists all important ADR locations including.
ADR retention policy – automatic purge
A retention policy allows you to specify how long to keep the diagnostic data. ADR incidents are controlled by two different policies:
The incident metadata retention policy controls how long the metadata is kept around. This policy has a default setting of one year (default is 8760 = 365 days = 1 year). Here are the components that currently get purged by LONGP_POLICY:
ALERT — files in the ./alert directory
INCIDENT – files in the ./incident/incdir_ directory.
SWEEP — files in the ./sweep directory
STAGE — files in the ./stage directory
HM — files in the ./hm directory and metadata in the HM schema
The incident files and dumps retention policy controls how long generated dump files are kept around. This policy has Read more »
Since Oracle Database 11g, all traces, incident dumps and packages, the alert log, Health Monitor reports, core dumps, and more files are stored in the ADR, a file-based Automatic Diagnostic Repository for simplified database diagnostic data management. In spite of that new 11g feature, the housekeeping or purging of those files got not considerably simplified. Unfortunately the new Oracle utility ADRCI can not purge all the files that Oracle database generates in ADR. Thus, you should use the other Unix OS methods and tools to accomplish Oracle 11g database housekeeping. I’ve already described the Unix log rotation mechanism. This time we will talk about Oracle System Audit files and how to purge them regularly.
SYS, SYSDBA or SYSOPER connections to the Oracle database are always audited. Sometimes this can lead to the creation of an excessive number of audit files. When this is considered to be a problem it cannot be solved at the database side and it must be investigated why the ‘client’ applications including Oracle Enterprise Manager (OEM) Components and agents are connecting so frequently as SYSDBA/SYSOPER. So this type of auditing mandatory in the Oracle database and can NOT be turned off. However the amount of audited information depends on AUDIT_SYS_OPERATIONS parameter that allows the addition audit of all statements issued by SYS/SYSDBA/SYSOPER in the same OS audit trail file. In case the parameter AUDIT_SYS_OPERATIONS=TRUE Oracle audits not only SYS/SYSDBA/SYSOPER connection details but also their SQL operations. In any case the audit files with the name like <sid>_ora_<spid >_<instance#>.aud are created in audit_file_dest location on Unix.
sql*plus> show parameter audit
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
audit_file_dest string /opt/app/oracle/admin/orcl/ adump
audit_sys_operations boolean FALSE
I share below one of the methods of purging those audit files on Unix using a simple Unix command based on following: Read more »
I was a part of a unique Oracle training seminar that happened in Munich, Germany last week.
This is a Day of Real World Performance Tour seminar given by: Andrew Holdsworth, Tom Kyte, Graham Wood. These 3 well known Oracle experts are key members of the Real World Performance Team that consist of about 25 database experts who focus on Oracle database performance tuning, creating live demos and teaching customers how Oracle perform and how they should write the optimal code.
The format of this Oracle event is very unique. The entire seminar day Andrew, Tom and Graham are continuously on stage sharing in the conversational format very unique technical material debating (sometimes in a funny form) with each other and the audience. There are three screens in the seminar room and every instructor drives his own projector. They show the quite interesting live demos loading terabytes of data and manipulating with billions of rows in just several minutes on their own Exadata systems.
Interesting enough that these guys consider performance tuning aspects of the same problem from different angels sometimes contradicting each other but finally giving concrete solutions and best practices to follow. After those tests and lessons learned you definitely change your mind in terms of database and application performance tuning.
This training seminar makes interesting not only the unique technical material but also all three very different personalities of the instructors and their backgrounds. Tom has never been a DBA but he represents mainly Read more »
My next post is an answer to the below question about Oracle SQL CASE expression and the way how it manipulates with NULL values compared to DECODE function. This topic in fact is very common in Oracle database developers’ area.
Could you please help me to clarify below mention doubt.
select decode (null,null,’true’,'false’) as value from dual
select case null when null then ‘true’ else ‘false’ end as Case_Test from dual
OUTPUT :- FALSE
Why above mention queries are giving different output and what is the difference between case and decode?
CASE expression treats NULLs in Oracle database SQL a bit differently compared to DECODE. You have to understand that the Boolean expressions usually result in TRUE or FALSE , but NULLs introduce a third possible result which is : UNKNOWN . At the same time NULL is not the same as UNKNOWN. See an example below:
sum + NULL results in NULL (this is a Scalar expression.)
NULL = NULL results in UNKNOWN (this is a Boolean expression.)
sum < NULL results in UNKNOWN (this is a Boolean expression.)
Ok. Now let’s explore the Oracle SQL CASE expression itself.
CASE introduces two ways of conditional expressions: Simple CASE and Searched CASE. Read more »