Oracle ADDM shows Virtual Memory Paging on 10gR2 11gR2

March 9, 2012 by 7 Comments 

Problem description

I’ve recently notices a common issue on Oracle Enterprise Linux 5.x with Oracle database 10gR2 and 11gR2. The Oracle ADDM Performance Analysis showed the following alert as a top event  in both OEM console – Virtual Memory Paging and in the ADDM report – Significant virtual memory paging was detected on the host operating system.

Problem Analysis

From the first look I thought this was a virtual memory issue on the boxes but on the other hand I could not find any evidence at OS level of any problem with virtual memory paging. Additionally I was surprised that the issue occurred at the same time on different servers and Oracle versions. So, below I did some checks to prove there was no memory issue on these Linux boxes.


$ vmstat 5 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
0  1 471576 1246004 1508736 52959096    0    0   395    77    0    0  3  2 92  3  0
5  1 471576 1241264 1508736 52959104    0    0  2157   478 3902 12026  2  1 94  3  0
0  0 471576 1258284 1508736 52959120    0    0  2013   914 3954 12746  2  1 94  3  0
1  0 471576 1258532 1508736 52959140    0    0  1415  3188 5151 14083  2  1 95  2  0
6    2 471576 1216776 1508736 52959344    0    0  3210  1384 4901 13155  3  1 92  4  0

* There is no activity with respect to swap (si/so). It is 0 only.


$ sar -B
Linux 2.6.18-164.el5 (host)    03/09/2012
12:00:01 AM  pgpgin/s pgpgout/s   fault/s  majflt/s
12:10:01 AM    337.13    908.20  12189.45      0.01
12:20:01 AM   2764.88   2947.61  14416.18      0.00
12:30:02 AM   2947.02   1580.62  14432.52      0.00
12:40:01 AM   1019.20    703.31  13251.20      0.00

* Paging is usual on Linux systems and you should be fine as long as you have no considerable majflt/s.


$ free -m
total       used       free     shared    buffers     cached
Mem:         64459      63368       1090          0       1473      51724
-/+ buffers/cache:      10170      54288
Swap:        25023        460      24563

* Swap file is almost not used.

Linux, like many other versions of UNIX, no longer swaps an entire process from memory to disk. The virtual memory system is said to be a paging system, where parts of a process are paged to and from disk. When a process starts up, at least some pages of information are brought into memory: a page of text (code), a page of data, and an empty/zero page. Once the instructions or data the process is using are no longer contained in those pages, the virtual memory system will page-in more data and code as is necessary. This is known as a demand-paged virtual memory system. Page-ins is a normal feature of the system. Considerable page-outs happen as a consequence of not having enough memory.

Cause and Solution

If indeed no paging occurs on the OS level, there is no need for concern and the ADDM or OEM warning can safely be ignored on Oracle database 10gR2 (10.2.0.5) and 11gR2. In fact the real cause of the issue was the incorrect paging statistics showed through V$OSSTAT dictionary view.

If you want to suppress the wrong ADDM / OEM finding you have to apply following Oracle bug fixes:
for 10.2.0.5 – apply patch for Bug 12347332 for your Operating System.
for 11.2. –  apply Patch 10220118

Enjoyed this article? Please share it with others using the social site of your choice:

Comments

7 Responses to “Oracle ADDM shows Virtual Memory Paging on 10gR2 11gR2”
  1. arie says:

    thank you that was realy helpfull
    i was wondering myself why was oracle ADDM reporting on memory issue

  2. Claude Saikaly says:

    Thanks all,
    Is this patch available for IBM AIX on POWER Systems (64-bit) ? I couldn’t find it over the metalink ?

    Regards,
    Claude Saikaly

  3. Kirill Loifman says:

    Hi Claude
    For 11.2, 10220118 is an unpublished bug and you have to check with Oracle Support if a patch exists for your OS

  4. sigcle says:

    I think I quite hesitate about your statement :
    “Cause and Solution

    If indeed no paging occurs on the OS level, there is no need for concern and the ADDM or OEM warning can safely be ignored on Oracle databases 10gR2 (10.2.0.5) and 11gR2. In fact the real cause of the issue was the incorrect paging statistics showed through V$OSSTAT dictionary view.

    If you want to suppress the wrong ADDM / OEM finding you have to apply following Oracle bug fixes:
    for 10.2.0.5 – apply patch for Bug 12347332 for your Operating System.”

    why ?
    in one of our client 10.2.0.5 prod with Sun Solaris Spark, ADDM said : “Significant virtual memory paging was detected on the host operating system”
    and refer to few sources :
    https://forums.oracle.com/forums/thread.jspa?threadID=2177025

    the solution should be :
    first -> analyze vmstat, etc
    second -> consider to reduce your sga and pga in order to get breath for external os services process

    Plz check

    Regards,
    Sigcle
    Senior Oracle DBA – TelkomSigma – Indonesia

  5. Kirill Loifman says:

    Hi Sigcle
    In my post I did not say there could not be virtual memory issues. That’s why I gave a few ways to check if you really had memory issues.
    If not, and ADDM still show VIrtual memory problem, that will be most probably an Oracle bug.
    In fact in that case at oracle forum, I do not see Virtual memory issue.
    Paging as bit different topic than swapping.
    Hope this more explains my intention.
    – Kirill

  6. Glad to see this blog post out there, believe I just saw this on a system myself. Bug 10148787 does mention two other workarounds — disabling the rule with the _addm_skiprules underscore parameter (set to 35 for 10.2/11.1 and set to 36 for 11.2) or filtering out the check using a “finding directive”. This should supposedly work in an 11g database: exec dbms_addm.insert_finding_directive(null, ‘wrong osstat values’, ‘Virtual Memory Paging’, 1000000, 100);

  7. Brian says:

    Hi,

    Thanks for the post. I have the same warnings on two servers with 10.2.0.5 and RHEL 5.8. The numbers are of almost exactly the same magnitude as yours and I had previously been unable to find evidence of significant paging.

    Now I feel confident about the situation.

    Best Regards,
    Brian

Add a Comment

We welcome thoughtful and constructive comments from readers.
If you want your own picture to show with your comment?
Go get a Globally Recognized Avatar!

DBMS Blog Updates : Subscribe RSS RSS: Subscribe to Articles · Subscribe to Comments Subscribe RSS Receive site updates via email