How

To Tell If There Is An
OS Memory Shortage
And What To Do About It

Prepared by: Craig Shallahamer, OraPub

craig@orapub.com / @orapubinc

Session ID: 771

This presentation was given by Craig Shallahamer (craig@orapub.com) on

April 4, 2017 at the 2017 IOUG/Collaborate conference in Las Vegas, NV USA

About Me...
• Long time Oracle DBA
• Specialize in Oracle Database
performance and predictive analysis
• Performance researcher
• Blogger: A Wider View About Oracle
Performance Tuning
• Author: Oracle Performance Firefighting
and Forecasting Oracle Performance.
• Conference speaker
• Teacher and mentor
• Oracle ACE Director

2 ©OraPub, Inc. OS Memory Pressure

 OraPub
works
closely with
DBAs who
want to take
their Oracle
tuning skills
to the next
level.

So, you can

solve the
toughest
Oracle DB
problems.

3 ©OraPub, Inc. OS Memory Pressure

This presentation was given by Craig Shallahamer (craig@orapub.com) on

April 4, 2017 at the 2017 IOUG/Collaborate conference in Las Vegas, NV USA

$ vmstat 5 99999
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 0 70256 3248140 23208 8109560 0 0 0 16 4080 2541 49 3 48 0 0
3 0 70256 2662240 23216 8109552 0 0 0 8 4296 2748 52 3 45 0 0
3 0 70256 2075596 23216 8109560 0 0 0 16 4076 2515 49 3 48 0 0
5 0 70256 1488332 23216 8109560 0 0 0 16 4069 2550 49 3 48 0 0
3 0 70256 902184 23224 8109552 0 0 0 8 4066 2565 49 3 48 0 0
3 0 70256 315788 23224 8109560 0 0 0 16 4051 2498 49 2 49 0 0
5 0 72708 123696 852 3062600 0 0 24 16 4174 3348 49 3 48 0 0
3 0 72708 124596 848 2480620 0 0 58 16 4250 3502 49 4 47 0 0
3 0 124484 127728 848 1902356 28 25898 7996 25906 5277 3783 49 6 36 9 0
1 12 370384 124436 840 1636872 56 133936 7074 133956 5436 2712 20 9 36 34 0
0 11 556592 131980 832 1514816 36 82142 1412 82158 1704 1395 6 3 7 85 0
0 12 730012 118624 832 1426524 50 86728 592 86728 1117 1266 0 3 24 72 0
0 19 1399456 117944 968 1113736 82 87676 1650 87710 2788 1529 0 3 33 64 0
0 15 1579932 130496 1232 1114588 76 84508 1486 84516 2221 1309 0 3 41 56 0
1 19 1743744 120616 1220 1115060 86 81966 1066 81974 1913 1301 0 3 34 62 0
0 20 1909104 121668 1216 1115664 78 82732 922 82766 1984 1413 0 3 25 72 0
0 21 2072248 119444 1212 1116772 68 81602 900 81602 2272 1349 0 3 37 60 0
0 19 2221332 118320 1564 1116744 78 74576 898 74604 2028 1188 0 3 55 42 0
1 27 2339572 126120 1624 1116692 130 59312 832 59342 1356 1188 0 3 26 71 0
1 31 2479024 118868 1576 1117336 132 69610 478 69610 1442 1210 0 3 17 80 0
1 26 2629444 7205164 1540 1117792 138 75274 1108 75290 2114 1133 0 7 2 91 0
1 12 876392 23222448 1692 1129932 950 0 6874 16 1110 1695 0 8 28 65 0
0 4 873148 23205632 2384 1142624 2272 0 8374 16 858 2676 0 1 39 60 0

4 ©OraPub, Inc. OS Memory Pressure

 OraPub 3-Circle Analysis

Application

5 ©OraPub, Inc. OS Memory Pressure

This presentation was given by Craig Shallahamer (craig@orapub.com) on

April 4, 2017 at the 2017 IOUG/Collaborate conference in Las Vegas, NV USA

OraPub 3-Circle Analysis

Oracle

Application

6 ©OraPub, Inc. OS Memory Pressure

 OraPub 3-Circle Analysis

Oracle O/S

Application

7 ©OraPub, Inc. OS Memory Pressure

This presentation was given by Craig Shallahamer (craig@orapub.com) on

April 4, 2017 at the 2017 IOUG/Collaborate conference in Las Vegas, NV USA

We need to understand how Oracle

processes use memory and how the
OS satisfies this need, at least a
high level.
For sure we need to know when the
OS is struggling trying to satisfy this
memory need.

8 ©OraPub, Inc. OS Memory Pressure

 User/Client
(sqlplus)
PGA
Memory
Foreground
(oracle)
Huge Pages

System Global Area (SGA)

Library Redo
Buffer Cache Shared Pool
Cache Buffer

dbwr ………. lgwr

DBFs Redo log

9 ©OraPub, Inc. OS Memory Pressure

This presentation was given by Craig Shallahamer (craig@orapub.com) on

April 4, 2017 at the 2017 IOUG/Collaborate conference in Las Vegas, NV USA

Understanding the numbers

Huge Page Space Being Used:
$ grep MemTotal /proc/meminfo = ((114688-906)*2048)/(1024*1024)
MemTotal: 264423332kB = 222.23 GB Huge Pages Being Used
$ grep Huge /proc/meminfo
AnonHugePages: 0 kB Shared Memory Segments Being Used:
HugePages_Total: 114688 = 239180449856/(1024*1024*1024)
HugePages_Free: 906 = 222.75 GB for SGA SHM Being Used
HugePages_Rsvd: 268
HugePages_Surp: 0
Hugepagesize: 2048 kB #463 -> 120.00 GB
#204 -> 102.75 GB
$ ipcs -m

------ Shared Memory Segments --------

key shmid owner perms bytes nattch status
0x55038600 0 root 644 256064 1 locked
0x97fd8798 46956545 oracle 660 4096 0
0x00000000 46989314 oracle 660 4096 0
0x96d25ea0 795672579 oracle 660 807403520 463
0x00000000 795705348 oracle 660 128043712512 463
0x09d0eb1c 1173454853 oracle 660 538968064 204
0x00000000 1173487622 oracle 660 109790101504 204
Source: MC Big
$ sysresv
12 ©OraPub, Inc. OS Memory Pressure
 The Memory Matrix

Sharable Memory
(oracle exec, SGA) Re a l Me m or y
(pmap: shm, lib, etc) (RSS, RES)
(pmap –x, top)

Process

Virtual Memory
Private Memory (sar, vmstat, iostat)
(sort area, variables)
(pmap: anon, stack)

14 ©OraPub, Inc. OS Memory Pressure

This presentation was given by Craig Shallahamer (craig@orapub.com) on

April 4, 2017 at the 2017 IOUG/Collaborate conference in Las Vegas, NV USA

The best way to know is to check.

15 ©OraPub, Inc. OS Memory Pressure

 The best way to know is to check.
$ pmap -x 53757
53757: oracleprod50 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 267508 63300 0 r-x-- oracle
0000000010b3d000 2564 1544 84 rw--- oracle
0000000010dbe000 196 116 116 rw--- [ anon ]
0000000012095000 264 148 148 rw--- [ anon ]
0000000060000000 4 0 0 r--s- [ shmid=0xe0d0003 ]
0000000060001000 3640 456 456 rw-s- [ shmid=0xe0d0003 ]
0000000061000000 7389184 506780 506780 rw-s- [ shmid=0xe0d8004 ]
0000000224000000 12740 8 8 rw-s- [ shmid=0xe0e0005 ]
0000000225000000 36 4 4 rw-s- [ shmid=0xe0e8006 ]
00000030a2400000 128 128 0 r-x-- ld-2.12.so
00000030a2622000 4 4 4 rw--- [ anon ]
00000030a2800000 4 4 0 r-x-- libaio.so.1.0.1
...
00007ffef4093000 296 180 180 rw--- [ stack ]
00007ffef41ba000 8 0 0 r---- [ anon ]
00007ffef41bc000 8 4 0 r-x-- [ anon ]
ffffffffff600000 4 0 0 r-x-- [ anon ]
---------------- ------ ------ ------
total kB 7758876 582028 512264
16 ©OraPub, Inc. OS Memory Pressure

This presentation was given by Craig Shallahamer (craig@orapub.com) on

April 4, 2017 at the 2017 IOUG/Collaborate conference in Las Vegas, NV USA

OraPub 3-Circle Analysis

Oracle O/S

Application

17 ©OraPub, Inc. OS Memory Pressure

 When all is good…
When all is bad…
When we see it all…

18 ©OraPub, Inc. OS Memory Pressure

This presentation was given by Craig Shallahamer (craig@orapub.com) on

April 4, 2017 at the 2017 IOUG/Collaborate conference in Las Vegas, NV USA

Oracle throughput - exec/sec

SQL> / SQL> l
1 declare
4785.8 exec/sec 2 stat_value_var number;
PL/SQL procedure successfully
3 completed.
sleep_time_var number := 5;
4 stat_persec_var number;
5 begin
SQL> / 6 select value
4740 exec/sec 7 into stat_value_var
8 from v$sysstat
PL/SQL procedure successfully
9 completed.
where name = 'execute count';
10
11 dbms_lock.sleep( sleep_time_var );
SQL> / 12
4591.8 exec/sec 13 select (value-stat_value_var)/sleep_time_var
14 into stat_value_var
PL/SQL procedure successfully completed.
15 from v$sysstat
16 where name = 'execute count';
SQL> / 17
18 dbms_output.put_line( round(stat_value_var,2)||
4641 exec/sec 19 ' exec/sec');
PL/SQL procedure successfully
20* end; completed.

19 ©OraPub, Inc. OS Memory Pressure

 Demo
20 ©OraPub, Inc. OS Memory Pressure

This presentation was given by Craig Shallahamer (craig@orapub.com) on

April 4, 2017 at the 2017 IOUG/Collaborate conference in Las Vegas, NV USA

How do we know when there

is a OS memory problem?

Big drop in free memory

Throughput decreases or “freezes”

Swap outs show themselves as non-zero

Scan rate increases

Paging activity increases

21 ©OraPub, Inc. OS Memory Pressure

 Bottleneck! What now?

Remove non-essential high memory

consuming process.

Reduce Oracle’s memory requirements.

Tuning the high memory SQL.

Try to push less work through the
system.

Increase the OS’s memory capacity.

22 ©OraPub, Inc. OS Memory Pressure

This presentation was given by Craig Shallahamer (craig@orapub.com) on

April 4, 2017 at the 2017 IOUG/Collaborate conference in Las Vegas, NV USA

OraPub 3-Circle Analysis

Oracle O/S

Application

23 ©OraPub, Inc. OS Memory Pressure

 Resource listing

Craig’s Blog & Website – Search: “temp”, “direct path read temp”

Toolkits – OSM and BloodHound

Presentations – Search OraPub.com: “temp”, ”direct path read
temp”, etc.

Books
– Oracle Performance Firefighting.
– Forecasting Oracle Performance.

OraPub Membership for premium content
– Webinars – one or two each month
– Video Seminars – any device, any time, high quality
– Learning paths, assessments and certificates, priority response

Live Virtual Classroom (LVC) Training
– Oracle Tuning Fastpath
– Tuning Oracle Using An AWR Report
– Oracle Buffer Cache Performance Analysis & Tuning
– Tuning Oracle Using Advanced Active Session History (ASH) Strategies

24 ©OraPub, Inc. OS Memory Pressure

This presentation was given by Craig Shallahamer (craig@orapub.com) on

April 4, 2017 at the 2017 IOUG/Collaborate conference in Las Vegas, NV USA

Thank You!

25 ©OraPub, Inc. OS Memory Pressure

 How To Tell If There Is A Memory
Shortage And What To Do About It

Prepared by: Craig Shallahamer, OraPub

craig@orapub.com / @orapubinc

Session ID: 771

This presentation was given by Craig Shallahamer (craig@orapub.com) on

April 4, 2017 at the 2017 IOUG/Collaborate conference in Las Vegas, NV USA