You are on page 1of 48

Waits Defined

Kyle Hailey http://perfvision.com

Top 36 Foreground Waits


1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. db file sequential read log file sync db file scattered read latch free enqueue SQL*Net more data from client direct path read direct path write buffer busy waits SQL*Net more data to client log buffer space log file switch completion library cache pin SQL*Net break/reset to client io done file open free buffer waits db file parallel read 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. write complete waits library cache lock SQL*Net more data from dblink log file switch (checkpoint incomplete) library cache load lock row cache lock local write wait sort segment request process startup unread message file identify pipe put switch logfile command SQL*Net break/reset to dblink log file switch (archiving needed) Wait for a undo record direct path write (lob) undo segment extension

#.2

minus OPS, RAC, PQ, Resource Mgr

http://www.oraperf.com

Copyright 2006 Kyle Hailey

#.3

Tuning Methodology
Machine
Run

queue (CPU)

reduce CPU usage or add CPUs Reduce memory usage or add memory

Paging

Oracle

Waits >> CPU

Tune waits

We are going to concentrate here on WAITS


Copyright 2006 Kyle Hailey

#.4

For Analysis
CPU Time
For

comparision with Wait Time

Wait Time Top Waits


If

Wait Time >> CPU Time, need top waits detailed analysis of some waits detailed analysis of some waits
Copyright 2006 Kyle Hailey

Wait Arguments p1, p2, p3


For

SQL that caused waits


For

#.5

Wait and CPU Data Sources


I Statspack
Top 5 Timed Events

II 10g ASH
OEM ASH Report : ashrpt.sql Custom queries on v$active_session_history

III V$session & v$session_wait


Custom queries

Copyright 2006 Kyle Hailey

#.6

I Statspack
Look at Top 5 Timed Events
Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ------------------buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51

Copyright 2006 Kyle Hailey

II OEM 10g Performance Tab

#.7

Copyright 2006 Kyle Hailey

#.8

II OEM 10g Zoom-In

Copyright 2006 Kyle Hailey

II Top Activity Page

#.9

Copyright 2006 Kyle Hailey

#.10

II 10g v$active_session_history
The best source Only in 10g Can simulate in any version since v7

Copyright 2006 Kyle Hailey

II Querying ASH

#.11

Top 5 Timed Events Count ------------------------------------ --------log file parallel write 8 CPU 7 select 3 event as "TOP 5 Timed Events", control file parallel write null event 3 cnt as "Count" from ( log file sync 2 select decode(ash.session_state, 'ON CPU','CPU',en.name) event , count(decode(ash.session_state, 'ON CPU','CPU',en.name)) cnt from v$active_session_history ash, v$event_name en where en.event# (+) = ash.event# and SAMPLE_TIME > sysdate - (5/(24*60)) group by decode(ash.session_state,'ON CPU','CPU',en.name) order by count(decode(ash.session_state, 'ON CPU','CPU',en.name)) desc ) where rownum < 6; Copyright 2006 Kyle Hailey

#.12

II Querying ASH for P1,P2,P3


select session_id, event, p1, p2, p3

SID EVENT
------------------------236 db file scattered read 245 db file scattered read 245 db file scattered read 248 db file scattered read

P1 P2 P3
--- --- --3 35291 16 1 13646 2 1 8457 7 1 42226 8

from v$ash

248 read by other session 1 13646 1

245 db file sequential read Copyright 1 13591 1 2006 Kyle Hailey

#.13

II ASH Report
ASH report @?/rdbms/admin/ashrpt.sql Pick interval over last 7 days !
1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time ***

Copyright 2006 Kyle Hailey

#.14

III V$session_wait
Moment in Time data
select decode(w.wait_time, 0, w.event , 'CPU') as "TOP 5 Timed Events, count(*) from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply') group by decode(w.wait_time, 0, w.event , 'CPU') order by count(*) desc; Copyright 2006 Kyle Hailey

#.15

III V$session_wait
Top 5 Timed Events ----------------------------------enq: US contention CPU buffer busy waits latch: row cache objects COUNT ---------4 3 1 1

Copyright 2006 Kyle Hailey

#.16

III V$session_wait
col status for a35 select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event , 'CPU') as status, w.p1, w.p2, w.p3 from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply');
Copyright 2006 Kyle Hailey

#.17

III V$session_wait
Moment in Time data
SID SQL_HASH STATUS ---------- --------------------------------------------234 82347421 CPU 235 3336613934 enq: US - contention 236 1772152815 enq: US - contention 238 2750335498 enq: US - contention 240 343101472 enq: US - contention 246 1782401401 enq: US - contention 248 3333220954 CPU 252 323960517 enq: US - contention 260 1272059733 CPU P1 P2 P3 ------------------- ---------1431502854 39 0 1431502854 44 0 1431502854 42 0 1431502854 44 0 1431502854 44 0 1431502854 44 0 1650815232 1 0 1431502854 44 0 1431502854 44 0

Copyright 2006 Kyle Hailey

#.18

Waits we will Ignore


Background Idle RAC Resource Manager

Copyright 2006 Kyle Hailey

#.19

Background Waits
Filter Out Background Waits
Statspack ASH : SESSION_TYPE='FOREGROUND V$session_wait : type='USER'

Copyright 2006 Kyle Hailey

#.20

Background Waits
ASH
Avoid

Background waits in ASH with

Select from v$active_session_history where SESSION_TYPE='FOREGROUND' V$session_wait joined to v$session


select from v$session s, v$session_wait w where w.sid=s.sid and s.type='USER'
Copyright 2006 Kyle Hailey

#.21

Idle Waits
Filter Out 10g
where

wait_class != Idle Create a list

9i

Select name from v$event_name where wait_class=Idle;


a list with
Documentation List created from 10g
Copyright 2006 Kyle Hailey

Create

#.22

Parallel Query Waits


Filter Out Parallel Query Wait events are unusable
Save

waits are both idle and waits

Parallel Query Waits start with PX or KX


PX

Deq: Par Recov Reply PX Deq: Parse Reply

Copyright 2006 Kyle Hailey

#.23

RAC Waits
You are on your own Check documentation If you are not using RAC then no worries 10g

Select event from v$event_name where wait_class=Cluster;


9i

RAC and OPS waits usually contain the word global

Copyright 2006 Kyle Hailey

#.24

Resource Manager Waits


Resource manager throttles user
Creates

wait Obfuscates problems

10g
select name from v$event_name where wait_class='Scheduler';

Copyright 2006 Kyle Hailey

#.25

Wait Arguments: P1,P2,P3


Each Wait has a 3 parameters P1,P2,P3 Give detailed information Meaning different for each wait Meaning definitions in V$event_name
Select name, parameter1, parameter2, parameter3 from v$event_name;
Copyright 2006 Kyle Hailey

#.26

Wait Arguments Example


NAME PARAMETER1 PARAMETER2 PARAMETER3 ----------------------------------- --------------- --------------- --------------latch: cache buffers chains free buffer waits buffer busy waits latch: redo copy log buffer space switch logfile command log file sync db file sequential read enq: TM - contention undo segment extension enq: TX - row lock contention row cache lock library cache pin library cache load lock pipe put address file# file# address number block# block# number tries set-id# class# tries

buffer# file# name|mode segment# name|mode cache id handle address object address handle address

block# object # usn<<16 | slot mode pin address lock address record length

blocks table/partition sequence request 100*mode+namesp 100*mask+namesp timeout

Copyright 2006 Kyle Hailey

#.27

Wait Tree
Buffer Cache I/O Locks Waits

Library Cache
Redo SQL*Net Undo
Copyright 2006 Kyle Hailey

Top 12 Waits
NAME Count % Total

#.29

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12.

db file sequential read 23,850.00 log file sync 20,594.00 db file scattered read 15,505.00 latch free 11,078.00 enqueue 7,732.00 SQL*Net more data from client 7,510.00 direct path read 5,840.00 direct path write 4,868.00 buffer busy waits 4,589.00 SQL*Net more data to client 3,805.00 log buffer space 2,990.00 log file switch completion 2,878.00

11.67% 10.08% 7.59% 5.42% 3.78% 3.67% 2.86% 2.38% 2.25% 1.86% 1.46% 1.41%

Above is over 80% of wait times reported

Anjo Kolk

Copyright 2006 Kyle Hailey www.oraperf.com

#.30

Top 36 Foreground Waits


1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. db file sequential read log file sync db file scattered read latch free enqueue SQL*Net more data from client direct path read direct path write buffer busy waits SQL*Net more data to client log buffer space log file switch completion library cache pin SQL*Net break/reset to client io done file open free buffer waits db file parallel read 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. write complete waits library cache lock SQL*Net more data from dblink log file switch (checkpoint incomplete) library cache load lock row cache lock local write wait sort segment request process startup file identify pipe put switch logfile command SQL*Net break/reset to dblink log file switch (archiving needed) Wait for a undo record direct path write (lob) undo segment extension undo segment tx slot

Copyright 2006 Kyle Hailey minus OPS, RAC, PQ, Resource Mgr

Waits 1-9
1.
2. 3. 4. 5. 6. 7. 8. 9.

#.31

db file sequential read


Tune SQL, speed up disks (5-15ms), increase buffer cache

log file sync


Commit less, put redo logs on faster disks

db file scattered read


FTS - Tune SQL, add indexes, speed up disks (5-15ms)

latch free need p1,p2,p3 enqueue need p1,p2,p3 and SQL SQL*Net more data from client
Usually OK, reduce data transferred, possible Network problems

direct path read


sorts or PQO - tune IO, sort less

direct path write


direct path load or temp io, improve disk speed

buffer busy waits need p1,p2,p3 and SQL


Copyright 2006 Kyle Hailey

Waits 10-18
10. SQL*Net more data to client
Usually OK, reduce amount of data transferred, possible Network tuning needed

#.32

11. log buffer space


Increase log buffer

12. log file switch completion


Increase log file sizes

13. library cache pin


Reduce parsing the same cursor concurently

14. SQL*Net break/reset to client


Check for errors in sql statement

15. io done
Io done by IO slaves, oracleIxxx could also be an idle wait sort of worthless

16. file open


Reduce logons/logoffs or keep log files open

17. free buffer waits


Increase buffer cache size

18. db file parallel read


Tune sql, tune io subsystem, increase buffer cache
Copyright 2006 Kyle Hailey

Waits 19-27
19. write complete waits
Increase buffer cache (or tune DBWR)

#.33

20. library cache lock


Avoid hard parsing same cursor at same time

21. SQL*Net more data from dblink


Reduce data transfer, check net response

22. log file switch (checkpoint incomplete)


Add log files (or increase log file size)

23. library cache load lock


Avoid hard parsing same cursor at same time

24. row cache lock


need p1

25. local write wait


truncates / reduce cache size

26. sort segment request


SMON busy, process rollback problem

27. process startup


If MTS, increase min servers and dispatchers in init.ora
Copyright 2006 Kyle Hailey

#.34

Waits 28-36
28. file identify
Keep log files open, reduce checkpoints

29. pipe put


Speed up pipe readers

30. switch logfile command


Avoid switching log files

31. SQL*Net break/reset to dblink


Check for errors in sql statement sent

32. log file switch (archiving needed)


Archive log running out of space

33. Wait for a undo record


??

34. direct path write (lob)


Improve IO, reduce lob write size

35. undo segment extension


Use UNDO or with RBS, increase RBS size, avoid OPTIMAL

36. undo segment tx slot


Use UNDO, increae # of RBS segs
Copyright 2006 Kyle Hailey

#.35

Difficult Waits
Latches Locks Buffer Busy Row Cache Lock

Copyright 2006 Kyle Hailey

#.36

Concepts: Latches
Protect areas of memory from concurrent use Light weight locks
Usually

a held in a bit in memory Often supported by atomic processor call Fast and cheap Gone if memory is lost

Often used in cache coherency management


Changes

to a datablock

Exclusive Generally
Sharing

reading has been introduced for some latches


Copyright 2006 Kyle Hailey

#.37

Finding Latches
latch free

Covers many latches, find the problem latch by 1. select name from v$latchname where latch# = p1;
OR
2.

Find highest sleeps in Statspack latch section

In 10g, important latches have a wait event


latch: cache buffers chains latch: shared pool latch: library cache
Copyright 2006 Kyle Hailey

#.38

Latches : Find Highest Sleeps


Use Statspack (or AWR report)
Latch Name ------------------shared pool Get Requests Misses Sleeps Spin Gets Sleep1

-------------- ----------- ----------- ---------- ----3,044,462 184,517 19,098 169,226 0

library cache
kks stats row cache object library cache lock library cache pin enqueues enqueue hash chains

2,242,805
263,464 434,844 666,668 741,963 133,577 133,751

79,554
2,842 955 734 635 373 89

15,198
1,774 147 54 42 40 19

66,497
1,152 816 680 593 333 70

0
0 0 0 0 0 0

Copyright 2006 Kyle Hailey

Important Latches
Cache Buffers Chains

#.39

Hot blocks, distribute activity, reduce update time, avoid select for update

Cache Buffers LRU Chain

Too much activity , uses multiple buffer caches and/or increse LRU latches

Shared Pool

Too much hard parsing, too small a shared pool

Library Cache Latch

Hard Parsing

Use bind variables Uses cursor_sharing=force Increase shared pool size if reloads are high

Avoid invalidations
Hold cursors open Use session_cached_cursors

Soft Parsing

Use cursor_space_for_time
Copyright 2006 Kyle Hailey

#.40

Concepts: Locks
Enqueue wait covers all locks pre 10 Protect data against concurrent changes Lock info written into data structures
Block

headers Data blocks Written in cache structures

Shareable in compatible modes

Copyright 2006 Kyle Hailey

Finding Locks
Statspack not much help V$session_wait needs lots of decoding
P1

#.41

tells Lock Type and Mode

P2,P3 give more data

Usually Need SQL to solve


select sid, event, chr(bitand(P1,-16777216)/16777215)|| chr(bitand(P1,16711680)/65535) as "Type", mod(p1,16) as "mode" SID EVENT Ty mode P2 P3 from v$session_wait where event = 'enqueue; --- ------- -- ---- ------- ---240 enqueue TX 6 2686995 433
Copyright 2006 Kyle Hailey

#.42

Locks 10g
10g breaks Enqueues out
enq: enq: enq: enq: enq: enq:

HW TM TX TX TX UL

contention contention allocate ITL entry index contention row lock contention contention

Configuration Application Configuration Concurrency Application Application

Copyright 2006 Kyle Hailey

#.43

Locks : TM & TX

select * from v$lock where type in ('TX', 'TM'); SID --151 135 151 135 TY ID1 ID2 LMODE REQUEST CTIME BLOCK -- ---------- ---------- ----- ------- ----- ----TX 589855 1592 0 6 4049 0 TM 53737 0 3 0 4058 0 TM 53737 0 3 0 4049 0 TX 589855 1592 6 0 4058 1

Copyright 2006 Kyle Hailey

#.44

Buffer Busy Cases


Undo Header - Not enough rollback segments Undo Block hot spot in RBS Data Block
Table Block w/ DML : Table lacks Free lists Table I/O : Multiple sessions reading same block into data cache Note: in 10g, becomes read by other session Index : has hot block, partition or change SQL

Segment Header add free Lists (with Datablock) Free List add free lists groups
Copyright 2006 Kyle Hailey

#.45

Buffer Busy Waits


Statspack
Buffer wait Statistics for DB: ORA9 Instance: ora9 Snaps: 2 -3 -> ordered by wait time desc, waits desc Tot Wait Avg

Class

Waits

Time (s) Time (ms)

------------------ ----------- ---------- --------data block undo header 1,490 2 2 0 1 0

Copyright 2006 Kyle Hailey

#.46

Buffer Busy Solutions


IF Buffer Busy Waits high
First

look at v$waitstat

Class Time/Count

SQL> select

* from v$waitstat;

CLASS COUNT TIME ------------------ ---------- ---------data block 1 1 segment header 0 0 free list 0 0 undo header 0 0 undo block 0 0
Copyright 2006 Kyle Hailey

#.47

Row Cache Lock


Need p1 to see the cache type
SQL> select cache#, parameter from v$rowcache; CACHE# PARAMETER ---------- -------------------------------1 dc_free_extents 4 dc_used_extents 2 dc_segments 0 dc_tablespaces 5 dc_tablespace_quotas 6 dc_files 7 dc_users 3 dc_rollback_segments 8 dc_objects 17 dc_global_oids 12 dc_constraints

Copyright 2006 Kyle Hailey

#.48

Row Cache Lock


Statspack
^LDictionary Cache Stats for DB: ORA9 Instance: ora9 Snaps: 1 -2 ->"Pct Misses" should be very low (< 2% in most cases) ->"Cache Usage" is the number of cache entries being used ->"Pct SGA" is the ratio of usage to allocated size for that cache Get Pct Scan Pct Mod Final Cache Requests Miss Reqs Miss Reqs Usage ----------------- --------- ------ ------- ----- -------- ---------dc_object_ids 45 0.0 0 0 958 dc_objects 89 0.0 0 0 1,129 dc_segments 69 0.0 0 0 807 dc_tablespaces 12 0.0 0 0 13 dc_usernames 22 0.0 0 0 19 dc_sequences 120,003 0.0 0 120,003 5

Copyright 2006 Kyle Hailey

#.49

Areas of Contention
Buffer Cache Disk I/O Library Cache Redo SQL*Net Undo

We will go through these areas and the wait events that are relevant to them.

Copyright 2006 Kyle Hailey