Академический Документы
Профессиональный Документы
Культура Документы
Troubleshooting in Production
Filip Hanik
SpringSource
2009
Copyright 2007 SpringSource. Copying, publishing or distributing without express written permission is prohibited.
Topics in this Session
2
Topics in this Session
3
The JVM process heap
OS Memory (RAM)
Everything else…
4
Storing data in memory
5
JVM Process Heap
6
Gotcha #1
7
Java Object Heap Allocation
8
Java Object Heap
Young Generation
Old Generation
• Young Generation
– All new objects are created here
– Only moved to Old Gen if they survive one or more minor
GC
• Sized using
– -Xmn – fixed value
– -XX:NewRatio=<value> - dynamic sizing
– -XX:MaxNewSize/-XX:NewSize – similar to -Xmx/-Xms
• Survivor Spaces
– 2, used during the GC algorithm (minor collections)
– Mainly to alleviate fragmentation
10
Young Generation
Young size(-XX:NewRatio )
Survivor Ratio(-XX:SurvivorRatio )
New Objects
2Mb default
To From
64Kb default
11
Old Generation
Tenured Space
5Mb min 44Mb max (default)
12
Java Heap Space
• Solution
– Increase -Xmx if possible
– Add -XX:+HeapDumpOnOutOfMemoryError
– Fix memory leak if application is consuming more
memory than expected
• Side effects
– Increasing -Xmx can have other side effects
13
JVM Process Heap
14
Permanent Space
• Permanent Generation
– Permanent Space (name for it)
– 4Mb initial, 64Mb max
• Stores classes, methods and other meta data
– -XX:PermSize=<value> (initial)
– -XX:MaxPermSize=<value> (max)
• Common OOM for webapp reloads
– Separate space for pre-historic reasons
– Early days of Java, class GC was not common, reduces
size of the Java Heap
– Does not exist in IBM JVM
15
Permanent Space
16
Socket Buffers
17
Thread Stacks
18
Thread Stacks
• To disable
– -XX:-UseGCOverheadLimit
• To enable
– -XX:+UseGCOverheadLimit
20
GC: How It Works
From To
Tenured Space
4. Next time Eden is full
1.
3.
2. New
Copy
Copy
When object
from Eden
EDEN is
surviving created
to
is 2
objects
fullnd
– into
minor 1
collection
st
5. If 2 fills and objects remain in Eden or 1st
nd
Copy from
survivor 1
space
st
to 2 nd
These get copied to the tenured
21
GC: Debugging it
• -Xloggc:%CATALINA_BASE%\logs\gc.log
• -XX:+PrintGCDetails
• -XX:+PrintGC
• -XX:+PrintGCApplicationStoppedTime
• -XX:+PrintGCTimeStamps
• -XX:+PrintHeapAtGC
22
Troubleshooting steps
23
Tomcat Logs
24
Tomcat Logs
25
Tomcat Logs
26
Tomcat Logs
27
Tomcat Logs
• Found in catalina.2008-03-28.log
– Uncaught application exception
28
Java Stack Traces
29
Java Stack Traces
31
Viewing Requests
32
Seeing traffic
• RequestDumper Valve
– Valve that spits out everything, similar to mod_dumpio
– Poorly designed, it breaks out the output into multiple
log statement
– Not recommended
33
Seeing traffic
34
Seeing traffic
• Ability to map
– Request to error
– Error to a time frame
– Error to a client
35
Thread dumps
36
Thread dumps
• On Windows Ctrl+Break
– JDK 1.6+ you have jstack to help
• Tanuki Wrapper
– telnet <host:port> D
38
The tale of Thread dumps
39
The tale of Thread dumps
• Thread information
• First line for each thread contains critical info
40
The tale of Thread dumps
• Thread information
– CPU usage very high, caused by a spinning thread, but
which one?
– On linux for example, one can get CPU usage per thread
42
The tale of Thread dumps
43
The tale of Thread dumps
44
The tale of Thread dumps
45
OS utilities
46
OS utilities
47
OS utilities
• File descriptors
– Sockets
– Open files
• On Linux – lsof –p <process id>
– Will list all open FD
– This will often put you on the right path on what is
going wrong
– Solaris – pfiles
48
OS utilities
• Active Connections
– Can be HTTP/AJP connections
– Useful to track database or other connections
• On Unix
– Able to track socket buffers and their usage
– Send buffer filling up, slow client or bad network
– Receive buffer filling up, application is not reading fast
enough
49
OS utilities
• Linux
– nmon
– Very nice system stats collector
– CPU,Memory,disk,network and more
• Windows
– Expand your task manager
– It reports as much as you need (thread counts, IO
activity, virtual vs residential memory
50
JMX
• jconsole
– Utility that comes with the JDK
– Lets you attach to a JVM and get information
51
So far…
52
Profilers
• My preferences
– www.yourkit.com
– Very biased, as a Tomcat developer, I get it for free
53
Profilers
54
Summary
55