Weblogic Troubleshooting Tools

WebLogic Server along with the JVM offers various ways to collect logs, debug traces, diagnostic images,
thread dumps, and much more. However, useful data can easily get buried with lots of noise, unneeded
data that won't help resolving a particular problem.
Performance Issues
We will start with performance issues such as long running requests or server hangs. The best way to
narrow down where the issue lies is to dump the java threads and parse them through a thread dump
analyzer tool such as ThreadLogic. Looking at the server log files, or enabling some debugging features
would be premature since the first step is to identify the thread(s) that are stuck to determine what is
being executed and relevant JEE resources. The log could also alert about stuck threads once they have
been stuck for over the maximum allowed time (see MaxStruckThreadTime) but unless thread dumps are
examined, it won't be possible to determine possible patterns of issues and threads' relationship around
locked objects.
Memory Leaks
Another common performance issue involves memory leaks. If you encounter OutOfMemory java.lang
exceptions you should issue a heap dump and analyze it with a tool such as VisualVM or the Eclipse
MAT. See my post on Heap Dumps analysis for more details. Heap dumps are snapshots of memory
certain times. Heap dumps are to the Java process memory (non native) what thread dumps are to Java
process threads. Enabling -verbose:gc is also an effective way to monitor heap usage at runtime. The
output of the GC stats can be redirected to a file so that not to clog the server logs. This can be done by
setting -Xloggc with HotSpot JVM. It's recommended to include -XX:+PrintGCDetails and -XX:
+PrintGCTimeStamps to log time stamps and detailed GC activities as well. Troubleshooting guide for
HotSpot VM details these flags among others useful flags for diagnosis purposes.
Applications, 3rd parties, and configuration issues
The following chart summarizes what could be collected to debug each of the listed WLS component.
This list is by no means exhaustive. Many more debug flags could be used for these listed
components. This chart contains examples of useful debug options and links to additional resources.
For the following, Redirect stdout and stderr logging should be enabled. This can be done via the
Administration Console in the advanced section of the servers logging tab or at startup in the java
command line with -Dweblogic.StdoutDebugEnabled=true. In addition, the severity level of the log file
should be set to debug.
WLS
comp.
or 3rd
party
resource
EJB/Web
Serv.

Examples of Debug options and flags

MyOracleSupport
related resources
(login required)

-Dweblogic.webservice.verbose=true

Troubleshooting EJBs
Issues

How to enable Debugging

for Weblogic EJB
-Dweblogic.webservice.client.verbose=true

How To Debug WLS

To log SOAP requests and response messages

Application Container
Problems

Trace in servers log files

Troubleshooting Oracle
Web Services 11g
-Dweblogic.debug.DebugJMSStore
To log info on load, store events and transaction records into servers
JMS

log files

Troubleshooting JMS Info

Center

See Debugging JMS for more JMS debugging options

-Dssl.debug=true
SSL

Troubleshooting SSL

-Dweblogic.security.SSL.verbose=true

Security
Trace in servers log files
-Dweblogic.debug.DebugReplication for high level replication info

Clustering

-Dweblogic.debug.DebugClusterAnnouncements to log info on

Debug RJVM or Intra-

announcement, StateDump and attributes messages sent or received

Cluster Communication

by multicast

Problems

Trace in servers log files

weblogic.JTAXA & weblogic.JTA2PC for runtime issues, example:
JAVA_OPTIONS="-Dweblogic.Debug=weblogic.JTAXA,
JTA

weblogic.J

TA2PC"

Investigating Transaction
Problems

Trace in servers log files

-Dweblogic.debug.DebugJDBCSQL=true
to print information about JDBC methods invoked with their
JDBC

arguments, return

values, and thrown exceptions

See JDBC Debugging Scopes under Monitoring JDBC Resources

Investigating JDBC Issues

JDBC Debugging Scopes

Trace in servers log files

-Doracle.jdbc.Trace=true to

enable jdbc logging

Oracle JDBC Create OracleLog.properties

Driver (*)
Trace is file defined in oracle.jdbc.LogFile
Deployment

Set the following options to true to report which

Enable Oracle JDBC

Driver Debug in WLS

Deployment Problems -

classes are getting loaded

-Dweblogic.utils.classloaders.GenericClassLoader.Verbose
-Dweblogic.utils.classloaders.ChangeAwareClassLoader.Verbose
-Dweblogic.utils.classloaders.ClasspathClassFinder
(Classloader) -Dweblogic.utils.classloaders.DefaultFilteringClassLoader.Verbose

Enabling Debug

-Dweblogic.utils.classloaders.FilteringClassLoader.Verbose
-Dweblogic.utils.classloaders.FilteringClassLoader.ResourceDump

Trace in servers log files

Debug="ALL" in the proxy configuration file
(iisproxy.ini, httpd.conf or obj.conf)
Proxy plug-in

Common Diagnostic
Process for Proxy Plug-In

Trace in file set in WLLogFile defined inside proxy configuration

Problems

file

(*) Enabling JDBC Driver debug flags is verbose so it's recommended to set logging properties to
appropriate levels such as FINE, SEVERE, FINEST, INFO etc. based upon the debugging requirement.

WLDF
The Weblogic Diagnostic Framework can be used to collect metrics, setup watch and notifications, and to
define instrumentation. With WLS 12.1.2, you can set the level (Low, Medium or High) of built-in
modules that will gather metrics, or disable them completely. WLDF collects runtime statistics on JDBC,
JTA and JVM and more. You can then use the Monitoring Dashboard to view and navigate through the
collected data. WLDF is not an alternative to debug options. Debug options activate tracing that is coded
as part of WLS whereas WLDF collects and monitors runtime mbeans values and can activate
notifications with defined rules. They are used for different purposes.
The WLDF built-in configurations (Low, Medium, & High) can be cloned into a new configuration, and
used as templates for creating custom WLDF configurations. The Low volume built-in is enabled by
default in production mode. One new feature with WLS 12.1.2 is Runtime control which gives the ability
to activate or deactivate diagnostic system modules dynamically at runtime wihtout making any domain
configuration change.
JFR
JFR recording data can also be captured through WLDF diagnostic images based on WLDF watch rules
so that JVM runtime system information can be analyzed along with recording of WLS components
diagnostic data. Once extracted from the diagnostic image capture, the JFR file can be analyzed
using Java Mission Control.
WLST
WLST diagnostic commands can be used to extract data from the diagnostic archive (event and
harvested metric data) in either XML form or CSV, based on the function you use. You can also dump the
diagnostics data from a harvester to a local file.

The following WLST functions allow to capture an image and to download image files from a remote
system over WLST:

captureAndSaveDiagnosticImage()
saveDiagnosticImageCaptureFile()
saveDiagnosticImageCaptureEntryFile()
One major difference between the diagnostic framework and the debug options is that debugs are
enabled so that an issue can be reproduced and traced whereas the diagnostic image capture is
used as a server-level dump for post-failure analysis in a similar way heap dumps are used post
memory leak failures.

___________________________________________________________________________
JSTACK
jstack can be use to dump the java threads. While working with WebLogic Server, you can dump the java threads with various tools such as
WLST. jstack can do just that but differs from <ctrl>+<break> or kill -3 since it doesn't dump a heap summary along with the threads.
In addition, jstack can print lock information with -l option and the list of synchronizers that may be exclusively owned by a thread. Such
information can be very useful while monitoring concurrent access by threads.
jstack comes as part of the JDK installed with WebLogic Server. With instances of WLS running, you can open a new shell, execute the
setDomain script and run jstack or any tool that's part of the JDK package. Thread dumps taken by jstack can be redirected to files, e.g.
jstack <pid> > myTD.txt.
PIDs can be obtained on Solaris/Linux with the ps (report process status) command, e.g. ps -ef | grep java, and from the task managed on
Windows. In task manager you might need to go to View and Select Columns to add PID or process identifiers.

______________________________________________________________________

Thread dumps

Introduction
A Java thread dump is a snapshot what every thread in the JVM is doing at a particular point in time. Each thread in
the JVM is listed with it's name and id, it's current state and the Java call stack showing what monitor it has locked or
is waiting on. This is especially useful if your Java application sometimes seems to hang when running under load, as
an analysis of the dump will show where the threads are stuck, either by deadlock or other thread contention.

How to get a thread dump

Unix and Linux
On most flavors of Unix kill -3 <pid> will generate a thread dump. To get the java process pid use ps -ef | grep java.
The dump is directed to stdout so redirecting to a file is neccessary. Since 1.6 the Sun/Oracle JDK has the jstack
utility to do the same thing.

Enter the following at the command line at startup:

bin\jrcmd.exe <pid> print_threads

The thread dump appears at the command line.

Windows:
If the application was started from a command window you can select the window and use CTRL+Breakbut the
output will likely overflow the window. Jstack is available if your using the Sun JDK. Otherwise use PsExec.exe
( http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx ) to execute SendSignal.exe.

How many thread dumps do I need

Often a single thread dump will reveal the problem, this is especially true with deadlocks where two or more threads
are waiting for locks obtained by each other. But in other cases, such as threads waiting on other processes, like IO
from a database, you won't be able to detect it with a single thread dump. In this case multiple thread dumps taken
over time will show the same thread waiting for the process to complete during that time span.

Understanding Thread States

Before starting on dissecting thread dumps it is very important to get a good understanding of the
common thread states.
NEW : A thread is in a new state when it has just been created and the start() method hasn't been
invoked
RUNNING : A thread is in a runnable state when the start method has been invoked and the JVM
actively schedules it for execution
BLOCKED / WAITING FOR MONITOR ENTRY: Thread enters this state when it isn't able to
acquire the necessary monitors to enter a synchronized block. A thread waits for other threads to reliquish
the monitor. A thread in this state isnt doing any work
WAITING : A thread enters this state when it relinquishes its control over the monitor by calling
Object.wait(). It waits till the time other threads invoke Object.notify() or Object.notifyAll(). In this state
too the thread isnt doing any work
TIMED WAITING : A thread enters this state when it relinquishes its control over the monitor by
calling Object.wait(long). It waits till the time other threads invoke Object.notify() / Object.notifyAll() or
when the wait time expires. In this state too the thread isnt doing any work
SLEEPING / WAITING ON CONDITION: A thread sleeps when it calls Thread.sleep(long). As
the name suggests the thread isnt doing in this state either
Terminated
Other
As you can see from the above list a thread is only doing active work when it is in the RUNABBLE state. In
all other states the thread is just wasting CPU cycles. For maximum throughput from an application it is
necessary to ensure that the maximum number of threads are in the RUNNABLE state and fewer threads
in WAITING FOR MONITOR ENTRY, SLEEPING, WAITING or TIMED WAITING state.

Threads present in a typical weblogic thread dump

JVM threads { GC threads, Code optimisation etc }
ListenThread.Default
Weblogic.socket.Muxer
Weblogic.kernel.System
Weblogic.admin.RMI

 Weblogic.admin.HTTP
Weblogic.kernel.Defaul
VDE Transaction Processor Thread
Weblogic.health.CoreHealthMonitor
Miscellaneous Threads ( Timer, Security etc. )

Analyzing the thread dumps

You can read a thread dump with a text editor, and many people use only this.

The Beginning of The Thread Dump

The thread dump starts with the date and time of the dump, and the version number of the
JRockit JVM used (see Listing 20-1).
Listing 20-1 The initial information of a thread dump
===== FULL THREAD DUMP ===============

Wed Feb 21 13:46:45 2007

BEA JRockit(R) R27.1.0-109-73164-1.5.0_08-20061129-1428-windows-ia32

Format of thread dumps

Full thread dump Java HotSpot(TM) Client VM (1.5.0_04-b05 mixed mode, sharing):
"Thread-1" prio=5 tid=0x00a995d0 nid=0x1300 in Object.wait()[0x02d0f000..0x02d0fb68]
at java.lang.Object.wait(Native Method)
- waiting on <0x22aaa8d0> (a org.tw.testyard.thread.Drop)
at java.lang.Object.wait(Unknown Source)
at org.tw.testyard.thread.Drop.take(Drop.java:14)
- locked <0x22aaa8d0> (a org.tw.testyard.thread.Drop)
at org.tw.testyard.thread.Consumer.run(Consumer.java:15)
at java.lang.Thread.run(Unknown Source)
"Thread-0" prio=5 tid=0x00a88440 nid=0x6a4 waiting on condition[0x02ccf000..0x02ccfbe8]
at java.lang.Thread.sleep(Native Method)
at org.tw.testyard.thread.Producer.run(Producer.java:24)
at java.lang.Thread.run(Unknown Source)
In the thread dump snippet above you can observe the following

The thread dump starts with "Full thread dump", followed by a list of threads currently being
executed.
2.
There are 2 application threads called Thread-1 and Thread-0. These are the default names which
the JVM handed over to the threads.
3.
"Thread-1" is waiting for a notification after it called Object.wait() on the Drop object.
4.
Similarly, "Thread-0" is sleeping on a condition after it called Thread.sleep
5.
At this particular instant, there are no threads in the runnable state and hence the application
isn't doing any real work.
1.

Although thread dumps also lists the state of the system threads we are not going to look deeper into
system threads.

Thread Dump: There are 3 tool are available in the market to analysis the thread dump.
1.Samurai.
2.Thread Dump Analyzer
3.Thread Logic.

1. Samurai:
Samurai description
A GUI based tail, thread dump analysis tool
Samurai is a tabbed GUI tail application. Samurai will colors idle threads in gray, blocked threads in red and all the running threads in green.
There are three result views and Samurai shows "Table view" by default. In many case you are just interested in the table view and the sequence
view.
Use the table view to decide which thread needs be inspected, the sequence view to understand the thread's behavior. You should take care
especially threds always in red.
Here are some key features of "Samurai":
Picks thread dumps from std/stderr output and colorize them.
Visualize the "-verbose:gc" log.
Works as a GUI based "tail -f"
Requirements:
JDK 1.4 or later

_______________________________________________________________________________

Thread Logic:
Download: https://java.net/projects/threadlogic/downloads
Thread Dump Analysis is a key tool for performance tuning and troubleshooting
of Java based applications. The current set of TDA tools (Samurai/TDA) dont
mine the thread dumps or provide a more detailed view of what each thread is
doing while just limiting themselves to reporting the state (locked/waiting/running)
or the lock information.

Once a thread dump is parsed and threads details are populated, each of the
thread is then analyzed against matching advisories and tagged appropriately.
The threads are also associated with specific Thread Groups based on
functionality or thread group name. Both the advisories and grouping are
managed via xml definition files which can be modified or extended.
Each of the advisory has a health level indicating severity of the issue found,
pattern, name, keyword and related advice.
Thread Groups
One can see the thread groups are divided into two buckets - WLS and non-WLS
related threads. The JVM threads, LDAP and other unknown custom threads go
under the non-WLS bucket while all the WLS, Muxer, ADF, Coherence, Oracle,
SOA, JMS, Oracle Adapter threads are all under the WLS bucket. The
classification can be changed by modifying the GroupsDefn xml files.

The health levels (in descending of severity) are FATAL (meant for Deadlocks,
STUCK, Finalizer blocked etc), WARNING, WATCH (worth watching), NORMAL
and IGNORE. Based on the highest severity of threads within a group, that
health level gets promoted to the Thread Group's health level and same is
repeated at the thread dump level. There can be multiple advisories tagged to a
Thread, Thread Group and Thread Dump.

Each of the advisory gets triggered based on either call execution patterns
observed in the thread stack or presence of other conditions (thread blocked or
multiple threads blocked for same lock can trigger BlockedThreads Advisory).
Sometimes a thread might be tagged as IGNORE or NORMAL based on its

execution logic or might be tagged more specifically as involved in JMS send or

receive client or a Servlet thread. The advisories are generated based on an
advisory xml map that is extensible.

___________________________________________________________________________

Heap Dump

Heap dump:

A Heap Dump is a snapshot of the Java process heap memory at a given time. It's a useful tool to
troubleshoot memory leaks and other memory related issues. Heap dumps are to memory what thread
dumps are to java threads.
There are different ways to dump the java heap used by a Java application such as WebLogic Server.

Heap dumps can be triggered automatically by JVMs such as HotSpot or JRockit when
an OutOfMemory event occurs.
Example:
java -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=D:\myApp\hprof-dumps myApp
This command will generate a heap dump file in the defined heap dump directory when an OOM is
thrown.
The file will be in HPROF binary format (*.hprof). If no path is defined the file will be created in the
directory from which the java process was launched. The format of the file with JRockit is
jrockit_pid,hprof, pid being the process ID of this JVM instance. With HotSpot the format of the file will be
java_pidpid.hprof.

Example:
java.lang.OutOfMemoryError: Java heap space

Dumping heap to D:\java_pid4148.hprof ...

Heap dump file created [24921731 bytes in 1.400 secs]
For WebLogic Server these
properties, HeapDumpOnOutOfMemoryError

and HeapDumpPath(optional), can be

added directly to the startup script that contain the Java command that launches WLS.
On Windows the script (with default name startWebLogic.cmd) lives in the domain
bin directory and the parameters can be directly added to the existing JAVA_OPTIONS.
With a JRockit VM process the following command can also be used to generate a heap dump.
Example:
jrcmd pid hprofdump filename=name_of_dump_file
However if you are using this command against a non JRockit process the following error will come up:
java.io.IOException: Command failed in target VM
You can also manually generate a heap dump with tools such as jmap or VisualVM.

Example:
jmap -heap:live,format=b,file=filename pid
This command will generate a heap dump file in binary format and will only count live objects (the live
option is optional). Live objects are used by applications and generally reside in the Young Generation
part of the heap.
As illustrated in the screenshot below, generating a heap dump with VisualVM is very simple. VisualVM is
shipped as part of the HotSpot JVM (/bin/jvisualvm.exe). However, VisualVM can connect to java
processes that started with other JVM such as JRockit.

Once generated, the heap dump will be parsed as shown below:

VisualVM enables you to browse heap dumps as shown above with the Classes view where you can
easily identify the number and percentage of instances referred by each class.
jhat can also be used to analyze heap dump files.
Example:
jhat dumpfile
Reading from dumpfile...
Dump file created Fri May 17 15:25:41 PDT 2013
Snapshot read, resolving...
Resolving 580390 objects...
Chasing references, expect 116
dots.............................................
.......................................................................
Eliminating duplicate
references................................................
....................................................................
Snapshot resolved.
Started HTTP server on port 7000
Server is ready.
Then you can open http://localhost:7000/ to navigate through the content of the dump file to
identify objects with high number of occurrences.

Finally, as shown below, the OpenSource Eclipse MAT offers a very rich GUI that includes histograms and
leak hunter actions, overview and leak suspect reports and much more.

____________________________________________________________________________________
______

jhat - Java Heap Analysis Tool

Synopsis
Parameters
Description
Options
See Also

SYNOPSIS
jhat [ options ] <heap-dump-file>

PARAMETERS
options
Options, if used, should follow immediately after the command name.
heap-dump-file

Java binary heap dump file to be browsed. For a dump file that contains multiple heap dumps, you may specify which
dump in the file by appending "#<number> to the file name, i.e. "foo.hprof#3".

DESCRIPTION
The jhat command parses a java heap dump file and launches a webserver. jhat enables you to browse heap dumps
using your favorite webbrowser. jhat supports pre-designed queries (such as 'show all instances of a known class
"Foo"') as well as OQL (Object Query Language) - a SQL-like query language to query heap dumps. Help on OQL is
available from the OQL help page shown by jhat. With the default port, OQL help is available at
http://localhost:7000/oqlhelp/
There are several ways to generate a java heap dump:

Use jmap -dump option to obtain a heap dump at runtime;

Use jconsole option to obtain a heap dump via HotSpotDiagnosticMXBean at runtime;
Heap dump will be generated when OutOfMemoryError is thrown by specifying -XX:
+HeapDumpOnOutOfMemoryError VM option;
Use hprof.
NOTE: This tool is experimental and may not be available in future versions of the JDK.

OPTIONS
-stack false/true
Turn off tracking object allocation call stack. Note that if allocation site information is not available in the heap dump,
you have to set this flag to false. Default is true.
-refs false/true
Turn off tracking of references to objects. Default is true. By default, back pointers (objects pointing to a given object
a.k.a referrers or in-coming references) are calculated for all objects in the heap.
-port port-number
Set the port for the jhat's HTTP server. Default is 7000.
-exclude exclude-file
Specify a file that lists data members that should be excluded from the "reachable objects" query. For example, if the
file lists java.lang.String.value, then, whenever list of objects reachable from a specific object "o" are calculated,
reference paths involving java.lang.String.value field will not considered.
-baseline baseline-dump-file
Specify a baseline heap dump. Objects in both heap dumps with the same object ID will be marked as not being
"new". Other objects will be marked as "new". This is useful while comparing two different heap dumps.
-debug int
Set debug level for this tool. 0 means no debug output. Set higher values for more verbose modes.
-version
Report version number and exit.
-h
Output help message and exit.
-help
Output help message and exit.
-J<flag>
Pass <flag> to the Java virtual machine on which jhat is run. For example, -J-Xmx512m to use a maximum heap size
of 512MB.

_____________________________________________________________________________
CORE DUMP
Core dump is usually a Binary file which gets generrated by the operating system when JVM or any other
process crashes. some times it also happens that the jvm will not be able to generate the crash dump.

Path:
Windows: c:\Documents and settings\All Users\Application Data\Microsoft\Dr Watson.
By default in Unix based operating systems the core dump files are created in the directory where the Java
program/server was started even sometimes it is generated in the "/Tmp" directory of the operating system.
but we can change the location to create crash/Heap dump.
-XX:HeapDumpPath=/opt/app/Prefered Location and -XX:+Heap Dump jvm Options.
Filename:
Sund JDK: "hs_err_pid<weblogicPID>.log
for JRockit Jvm also generates a textual file with name "*.dump"
Flags:
Sun JDK JAVA_OPTIONS -XX:+showMessageBoxOnError.
JRockit JAVA_OPTIONS -Djrockit.waitonerror.
Cause JVM Crash.
1.Native Code causes the JVM crash (c/C++,JNI-Javanative interface APIs)
2.JDBC Drivers Specially the Native drivers
3.JVM code optimization
4.Less Memory availability for native area of a java process.
5.Application Server Native Performance Pack Libraries.
6.JVMs library itself can cause the Crash.
7.High CPU Utilization by the threads.
Tools.
1. window's : Dr.Watson start->run->drwtsn32.
2. Solaris : Pstack and Pmap
3. Linux : lsstack