Velikanovs - Performance Tuning and Troubleshooting For Oracle OC4J

Performance Tuning and Troubleshooting for Oracle OC4J
Version 1.0
Jurijs Velikanovs
http://otn.oracle.com/ocm
Introduction....................................................................................................................2
OC4J Internals ...............................................................................................................3
Physical structure .......................................................................................................3
Memory structures .....................................................................................................3
Processes ....................................................................................................................3
OC4J Troubleshooting ...................................................................................................9
Classification of Problem Areas ................................................................................9
Possible Approaches & Information Sources ..........................................................10
Java “Full thread dump” ..........................................................................................11
How to get Full thread dump? .............................................................................11
The interpretation of Full thread dump................................................................13
Useful Java runtime parameter ............................................................................16
Java DEADLOCK problem .................................................................................17
Log files ...................................................................................................................18
OC4J output file...................................................................................................18
OC4J default-web-access.log...............................................................................19
OS Monitoring commands.......................................................................................19
OS Tracing commands.............................................................................................20
Apache side information ..........................................................................................22
Apache Status page ..............................................................................................22
Apache log files ...................................................................................................22
Java code decompilation ..........................................................................................23
Debugging Switches ................................................................................................23
Documentation, External searches and Metalink.....................................................24
Documentation.....................................................................................................24
External searches .................................................................................................24
OC4J Performance .......................................................................................................26
Our Findings ............................................................................................................26
Java Workers........................................................................................................26
OC4J Processes....................................................................................................26
Memory................................................................................................................27
Stresstesting .........................................................................................................28
Response Time Monitoring by DMS.......................................................................29
Response Time Breakdown Example ..................................................................29
DMS Sensors Classification ................................................................................30
Collecting and Reflecting DMS Statistics ...........................................................30
Appendix A. How to diagnose opened files, sockets, pipes by OC4J process? ..........32
Introduction
There is almost no information about OC4J troubleshooting and performance tuning.
This paper covers available methods of OC4J performance and availability
troubleshooting. The information provided is based on several years of author’s
experience working with this technology. The main question paper is trying to cover
is how to troubleshoot OC4J production environments, where it is almost impossible
to use the traditional Java troubleshooting methods such as Java profilers because of
the significant overhead.
The first part of the paper describes the internals of OC4J. It is vital to understand
OC4J main processes (threads) and their responsibilities to be able to troubleshoot
OC4J related problems successfully. What are the main thread (process) types? What
tasks are they responsible for? What are the relationships between different types of
the processes? “OC4J Internals” part of this paper gives the answers to these
questions.
There is no any single method which would give the complete information about the
problem within OC4J environment. The “OC4J Troubleshooting” section gives you
the information about available methods. Use all available methods to gather the
sufficient troubleshooting data for further analysis, localization of the problem area
and elimination of the problem.
“OC4J Performance” describes author’s main findings in the context of OC4J
performance tuning and available performance monitoring capabilities.
Note: As English is not the author’s native language, author apologizes for possible
"syntax errors" in this document.
OC4J Internals
Warning!!! Please note that all the information provided in this section is based on author’s experience
working with OC4J processes. Java “Full thread dump” and Java classes decompilation were the main
methods to get the information. Because of that and because the fact that Oracle Corporation doesn’t
publish OC4J architecture publicly, the OC4J architecture described below possibly is not 100%
accurate. Author hopes that it is correct in main terms and will be useful for reader to understand better
how to troubleshoot and tune OC4J.
Physical structure
OC4J physical structure (source, configuration and application files structure) is
described in the Oracle documentation (See. OC4J Standalone User's Guide1).
However you can find useful information about how to diagnose opened files,
sockets, pipes by OC4J process in the appendix (see. Appendix A. How to diagnose
opened files, sockets, pipes by OC4J process?).
Memory structures
Memory structures of OC4J process are not covered in this paper in details. They are
highly dependent on the application you are using. For sure there are OC4J specific
memory structures for storing such information as HTTP session list, managing
threads and connections pools etc. as well. But they are not covered in this paper.
There will be additional paper on OC4J memory structures in the future probably.
Processes
For troubleshooting and performance purposes it is extremely important to understand
the architecture of OC4J instance (process). What are the main treads? What are their
responsibilities? OC4J process architecture is a matter of a great interest in a context
of this paper.
OC4J is written entirely in Java language and it is a Java application. Not depending
on the configuration you are using, JRE2 is used to run OC4J instance. The command
below is used to start-up any OC4J instance:
<jre location>/bin/java <1 set of parameters> -jar oc4j.jar <2 set of
parameters>
Any java process is multithreaded. It is vital to know multithreading concepts for
understanding OC4J internals. Multithreading is the way how many tasks (threads)
can be executed within a single java process. Threads are executed in the same
context and have access to all process memory structures. Different operational
systems have different implementations of multithreading within the single process.
In most of the operational systems you will see OC4J as single process. There are
different possibilities on different OS to monitor threads within the single process.
Sun Solaris for example realizes multithreading through light-weight processes.
Light-weight processes can be monitored easily. You can use prstat -Lp <pid> to see
CPU utilization by different threads within the single process. Unlike most of the OS
1
The physical structures of OC4J is the mostly the same despite OC4J version. Author recommends to
read “Oracle® Application Server Containers for J2EE Standalone User's Guide, 10g Release 2
(10.1.2), Part No. B14361-01” chapter “Advanced Configuration and Development”
2
There is JDK used actually, OC4J instance for JSP executing compile any JSP to an *.java file and
*.calss file afterwards.
Linux realizes multithreading by running each process thread as dedicated process.
You can monitor each thread using regular OS utilities such as ps, strace, pstree etc.
There is an OC4J single process on the Figure 1 “OC4J instance” below. The OC4J
instance consists of the processes or threads (quadrangles) and memory structures
(ovals).
ONS
Thread
OC4JMonitor
new req Thread
PORT: 3001
AJP
Connection ApplicationServerThread
Listener Thread
ApplicationServerThread
Java native
AJP 1.3 ApplicationServerThread
... ONet
Memory Structures
Application objects
Http Session List ... JDBC Connection pool
Java native
RMI
RMIServer
Server TaskManager VM Periodic Task
3101
PORT: 3101
TaskManager VM Thread VMThread
Periodic Task
VM Thread
Thread
JMSServer
3201
Signal Dispatcher Finalizer

PORT: 3201 Signal Dispatcher Finalizer
JMSServer
JMSServer Suspend Checker Reference
JMSServer Suspend
JMSServer
Thread ThreadChecker Reference
Handler
Thread Thread Handler
Figure 1 “OC4J instance”
Each group of processes is responsible for different tasks executing within OC4J
instance. Let’s go through OC4J process groups and describe their responsibilities:
Java native threads

Group processes (Java HotSpot 1.3.1): VM Thread, VM Periodic Task Thread,
Reference Handler, Signal Dispatcher, Suspend Checker Thread, Finalizer
This group of the threads exists in any java application process. They are initialized
during a start-up of java application and are responsible for generic java routines, such
as: initializing and terminating other threads, freeing unused memory etc.
OC4J Listener – threads

Group processes: HTTP/AJPConnectionListener, RMIServer (2), JMSServer
These threads handle incoming client requests coming through different protocols. In
case of Oracle AS configuration, you can use command listed below to detect ports
used by listener processes for each started OC4J process:
$ opmnadmin debug comp=pm
...
======== PM ========
PM requests processed: 7
PROCESS TABLE
UID PID FLAGS TYPE STATUS REF HTTP AJP RMI JMS
---------- ----- -------- ------- ------- ------ ----- ----- ----- -----
750407 20502 00000040 OC4J Alive 1 0 3002 3103 3203
619335 4535 00000040 OC4J Alive 1 0 3003 3104 3204
488263 4533 00000040 OC4J Alive 1 0 3001 3102 3202
226119 15875 00000040 OC4J Alive 1 0 3000 3101 3201
160583 15513 00000040 Apache Alive 1 7780 0 0 0
95047 15456 00000040 Generic Alive 1 0 0 0 0
$
HTTP/AJPConnectionListener – depending on configuration you will have one of

the listeners (HTTP or AJP).
HTTPConnectionListener appears if you run standalone OC4J process. You can set
port on which this process accepting requests in http-web-site.xml file:
OH/j2ee/<comp_name>/config/http-web-site.xml
<web-site port="8888" ...>
...
</web-site>
AJPConnectionListener is the main listener process in the typical Oracle AS
configuration. Normally3 the most requests come in through this process. It accepts
clients’ requests forwarded by Apache child processes through AJP protocol. AJP –
Apache JServ Protocol4. A AJP port negotiated during OC4J process start up. The
OPMN process informs Apache mod_oc4j component on AJP chosen port. You can
set the range of the AJP ports in opnm.xml file5 for each OC4J component:
OH/opmn/conf/opmn.xml
<oc4j ... >
...
<port ... ajp="3001-3101" .../>
...
</oc4j>
HTTP/AJPConnectionListener process fulfils the default-web-access.log file.
OH/j2ee/<comp_name>/log/<comp_name>_default_island_<proc_num>/default
-web-access.log
This file is the analogue of the Apache’s accecc_log log file where OC4J process
writes information about each clients’ request.
HTTP/AJPConnectionListener process doesn’t process clients’ requests. It accepts
and assigns each request to available ApplicationServerThread process for further
processing. After requests had been assigned ApplicationServerThread communicates
with Apache child process directly.
3
Unless you run OC4J for EJB calls processing
4
See http://jakarta.apache.org/tomcat/connectors-doc-archive/jk2/common/AJPv13.html#Intro for
more information about AJP.
RMIServer (2) – There are two processes. First one accepts clients’ RMI requests
through public network interfaces (external IP addresses). Second listener process is
mainly used for administrating requests processing. Independently of configuration
you are using (standalone or Oracle AS) the OC4J process is listening on
127.0.0.1:RMIport for shutdown/reload and other administration commands. Even if
you are using Oracle AS configuration, a OMPN calls admin.jar to contact OC4J
process through RMI port. For example the command issued by the OPMN to
shutdown OC4J process listed below6:
/u01/app/oracle/product/J2EE01/jdk/bin/java
-DOPMN=true
-Doracle.home=/u01/app/oracle/product/J2EE01
-jar
admin.jar
ormi://localhost:3104
admin
null
-shutdown
Public RMI listener accepts RMI requests to call methods of the objects located inside
OC4J. In general terms this protocol enables to make distributed java calls to objects
located inside OC4J. Description of RMI protocols is out of the scope of this paper.
As AJP listener thread RMIServer thread doesn’t process calls. It just assigns request
to available ApplicationServerThread threads. Used RMI ports are negotiated7 during
the start-up of the OC4J process. You can set a range of ports in the opmn.xml file:
<oc4j ... >
...
<port ... rmi="3101-3200".../>
...
</oc4j>
JMSServer – It is like RMI listeners from technical point of view. Java Message
Service (JMS) is one of the required protocols by J2EE specification. JMS enables to
send and receive data and events between containers asynchronously.
ApplicationServerThread – is a group of threads which processes users’ requests.
Basically these treads execute application code, compile and execute JSP,
communicate with database and send a response to the Apache child processes. They
are workers who are doing most the job within OC4J container.
The worker threads are linked up to the thread pool managed by TaskManager thread.
Listeners assign users’ requests to the available ApplicationServerThread threads.
You can define minimal and maximal number of the worker threads in server.xml file.
OH/j2ee/<comp_name>/config/server.xml
<application-server ...>
...
6
You can see it in the Process Manager log file OH/opmn/logs/ipm.log with 6 logging level switched
on. You can change the PM logging level in the OH/opmn/conf/opmn.xml file:
<log-file path="OH/opmn/logs/ipm.log" level="6"/>
PS Logging overhead is minimal.
7
Unless you use stand alone configuration then port is configured in rmi.xml
<global-thread-pool min="100" max="300" queue="30" keepAlive="-1" cx-
min="100" cx-max="300" cx-queue="30" cx-keepAlive="-1" debug="true"/>
...
</application-server>
min, max, queue, keepAlive parameters are not mandatory. If you do not set them
OC4J creates a single ApplicationServerThread in the thread pool during the start-up
and adds additional threads on an as-needed basis. Threads allocation is an expensive
task. OC4J spends CPU resource to allocate/deallocate worker threads. You can pre-
allocate threads in the thread pool by setting min parameter for better performance.
OC4J allocates min threads during the start-up. TaskManager never reduces the
number of threads in the pool less then min value during OC4J life time. If there is no
available thread in the pool OC4J puts new request in the queue. In case the queue
length reaches a limit set by queue parameter OC4J spawns a new
ApplicationServerThread thread. OC4J newer creates more threads then set by max
parameter. If thread is in an idle state longer then keepAlive parameter value (in milli-
seconds) TaskManager removes the thread from the pool.
If you set debug parameter to true TaskManager process prints the thread pools
information every time of the execution.
1108429632732:ThrP[100] ThrQ[0]
1108429632732:CxP[100] CxQ[0] - WebC[0] RmiC[0]
1108429633742:ThrP[100] ThrQ[0]
1108429633742:CxP[100] CxQ[0] - WebC[0] RmiC[0]
1108429634752:ThrP[100] ThrQ[0]
1108429634752:CxP[100] CxQ[0] - WebC[0] RmiC[0]
TaskManager
TaskManager process is responsible for regular tasks execution. You can change the
default execution frequency (1000 = 1 sec) by setting parameter in the server.xml file:
<application-server ... taskmanager-granularity="10000" ...>
In addition to thread pool housekeeping task TaskManager has other tasks.
TaskManager for example is responsible for removing expired http and JDBC
sessions. You can monitor different tasks statistics using dmstool utility:
$ dmstool -table oc4j_task | grep 3001 | grep completed
.../TASK[1]_com.evermind.server.ApplicationServerTask/run().completed 3811 ops
.../TASK[2]_com.evermind.server.jms.JMSTask/run().completed 3811 ops
.../TASK[3]_com.evermind.server.rmi.RMITask/run().completed 3811 ops
.../TASK[4]_com.evermind.server.rmi.RMITask/run().completed 3811 ops
.../TASK[5]_default/run().completed 3811 ops
.../TASK[6]_com.evermind.sql.DriverManagerXADataSource/run().completed 3811 ops
.../TASK[7]_com.evermind.sql.OrionPooledDataSource/run().completed 3811 ops
.../TASK[8]_com.evermind.sql.OrionCMTDataSource/run().completed 3811 ops
.../TASK[9]_BC4J/run().completed 3811 ops
.../TASK[10]_PANDA/run().completed 3811 ops
.../TASK[11]_com.evermind.server.http.SessionTimeoutTask/run().completed 3811 ops
$
Note: There is another set of parameters with prefix cx-. Documentation refers to this
set of parameters as parameters for second type of pool – conection pool. Author was
unable to find practical differences between these two types of pools.
OC4JMonitorThread+
OC4JMonitorThread and two Thread type treads are responsible for a communication
with ONS (Oracle Notification Service) process. One of Thread type treads makes
connection to the ONS process immediately after container start-up. Those threads are
responsible for sending and receiving notifications from other Oracle AS processes.
OC4J Troubleshooting
The troubleshooting in OC4J environment is connected strongly to a performance
tuning: often the root cause of the performance problems is serialization (bugs)
introduced in the container, framework or application code, often users claim that
OC4J system hangs when there are the performance problems within a database tier.
It is vital to have and to understand the available troubleshooting methods for
implementing and maintaining OC4J based systems successfully.
Using information and troubleshooting methods described in this paper you will be
able to identify the areas of the problems, to gather all the necessary information and
to provide it to the developers of the applications or to Oracle Support. In most cases
there is necessity to correct the code of the application, used framework or even
OC4J. So, it is extremely important to localize the problem area, gather all necessary
information and provide it to the appropriate organization.
Classification of Problem Areas

At the beginning of troubleshooting process it is important to be aware of possible
types of problems. Figure 2 highlights possible problem areas within OC4J
environments.
OC4J
JVM(java)
4.
PORT: 3001
AJP
JMSServer
JMSServer
Connection
Listener ApplicationServerThread
New Request
as
Apache
sig
n ApplicationServerThread
1.
TaskManager
3. DB
2. ApplicationServerThread
JDK
OS
Figure 2
1. OC4J container inside. This type of problems can be introduced by infinitive

loops, suboptimal application code, bugs of underlying software, serialization
introduced within used algorithms etc. The threads8 can wait on the objects
locked by other system of application threads. The Java deadlocks situation is
possible within a single OC4J process. The problems of this type can have the
spinning characteristics, when the thread consumes a lot of CPU resources, as
hanging ones (when the thread is waiting on some locked resources).
2. Interface to Apache processes. Sometimes threads can hang due to problems
with interface to Apache processes. The situation when OC4J thread and
8
In a context of this section author uses terms “application thread” or thread to reference
ApplicationServerThreads (see. OC4J internals) which are worker threads within OC4J environment.
Apache child process send the AJP messages to each other in the infinitive
loop can happen. The threads can have problems while receiving or sending
the information to Apache processes.
3. Interface to the database. Often the database problems (such as enqueues,
long running queries or DML operations, parsing or hanging cases) are
interpreted as OC4J problems. It is important to have the possibilities to
troubleshoot and to identify problem areas.
4. 500 Internal Errors or OC4J unavailability. Unlike previous types of
problems this one can be identified easily. If there are problems with AJP
(HTTP, RMI, JMS) Listener thread the whole OC4J process is not available
for application users’ requests. Users will receive “500 Internal Error”
message on the screens of the browsers. The problems of this type as well as
described above will be useful in further investigation and troubleshooting.
Possible Approaches & Information Sources

There are no many methods of OC4J process troubleshooting. No one of them will
give full information about your problem. You have to use a combination of methods
to diagnose OC4J related problems successfully. You can identify the most of CPU
consuming threads using OS Monitoring commands for example when use Java “Full
thread dump” to get an information of code to be executed and navigated to the
source of problem by using Java code decompilation.
Author identified and successfully used the methods and the additional information
sources listed below:
• Java “Full thread dump” – often underestimated method for Java processes
troubleshooting. This is the best source of troubleshooting information for
Java processes. For example, Java “Full thread dump” can give the
information of why the OC4J process is no longer responding. This method is
extremely effective in combination with OS monitoring commands. It is vital
to understand Java “Full thread dump” for OC4J troubleshooting. You do not
have to be a Java developer to make a preliminary analysis of dumps and to
make the conclusions. This method is described in this paper in more details as
well as other methods listed.
• Log files – there is a couple of useful OC4J log files you can use for
troubleshooting purposes.
• OS Monitoring commands – the OS commands are effective in the
combination with Java “Full thread dump”. You identify top resources
consuming Java threads by OS commands, then investigate OC4J process Java
dump to determine the state of the tread and executed code. You can use also a
reverse process to highlight the suspicious threads by analyzing Java “Full
thread dump” to monitor such threads by OS commands in order to get
knowledge of the resource consumption.
• OS Tracing commands – in the same time you can diagnose problems by using
this type of troubleshooting. You can use OS Tracing to attach to the
suspicious thread and to see thread’s outgoing calls to the OS. This method is
also useful in case you are unable to get Java “Full thread dump”.
• Apache side information – Apache monitoring tools and log files can give you
the indirect information about the state of OC4J processes.
• Java code decompilation – you can decompile Java code for troubleshooting
purposes
• Documentation, External searches and Metalink – other useful OC4J
troubleshooting information sources are covered also in this paper.
As there are no many OC4J troubleshooting methods available, you have to use all
of them in combination to be successful in your troubleshooting efforts. Ensure
automatic information gathering and publishing in case of problems. Make this
information available to the developers and to the support organization
representatives for the effective problem solving process.
Java “Full thread dump”

Most of JRE/JDK environments9 have built-in mechanism to dump a snapshot of all
Java threads’ states. This dump is called full thread dump. For each Java thread, you
get a Java thread’s name, a thread’s ID, and a Java call stack of the thread. Java-level
deadlocks information is also displayed in the dump. This section explains how to get
and to interpret full thread dumps of the OC4J process.
How to get Full thread dump?

There are four steps you have to execute to get OC4J Java “Full thread dump”:
1. Get OC4J process PID (OS process identification)
2. Locate OC4J output file
3. Send SIGQUIT (Quit from keyboard) signal to the process
4. Check the output file
Getting OS PID of OC4J

You can use ps, pstree or opmnadmin 10utilities to get OC4J process OS PID.
$ opmnadmin debug comp=pm
HTTP/1.1 200 OK
Content-Length: 677
Content-Type: text/html
Response:
======== PM ========
PM requests processed: 5
PROCESS TABLE
UID PID FLAGS TYPE STATUS REF HTTP AJP RMI JMS
---------- ----- -------- ------- ------- ------ ----- ----- ----- -----
456236 13512 00000040 OC4J Alive 1 0 3003 3104 3204
390700 13511 00000040 OC4J Alive 1 0 3002 3103 3203
521772 15874 00000040 OC4J Alive 1 0 3001 3102 3202
259628 13415 00000040 OC4J Alive 1 0 3000 3101 3201
194092 13338 00000040 Apache Alive 1 7780 0 0 0
128556 13281 00000040 Generic Alive 1 0 0 0 0
9
Java HotSpot version from Sun Microsystems used by all versions of OC4J and hashave Java “Full
thread dump” build-in functionality.
10
opmnadmin is Oracle AS utility. Starting from 9.0.4 version of Oracle AS it is renamed to opmnctl.
Output format differs in deferent Oracle AS versions.
opmnadmin command displays PID-s as well as other useful information about each
component managing bu OPMN process.
Locating OC4J output file

Use ps or command like that for your OS to find out the output file name:
$ ps -fw --width 4000 -p 16114
UID PID PPID C STIME TTY TIME CMD
oracle 15874 15902 0 23:40 ? 00:00:00
/u01/app/oracle/product/J2EE01/jdk/bin/i386/native_threads/Java -Xms512M -Xmx512M -
Doracle.ons.oraclehome=/u01/app/oracle/product/J2EE01 -
Doracle.ons.oracleconfighome=/u01/app/oracle/product/J2EE01 -Doracle.ons.clusterid=1 -
Doracle.ons.instanceid=10.130.2.9.20dcd9.fdc4ac6949.-8000 -
Doracle.ons.indexid=PANDY.default_island.1 -DOPMN=true -jar oc4j.jar -config
/u01/app/oracle/product/J2EE01/j2ee/PANDY/config/server.xml -properties -instance -
vmId C_10.130.2.9.20dcd9.fdc4ac6949.-
8000#J2EE01.ap1.voava.lv#521772#PANDY#PANDY#default_island -ajp 3001 -rmi 3102 -jms
3202
$ ps -wo args --width 4000 -p 16114 | grep -v COMMAND | awk -F"oracle.ons.indexid="

'{print $2}' | awk '{print $1}'
PANDY.default_island.1
In the example above the output file name is PANDY.default_island.1, where PANDY
is an OC4J instance name, 1 is an index of the OC4J process and island is a name of
an island. Typically the output file is located in OH/opmn/logs directory.
Sending SIGQUIT signal

Execute kill -3 <pid> command to send SIGQUIT (Quit from keyboard) signal to the
JVM. As Java “Full thread dump” is a generic JRE functionality, the JVM which
runs the OC4J process spools dump to the standard output file you located in previous
step.
Check the output file
Check the log file. You will see output like provided below.
Full thread dump:
"ApplicationServerThread-5" prio=1 tid=0x8450488 nid=0x576d waiting on monitor [0x6877f000..0x6877f870]

at Java.lang.Object.wait(Native Method)
- waiting on <437be538> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode)
at EDU.oswego.cs.dl.util.concurrent.SynchronousChannel.poll(SynchronousChannel.Java:353)
- locked <437be538> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.Java:767)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.Java:801)
at Java.lang.Thread.run(Thread.Java:479)
"Thread-10" prio=1 tid=0x8162568 nid=0x5738 waiting on monitor [0..0xbffe9b08]
"OC4JMonitorThread" daemon prio=1 tid=0x83c00b0 nid=0x576a waiting on monitor [0x698ff000..0x698ff870]

- waiting on <436e2b28> (a Java.lang.Object)
at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.Java:210)
- locked <436e2b28> (a Java.lang.Object)
at oracle.ons.NotificationQueue.dequeue(NotificationQueue.Java:183)
at oracle.ons.Subscriber.receive(Subscriber.Java:136)
at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.Java:419)
...
"VM Thread" prio=1 tid=0x80821e8 nid=0x4dd runnable
"VM Periodic Task Thread" prio=1 tid=0x808cb98 nid=0x4e0 waiting on monitor

"Suspend Checker Thread" prio=1 tid=0x808d518 nid=0x4e1 runnable
Java “Full thread dump” begins with a header Full thread dump. There is a list of all
the threads after the header.
The interpretation of Full thread dump
As mentioned above Java “Full thread dump” begins with the header, the list of all
the threads follows. Optionally there is Java-level deadlocks information if detected
by the JVM at the end of the threads dump.
This is an example of the available information about the single thread in the dump:
"OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor
[0xa53ff000..0xa53ff89c]
The most interesting parts are the following:

• OC4JMonitorThread – thread’s name assigned by Java application (by OC4J
in our case). OC4J assigns the unique meaningful names to the threads at the
moment when they are created. Thread.currentThread().setName During the
creation of the thread Java procedure is called for this purpose. If application
doesn’t call mentioned function JVM assigns Thread-XX name to the thread.
• nid=0x3ef2 – thread’s ID in a hexadecimal notation. This ID corresponds to
OS thread’s ID. This way you can map the thread dump information to the
threads you can monitor using OS utilities. It is very important to have such a
possibility. You can get the complete information about particular Java thread
in case of problems.
• waiting on monitor - execution state of the thread. The typical threads’ states
are the following: runnable, waiting on monitor, waiting for monitor entry. In
this document in the context of OC4J there is more detailed description of
each state.
• oracle.ons.NotificationQueue.internalDequeue – the last part is the current
stack trace snapshot of the thread. You can see all functions called by the
thread. The most upper part of the stacks corresponds to the current Java
function executed by the thread. In most cases you can understand what is the
thread doing using the stack information and can find the way for further
investigation of the problem.
Different types of the threads within OC4J container

In most cases you will see the following threads’ types executing OC4J applications:
OC4J Listeners - AJPConnectionListener, HTTPConnectionListener, RMIServer(2),
JMSServer
OC4J Workers - ApplicationServerThread
Other OC4J - OC4JMonitorThread, TaskManager, Thread
JAVA native - VM Thread, VM Periodic Task Thread, Reference Handler, Signal,
Dispatcher, Suspend Checker Thread, Finalizer
Each thread’s type is described in the “OC4J Internals” part of this document.
Different states of the threads

In Java “Full thread dump” there is the information about the state of each thread.
Author has seen three states of the threads so far:
• runnable – this state indicates that the thread executes an application code or
is waiting to receive a data from the socket. This state is typical for all OC4J
Listeners threads. If some of Listeners threads are in other state for a longer
time you probably have a problem and have to investigate it. Workers threads
in this state either execute Java application code or waiting on the socket for
the data. For example worker thread can wait on database query to be
executed. You can identify easily what is doing the thread looking on current
thread’s stack.
• waiting on monitor - waiting for some event or notification to occur. Often in
OC4J this state indicates that the thread is idle – thread is waiting for a task to
be executed. Currant call (shown in java stack) for thread in this state usually
is Java.lang.Object.wait(Native Method) that correspond to wait() or sleep()
functions in used in the Java code. This state is typical for all non Listener
threads in idle or sleeping state. Worker threads in this state are idle and
available to assign a task. TaskManager thread is waiting in this state for the
next execution time (calls sleep() function).
• waiting for monitor entry – thread in this state is waiting for a object locked by
the another thread. This is not normal state for the thread. If there are many
threads waiting on the same object this indicates clearly a bottle neck in the
system. You have to investigate this situation. Note the threads’ stacks, this
will show you the place in the code where the problem occurs; note all the
information available for the thread holding the lock.
Some threads examples.

Example 1
"OC4JMonitorThread" daemon prio=1 tid=0x679a3518 nid=0x573 waiting on monitor
[0xa53ff000..0xa53ff89c]
This is the typical state for OC4JMonitorThread OC4J thread. This thread is waiting
for a work to be processed. Java stack indicates that this thread executes ONS (Oracle
Notification Service) connected routines.
Example 2
"Thread-9" daemon prio=1 tid=0x67a02920 nid=0x571 runnable [0xa57ff000..0xa57ff89c]
at Java.net.SocketInputStream.socketRead(Native Method)
at Java.net.SocketInputStream.read(SocketInputStream.Java:85)
at oracle.ons.InputBuffer.readMoreData(InputBuffer.Java:267)
at oracle.ons.InputBuffer.getNextString(InputBuffer.Java:222)
at oracle.ons.ReceiverThread.run(ReceiverThread.Java:225)
The thread is waiting to receive the data from the ONS process. This thread connects
to the ONS process immediately after OC4J starts-up and is waiting for the messages.
As soon as message arrives the thread assigns OC4JMonitorThread thread to process
this message.
Example 3
"ApplicationServerThread-5" prio=1 tid=0x6794da88 nid=0x43f waiting on monitor
[0xb15ff000..0xb15ff89c]
at EDU.oswego.cs.dl.ut...concurrent.BoundedBuffer.poll(BoundedBuffer.Java:170)
at EDU.oswego.cs....concurrent.PooledExecutor.getTask(PooledExecutor.Java:768)
at EDU.oswego....concurrent.PooledExecutor$Worker.run(PooledExecutor.Java:802)
This is the typical state for ApplicationServerThread OC4J thread. This thread is
waiting for the work to be processed. OC4J Listener threads assign work to idle
worker threads.
Example 4
"AJPConnectionListener [0.0.0.0/0.0.0.0:3001]" prio=1 tid=0x67911828 nid=0x439
runnable [0xb19ff000..0xb19ff89c]
at Java.net.PlainSocketImpl.socketAccept(Native Method)
at Java.net.PlainSocketImpl.accept(PlainSocketImpl.Java:463)
at Java.net.ServerSocket.implAccept(ServerSocket.Java:238)
at Java.net.ServerSocket.accept(ServerSocket.Java:217)
at com.evermind...http.AJPConnectionListener.run(AJPConnectionListener.Java:58)
at EDU.oswego.c...concurrent.PooledExecutor$Worker.run(PooledExecutor.Java:804)
AJPConnectionListener is one of the OC4J listener threads. The state shown above is
typical for any OC4J Listener thread. Call to the AJP function shown in the thread
Java stack indicates that this thread is listening on the APJ port to accept new
connections. It assigns any received message addressed to available
ApplicationServerThread thread to put it to the queue. If this thread is in another state
for a longer (more then for a couple of seconds) time this is a clear indication of the
problem, you have to investigate it.
Example 5
"ApplicationServerThread-77" prio=1 tid=0x8391048 nid=0x4a75 runnable
[0xb4dff000..0xb4dff89c]
at com.evermind.io...nputStream.readChunk(SingleReadBufferInputStream.Java:116)
at com.evermind.io...ufferInputStream.read(SingleReadBufferInputStream.Java:35)
at com.evermind.server.http.AJPRe...ler.initRequest(AJPRequestHandler.Java:345)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.Java:131)
It is a problem possible to occur on Apache interface. Worker thread is reading or
waiting for the information to be received from the socket. The stack trace indicates
clearly that the ApplicationServerThread thread is reading AJP data from the socket
connected to Apache child process. It is not always indicating the problem. However
if the thread utilizes the CPU resource for a longer time this obviously is the problem.
Convert nid=0x4a75 to the decimal value (19061) and monitor the thread from the
OS. If you are on Linux you can use ps command to monitor whether the thread is
utilizing CPU constantly:
$ while [ 1 ] ; do ps -p 13285 -o pcpu | grep -v CPU ; sleep 1 ; done
25.3
25.3
25.3
See “Error! Reference source not found.” and “Error! Reference source not
found.” for more details.
Example 6
"ApplicationServerThread-71" prio=1 tid=0x8379130 nid=0x4a6f runnable
[0xb59fe000..0xb59ff89c]
at oracle.net.ns.Packet.receive(Unknown Source)
at oracle.net.ns.DataPacket.receive(Unknown Source)
at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.Java:931)
at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.Java:893)
at oracle.jdbc.ttc7.Oall7.receive(Oall7.Java:375)
at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.Java:1986)
at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.Java:1144)
at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.Java:2484)
at oracle.jdbc.driver...atement.doExecuteWithTimeout(OracleStatement.Java:2850)
at oracle.jdbc.driver...atement.executeUpdate(OraclePreparedStatement.Java:609)
at oracle.jdbc.driver...tatement.executeQuery(OraclePreparedStatement.Java:537)
at oracle.jbo.server.ViewObjectImpl.getQueryHitCount(ViewObjectImpl.Java:1522)
at oracle.jbo.server...llection.getEstimatedRowCount(QueryCollection.Java:1530)
at oracle.jbo.server...owSetImpl.getEstimatedRowCount(ViewRowSetImpl.Java:1482)
at oracle.jbo.server...bjectImpl.getEstimatedRowCount(ViewObjectImpl.Java:4596)
at _common._comp._CMN__...ent._jspService(_CMN__DataScrollerComponent.Java:115)
at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.Java:56)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.Java:317)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.Java:476)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.Java:390)
at Javax.servlet.http.HttpServlet.service(HttpServlet.Java:853)
at com.evermind.server.http...FilterChain.doFilter(ResourceFilterChain.Java:65)
at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.Java:283)
at com.evermind.server.http...patcher.invoke(ServletRequestDispatcher.Java:566)
at com.evermind.server.http...atcher.include(ServletRequestDispatcher.Java:119)
at com.evermind.server.http...r.include(GetParametersRequestDispatcher.Java:95)
at com.evermind.server.http...PageContext.include(EvermindPageContext.Java:280)
at oracle.jbo.html.jsp.datatags.ComponentTag.doStartTag(ComponentTag.Java:70)
at _skd._d001._D2260F__Browse._jspService(_D2260F__Browse.Java:249)
at com.evermind.server.http...FilterChain.doFilter(ResourceFilterChain.Java:65)
at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.Java:283)
at com.evermind.server.http...patcher.invoke(ServletRequestDispatcher.Java:566)
at com.evermind.server.http...orwardInternal(ServletRequestDispatcher.Java:306)
at com.evermind.server.http...ndler.processRequest(HttpRequestHandler.Java:767)
The problem on the DB interface is possible. Worker thread is reading or waiting for
the information from the database. It is clear from the stack that this thread is waiting
for a response from the database. You can see the name of Java server page which
made the call to the database _skd._d001._D2260F__Browse. If the thread is in this
state for a longer time you have to look on the database tier to investigate the reason
of a slow response time. If there are many threads waiting on the database it is the
bottleneck in your environment.
Useful Java runtime parameter

It is difficult to investigate the threads in waiting for monitor entry state. JVM doesn’t
reflect information about locked objects by default. Take a look on the example
below.
"ApplicationServerThread-18" prio=1 tid=0x692c40e0 nid=0x2f92 waiting for monitor
entry [0xbc1ff000..0xbc1ff89c]
at oracle.jbo.server.ViewRowSe...setRangeSize(ViewRowSetIteratorImpl.Java:376)
at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.Java:1659)
at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.Java:4626)
at lv.vovaa.panda.common.da...DataSourceCmn.doStartTag(DataSourceCmn.Java:189)
at _nsk._nsk001._M2801F__Edit._jspService(_M2801F__Edit.Java:150)
There is a problem inside the OC4J container. Worker thread is waiting on an object
blocked by the other thread. If the thread is in this state for a while or there are many
threads in this state this indicates clearly the problem in the application code. You are
unable to find the blocking thread and locked object using the information provided.
There is a very helpful and often underestimated Java generic parameter
“-XX:+JavaMonitorsInStackTrace”.
After you set this parameter in Java-option section of the opmn.xml file the JVM will
reflect information about locked objects by the threads the dump:
"ApplicationServerThread-96" prio=1 tid=0x69d6c8c0 nid=0xdce waiting for monitor entry
[0xb25ff000..0xb25ff89c]
at
oracle.jbo.server.ViewRowSetIteratorImpl.setRangeSize(ViewRowSetIteratorImpl.Java:376)
- waiting to lock <460e0338> (a oracle.jbo.JboSyncLock)
at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.Java:1659)
at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.Java:4626)
at lv.vovaa.PANDY.common.da...DataSourceCmn.doStartTag(DataSourceCmn.Java:189)
- locked <45f722d0> (a lv.vovaa.PANDY.common.datatags.DataSourceCmn)
at _nsk._nsk...Tickets__browse._jspService(_AP3220F__Tickets__browse.Java:105)
As you can see the JVM reflects the information about the objects locked by the
thread (object_id=45f722d0) as well as about the object which the thread is waiting
on (object_id=460e0338). It is possible to locate the thread which locked the object
using the information provided:
"ApplicationServerThread-55" prio=1 tid=0x69d512b8 nid=0xda4 runnable
[0xb77fe000..0xb77ff89c]
at oracle.net.ns.Packet.receive(Unknown Source)
...
at oracle.jbo.server.QueryCollection.executeQuery(QueryCollection.Java:549)
at oracle.jbo.server.ViewObjectImpl...yForCollection(ViewObjectImpl.Java:2600)
- locked <460e0338> (a oracle.jbo.JboSyncLock)
at lv...PboViewObjectImpl.pboExecuteQueryForCollection(PboViewObject.Java:324)
at lv...PboViewObjectImpl.executeQueryForCollection(PboViewObjectImp.Java:309)
at oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.Java:521)
...
The thread ApplicationServerThread-96 is waiting on the object
oracle.jbo.JboSyncLock id=460e0338 locked by the thread ApplicationServerThread-
55. Thread ApplicationServerThread-55 is waiting on database. It is clear that the
application (framework, OC4J) code introduces a serialization. One thread is waiting
on the other which is reading information from the database. Provide discovered
information to the development. In most cases this is the coding problem and it has to
be corrected. Otherwise the development should make a comment why this
serialization happened.
The “-XX:+JavaMonitorsInStackTrace” parameter is extremely useful for OC4J
application troubleshooting. According to author’s experience this parameter does not
add significant overhead to the OC4J tier and can be used in production environments.
Java DEADLOCK problem

JVM reflects information about Java deadlocks in Java “Full thread dump” if any
detected. However JVM is unable to solve this kind of problems as it happens in case
of Oracle Database deadlocks. OC4J process has to be restarted in case of Java
deadlock. Information about the existing deadlock reflected at the end of Java “Full
thread dump”.
Full thread dump:
"ApplicationServerThread-126" prio=1 tid=0x881f668 nid=0x5d59 waiting for monitor
entry [0xa19ff000..0xa19ff89c]
...
...
FOUND A JAVA LEVEL DEADLOCK:
----------------------------
"ApplicationServerThread-58":
waiting to lock monitor 0x80b9804 (object 0x4595b408, a Java.lang.Object),
which is locked by "ApplicationServerThread-51"
"ApplicationServerThread-51":
waiting to lock monitor 0x80b9564 (object 0x45955750, a
com.evermind.server.http.EvermindHttpSession),
which is locked by "ApplicationServerThread-58"
JAVA STACK INFORMATION FOR THREADS LISTED ABOVE:
------------------------------------------------
Java Stack for "ApplicationServerThread-58":
==========
at oracle.jbo.http.HttpContainer.setValue(HttpContainer.Java:352)
- waiting to lock <4595b408> (a Java.lang.Object)
As you can note there is the complete information about locked object and blocked
threads. Provide this information to the development and restart OC4J process.
You have to monitor regularly the availability of OC4J process and in case of Java
deadlock to restart OC4J. This is not happening automatically by default. Author
successfully used a shell scripting for this purposes.
Log files
There are a dozen of log files within the standard OC4J installation. Author would
like to pay your attention to two of them.
OC4J output file

We have discussed this file in the context of Java “Full thread dump” (see. “Locating
OC4J output file” to get location of the file). Apart from Java dump, a lot of
information is going to this file, for example:
• TaskManager11 process output – if you are switched on TaskManager’ debug
parameter you can monitor whether the process is functioning. The standard
report about the thread pool status appears in output file at the end of each
TaskManager working cycle:
1108429632732:ThrP[100] ThrQ[0]
1108429632732:CxP[100] CxQ[0] - WebC[0] RmiC[0]
1108429633742:ThrP[100] ThrQ[0]
1108429633742:CxP[100] CxQ[0] - WebC[0] RmiC[0]
1108429634752:ThrP[100] ThrQ[0]
1108429634752:CxP[100] CxQ[0] - WebC[0] RmiC[0]
If output doesn’t appear in the output file it indicate the problem in the
TaskManager process work.
•
OC4J process debug information – if you set any OC4J debug switches (e.g. -
Dajp.io.debug=true) the debug information appears in OC4J output file.
• Standard java application output – if application developer uses
System.out.println function in the application code the output appears in OC4J
output file.
As you can see there is a lot of useful information in this file. Unfortunately there is
not any clear indication of the event time in the file. Author finds useful to add
11
TaskManager process discussed in more details in “OC4J Internals” part of this paper
regularly a timestamp to the file. You can achieve this by adding line below 12to the
UNIX user crontab:
* * * * * echo "Timestamp: `date`" >>
/u01/app/oracle/product/J2EE01/opmn/logs/PANDY.default_island.1
OC4J process output file is one of the best sources of the troubleshooting information.
OC4J default-web-access.log
This file is an analogue of Apache access_log file. The information of each request
processed by OC4J process is reflected in the file. This information can be extremely
useful to track which OC4J process have processed the request in case you are using
configuration with many OC4J processes acting as single OC4J instance13.The
example of file location is seen below:
OH/ j2ee/PANDY/log/PANDY_default_island_1/default-web-access.log
OS Monitoring commands
Please note that multithreading implementation differs and strongly
depends on used OS. Monitoring commands used in this section are
relative to Linux OS. Look for equal functionality commands for your
OS.
As it was mentioned before each source of OC4J troubleshooting information does
not provide a complete view of the problem. You have to use all the appropriate
methods to get sufficient information to troubleshoot the problem.
By author’s experience the most useful sources of OC4J process runtime information
are Java “Full thread dump” and OS commands. Use “nid=0x3ef2” – thread’s id
provided in Java dump to connect Java dump information to the information gathered
by OS commands. When the problem occurs you can go from Java dump to OS
commands and to reverse the direction.
For example if during OC4J unavailability you observe the high utilization of CPU
(using vmstat, iostat, uptime commands), you can use top or ps -eo pcpu,pid,args |
sort -k1n | tail -10 OS commands to detect top CPU consumers. Note pid-s of the
processes and convert them to the hexadecimal values. Make a couple of Java “Full
thread dump” and navigate to the threads you have identified by OS command. Note
the information about the status of the thread (in most cases the state of top CPU
consumers’ threads will be runnable), java stack (provides the information about the
executed application code) and name of the thread. Provide information you gathered
to the developers for further investigation.
In other cases the reverse investigation path is more appropriate. Making Java “Full
thread dump” you observe that many Java threads are waiting in the waiting for
monitor entry state for the object locked by the tread in the runnable state. Use OS
command to identify if thread is consuming resources or is waiting to receive the
information from TCP/IP socket. For this purpose you can use ps command as shown
below:
$ while [ 1 ] ; do ps -p 13285 -o pcpu | grep -v CPU ; sleep 1 ; done
25.3
25.3
12
Replace the name of the output file to correct one for your environment
13
OC4J Performance section for information about multiple OC4J configuration
25.3
$ while [ 1 ] ; do ps -p 19273 -o vsz,rss | grep -v VSZ ; sleep 1 ;
done
1095372 258244
1095372 258244
1095372 258244
In case the thread is using CPU resource you can use OS tracing commands
(described in the next section) to get the additional information.
Author has found the OS monitoring commands listed below as the most useful for
OC4J process troubleshooting:
• System wide:
vmstat, iostat, uptime, pstree
• Top consumers of the resources
top, ps -eo pcpu,pid,args | sort -k1n | tail -10
• Particular process information
while [ 1 ] ; do ps -p <pid> -o pcpu | grep -v CPU ; sleep 1 ; done
while [ 1 ] ; do ps -p <pid> -o vsz,rss | grep -v VSZ ; sleep 1 ; done
OS Tracing commands
Many operational systems (OS-s) have tracing utilities. They are able to attach to
running process and to reflect all calls made by process to the OS. In some situations
OS processes’ tracing capabilities add invaluable information for problems
troubleshooting. Let us illustrate OS tracing commands’ possibilities within OC4J
troubleshooting context by the example.
You have noted for example that there are two processes consuming most of CPU
resources for a longer time using top command:
$ top -d 1
3:23pm up 7 days, 5:24, 4 users, load average: 1.36, 1.27, 1.49
1080 processes: 1076 sleeping, 3 running, 0 zombie, 1 stopped
CPU2 states: 5.0% user, 21.0% system, 0.0% nice, 72.1% idle
CPU3 states: 25.0% user, 64.0% system, 0.0% nice, 9.1% idle
...
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
27021 oracle 25 0 587M 587M 25096 R 65.7 15.2 244:22 java
26018 oracle 15 0 11228 10M 8184 S 36.0 0.2 165:02 httpd
9514 oracle 15 0 2232 2228 836 R 8.1 0.0 0:00 top
16047 root 15 0 1172 1172 852 S 6.3 0.0 15:11 cmahostd
1 root 15 0 524 524 452 S 0.0 0.0 0:04 init
One of the processes is the Apache child process, the second – java thread. From Java
dump you can see that the thread with id 27021 is in the runnable state. Java stack
indicates that this thread is sending AJP packages:
"ApplicationServerThread-31" prio=1 tid=0x69cd3c50 nid=0x698d runnable [0xba7ff000..0xba7ff89c]
at java.net.SocketOutputStream.socketWrite(Native Method)
at java.net.SocketOutputStream.write(SocketOutputStream.java:60)
at com.evermind.server.http.AJPBodyInputStream.refillReadBuffer(AJPBodyInputStream.java:130)
at com.evermind.server.http.AJPBodyInputStream.read(AJPBodyInputStream.java:102)
at com.evermind.server.http.EvermindServletInputStream.read(EvermindServletInputStream.java:193)
at com.evermind.server.http.EvermindHttpServletRequest.parsePostData(EvermindHttpServletRequest.java:2103)
at com.evermind.server.http.EvermindHttpServletRequest.parseParameters(EvermindHttpServletRequest.java:2047)
at com.evermind.server.http.EvermindHttpServletRequest.getParameter(EvermindHttpServletRequest.java:2133)
at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:203)
at _nsk._nsk001._SP0301F__Save._jspService(_SP0301F__Save.java:68)
at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56)
at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317)
at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476)
at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:306)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:767)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:166)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:72)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804)
at java.lang.Thread.run(Thread.java:479)
By tracing the java thread we got the following information:

$ strace –s1000 -p 27021
--- SIGSTOP (Stopped (signal)) ---
read(104, "\0224\0\2\0\0", 4096) = 6
send(104, "A", 1, 0) = 1
send(104, "B", 1, 0) = 1
send(104, "\0", 1, 0) = 1
send(104, "\3", 1, 0) = 1
send(104, "\6", 1, 0) = 1
send(104, "\37", 1, 0) = 1
send(104, "\372", 1, 0) = 1
read(104, "\0224\0\2\0\0", 4096) = 6
send(104, "A", 1, 0) = 1
send(104, "B", 1, 0) = 1
send(104, "\0", 1, 0) = 1
send(104, "\3", 1, 0) = 1
send(104, "\6", 1, 0) = 1
send(104, "\37", 1, 0) = 1
send(104, "\372", 1, 0) = 1
read(104, "\0224\0\2\0\0", 4096) = 6
send(104, "A", 1, 0) = 1
send(104, "B", 1, 0) = 1
send(104, "\0", 1, 0) = 1
send(104, "\3", 1, 0) = 1
send(104, "\6", 1, 0) = 1
send(104, "\37", 1, 0) = 1
send(104, "\372", 1, 0) = 1
read(104, "\0224\0\2\0\0", 4096) = 6
...
The process is sending continuously the string “AB\0\3\6\37\372” and is receiving
“\0224\0\2\0\0”. The apache process’s tracing gives us the following results:
[root@ap1 root]# strace –s1000 -p 26018
recv(30, "\3", 1, 0) = 1
recv(30, "\6\37\372", 3, 0) = 3
send(30, "\0224\0\2\0\0", 6, 0) = 6
recv(30, "AB\0", 4, 0) = 3
recv(30, "\3", 1, 0) = 1
recv(30, "\6\37\372", 3, 0) = 3
send(30, "\0224\0\2\0\0", 6, 0) = 6
recv(30, "AB\0", 4, 0) = 3
recv(30, "\3", 1, 0) = 1
recv(30, "\6\37\372", 3, 0) = 3
send(30, "\0224\0\2\0\0", 6, 0) = 6
recv(30, "AB\0\3", 4, 0) = 4
recv(30, "\6\37\372", 3, 0) = 3
send(30, "\0224\0\2\0\0", 6, 0) = 6
recv(30, "AB\0\3", 4, 0) = 4
recv(30, "\6\37\372", 3, 0) = 3
Obviously two processes send the same packages to each other in the infinitive loop.
This explains why these two processes are consuming a lot of CPU resources.
The information gathered using OS tracing capabilities was invaluable for the support
organization. After this information was provided to the development, the problem in
the code was identified and corrected.
Please note that OS tracing functionality highly depends on operational system you
use. The strace utility used in the example is Linux specific. Most of the OS-s have
tracing utilities and they can be very useful during OC4J troubleshooting scenarios.
Apache side information

Apache Status page
There is a build-in status page in the Apache server. During unavailability cases this is
the additional source of information. You navigate to the “status page” using URL
provided below14:
http://<server name>:<apache port>/server-status?refresh=15
Live example:
http://ap10.zont.lv:7780/server-status?refresh=15
Apache “status page” provides you with apache side information. There is the
information about Apache child processes such as: active processes count, processes
waiting for OC4J, available Apache child processes etc. This information helps you to
evaluate the situation correctly.
Apache log files

Apache log files are located in the $OH/Apache/Apache/logs directory. Author has
found that the information provided in the Apache log files can be misleading or
interpreted incorrectly in the context of OC4J troubleshooting.
Let us illustrate it by example. Often in the OC4J environment you can see messages
about 500 errors in the access_log Apache log file:
192.168.2.12 - - [01/Feb/2005:16:43:34 +0200] "GET
/PANDY/NSR/NSR003/NT0291F_Browse.jsp HTTP/1.1" 500 0
In the erorr_log file there are the sets of error messages:
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0080: After calling
Apache's ap_bwrite(), got an return value: -1 and the client could have aborted in the
middle of receiving the chunk data.
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0058: Failed to
handle response chunk data passed in from oc4j via ajp13.
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0035: After hdndling
ajp13 response message, got an invalid value: -2.
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0121: Failed to
service request with network worker: PANDY_15 and it is not recoverable.
[Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0013: Failed to call
destination: PANDY's service() to service the request.
These messages mean that user of the application had pressed Stop Internet Browser
button. Connection has been interrupted. After the response receiving from OC4J
Apache child process was unable to send it to the Browser.
As was demonstrated above Apache log files’ information can be misleading and
error messages in apache log files don’t always indicate the problem.
14
Please note that you have to allow access to the page by editing OH/Apacha/Apache/conf/httpd.conf
file.
Java code decompilation
Unlike units written in other languages Java classes can be decompiled easily. There
are many utilities available for this purpose15. As mentioned above OC4J (oc4j.jar)
and other related components (JDBC, BC4J, etc.) are written in Java and can be
decompiled.
For troubleshooting purposes this adds information for you investigation. You can
unzip the related archive files and decompile class files. Author has found useful to
decompile java components listed below:
OC4J
OH/j2ee/home/oc4j.jar
JDBC
OH/jdbc/lib/classes12.zip
OH/jdbc/lib/classes12dms.jar
DCM
OH/dcm/lib/dcm.jar
OH/dcm/lib/oc4j_deploy_tools.jar
OH/dcm/lib/oc4j_remote_deploy.jar
DMS
OH/lib/dms.jar
BC4J
OH/BC4J/redist/bc4j.ear
OH/BC4J/jlib/*
OH/BC4J/lib/*
Debugging Switches
OC4J debugging switches documented well in the Oracle AS 10g Documentation set
Containers for J2EE User's Guide
10g Release 2 (10.1.2)
Part No. B14011-01
B Additional Information
B.8 OC4J Command-Line Options and System Properties
If your OC4J environment is on the earlier version you can still use the book
mentioned above. Author has found that this is the first release of OC4J
documentation where the debugging switches described relatively well.
You set OC4J debugging switches in the OH/opmn/conf/opmn.xml file (java-option
section). There is the example of OC4J output file with AJP debugging switch turned
on (-Dajp.io.debug=true) shown below.
15
One of these utilities is available from http://www.kpdus.com/jad.html
Output file PANDA.default_island.1
1107387907360:ThrP[50] ThrQ[0]
1107387907360:CxP[20] CxQ[0] - WebC[0] RmiC[0]
AJP: Input request length is: 1
AJP: AJPOutputStream::write() of 1
AJP: AJPOutputStream::writeChunk(): 1
1107387908370:ThrP[50] ThrQ[0]
1107387908370:CxP[20] CxQ[0] - WebC[0] RmiC[0]
Each switch is useful in different situation. Some of available useful categories are:
ajp, http, jms, rmi, jdbc.
Please do not forget to test each change of the configuration in the test environment
before to implement it in the production.
Documentation, External searches and Metalink

Documentation
Author has found that 10g (10.1.2) Documentation set describes better OC4J related
questions comparing to previous realises of Documentation (9.0.2/3/4). The concepts
described in 10g documentation are valid for the earlier releases as well. You will get
a better understanding of OC4J reading 10g documentation set.
Author recommends to review books listed below for better understanding of OC4J
related questions and troubleshooting possibilities:
Oracle AS Documentation Library 10g Release 2 (10.1.2)
Containers for J2EE User's Guide (144)
Containers for J2EE Standalone User's Guide (134)
Server Administrator's Guide (542)
OPMN Administrator's Guide (158)
DCM Administrator's Guide (116)
Server Performance Guide (234)
Oracle AS Documentation Library 10g (9.0.4)
Oracle® Application Server 10g Best Practices (148)
In case you are using BC4J based application please note that most of information
regarding BC4J usage is located in JDeveloper Online Help system.
External searches
Unfortunately there is no much information about OC4J. Sometimes you can get
information you are interested in using www.google.com search capabilities.
http://otn.oracle.com has a couple of forums on OC4J related questions; unfortunately
the searching capabilities are very limited there. Try to ask question you are interested
in and probably a person from Oracle will answer you.
http://metalink.oracle.com does not have OC4J related forums and available
information is rather poor. There are no many technical notes related to OC4J
technology and OC4J troubleshooting. Author however has found very useful to
review Metalink Patches section of the Metalink to get the solution for some critical
scalability problems.
OC4J Performance
In addition to the troubleshooting methods described in “OC4J troubleshooting” part
this section describes author’s findings in the context of OC4J performance tuning
and available performance monitoring capabilities as well as other author’s
recommendations.
Our Findings
Java Workers
OC4J allows you to set a minimal number of ApplicationServerThread (Workers16),
as well as the maximum number of Workers and the size of requests’ queue. In the
ideal situation when workers do not block each other and the database responds
quickly to any request sent by Worker threads, the initial number of workers can be
set to the minimal value (20) or even to be unset17.
Author however has found that the setting of the minimal number to the highest value
improves OC4J instance availability, scalability and overall response time. If there is
a blocking situation in your environment and there is no available worker thread,
OC4J will put the requests of each user in a queue. OC4J will spawn the additional
Worker thread only when queue achieves the limit set in the configuration file. This
behavior can significantly limit OC4J process availability and performance.
Set the Worker thread’s parameters in the server.xml file (see. example below).
OH/j2ee/<comp_name>/config/server.xml
<application-server ...>
...
<global-thread-pool
min="100" max="300" queue="30" keepAlive="-1"
cx-min="100" cx-max="300" cx-queue="30" cx-keepAlive="-1"
debug="true"/>
...
</application-server>
Setting the min parameter to the high value you make OC4J free from resources’
intensive procedure of creating the new Workers, as well as from queue managing
routine. Reader can argue that the high number of unused threads adds the additional
overhead and utilization of CPU resources. Author however did not note the
significant overhead while setting the parameter in the production environments.
OC4J Processes
It is possible to startup the multiple OC4J processes, which act as one OC4J instance.
mod_oc4j (Apache module) will spread users’ requests across all the available OC4J
processes. Even if OC4J instance is located on a single host you can earn a significant
improvement in terms of system availability and performance by distributing the load
through multiple OC4J instances.
Set numProcs parameter in opmn.xml file for particular instance. There are three
processes for instance PANDY in the following example.
16
Workers – are the threads which process users’ requests and execute the application code. For the
additional information about this type of threads see “OC4J Internals”
17
If you do not set the initial number of Workers OC4J initializes 1 Worker thread at the startup
<oc4j maxRetry="3" instanceName="PANDY" gid="PANDY" numProcs="1">
<config-file
path="/u01/app/oracle/product/J2EE01/j2ee/PANDY/config/server.xml"/>
<java-option value="-server -Xms512M -Xmx512M"/>
<oc4j-option value="-properties "/>
<port ajp="3001-3100" jms="3201-3300" rmi="3101-3200"/>
<island id="default_island" numProcs="3"/>
</oc4j>
Please note that OPMN will start three dedicated OC4J processes. Each of them will
utilize its own memory areas. In the example above all the PANDY instance OC4J
processes will utilize 1.5 GB of RAM. Ensure that you have the sufficient memory
resources for all instances. Please note that each process acts independently, each
process will have separate DMS and BC4J statistics, as well as own AJP/RMI/JMS
ports.
The mod_oc4j however will send users’ requests to one particular OC4J process.
Routing information for each user session is stored in the JSESSIONID coke18.
Memory
Author has found reasonable to set the minimal size of memory available to the single
OC4J process equal to the maximum size. However this is not a strict
recommendation. You set memory usage parameters in opmn.xml file:
<oc4j …
<java-option value="-server -Xms512M -Xmx512M"/>
</oc4j>
Setting min=max JVM does not need memory to be allocated dynamically. Each
OC4J process reserves memory during startup.
You have to monitor regularly the usage of memory within OC4J process. For this
purpose you can set -verbose:gc java parameter, as well as dmstool functionality.
DMS utility allows you to monitor memory usage any time:
$ dmstool -table JVM | grep 3001 | grep Memory
/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.value 495796 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.minValue 495796.0 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.maxValue 522468.0 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.value 524032 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.minValue 524032.0 kbytes
/ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.maxValue 524032.0 kbytes
ap1.voava.lv*J2EE01-/u01/app/oracle/product
$
The single snapshot of memory metrics does not add any significant value. It might be
more effective to collect metrics regularly.
18
You can observe cookie by entering javascript:alert(document.cookie); string in the Internet
Browser address bar after you logged to OC4J application.
The above example reflects the typical behavior of OC4J JVM. Garbage Collector
(GC) frees the significant memory areas periodically. There is a lot of information
regarding the tuning of GC. For example you may find the following links useful:
http://java.sun.com/docs/hotspot/gc/
http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/
http://www.cons.org/cmucl/doc/gc-tuning.html
Grep OC4J log files for out of memory errors. If you allocated a significant amount of
memory it is possible that there are the memory leaks in the application code. Ask the
application support or the development organizations for memory allocation
recommendations.
In case of memory leaks Java Profilers19 such as JProfiler from Quest Software or
others can be used to locate and to find root of the problem. Be aware that Java
Profilers add significant overhead and are not usable normally in the production
environments.
Stresstesting
Often a need for the stresstesting is underestimated. However, according to author’s
experience the stresstesting is one of the best methods to tune OC4J application
performance. Weak areas can be identified easily. Author considers using the
automatic stresstesting tools. This will allow you to repeat the testing for several times
and to estimate the code correction results. It is almost impossible to test all the
functionality and dynamic behavior of system users. Include the most used forms and
functionality in the stresstesting scenarios. Repeat testing – performance the problem
correction cycles for several times until you get the appropriate results.
By implementing the stresstesting process you can avoid a lot of problems in future.
19
The profiling with Oracle JDeveloper is possible on Win32 platform only.
Response Time Monitoring by DMS
Apache OC4J
OHS [c] mod_plsql JVM [java]
handle
Send
Request B
Handle B ApplicationServerThread
Read
Resolve
Process B
JSP Processing
mod_oc4j Servlet Processing
DB
handle
Process E
Handle E
Handle B
Handle E
Request E
mod_perl
Recive mod_php
...
DMS Sensors DMS Sensors
DMS-client
DMS.JAR (java)
statistics reflecting
The figure above reflects the example of single user’s request processing. Apache as
well as OC4J and other Oracle AS components have the build-in DMS (Dynamic
Monitoring Services) sensors, which reflect different performance metrics. You can
monitor this statistics using dmstool.
Response Time Breakdown Example

$ dmstool -table ohs_server | grep time
.../Apache/connection.time 48123608767 usecs
.../Apache/request.time 15916131942 usecs
.../Apache/handle.time 15913988395 usecs
$
$ dmstool -table ohs_module | grep time | sort -k2n | tail -2
.../Apache/Modules/mod_dms.c/handle.time 8081655 usecs
.../Apache/Modules/mod_oc4j.c/handle.time 15905072698 usecs
$
$ dmstool -table oc4j_jspExec | grep time
.../oc4j/PANDY/WEBs/PANDY/JSPs/processRequest.time 17 563 793 msecs
$
$
$
$ dmstool -table "oc4j_jsp(threadsafe=true)" | grep time | sort -k2n | tail -5
.../oc4j/PANDY/WEBs/PANDY/NS0285F_searchForm.jsp/service.time 1 072 570 msecs
.../oc4j/PANDY/WEBs/PANDY/D2261F_PersonsBrowse.jsp/service.time 1 407 663 msecs
.../oc4j/PANDY/WEBs/PANDY/NS0291F_Browse.jsp/service.time 2 942 655 msecs
.../oc4j/PANDY/WEBs/PANDY/D2261F_Edit.jsp/service.time 3 151 184 msecs
.../oc4j/PANDY/WEBs/PANDY/D2260F_Browse.jsp/service.time 4 847 408 msecs
$
As it is shown in the example you can track Apache response time. Unfortunately
DMS do not have the jet capabilities as event 10046 within Oracle database. However
they can give you the answers to the questions like:
- Which handle (Apache module) has taken the most of the response time?
- How much time the JSP processing has taken?
- Which JSP has taken the longest time to execute?
DMS Sensors Classification

You can observe all the available DMS sensor types by entering dmstool -list -table
command. This command reflects thousands of different metrics (depending on
configuration). It is difficult to identify the important ones. There is the DMS metric
classification below to simplify the process of identification:
• Informative sensors – reflect a configuration information and some system
wide statistics
o ohs_child, ohs_responses,
o JVM, oc4j_ear, oc4j_opmn, oc4j_task,
o JDBC_Driver , JDBC_Connection
• Average performance sensors – reflect response time totals by modules,
applications, etc.
o ohs_server , ohs_module, ohs_virtualHost,
o oc4j_context, oc4j_web_module, oc4j_jspExec
• Units performance sensors – reflect response time statistics for each unit
o oc4j_servlet
o oc4j_jsp(threadsafe=true)
Collecting and Reflecting DMS Statistics

Be aware that the DMS Sensor statistics is incremental. You have to make the
periodical snapshots of statistics to get a time dimension for further analysis. Develop
the interface to access statistics easily. For example you may reflect the average
performance sensors graphically:
Or develop STATSPACK like interface to reflect Units performance metrics:
SYS:PROD> @sp_jsp
Completed Snapshots
Snap
Id Snap Started
------- -------------------
530 2005-02-03 12:33:26
531 2005-02-03 13:03:24
532 2005-02-03 13:33:27
549 2005-02-03 14:03:32
Specify the Begin and End Snapshot Ids

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Enter value for begin_snap: 532
Enter value for end_snap: 549
Saved in c:\temp\532_549.log
Snap Id Snap Time

------- ------------------
Begin Snap: 532 2005-02-03 13:33:27
End Snap: 549 2005-02-03 14:03:32
Elapsed: 30.08 (mins)
JSP ordered by Response time
JSP Name Response Execution Average

(sec) Time(sec) Count Response
------------------------------------------------------------------ --------- ---------- ----------
../3001/../NSK/NSK001/AP3220F_TicketImportProcessOneByOne.jsp 956.439 2 478.22
../3002/../NSK/NSK001/AP3220F_Tickets_browse.jsp 708.967 71 9.99
../3001/../NSK/NSK001/M2801F_Browse.jsp 533.389 74 7.21
../3003/../NSK/NSK001/P2600F_XMLImportProcess.jsp 471.106 1 471.11
../3003/../LRG/l001/L3000F_Browse.jsp 293.775 92 3.19
../3002/../NSK/NSK001/AP3221F_ApnAccountingDocFilter.jsp 291.382 6 48.56
../3002/../common/comp/CMN_DataFilterComponent.jsp 238.75 35 6.82
../3001/../NSK/NSK001/SP0301F.jsp 216.499 32 6.77
../3001/../SKD/d001/D2262F_Browse.jsp 208.709 7 29.82
../3001/../NSK/NSK001/SP0301F_filter.jsp 168.957 6 28.16
../3001/../NSK/NSK001/Z3601F_Save.jsp 80.127 30 2.67
../3001/../common/comp/CMN_LOVcomp.jsp 73.544 52 1.41
../3003/../NSK/NSK001/SP0301F.jsp 72.548 29 2.5
../3002/../NSK/NSK001/AP3221F_Save.jsp 68.509 91 .75
../3001/../common/comp/CMN_DataHandlerComponent.jsp 68.46 765 .09
../3002/../common/comp/CMN_DataHandlerComponent.jsp 57.144 722 .08
SYS:PROD>
Appendix A. How to diagnose opened files, sockets,
pipes by OC4J process?
It can be useful for diagnostic and troubleshooting purposes to see all files opened by
the process. For this purposes you can use lsof like utility20. The lsof utility lists all
files opened by the process. There is an example of lsof output for OC4J process
below (see Figure 3 lsof utility output example). Using mentioned utility you can see
easily all the log files, sockets, *.JAR files opened by OC4J process. It’s interesting to
see for example that OC4J process uses different *.JAR files from directories listed
below:
/u01/app/oracle/product/J2EE01/lib/dms.jar
/u01/app/oracle/product/J2EE01/opmn/lib/ons.jar
/u01/app/oracle/product/J2EE01/j2ee/home/oc4j.jar
/u01/app/oracle/product/J2EE01/j2ee/home/lib/http_client.jar
/u01/app/oracle/product/J2EE01/config/Queries.jar
...
20
lsof utility is available on Linux and Sun Solaris platforms. Look for similar utility on your platform.
Figure 3 lsof utility output example
COMM PI USER FD TYPE DEVIC SIZE NODE NAME
AND D E
java 8010 oracle 18u Ipv4 -1 79444636 TCP *:3001 (LISTEN)
java 8010 oracle 13u Ipv4 -1 79444643 TCP *:3201 (LISTEN)
java 8010 oracle rtd DIR 104,7 1024 2 /
java 8010 oracle 0r CHR 1,3 9511 /dev/null
java 8010 oracle mem REG 104,7 466557 60577 /lib/ld-2.2.4.so
java 8010 oracle mem REG 104,7 5774329 60490 /lib/libc-2.2.4.so
java 8010 oracle mem REG 104,7 65509 60494 /lib/libdl-2.2.4.so
java 8010 oracle mem REG 104,7 616450 60496 /lib/libm-2.2.4.so
java 8010 oracle mem REG 104,7 451085 60499 /lib/libnsl-2.2.4.so
java 8010 oracle mem REG 104,7 73929 60512 /lib/libnss_dns-2.2.4.so
java 8010 oracle mem REG 104,7 257535 60515 /lib/libnss_files-2.2.4.so
java 8010 oracle mem REG 104,7 351452 60523 /lib/libnss_nisplus-2.2.4.so
java 8010 oracle mem REG 104,7 561991 60526 /lib/libpthread-0.9.so
java 8010 oracle mem REG 104,7 263234 60528 /lib/libresolv-2.2.4.so
java 8010 oracle mem REG 104,8 120257 2998503 /u01/app/oracle/product/J2EE01/config/Queries.jar
java 8010 oracle cwd DIR 104,8 4096 3571783 /u01/app/oracle/product/J2EE01/j2ee/home
java 8010 oracle mem REG 104,8 1431995 3571807 /u01/app/oracle/product/J2EE01/j2ee/home/iiop.jar
java 8010 oracle mem REG 104,8 37265 3571808 /u01/app/oracle/product/J2EE01/j2ee/home/iiop_gen_bin.jar
java 8010 oracle mem REG 104,8 126451 3571810 /u01/app/oracle/product/J2EE01/j2ee/home/jaxb-rt-1.0-ea.jar
java 8010 oracle mem REG 104,8 938119 3571789 /u01/app/oracle/product/J2EE01/j2ee/home/jazn.jar
java 8010 oracle mem REG 104,8 45386 2769040 /u01/app/oracle/product/J2EE01/j2ee/home/lib/activation.jar
java 8010 oracle mem REG 104,8 726013 2769036 /u01/app/oracle/product/J2EE01/j2ee/home/lib/aqapi.jar
java 8010 oracle mem REG 104,8 515920 2769054 /u01/app/oracle/product/J2EE01/j2ee/home/lib/bcel.jar
java 8010 oracle mem REG 104,8 17978 2769032 /u01/app/oracle/product/J2EE01/j2ee/home/lib/connector.jar
java 8010 oracle mem REG 104,8 58142 2769033 /u01/app/oracle/product/J2EE01/j2ee/home/lib/cos.jar
java 8010 oracle mem REG 104,8 206289 2769041 /u01/app/oracle/product/J2EE01/j2ee/home/lib/crimson.jar
java 8010 oracle mem REG 104,8 12822 2769029 /u01/app/oracle/product/J2EE01/j2ee/home/lib/ejb.jar
java 8010 oracle mem REG 104,8 228847 2769037 /u01/app/oracle/product/J2EE01/j2ee/home/lib/http_client.jar
java 8010 oracle mem REG 104,8 104195 2769030 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jaas.jar
java 8010 oracle mem REG 104,8 1070096 2769038 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jasper.zip
java 8010 oracle mem REG 104,8 28404 2769031 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jaxp.jar
java 8010 oracle mem REG 104,8 6727 2769034 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jdbc.jar
java 8010 oracle mem REG 104,8 424080 2769039 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jem.jar
java 8010 oracle mem REG 104,8 27724 2769044 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jms.jar
java 8010 oracle mem REG 104,8 98496 2769047 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jndi.jar
java 8010 oracle mem REG 104,8 8674 2769048 /u01/app/oracle/product/J2EE01/j2ee/home/lib/jta.jar
java 8010 oracle mem REG 104,8 280984 2769049 /u01/app/oracle/product/J2EE01/j2ee/home/lib/mail.jar
java 8010 oracle mem REG 104,8 605174 2769001 /u01/app/oracle/product/J2EE01/j2ee/home/lib/ojsp.jar
java 8010 oracle mem REG 104,8 53389 2769051 /u01/app/oracle/product/J2EE01/j2ee/home/lib/servlet.jar
java 8010 oracle mem REG 104,8 4783072 3571813 /u01/app/oracle/product/J2EE01/j2ee/home/oc4j.jar
java 8010 oracle 7w REG 104,8 1114323 5030160 /u01/app/oracle/product/J2EE01/j2ee/PANDA/application-deployments/BC4J/PANDA_default_island_1/application.log
java 8010 oracle 9w REG 104,8 6690838 6340902 /u01/app/oracle/product/J2EE01/j2ee/PANDA/application-deployments/PANDA/PANDA_default_island_1/application.log
java 8010 oracle 35r REG 104,8 1438 2184291 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/ApnNskAdDiagnosis_AS.xml
java 8010 oracle 49r REG 104,8 1422 2184306 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/ApnNskForward_AS.xml
java 8010 oracle 32r REG 104,8 1433 2184453 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NmpNskAdClassify_AS.xml
java 8010 oracle 54r REG 104,8 1339 2184505 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NskAdDiagnosis_AS.xml
java 8010 oracle 28r REG 104,8 1359 2184528 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NskAdPatientGroups_AS.xm
java 8010 oracle 29r REG 104,8 1386 2184557 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NskCadAdsFkAssoc.xml
java 8010 oracle 38r REG 104,8 1418 2184596 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/entity/NskHpsAdsFkAssoc.xml
java 8010 oracle 37r REG 104,8 3761 2184998 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/nsk001/P2601F_NmpAdAttributes_V
java 8010 oracle 43r REG 104,8 1496 2185078 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/nsk001/SP0301F_NskAccountingDoc
java 8010 oracle 30r REG 104,8 1464 2185147 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/nsk001/SP0301F_NskXmlDocErrors_
java 8010 oracle 36r REG 104,8 6031 2185426 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/nsk/nsk002/P2631F_NskLcSubcriteria_
java 8010 oracle 34r REG 104,8 1387 2186115 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/skd/d001/D2262F_Persons_Det5_VL.xml
java 8010 oracle 52r REG 104,8 7517 2186174 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/skd/entity/SkdPersonDiagnoses_EO.xm
java 8010 oracle 45r REG 104,8 6657 2186205 /u01/app/oracle/product/J2EE01/j2ee/PANDA/applications/PANDA/PANDA/WEB-INF/classes/lv/vovaa/panda/skd/entity/SkdVcurrentPvaDoctors.xm
java 8010 oracle 10w REG 104,8 55981920 5095446 /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/default-web-access.log
java 8010 oracle 6w REG 104,8 133330 5095445 /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/global-application.log
java 8010 oracle 12w REG 104,8 42600 5095448 /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/jms.log
java 8010 oracle 15w REG 104,8 85101 5095449 /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/rmi.log
java 8010 oracle 5w REG 104,8 148705 5095444 /u01/app/oracle/product/J2EE01/j2ee/PANDA/log/PANDA_default_island_1/server.log
java 8010 oracle 11w REG 104,8 10 6865677 /u01/app/oracle/product/J2EE01/j2ee/PANDA/persistence/PANDA_default_island_1/transaction.state
java 8010 oracle mem REG 104,8 400813 2293843 /u01/app/oracle/product/J2EE01/javacache/lib/cache.jar
java 8010 oracle mem REG 104,8 1227663 5783844 /u01/app/oracle/product/J2EE01/jdbc/lib/classes12dms.jar
java 8010 oracle mem REG 104,8 1846926 5783845 /u01/app/oracle/product/J2EE01/jdbc/lib/nls_charset12.jar
java 8010 oracle txt REG 104,8 27090 3998180 /u01/app/oracle/product/J2EE01/jdk/bin/i386/native_threads/java
java 8010 oracle mem REG 104,8 3228357 6357412 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i18n.jar
java 8010 oracle mem REG 104,8 3968300 2441654 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/client/libjvm.so
java 8010 oracle mem REG 104,8 4323150 6357389 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libawt.so
java 8010 oracle mem REG 104,8 221718 6357381 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libjava.so
java 8010 oracle mem REG 104,8 214414 6357388 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libmlib_image.so
java 8010 oracle mem REG 104,8 45805 6357385 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libnet.so
java 8010 oracle mem REG 104,8 81155 6357380 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libverify.so
java 8010 oracle mem REG 104,8 100867 6357382 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/libzip.so
java 8010 oracle mem REG 104,8 60924 2441652 /u01/app/oracle/product/J2EE01/jdk/jre/lib/i386/native_threads/libhpi.so
java 8010 oracle mem REG 104,8 13887946 6357411 /u01/app/oracle/product/J2EE01/jdk/jre/lib/rt.jar
java 8010 oracle mem REG 104,8 86267 6357399 /u01/app/oracle/product/J2EE01/jdk/jre/lib/sunrsasign.jar
java 8010 oracle mem REG 104,8 4723739 3522887 /u01/app/oracle/product/J2EE01/jdk/lib/tools.jar
java 8010 oracle mem REG 104,8 47977 3719344 /u01/app/oracle/product/J2EE01/jlib/infratool.jar
java 8010 oracle mem REG 104,8 19055 3719459 /u01/app/oracle/product/J2EE01/jlib/javax-ssl-1_1.jar
java 8010 oracle mem REG 104,8 97158 3719348 /u01/app/oracle/product/J2EE01/jlib/jndi.jar
java 8010 oracle mem REG 104,8 50921 3719462 /u01/app/oracle/product/J2EE01/jlib/jssl-1_1.jar
java 8010 oracle mem REG 104,8 162059 3719351 /u01/app/oracle/product/J2EE01/jlib/ldapjclnt9.jar
java 8010 oracle mem REG 104,8 246743 3719457 /u01/app/oracle/product/J2EE01/jlib/netcfg.jar
java 8010 oracle mem REG 104,8 23443 3719364 /u01/app/oracle/product/J2EE01/jlib/ojmisc.jar
java 8010 oracle mem REG 104,8 50413 3719352 /u01/app/oracle/product/J2EE01/jlib/repository.jar
java 8010 oracle mem REG 104,8 260225 5783839 /u01/app/oracle/product/J2EE01/lib/dms.jar
java 8010 oracle mem REG 104,8 463863 5783772 /u01/app/oracle/product/J2EE01/lib/jsse.jar
java 8010 oracle mem REG 104,8 658404 5783805 /u01/app/oracle/product/J2EE01/lib/xmlparserv2.jar
java 8010 oracle mem REG 104,8 53464 6832352 /u01/app/oracle/product/J2EE01/opmn/lib/ons.jar
java 8010 oracle 1w REG 104,8 23185778 6144409 /u01/app/oracle/product/J2EE01/opmn/logs/PANDA.default_island.1
java 8010 oracle mem REG 104,8 297060 819453 /u01/app/oracle/product/J2EE01/soap/lib/soap.jar
java 8010 oracle mem REG 104,8 419100 6471908 /u01/app/oracle/product/J2EE01/sqlj/lib/runtime12ee.jar
java 8010 oracle mem REG 104,8 3717072 6471909 /u01/app/oracle/product/J2EE01/sqlj/lib/translator.jar
java 8010 oracle mem REG 104,8 142704 3276954 /u01/app/oracle/product/J2EE01/webservices/lib/wsdl.jar
java 8010 oracle mem REG 104,8 123656 3276958 /u01/app/oracle/product/J2EE01/webservices/lib/wsserver.jar
java 8010 oracle 22w REG 104,8 7154 2999584 /u01/app/PANDA/logs/panda-2005-01-30-08-28-14-958.log
java 8010 oracle mem REG 104,5 1144432 98829 /usr/lib/libstdc++-2-libc6.1-1-2.9.0.so
java 8010 oracle mem REG 104,5 160 427556 /usr/lib/locale/en_US/LC_ADDRESS
java 8010 oracle mem REG 104,5 21499 213158 /usr/lib/locale/en_US/LC_COLLATE
java 8010 oracle mem REG 104,5 173408 98491 /usr/lib/locale/en_US/LC_CTYPE
java 8010 oracle mem REG 104,5 370 131090 /usr/lib/locale/en_US/LC_IDENTIFICATION
java 8010 oracle mem REG 104,5 28 263082 /usr/lib/locale/en_US/LC_MEASUREMENT
java 8010 oracle mem REG 104,5 57 196715 /usr/lib/locale/en_US/LC_MESSAGES/SYS_LC_MESSAGES
java 8010 oracle mem REG 104,5 291 262181 /usr/lib/locale/en_US/LC_MONETARY
java 8010 oracle mem REG 104,5 82 427555 /usr/lib/locale/en_US/LC_NAME
java 8010 oracle mem REG 104,5 59 98484 /usr/lib/locale/en_US/LC_NUMERIC
java 8010 oracle mem REG 104,5 39 393342 /usr/lib/locale/en_US/LC_PAPER
java 8010 oracle mem REG 104,5 64 426055 /usr/lib/locale/en_US/LC_TELEPHONE
java 8010 oracle mem REG 104,5 2456 132545 /usr/lib/locale/en_US/LC_TIME
java 8010 oracle mem REG 104,5 82080 361727 /usr/X11R6/lib/libICE.so.6.3
java 8010 oracle mem REG 104,5 31544 361731 /usr/X11R6/lib/libSM.so.6.0
java 8010 oracle mem REG 104,5 1009336 361733 /usr/X11R6/lib/libX11.so.6.2
java 8010 oracle mem REG 104,5 51632 361741 /usr/X11R6/lib/libXext.so.6.4
java 8010 oracle mem REG 104,5 26788 361753 /usr/X11R6/lib/libXp.so.6.2
java 8010 oracle mem REG 104,5 306428 361759 /usr/X11R6/lib/libXt.so.6.0
java 8010 oracle mem REG 104,5 17712 361761 /usr/X11R6/lib/libXtst.so.6.1
java 8010 oracle 53u IPv4 - 988260755 TCP ap1.voava.lv:28475->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 50u IPv4 -1 9403290 TCP ap1.voava.lv:30001->prod2.vovaa.lv:1526 (ESTABLISHED)
java 8010 oracle 16u IPv4 -1 79444641 TCP ap1.voava.lv:3101 (LISTEN)
java 8010 oracle 14u sock 0,0 3215522651 can't identify protocol
java 8010 oracle 17u IPv4 -1 79444637 TCP localhost:3101 (LISTEN)
java 8010 oracle 19u IPv4 -1 79444635 TCP localhost:64214->localhost:7100 (ESTABLISHED)
java 8010 oracle 3r FIFO 0,0 3215522636 pipe
java 8010 oracle 4w FIFO 0,0 3215522636 pipe

Velikanovs - Performance Tuning and Troubleshooting For Oracle OC4J

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Velikanovs - Performance Tuning and Troubleshooting For Oracle OC4J

Загружено:

Авторское право:

Доступные форматы

Performance Tuning and Troubleshooting for Oracle OC4J

Http Session List ... JDBC Connection pool

Signal Dispatcher Finalizer

Figure 1 “OC4J instance”

Java native threads

OC4J Listener – threads

HTTP/AJPConnectionListener – depending on configuration you will have one of

Classification of Problem Areas

1. OC4J container inside. This type of problems can be introduced by infinitive

Possible Approaches & Information Sources

Java “Full thread dump”

How to get Full thread dump?

Getting OS PID of OC4J

Locating OC4J output file

$ ps -wo args --width 4000 -p 16114 | grep -v COMMAND | awk -F"oracle.ons.indexid="

Sending SIGQUIT signal

Check the output file

"ApplicationServerThread-5" prio=1 tid=0x8450488 nid=0x576d waiting on monitor [0x6877f000..0x6877f870]

"Thread-10" prio=1 tid=0x8162568 nid=0x5738 waiting on monitor [0..0xbffe9b08]

"OC4JMonitorThread" daemon prio=1 tid=0x83c00b0 nid=0x576a waiting on monitor [0x698ff000..0x698ff870]

"VM Periodic Task Thread" prio=1 tid=0x808cb98 nid=0x4e0 waiting on monitor

The most interesting parts are the following:

Different types of the threads within OC4J container

Different states of the threads

Some threads examples.

Useful Java runtime parameter

Java DEADLOCK problem

OC4J output file

By tracing the java thread we got the following information:

Apache side information

Apache log files

Documentation, External searches and Metalink

DMS Sensors DMS Sensors

Response Time Breakdown Example

DMS Sensors Classification

Collecting and Reflecting DMS Statistics

Specify the Begin and End Snapshot Ids

Snap Id Snap Time

JSP ordered by Response time

JSP Name Response Execution Average

Вам также может понравиться