[February, 2011]

IBM WebSphere Application Server v7.0 Performance Optimization Scripts

Document version [1.1]

David W. Hare

Introduction
Over the past year, we have made several changes to the performance tuning scripts that were discussed in the previous iteration of this document and the corresponding download package. Since that time, the scripts have been integrated into the IBM WebSphere Application Server service stream and have been refactored to use the application servers property file configuration features (starting with V7.0.0.9). Furthermore, in V7.0.0.11, support was added to the Profile Management Tools to apply one of the standard performance tuning templates to a standalone application server profile as part of the profile creation process. In the latest service refresh of WebSphere Application Server (V7.0.0.15), the production tuning template has been renamed to peak and the scripts have been updated to capture the latest best practices and tuning recommendations. This document discusses the latest version of the performance tuning scripts that are delivered in V7.0.0.15. The corresponding download package not only contains the latest version(s) of the property file configuration based performance tuning scripts, but also contains the original jython-based scripts. The older jython-based scripts remain as code samples for those that wish to construct their own jython based tuning scripts from scratch. Similarly, the new property file configuration based scripts can also be used as templates for your own custom tuning templates.

Property File Configuration Scripts

As previously mentioned, the new, property file configuration based scripts that were introduced in V7.0.0.9 consist of a single jython-based script and two standard property file based tuning templates. These standard tuning templates are designed to target individual server instances or server instances within a cluster and apply some of the most common WebSphere Application Server tuning parameters to suit one of the following two environments: Peak Applies tuning parameters that are well suited for performance test and proof of concept (POC) environments where peak runtime performance is desired. This template was previously referred to as Production, but has been renamed to emphasize the need for further testing before using this script, or customized versions of this script to tune your production environment. Development Tunes the server for a development environment where frequent application updates are performed and system resources typically constrained.

A third tuning script (Default) is also provided that will return the server configuration to the standard out-of-the-box defaults.

The tuning parameters and values applied by these templates may not result in optimal performance for your specific application. These templates should be viewed as a recommended starting point for improving application server performance. We highly recommend conducting your own performance evaluation and tuning exercise in order to fine tune the server for your application. Furthermore, some parameters tuned by these scripts (most notably the ORB Pass-By-Reference) setting may impact the functionality of your application. So again, we cannot overstress the importance of fully testing your application with these settings applied.

Running the Scripts Manually

The tuning script files are located in the following directory:
<WAS_HOME>/scriptLibraries/perfTuning/V70

In this directory you will find the following jython tuning script and properties files corresponding to the three templates discussed in the introduction. applyPerfTuningTemplate.py peak.properties (formerly production.properties prior to V7.0.0.15) development.properties default.properties

To apply a tuning template to your server profile, execute the following command:
<WAS_HOME>/profiles/<PROFILE>/bin/wsadmin -f ../../../scriptLibraries/ perfTuning/V70/applyPerfTuningTemplate.py -node <node name> -server <server name> -templateFile <template file name>

The -node and -server options are needed to target a specific server; however, the -cluster option can be used instead to target all of the servers in a given cluster by using the following script arguments:
-cluster <cluster name>

The -templateFile parameter is used to specify the desired tuning template. The tuning templates can be applied to a server profile at any time. However, to benefit from the Data Source tuning described in the next section, the scripts should be executed after your application and its required resources have been installed. Application server tuning script

Running the Scripts During Profile Creation

In WebSphere Application Server v7.0.0.11 and beyond, support was added to the Profile Management Tool (PMT) and the manageprofiles command-line utilities to apply tuning to a basic, standalone application server environment. Profile Management Tool Below is a screen shot of the Profile Management Tool GUI demonstrating the ability to select and apply one of the pre-defined tuning templates. In order to reach this screen, you must select the Advanced profile creation option, otherwise Standard is selected as the default and no tuning is applied.

manageprofiles command line

Performance tuning settings can also be applied to a standalone application server environment when created using the manageprofiles command line tool. The following argument can be added to the manageprofiles command to apply one of the pre-defined templates.
-applyPerfTuningSetting standard | peak | development

Please note that if the -isDeveloperServer argument is also used on the same command line with the -applyPerfTuningSetting option, the tuning settings may override the isDeveloperServer settings. For more information, please consult the following references. Manageprofiles command Managing profiles using the graphical user interface

Tuning Performed By Scripts

The following table details the specific tuning performed by each of the two templates and compares this to the original out-of-the-box defaults. If one of the listed parameters is omitted or configured back to the server defaults for a particular tuning template, the cell will be left blank in the table. There are also a few subtle tuning differences on the Solaris, HP-UX, and zOS platforms that are called out individually below the table. Further information regarding each tuning parameter and the impact each has on the server performance is discussed in the next section. Parameter JVM Heap Size (MB) Verbose GC*** JVM Diagnostic Trace (Generic JVM Arguments) Optimized JAXB Processing (Generic JVM Arguments) TCP Channel maxOpenConnect ions**
20000

Server Default
50 min / 256 max disabled

Peak (formerly known as Production)

512 min / 512 max

Development
256 min / 512 max

enabled -Xtrace:none

enabled -Xtrace:none

Dcom.ibm.xml.xlxp.jaxb.opti.l evel=3

Dcom.ibm.xml.xlxp.jaxb.opti.l evel=3

500

500

TCP Channel listenBacklog** HTTP (9080) and HTTPS (9443) Channel maxKeepAliveRe quests Development Mode Server Component Provisioning PMI Statistic Set*** Authentication Cache Timeout * Data Source Connection Pool Size * Data Source Prepared Statement Cache Size* ORB Pass-byReference Thread Pools (Web Container, ORB, Default)

511

128

128

100

10000

10000

disabled

enabled

disabled

enabled

disabled

basic

none

none

10 minutes

60 minutes

60 minutes

1 min / 10 max

10 min / 50 max

10

50

disabled

enabled

enabled

50 min / 50 max, 10 min / 50 max, 20 min / 20 max 60 900

5 min / 10 max (for all pools listed)

Web Server Pluggin ServerIOTimeout* *

900

*Indicates items that must exist in the configuration to be tuned. Any new items will be created using the standard server defaults ** Indicates items which were added to the scripts in V7.0.0.15 *** Indicates items which were modified from previous versions of the tuning scripts. For more details, please consult the Tuning Parameter Descriptions section.

Solaris On the Solaris platform, the following Generic JVM arguments are used for Peak and Development: -XX:-UseAdaptiveSizePolicy -XX:+UseParallelGC -XX:+AggressiveOpts -XX: +UnlockDiagnosticVMOptions -server -Dcom.ibm.xml.xlxp.jaxb.opti.level=3 HP-UX On the HP-UX platform, the following Generic JVM arguments are used for Peak and Development: -XX:+AggressiveOpts -XX:+ForceMmapReserved -XX:SurvivorRatio=16 -Xoptgc -XX:+UseParallelGC -Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.DevPollSelectorProvider -XX:-ExtraPollBeforeRead -XX:+UseSpinning -Dcom.ibm.xml.xlxp.jaxb.opti.level=3 zOS On the zOS platform, the default JVM heap sizes are different than those on the other distributed platforms: Default minimum heap size: Default maximum heap size: 256 MB 512 MB

Tuning Parameter Descriptions

JVM Heap Size The JavaTM Virtual Machine (JVM) heap size is typically one of the first parameters tuned in any WebSphere Application Server environment. The overall goal of tuning the heap size is to not only reduce the frequency of garbage collections, but also minimize the duration of each garbage collection cycle. This maximizes the number of cycles granted to the server to perform application work. The default size of 50 MB minimum and 256 MB maximum is generally too small for most applications. Increasing the heap size to 512 MB minimum and maximum generally reduces the frequency of garbage collections and eliminates the overhead incurred by the JVM for dynamically managing the heap size. For the Development profile where optimal performance is not a key factor, the minimum is left at 256 MB to reduce the overall size of the JVM process. Java virtual machine settings Tuning the IBM virtual machine for Java Tuning the HotSpot Java virtual machines (Solaris & HP-UX)

Verbose GC The verbose garbage collection output generated by the JVM is an essential component in fine tuning the JVM heap size and garbage collection policy. When enabled, this information is written to the application servers native_stderr.log. Since there is very little performance overhead associated with verbose garbage collection and the potential benefit(s) from having this information can be a crucial piece of diagnostic information, this setting is enabled in both the Peak and Development templates. The only downside is the risk of filling your hard disk if the log file size is not monitored. Note: The IBM J9 JDK does provide the ability to roll the verbose GC log files based on the number of GC cycles using the following generic command line option. For further details, please consult the IBM JDK Diagnostic Guide. In previous versions of these scripts, verbose GC was only enabled on the development template. -Xverbosegclog(:<file>(,<X>,<Y>)) This policy is rolling through <X> log files named with <file> pattern, containing <Y> GC cycles. The file name pattern can be something like verbosegc.%Y%m%d.%H%M%S.%pid.txt Java virtual machine settings IBM Java Diagnostics Guide 6 Garbage Collector command-line options JVM Diagnostic Trace The low-level JVM diagnostic trace facilities are rarely needed and can be disabled to eliminate associated overhead. Other diagnostic capabilities like the ability to generate javacores and heap dumps are not disrupted. Optimized JAXB Processing - This property controls whether optimization methods are enabled for Java Architecture for XML Binding (JAXB) unmarshalling (deserialization) and marshalling (serialization). A value of 0 signifies that optimization methods are not enabled. A value of 1 enables unmarshalling optimization methods only, and is the default setting. A value of 2 enables marshalling optimization methods only. And a value of 3 enables both unmarshalling and marshalling optimization methods. This value increases throughput for Web services and applications that use JAXB directly. Java virtual machine custom properties Maximum Keepalives Per Request This parameter adjusts the number of requests a keepalive connection can service before the server forces the connection to be closed. This capability is used to prevent denial of service attacks; however, the default value of 100 is relatively small. Increasing this parameter has a significant impact on SSL connections where the initial SSL handshake is a costly operation. HTTP transport channel settings

Development Mode This feature reduces the time needed to start the application server by adding the Xquickstart and -Xverify:none parameters to the server JVM command line. The quickstart option directs the JVM to perform class optimization at a lower level, providing an improvement in JVM startup time. The verify:none parameter directs the JVM to skip class verification which can also provide benefits to JVM startup time. Application server settings Server Component Provisioning This feature can potentially reduce the startup time and memory footprint of the application server by allowing internal server components to start as needed. For instance, if an application consists of Java Servlets and JavaServer Pages (JSPTM) only, there is no need for the server to start the EJB container. Application server settings Performance Monitoring Infrastructure (PMI) Statistic Set The PMI service is enabled by default and a common set of statistics (basic) are also enabled out-ofthe-box to provide common performance statistics for the server and installed applications. This service is generally useful for performance tuning and problem determination, but does add additional overhead. Consequently, we recommend leaving this service enabled and changing the statistic set to none. This will effectively disable all PMI counters, but provides the flexibility to enable them at a later time without having to restart the server. If you do enable these counters, we recommend selectively enabling only the counters you need in order to reduce overhead. Note: In previous versions of these scripts, the entire PMI service was disabled. Enabling PMI data collection Authentication Cache Timeout The authentication cache timeout controls how often authentication information is refreshed within the cache. Obtaining authentication information from an LDAP server or other native authentication mechanisms is an expensive operation. Increasing the authentication cache timeout reduces the frequency of these costly updates. Authentication cache settings Data Source Connection Pool Size The minimum and maximum connection pool size should be increased to reduce time spent waiting to obtain a datasource connection from the connection pool. In this case, the maximum pool size is set to correspond to the default maximum Web Container and ORB thread pool size of 50. In clustered environments, this setting may overwhelm the database server and should be adjusted accordingly.

Connection pool settings Data Source Prepared Statement Cache Size The prepared statement cache optimizes the processing of prepared and callable statements by caching them in a least-recently-used (LRU) cache. Most applications will utilize more than 10 prepared statements. Consequently, the scripts for the performance test and production environments increase this value to 50. The Tivoli Performance Viewer can be used to monitor LRU discards from the prepared statement cache to determine if the cache size should be increased. Data access tuning parameters WebSphere Application Server data source properties ORB Pass-By-Reference This feature allows the server to use pass-by-reference semantics instead of pass-by-value semantics for Enterprise JavaBeanTM (EJB) invocations where the EJB client and remote target exist within the same JVM. This optimization essentially treats EJBs implementing a remote interface as local and avoids the requisite object copy. In some cases, this can improve performance by over 50%. It is also important to note that this setting can cause unexpected behavior in your application. Since object references are passed as arguments instead of deep copies of the objects, any changes to the object will be made on the object itself (not the copy). Consequently, we strongly advise rigorous testing to ensure this performance optimization does not cause unexpected results in your application. Object Request Broker service settings Thread Pool Size The default size of the various threads pools within the application server are generally well suited for Production and Test environments. However, for Development environments, the minimum and maximum sizes for the Web Container, ORB, Default thread pools can be reduced to reduce the memory footprint requirements of the server. Thread pool settings TCP Channel Max Open Connections This parameter controls the maximum number of allowed connections to the server. The default value of 20000 is relatively high and could lead to stalled websites under failure conditions because the server continues to accept incoming connections, increasing the servers work backlog. Tuning transport channel services TCP Channel Listen Backlog The TCP channel listen backlog property controls the maximum number of outstanding connection requests that the operating system can buffer while waiting to accept the connections. If the listen backlog has been filled with pending connection requests, any additional connection requests

will be rejected. This property in conjunction with the Max Open Connections property can be effectively used to manage connections to the server. TCP Transport Channel custom properties Web Server Plug-in Server IO Timeout Also referred to as the read/write timeout, this parameter dictates how long the plug-in will wait to send a request to or receive a response from the application server. For applications that require a significant amount of backend processing (i.e. database intensive workloads, applications with complex business logic, etc.), the default value of 60 seconds can prematurely timeout the request which will then be retried by redirecting it to another application server. The suggested starting value of 900 seconds should avoid this scenario for most applications; however, this should be further tuned based on the characteristics of your application. Application Server property setting for a Web server plug-in

Additional Parameters to Consider

In the original jython-based version of the tuning scripts, the following two parameters were tuned based on the target environment. Unfortunately, the property file configuration feature does not support tuning these parameters since they are application specific. If you would still like to tune these parameters, you can do so using the following three methods: Application settings found in the Administration Console WebSphere application deployment descriptors Jython wsadmin code snippets in the previous version of the tuning scripts

Override Application Class Reload Interval By default, the application server scans for application class changes every 3 seconds. This is good for development environments where application changes are expected on a frequent basis, but may not be necessary in other environments. For instance, in a test environment less frequent application changes are expected, the default value can be overridden and increased to 60 seconds. In a production environment, this feature can be disabled all together by overriding the default value and setting the reload interval to 0. This is extremely useful when trying to reduce overall cpu cycles consumed by an idle application server. Class loading and update detection settings JSP Reload Interval This feature is similar to the application class reload interval, but extends to an applications JSP files. JavaServer Pages (JSP) runtime reloading settings


Copyright IBM Corporation 2009 All Rights Reserved. IBM, the IBM (logo), and WebSphere are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. Solaris, Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. References in this publication to IBM products or services do not imply that IBM intends to make them available in all countries in which IBM operates. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATIONPROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTYOF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUTNOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR APARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. The information in this publication is provided AS IS without warranty. Such information was obtained from publicly available sources, is current as of January 2009, and is subject to change. Any performance data included in the paper was obtained in the specific operating environment and is provided as an illustration. Performance in other operating environments may vary. More specific information about the capabilities of products described should be obtained from the suppliers of those products.