IBM WebSphere Application Server v7.0 Performance Optimization ...

[February, 2011]
IBM WebSphere

Application Server
v7.0

Performance

Optimization Scripts

Document version [1.1]
David W. Hare
Introduction
Over the past year, we have made several changes to the performance tuning

scripts that were discussed in the previous iteration of this document and the

corresponding download package. Since that time, the scripts have been integrated

into the IBM WebSphere Application Server service stream and have been

refactored to use the application server’s property file configuration features

(starting with V7.0.0.9). Furthermore, in V7.0.0.11, support was added to the Profile

Management Tools to apply one of the standard performance tuning templates to a

standalone application server profile as part of the profile creation process. In the

latest service refresh of WebSphere Application Server (V7.0.0.15), the “production”

tuning template has been renamed to “peak” and the scripts have been updated to

capture the latest best practices and tuning recommendations.
This document discusses the latest version of the performance tuning scripts that

are delivered in V7.0.0.15. The corresponding download package not only contains

the latest version(s) of the property file configuration based performance tuning

scripts, but also contains the original jython-based scripts. The older jython-based

scripts remain as code samples for those that wish to construct their own jython

based tuning scripts from scratch. Similarly, the new property file configuration

based scripts can also be used as templates for your own custom tuning templates.
Property File Configuration Scripts
As previously mentioned, the new, property file configuration based scripts that

were introduced in V7.0.0.9 consist of a single jython-based script and two standard

property file based tuning templates. These standard tuning templates are designed

to target individual server instances or server instances within a cluster and apply

some of the most common WebSphere Application Server tuning parameters to suit

one of the following two environments:
•
Peak – Applies tuning parameters that are well suited for performance test

and proof of concept (POC) environments where peak runtime performance is

desired. This template was previously referred to as “Production”, but has

been renamed to emphasize the need for further testing before using this

script, or customized versions of this script to tune your production

environment.
•
Development – Tunes the server for a development environment where

frequent application updates are performed and system resources typically

constrained.
A third tuning script (Default) is also provided that will return the server

configuration to the standard out-of-the-box defaults.
The tuning parameters and values applied by these templates may not result in

optimal performance for your specific application.
These templates should be

viewed as a recommended starting point for improving application server

performance. We highly recommend conducting your own performance

evaluation and tuning exercise in order to fine tune the server for your

application.

Furthermore, some parameters tuned by these scripts (most notably

the ORB Pass-By-Reference) setting may impact the functionality of your

application. So again, we cannot overstress the importance of fully testing your

application with these settings applied.
Running the Scripts Manually
The tuning script files are located in the following directory:
<WAS_HOME>/scriptLibraries/perfTuning/V70
In this directory you will find the following jython tuning script and properties files

corresponding to the three templates discussed in the introduction.
•
applyPerfTuningTemplate.py
•
peak.properties (formerly production.properties prior to V7.0.0.15)
•
development.properties
•
default.properties
To apply a tuning template to your server profile, execute the following command:
<WAS_HOME>/profiles/<PROFILE>/bin/wsadmin -f ../../../scriptLibraries/

perfTuning/V70/applyPerfTuningTemplate.py
-node <node name> -server <server name>

-templateFile <template file name>
The “-node” and “-server” options are needed to target a specific server; however,

the “-cluster” option can be used instead to target all of the servers in a given

cluster by using the following script arguments:
-cluster <cluster name>
The “-templateFile” parameter is used to specify the desired tuning template.
The tuning templates can be applied to a server profile at any time. However, to

benefit from the Data Source tuning described in the next section, the scripts should

be executed after your application and its required resources have been installed.
Application server tuning script
Running the Scripts During Profile Creation
In WebSphere Application Server v7.0.0.11 and beyond, support was added to the

Profile Management Tool (PMT) and the manageprofiles command-line utilities to

apply tuning to a basic, standalone application server environment.
Profile Management Tool
Below is a screen shot of the Profile Management Tool GUI demonstrating the ability

to select and apply one of the pre-defined tuning templates. In order to reach this

screen, you must select the “Advanced profile creation” option, otherwise

“Standard” is selected as the default and no tuning is applied.
manageprofiles command line
Performance tuning settings can also be applied to a standalone application server

environment when created using the manageprofiles command line tool. The

following argument can be added to the manageprofiles command to apply one of

the pre-defined templates.
-applyPerfTuningSetting

standard | peak | development
Please note that if the -
isDeveloperServer
argument is also used on the same

command line with the -
applyPerfTuningSetting
option, the tuning settings may

override the
–isDeveloperServer
settings.
For more information, please consult the following references.
Manageprofiles command
Managing profiles using the graphical user interface
Tuning Performed By Scripts
The following table details the specific tuning performed by each of the two

templates and compares this to the original out-of-the-box defaults. If one of the

listed parameters is omitted or configured back to the server defaults for a

particular tuning template, the cell will be left blank in the table. There are also a

few subtle tuning differences on the Solaris, HP-UX, and zOS platforms that are

called out individually below the table. Further information regarding each tuning

parameter and the impact each has on the server performance is discussed in the

next section.
Parameter
Server

Default
Peak (formerly known

as Production)
Development
JVM Heap Size

(MB)
50 min / 256

max
512 min / 512 max
256 min / 512 max
Verbose GC***
disabled
enabled
enabled
JVM Diagnostic

Trace
(Generic JVM

Arguments)
-Xtrace:none
-Xtrace:none
Optimized JAXB

Processing

(Generic JVM

Arguments)
-
Dcom.ibm.xml.xlxp.jaxb.opti.l
evel=3
-
Dcom.ibm.xml.xlxp.jaxb.opti.l
evel=3
TCP Channel

maxOpenConnect
ions**
20000
500
500
TCP Channel

listenBacklog**
511
128
128
HTTP (9080) and

HTTPS (9443)

Channel

maxKeepAliveRe
quests
100
10000
10000
Development

Mode
disabled
enabled
Server

Component

Provisioning
disabled
enabled
disabled
PMI Statistic

Set***
basic
none
none
Authentication

Cache Timeout *
10 minutes
60 minutes
60 minutes
Data Source

Connection Pool

Size *
1 min / 10

max
10 min / 50 max
Data Source

Prepared

Statement Cache

Size*
10
50
ORB Pass-by-
Reference
disabled
enabled
enabled
Thread Pools
(Web Container,

ORB, Default)
50 min / 50

max,
10 min / 50

max,
20 min / 20

max
5 min / 10 max (for all pools

listed)
Web Server

Pluggin
ServerIOTimeout*
*
60
900
900
*Indicates items that must exist in the configuration to be tuned. Any new items will be created using the standard

server defaults
** Indicates items which were added to the scripts in V7.0.0.15
*** Indicates items which were modified from previous versions of the tuning scripts. For more details, please

consult the Tuning Parameter Descriptions section.
Solaris
On the Solaris platform, the following Generic JVM arguments are used for Peak and

Development:
-XX:-UseAdaptiveSizePolicy -XX:+UseParallelGC -XX:+AggressiveOpts -XX:
+UnlockDiagnosticVMOptions -server -Dcom.ibm.xml.xlxp.jaxb.opti.level=3
HP-UX
On the HP-UX platform, the following Generic JVM arguments are used for Peak and

Development:
-XX:+AggressiveOpts -XX:+ForceMmapReserved -XX:SurvivorRatio=16

-Xoptgc -XX:+UseParallelGC

-Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.DevPollSelectorProvider

-XX:-ExtraPollBeforeRead -XX:+UseSpinning

-Dcom.ibm.xml.xlxp.jaxb.opti.level=3
zOS
On the zOS platform, the default JVM heap sizes are different than those on the

other distributed platforms:
Default minimum heap size:
256 MB
Default maximum heap size:
512 MB
Tuning Parameter Descriptions
JVM Heap Size
– The Java
TM
Virtual Machine (JVM) heap size is typically one of the

first parameters tuned in any WebSphere Application Server environment. The

overall goal of tuning the heap size is to not only reduce the frequency of garbage

collections, but also minimize the duration of each garbage collection cycle. This

maximizes the number of cycles granted to the server to perform application work.

The default size of 50 MB minimum and 256 MB maximum is generally too small for

most applications. Increasing the heap size to 512 MB minimum and maximum

generally reduces the frequency of garbage collections and eliminates the overhead

incurred by the JVM for dynamically managing the heap size. For the Development

profile where optimal performance is not a key factor, the minimum is left at 256 MB

to reduce the overall size of the JVM process.
Java virtual machine settings
Tuning the IBM virtual machine for Java
Tuning the HotSpot Java virtual machines (Solaris & HP-UX)
Verbose GC
– The verbose garbage collection output generated by the JVM is an

essential component in fine tuning the JVM heap size and garbage collection policy.

When enabled, this information is written to the application server’s

native_stderr.log. Since there is very little performance overhead associated with

verbose garbage collection and the potential benefit(s) from having this information

can be a crucial piece of diagnostic information, this setting is enabled in both the

Peak and Development templates. The only downside is the risk of filling your hard

disk if the log file size is not monitored.
Note: The IBM J9 JDK does provide the ability to roll the verbose GC log files based

on the number of GC cycles using the following generic command line option. For

further details, please consult the IBM JDK Diagnostic Guide. In previous versions of

these scripts, verbose GC was only enabled on the development template.
-Xverbosegclog(:<file>(,<X>,<Y>))
This policy is rolling through <X> log files named with <file> pattern,

containing <Y> GC cycles. The file name pattern can be something like

verbosegc.%Y%m%d.%H%M%S.%pid.txt
Java virtual machine settings
IBM Java Diagnostics Guide 6 – Garbage Collector command-line options
JVM Diagnostic Trace
– The low-level JVM diagnostic trace facilities are rarely

needed and can be disabled to eliminate associated overhead. Other diagnostic

capabilities like the ability to generate javacores and heap dumps are not disrupted.

Optimized JAXB Processing
- This property controls whether optimization

methods are enabled for Java Architecture for XML Binding (JAXB) unmarshalling

(deserialization) and marshalling (serialization). A value of 0 signifies that

optimization methods are not enabled. A value of 1 enables unmarshalling

optimization methods only, and is the default setting. A value of 2 enables

marshalling optimization methods only. And a value of 3 enables both unmarshalling

and marshalling optimization methods. This value increases throughput for Web

services and applications that use JAXB directly.
Java virtual machine custom properties
Maximum Keepalives Per Request
– This parameter adjusts the number of

requests a keepalive connection can service before the server forces the connection

to be closed. This capability is used to prevent denial of service attacks; however,

the default value of 100 is relatively small. Increasing this parameter has a

significant impact on SSL connections where the initial SSL handshake is a costly

operation.
HTTP transport channel settings
Development Mode
– This feature reduces the time needed to start the

application server by adding the –Xquickstart and -Xverify:none parameters to the

server JVM command line. The quickstart option directs the JVM to perform class

optimization at a lower level, providing an improvement in JVM startup time. The

verify:none parameter directs the JVM to skip class verification which can also

provide benefits to JVM startup time.
Application server settings
Server Component Provisioning
– This feature can potentially reduce the startup

time and memory footprint of the application server by allowing internal server

components to start as needed. For instance, if an application consists of Java

Servlets and JavaServer Pages (JSP
TM
) only, there is no need for the server to start

the EJB container.
Application server settings
Performance Monitoring Infrastructure (PMI) Statistic Set
– The PMI service

is enabled by default and a common set of statistics (basic) are also enabled out-of-
the-box to provide common performance statistics for the server and installed

applications. This service is generally useful for performance tuning and problem

determination, but does add additional overhead. Consequently, we recommend

leaving this service enabled and changing the statistic set to none. This will

effectively disable all PMI counters, but provides the flexibility to enable them at a

later time without having to restart the server. If you do enable these counters, we

recommend selectively enabling only the counters you need in order to reduce

overhead.
Note: In previous versions of these scripts, the entire PMI service was disabled.
Enabling PMI data collection
Authentication Cache Timeout
– The authentication cache timeout controls how

often authentication information is refreshed within the cache. Obtaining

authentication information from an LDAP server or other native authentication

mechanisms is an expensive operation. Increasing the authentication cache timeout

reduces the frequency of these costly updates.
Authentication cache settings
Data Source Connection Pool Size
– The minimum and maximum connection

pool size should be increased to reduce time spent waiting to obtain a datasource

connection from the connection pool. In this case, the maximum pool size is set to

correspond to the default maximum Web Container and ORB thread pool size of 50.

In clustered environments, this setting may overwhelm the database server and

should be adjusted accordingly.
Connection pool settings
Data Source Prepared Statement Cache Size
– The prepared statement cache

optimizes the processing of prepared and callable statements by caching them in a

least-recently-used (LRU) cache. Most applications will utilize more than 10

prepared statements. Consequently, the scripts for the performance test and

production environments increase this value to 50. The Tivoli Performance Viewer

can be used to monitor LRU discards from the prepared statement cache to

determine if the cache size should be increased.
Data access tuning parameters
WebSphere Application Server data source properties
ORB Pass-By-Reference
– This feature allows the server to use pass-by-reference

semantics instead of pass-by-value semantics for Enterprise JavaBean
TM
(EJB)

invocations where the EJB client and remote target exist within the same JVM. This

optimization essentially treats EJBs implementing a remote interface as local and

avoids the requisite object copy. In some cases, this can improve performance by

over 50%. It is also important to note that this setting can cause unexpected

behavior in your application. Since object references are passed as arguments

instead of deep copies of the objects, any changes to the object will be made on the

object itself (not the copy). Consequently, we strongly advise rigorous testing to

ensure this performance optimization does not cause unexpected results in your

application.
Object Request Broker service settings
Thread Pool Size
– The default size of the various threads pools within the

application server are generally well suited for Production and Test environments.

However, for Development environments, the minimum and maximum sizes for the

Web Container, ORB, Default thread pools can be reduced to reduce the memory

footprint requirements of the server.
Thread pool settings
TCP Channel Max Open Connections
– This parameter controls the maximum

number of allowed connections to the server. The default value of 20000 is

relatively high and could lead to stalled websites under failure conditions because

the server continues to accept incoming connections, increasing the servers work

backlog.
Tuning transport channel services
TCP Channel Listen Backlog
– The TCP channel listen backlog property controls

the maximum number of outstanding connection requests that the operating

system can buffer while waiting to accept the connections. If the listen backlog has

been filled with pending connection requests, any additional connection requests

will be rejected. This property in conjunction with the Max Open Connections

property can be effectively used to manage connections to the server.
TCP Transport Channel custom properties
Web Server Plug-in Server IO Timeout
–Also referred to as the read/write

timeout, this parameter dictates how long the plug-in will wait to send a request to

or receive a response from the application server. For applications that require a

significant amount of backend processing (i.e. database intensive workloads,

applications with complex business logic, etc.), the default value of 60 seconds can

prematurely timeout the request which will then be retried by redirecting it to

another application server. The suggested starting value of 900 seconds should

avoid this scenario for most applications; however, this should be further tuned

based on the characteristics of your application.
Application Server property setting for a Web server plug-in
Additional Parameters to Consider
In the original jython-based version of the tuning scripts, the following two

parameters were tuned based on the target environment. Unfortunately, the

property file configuration feature does not support tuning these parameters since

they are application specific. If you would still like to tune these parameters, you

can do so using the following three methods:
•
Application settings found in the Administration Console
•
WebSphere application deployment descriptors
•
Jython wsadmin code snippets in the previous version of the tuning scripts
Override Application Class Reload Interval
– By default, the application server

scans for application class changes every 3 seconds. This is good for development

environments where application changes are expected on a frequent basis, but may

not be necessary in other environments. For instance, in a test environment less

frequent application changes are expected, the default value can be overridden and

increased to 60 seconds. In a production environment, this feature can be disabled

all together by overriding the default value and setting the reload interval to 0. This

is extremely useful when trying to reduce overall cpu cycles consumed by an idle

application server.
Class loading and update detection settings
JSP Reload Interval
– This feature is similar to the application class reload interval,

but extends to an application’s JSP files.
JavaServer Pages (JSP) runtime reloading settings
®
© Copyright IBM Corporation 2009
All Rights Reserved.
IBM, the IBM (logo), and WebSphere are trademarks or registered trademarks of

International Business Machines Corporation in the United States, other countries, or both.
Solaris, Java and all Java-based trademarks and logos are trademarks of Sun Microsystems,

Inc. in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
References in this publication to IBM products or services do not imply that IBM intends to

make them available in all countries in which IBM operates. The following paragraph does

not apply to the United Kingdom or any other country where such provisions are inconsistent

with local law:
INTERNATIONAL BUSINESS MACHINES CORPORATIONPROVIDES THIS PUBLICATION "AS IS"

WITHOUT WARRANTYOF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUTNOT

LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR

FITNESS FOR APARTICULAR PURPOSE.
Some states do not allow disclaimer of express or implied warranties in certain transactions,

therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are

periodically made to the information herein; these changes will be incorporated in new

editions of the publication. IBM may make improvements and/or changes in the product(s)

and/or the program(s) described in this publication at any time without notice.
Information concerning non-IBM products was obtained from the suppliers of those products,

their published announcements or other publicly available sources. IBM has not tested those

products and cannot confirm the accuracy of performance, compatibility or any other claims

related to non-IBM products. Questions on the capabilities of non-IBM products should be

addressed to the suppliers of those products.
The information in this publication is provided AS IS without warranty. Such information was

obtained from publicly available sources, is current as of January 2009, and is subject to

change. Any performance data included in the paper was obtained in the specific operating

environment and is provided as an illustration. Performance in other operating environments

may vary. More specific information about the capabilities of products described should be

obtained from the suppliers of those products.