J2EE Applications Performance Management

quicksandwalleyeInternet and Web Development

Oct 31, 2013 (3 years and 7 months ago)

96 views

Kenneth Merkel

Systems Engineer

J2EE Applications Performance Management


Copyright ©2003 Wily Technology, Inc.

2

Agenda


J2EE Applications Performance Management


Challenges in Managing Web/J2EE
Applications


Methodology & Goals of Performance
Management


Performance Management Approaches /
Techniques


Real Issues


Introscope
TM

Platform Technology Overview


Copyright ©2003 Wily Technology, Inc.

3

Why Is Web Application Performance
Management So Complex?

Customers

Internet

Web Servers

Database

CICS/IMS on
Mainframe

MQ /

Tuxedo

Back
-
End

Servers

Web Application Servers

Mission
-
Critical
Java Applications

Copyright ©2003 Wily Technology, Inc.

4

Why Is Java Application Performance
Management So Complex?

JVM

Application Server

Java/J2EE Application

Web
Application
Servers

Customers

Internet

Web Servers

Database

CICS/IMS on
Mainframe

MQ /

Tuxedo

Back
-
End

Servers

Copyright ©2003 Wily Technology, Inc.

5

JVM


Application Server

J2EE Application

Account

Info

Servlet

Account

EJB

MQSeries
Connector

CICS

Connector

JDBC

Driver

Typical J2EE Architecture

File and Network I/O

Memory

JDBC Pools

EJB Pools

Thread Pool

Database

CICS Server

Back
-
End

Systems

MQ

LDAP Server

Login

JSP

JNDI

Login
Request

Response

Account

Transaction

Servlet

Transaction
Request

Response

Copyright ©2003 Wily Technology, Inc.

6

Common Performance Problems


Bottlenecks on resources


3
rd

party interaction points


Configuration problems


Memory leaks


Deadlocks and threading problems


Sloppy code

Copyright ©2003 Wily Technology, Inc.

7

What is Performance Measurement and
Analysis?



Different definitions for
different parts of your project…



Development


Profiling


Logging


Unit Testing



Staging/QA


Functional Testing


Performance/Load Testing


Defect/Performance Tracking



Production


Audit Trails


Availability


Service Level Agreements (SLAs)


Sitemining

Copyright ©2003 Wily Technology, Inc.

8

Web Application Performance
Management Methodology

Analyze

1.

Business Analysis

2.

System Architecture Analysis

3.

Application Analysis

4.

Establish Baseline

Establish performance

criteria

Apply criteria

Test in QA

5.
Functional & Stress Test

Manage in Production

6.
Production Deployment

7.
Real
-
time Application Monitoring

8.
Event Management

9.
Problem Resolution

10.
Application Server Tuning

11.
Continuous Performance
Management


Met criteria

Maintain criteria

Failed criteria

New criteria requirements

Copyright ©2003 Wily Technology, Inc.

9

What do I analyze?


Think in terms of
user transactions

or
use
cases

for your application



Typical examples of a single transaction
might be…


Logging in


Looking Up Account Balance


Transferring Funds



Look at resource usage by use
-
case
(especially high concurrency points)



Stay focused on what your customers
do
with your application… why tune something
unrelated?


Copyright ©2003 Wily Technology, Inc.

10

What am I looking for?


Resource usage/abuse


Using Exceptions to handle non
-
exceptional
cases.


Synchronization/Contention Costs


Collection Load Factors


Proper pattern usage



Intra
-
component …

Copyright ©2003 Wily Technology, Inc.

11

… And what about the application server?

Application Server
Queues


Web Container


EJB Container


ORB Thread Pool


Datasource
Queuing

Memory
considerations



Garbage collection



Memory Leaks

Connection Pools


JDBC Statement Types


Prepared Statement
Cache


Exceptions



Messaging


JMS


MQ


Application Server
Exceptions



Copyright ©2003 Wily Technology, Inc.

12

Now how do I get this info?


Logging


Profiling


ByteCode Instrumentation

Copyright ©2003 Wily Technology, Inc.

13

Approach #1: Logging


How does it work?


You log (Log4J, JDK 1.4
Logging, ..)


They log (EPM, JMX, PMI
..)


How can you use it?


In a file…


Into a monitor...



Pros (of
you

log):


Produce
any
statistic


Send stat
anywhere


Cons (of
you

log):


You write it yourself


Can not log into
components you did not
write (e.g. WebLogic,
WebSphere)


Hard or impossible to
change once application is
deployed


While logging is powerful and flexible, it can be very
costly in terms of IO performance

Copyright ©2003 Wily Technology, Inc.

14

Approach #2: Profiling


How does it work?




Java Virtual Machine Profiler
Interface (JVMPI)



What can it get you?


CPU usage


Clock time


CPU time


Memory usage


Navigating a Java Heap
Reference Graph


Watching the Garbage
Collector (temporaries, etc..)


Pros:


Lots of Information


Detailed Memory Data



Cons:


Lots of Information


Lots of overhead (not
usually possible under
load)

Copyright ©2003 Wily Technology, Inc.

15

Approach #3:

Bytecode Instrumentation


How does it work?


In
-
memory or Out
-
of
-
memory post
-
processing of
byte code (e.g. JAR or ZIP
files)


“Logging without source
code changes”


How can you use it?


Like you do logging
or

profiling, but…


Without the same level of
detail about memory
utilization


For example:


average response times,


call count,


call rate,


stalled thread counts



Pros:


Add or remove on demand


Low overhead


Watch anything (even
components you don’t
own, like the application
server)


Cons:


Limited options on what
kind of tracing can be
done


Doing it for free would be
hard

Copyright ©2003 Wily Technology, Inc.

16

So which technique is best for me?

Depends on your performance goals…


What impact does the technique have in
production?


How much overhead can you tolerate?


What are your performance goals?


Is source code modification feasible?


Do I need instance level metrics?


How much data will the technique produce?


How hard is the data to interpret?



Copyright ©2003 Wily Technology, Inc.

17

Best advice…


Think through performance goals

early
-

setting target benchmarks/baselines.



Implement a methodology

that allows
measurement of application performance against
benchmarks and baselines.



Use the right tools

when/where possible.
Preferably tools that generate actionable data and
allow for proactive management.



Build
repeatable

and
automated

test scripts.



Remember the key to J2EE Application
performance management is
constant vigilance
.

Copyright ©2003 Wily Technology, Inc.

18

Actual Issues

All Systems Green

Copyright ©2003 Wily Technology, Inc.

19

Additional Examples of Performance
Issues and Improvement Points


Back
-
End and Concurrency Issue


Garbage Collection and Memory Utilization


Synchronization and Script Bug Issue


Load Generation Test Performance Data and Component
Unit Test Counts (Weighted Averages)

Copyright ©2003 Wily Technology, Inc.

20

Info regarding 24 instances of an app servlet

Copyright ©2003 Wily Technology, Inc.

21

Slow leak in connections to MQSeries?

Copyright ©2003 Wily Technology, Inc.

22

History of the number of active MQ
connections settles into average of 26

Copyright ©2003 Wily Technology, Inc.

23

One server starts to back up execute threads

Copyright ©2003 Wily Technology, Inc.

24

Different view showing concurrency problems

Copyright ©2003 Wily Technology, Inc.

25

Transaction Tracer shows slow point

Copyright ©2003 Wily Technology, Inc.

26

Memory Usage (The Good)

Copyright ©2003 Wily Technology, Inc.

27

Memory Usage (The Bad)

Copyright ©2003 Wily Technology, Inc.

28

Memory Usage (The Ugly)

Copyright ©2003 Wily Technology, Inc.

29

Caused a component slowdown

Copyright ©2003 Wily Technology, Inc.

30

Isolated to single EJB via Blame

Copyright ©2003 Wily Technology, Inc.

31

LeakHunter isolates leak in
each transaction that is run.

Copyright ©2003 Wily Technology, Inc.

32

Synchronization Issue

Copyright ©2003 Wily Technology, Inc.

33

Copyright ©2003 Wily Technology, Inc.

34

Copyright ©2003 Wily Technology, Inc.

35

Copyright ©2003 Wily Technology, Inc.

36

Copyright ©2003 Wily Technology, Inc.

37

Copyright ©2003 Wily Technology, Inc.

38

Wily Introscope

Technology Overview

Copyright ©2003 Wily Technology, Inc.

39

Introscope Agent


Gathers
All

Data Required to Manage
a J2EE Application


Introscope Probes


JMX


PMI


CPU Stats


JVM Stats




Enables Management of the Whole
Application…


Application Code Performance


Application Server Health and Configuration


Back
-
End System Performance


Network/Infrastructure Health




Runs Inside the JVM


Negligible Overhead


Benign (Fault Tolerant)


Easy to Install


Copyright ©2003 Wily Technology, Inc.

40

Introscope Probes



-
Patented Bytecode Instrumentation Technology

Servlet

10100101110101
01001010001011
01101111001101
01010000100011
11011101111101
01010100000011
11010101001010
00111111101010
10100101001001
01000010000011

Timer and Counter
“Probes” inserted
into Java bytecode.

No Source Code
modifications required!

Copyright ©2003 Wily Technology, Inc.

41

Metrics Provided by

Introscope Probes


Average Response Times


Run time for Servlet Execution, Method Call, JDBC Call, Etc.


Measured in milliseconds


Average value over small fixed time interval

Rates


Invocations Per Second of Object, Method, or any J2EE
Component


File
-
system and Network I/O Bytes Per Second

Counts


Total Number of Servlet Invocations


Number of open Database connections


Etc.

Copyright ©2003 Wily Technology, Inc.

42

Introscope Probes



-
Provide the Agent with Performance Metrics


Introscope


Agent


Introscope Agent “listens” to
Probes



Probes Can be Installed in Any
Java Code


Application


Application Server


Back
-
End System Connector Libraries


CICS, MQSeries, JDBC


3
rd

Party Applications and Libraries




Provides Low
-
Overhead Visibility
into Java Performance

Copyright ©2003 Wily Technology, Inc.

43

Introscope Enterprise Manager


-
The Central Processing Unit



Introscope


Enterprise Manager


Processes and Manages Data Gathered
by Introscope Agents



Processes Alerts, Database Storage,
Systems Management Integration, etc



Stand
-
alone Java Application. Runs
Anywhere…


UNIX (AIX, Solaris, HPUX)


Windows NT


OS/390


AS400


Linux




Agent

Agent

Agent

Copyright ©2003 Wily Technology, Inc.

44

Introscope Alerts Enable Actionable
Response to Changes in Performance


Easily Configurable Thresholds


Caution and Danger Levels


Configured in GUI Workstation


Maximums or Minimums



Available for Any Metric in System


Response Times, Rates, Counts, …


Servlets, EJBs, Methods, …



Flexible/Powerful Event Manager


Console Alert Dialog Boxes


Emails


Shell Scripts (executed on EM machine)


Any combination of Events for either
Threshold


Copyright ©2003 Wily Technology, Inc.

45

Historical Performance Reporting

-
Enterprise Manager Preserves Data in RDBMS



Introscope


Enterprise Manager


Enterprise Manager can write
any or all metrics to a RDBMS


Configurable frequency and
volume of data output


Run Batch Summary Reports
on Historical Data


Interrogate Historical Data in
Introscope Workstation


System Performance Reports

Historical & Trend

Performance Data

Historical

Data

Copyright ©2003 Wily Technology, Inc.

46

Systems Mgmt Framework
Integration

-
SNMP MIB and Alerts Make Integration Seamless



Introscope


Enterprise Manager

Operations Center

Alerts


Enterprise Manager collects
and analyzes data


Enterprise Manager PUSHES
SNMP Alert to SNMP Manager


Enterprise Manager publishes
live application metrics in
Dynamic SNMP MIB


SNMP Manager displays SNMP
Alert


SNMP Manager reads MIB and
PULLS live performance data
and displays graphically

Copyright ©2003 Wily Technology, Inc.

47

Introscope Workstation

-
Intuitive Visualization of Performance Data


The Console


Dashboards


Customized Display of Multiple
Metrics


Different Dashboards for
Different Users



The Explorer


Hierarchical View of Application


Management Modules


Containers for Alerts, Dashboards,
Calculators, Groupings, etc.



PowerPacks




App Specific Metrics and Views for
3
rd

Party Apps and AppServers


Copyright ©2003 Wily Technology, Inc.

48

Alerts & Events

Performance Data via SNMP MIB

Introscope Architecture

===================================

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586


Capacity Planning

Tr end Repor ts

===================================

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586


===================================

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586

02:00 StockTrading:Serlvet42 318 586


Historical Charts

Availability Reports

System Performance Reports

Systems Management Integration

J2EE

Application

Server

Historical & Trend

Performance Data

Historical

Data

Stand
-
Alone

Java

Application

Agent

Agent

Java

Application

Server


Introscope


Workstation


Introscope


Enterprise Manager


Introscope


Agent

Agent

Copyright ©2003 Wily Technology, Inc.

54

For more info on
achieving World Class
Java Application
management

Wily Technology, Inc.

8000 Marina Blvd, Suite 700

Brisbane, CA 94005

http://www.wilytech.com

1 888 GET WILY


sales@wilytech.com



(415) 562
-
2000


All Systems Green

Ray Beecher


(972) 830
-
9033

rbeecher@wilytech.com

Kenneth Merkel


(713) 669
-
9239

kmerkel@wilytech.com