Web application Performance Testing Guidelines

hipshorseheadsServers

Nov 17, 2013 (3 years and 10 months ago)

109 views

Performance Testing

Best Practices



May 18
th

2010

Tuli Nivas

User
perceived
performance

Stability

Scalability

Efficiency

Assessing release readiness

Assessing infrastructure adequacy

Assessing adequacy of developed software

Improving the efficiency of performance tuning


Understand
Business/Application
Needs


Identify Acceptance
Criteria

Identify Test
Environment

Plan & Design Tests

Execute the Tests

Analyze Results,
Report & Retest

Project vision

Purpose of the entire system

Customer/User expectations

Business Drivers

Reasons for testing performance

Value that performance testing brings to the project

Process

Compliance criteria

Understanding the
Business/Application
Needs


Business requirements


Contractual obligations


Performance test
goals/requirements


Key business indicators

Understand
Business/App
lication Needs


Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Understand
Business/App
lication Needs


Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest



Determine if the application complies with SLAs



Detect bottlenecks to be tuned



Determine performance characteristics for various
configurations




Provide input data for scalability and capacity
-
planning
efforts




Determine if the application is ready for deployment to
production

A Web site is responsible for conducting online surveys
with 2 million users in a one
-
hour timeframe. The site
infrastructure was built with wide area network (WAN)
links all over the world. The site administrators want to
test the site’s performance to ensure that it can sustain 2
million user visits in one hour.


Simulate one user transaction scripted with 2 million total virtual users in
one hour distributed among two datacenters, with 1 million active users at
each data center.



Simulate the peak load of 2 million user visits in a one
-
hour period.



Test for 100
-
percent coverage of all survey types.



Monitor for relevant component metrics: end
-
user response time, error
rate, database transactions per second, and overall processor, memory,
network and disk status for the database server.



Test the error rate to determine the reliability metrics of the survey system.



Test by using firewall and load
-
balancing configurations.

Understand
Business/App
lication Needs


Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Identifying Acceptance
Criteria


Client expectations


Risks to be mitigated


Business requirements


Contractual obligations


Performance testing success criteria


Key areas of investigation


Key performance indicators

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Factors to
Consider

Business
Requirements

User
Expectations

SLAs

Resource
Utilization
Targets

Realistic
Workload
Models

Anticipated
Load Conditions

Conditions of
System Stress

Pervious
Releases of the
Application

Component/

Interfaces
Activities

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Response Time

Logical I/Os

Throughput (TPS)

Resource Utilization

Service Restarts/Aborts

Concurrent Threads/ Thread Pool Size

Queue Depth

Requests /Responses

Connections

Core Dumps

Existence Time

CPU Time

Server CPU Usage

Process CPU Usage

Server Memory Usage

Process Memory Usage

Network Usage

Disk Space Usage

Disk Busy

Requests Received

Responses Sent

Error Responses

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Request
Throughput

E2E
Maximum
Response
Time

E2E
Minimum
Response
Time

E2E Average
Response
Time

Response
Time
Deviation

Error
Percentage

Different
Error Types

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

CPU Utilization

Process CPU Utilization

Memory Utilization

SWAP Space/VM Utilization

Network I/O

Disk I/O

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

sar

Example usage
: (
sar

-
d
-
r
-
q
-
n SOCK
-
b
-
B
-
u 120 6 > sarSample.txt &);




-
d : I/O transfers



I/O Read



I/O Write




-
r : Memory Usage data




-
q : Queue Length Data


Number of processes in the list


Average load for last 1, 5 and 15 minutes



-
n SOCK : Socket Usage Data



-
b : Buffer to disk TPS



Buffer Read


Buffer Write



-
B : Page Faults Data


-
u : Total CPU Utilization



120: # of
secs

between each interval


6 : # of requests to generate.

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

nmon

Example usage
:
nmon


ft

s 15

c 250



-
f
-
s : sample time in seconds


-
t : Include top process in the output


-
c : number of samples

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Yaketystats


Example usage:
http://plabptl020.dev.sabre.com/yaketystats/jart/index.php?pl=IndividualServers/plab202




Collector


is the client. That collects stats and sends them to the server.


Stuffer


is the server that accepts stats from the client and puts them in a file system. Once every 5
minutes it puts “stuffs” these stats to the RRD file.

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Slow Responses from Interfacing Components

Depleting/Exceeding Resources Utilization Threshold

Exceeding Timeout/TTL

System Exceptions

Unresponsive Interfacing Systems

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest


Heap Size


Garbage Collection


Thread Count


Number of Exceptions by Type


Number of Instances


Number of Open Connections


Queue Size


Thread Pools/Groups

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

JConsole

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

VisualGC

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

YaketyStats

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest



HPJmeter

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Maximum
Configured DB
Connections


Percentage of
Utilized DB
Connections

Number of Sessions

Response Time

Errors

Resource
Utilization

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest


Identify
Issue


Assess Key
Risk Areas


Assess
Likelihood
& Impact


Rank Risks


Select
Strategy


Implement
Strategy


Monitor,
Evaluate &
Adjust

Risk Identification

Risk Analysis

Risk Mitigation


ID


Category


Description

Probability of failure (%)

Impact

(H/M/L)

Confidence Level

(H/M/L)

Tolerance Level

(H/M/L)

1. Comment on risk mitigation

2. Assess the proposed plan

3. Comment on practicality of the solution

1


Hardware


New Blade Servers


HP BL 460c G6s


L

L

H

H

Scalability test, load test & failover test

2


Operating System


New operating System


RH5u5

L

M

H

H

QA E2E full regression test, critical path load test,
scalability test & soak test

3


Middleware


New Web Server


Apache HTTP Server

New Service Container


Apache Tomcat

L

M

M

M

Soak test & failover test

4


Software Application


New Software Platform


TIBCO
PortalBuilder

5.2

L

M

M

M

Soak test & failover test

5


Database


New Database


Oracle 11g

L

M

M

H

SLA for Oracle database


soak test

6


Network Bandwidth


N/A











7


Network Protocol


N/A











8


Others


Akamai



JAR File Update

Backup Emulator

M

M

M

M

Performance test to confirm failure, impact,
confidence & tolerance level

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Identifying the Test
Environment


Logical & physical production
architecture


Logical & physical test architecture


Available tools


Test vs. production environment


Environment related concerns


Additional tools requirement
determination

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Factors to
Consider

Hardware

Configuration

Hardware
Specification

Network

Network
Architecture

End User
location

Load Balancing
Implication

Tools

Load Driver
Limitations

Impact of
Monitoring
Tools

Software

Software
License
Constraints

Software on
Shared/Virtual

Environment

Logging Levels

External
Factors

Interaction with
Other Systems

Batch Processes/

Updates

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Planning & Designing
Tests


Available Application Features
and/or Components


Application Usage Scenarios


Performance Acceptance Criteria


Test execution prerequisites


Tools & resources required


Application workload model


Test data required


Tests ready to be implemented

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

What suits one customer


might not suit the next

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Identify Key Usage Scenarios

Determine Navigation Paths

Identify User Data & Variance

Determine Distribution of Scenario

Identify Target Load Levels

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest


Various loads & traffic patterns


Identify breaking point

Load Test


Production level load


MTBF

Soak Test


Loads over anticipated levels

Impulse Test


Verify redundancy mechanisms

Failover Test


Evaluate behavior in ways not intended

Negative Test


Verify load balancing mechanisms

Load Balancing Test


Evaluate behavior with depleted resources/disconnected
components

Destructive Test

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

0
20
40
60
80
100
120
140
160
180
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33
35
37
39
41
43
45
47
49
Peak Load

3X

Peak Load

0.2X

Peak Load

Peak Load

33

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

0
20
40
60
80
100
120
140
160
180
1
6
11
16
21
26
31
36
41
46
0
20
40
60
80
100
120
140
160
180
1
6
11
16
21
26
31
36
41
46
34

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Executing the Tests


Task execution plan


Available tools/environment


Available application
features/components


Validated, executable tests



Test execution results

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Reset the System (Unless Required Otherwise)

Smoke Test to Ensure Script & Performance Counters Setup
Properly


Ensure Script Represents the Workload Model

Ensure Correct Metrics Collection

Ensure Test Configuration for Any Business/Performance
Indicators

Validate Test Environment Configuration


Validate Test Data

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Coordinate Test Execution &
Monitoring

Begin Test Execution

During Test Run


Validate
Monitors, System & Data

After Test


Review Results

Archive Test Results & Data for
Future Reference

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Analyzing Results,
Reporting & Retesting


Test execution results


Performance Acceptance Criteria


Risks, concerns & issues



Results analysis


Recommendations


Reports

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Principles of Data Analysis & Reporting

Report early, often, visually & intuitively

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Principles of Data Analysis & Reporting

Report

early, often, visually & intuitively

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Use the right statistics

Statistical
Outliers

Histograms

Averages
with
Minimum/

Maximum

Percentiles

Standard
Deviation

Correct Data
Visualization

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Principles of Data Analysis & Reporting

Report

early, often, visually & intuitively

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Use the
right statistics

Consolidate

data

correctly & effectively

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Principles of Data Analysis & Reporting

Report

early, often, visually & intuitively

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest

Use the
right statistics

Consolidate

data

correctly & effectively

Customize reports

for intended audience

Create

concise reports

Make

raw data
available

When you fix one bug

you introduce several new ones

Understand
Business/App
lication

Needs



Identify
Acceptance
Criteria

Identify Test
Environment

Plan &
Design Tests

Execute the
Tests

Analyze
Results,
Report &
Retest