Load Testing WebIntelligence

watermelonroachdaleInternet and Web Development

Jul 30, 2012 (5 years and 3 months ago)

197 views

Load Testing WebIntelligence

Kris Kelso, Renal Care Group, Inc.

Slide
3

About Renal Care Group

Load Testing Methodology

Load Testing Tools

Monitoring & Analysis

Our Scenario

Execution

Lessons Learned / Other Opportunities

Q&A

Topics

Slide
4

Who We Are


A specialized dialysis services company


Provides care to patients with kidney disease

31,900 patients

450 outpatient facilities

210 hospitals

9,200 associates

NYSE: RCI

http://www.renalcaregroup.com



Slide
5

Renal Care Group Locations

Slide
6

How We Use Business Objects Products

5,000+ end users

Primarily full
-
client BusinessObjects reports


Small development team (less than 10)


Extensive use of prompts


Prompts are standardized across reports

Custom application


“InfoCentral”


Written in Java


Uses WebIntelligence SDK (Developer Suite)


Limited use of the WebIntelligence Java Report Panel

500+ scheduled reports per day via Broadcast Agent


Custom application to bulk
-
schedule reports: BCA Xpress


Slide
7

About Renal Care Group

Load Testing Methodology

Load Testing Tools

Monitoring & Analysis

Our Scenario

Execution

Lessons Learned / Other Opportunities

Q&A

Topics

Slide
8

Load Testing

What is load testing?


An automated set of exercises against a system


A consistent, repeatable load

When should it be done?


A new system is put in place


Significant changes are made to an existing system

Where should it be done?


Testing environment that is similar to production or scaled
measurably


Production environment during go
-
live validation

Why bother?


Identify and resolve problems related to performance under load,
which are usually not seen in functional tests (less than 10 users)

Slide
9

Testing Methodology 1/2

Define business goal


Number of users and/or transactions vs. time


System stability / length of uptime


Redundancy / failover

Validate server configuration


Same hardware and software platform as production system


Perform functional testing first

Design test scenario / create test script


Scripting process will vary with different tools

Slide
10

Testing Methodology 2/2

Incremental phased testing


Execute test


Troubleshoot problems


Repeat test


Advance to next test

Monitor tests


System resources


User and transaction counts


Load balancing

Analyze and publish results


Impress your boss with charts
and graphs!

Slide
11

About Renal Care Group

Load Testing Methodology

Load Testing Tools

Monitoring & Analysis

Our Scenario

Execution

Lessons Learned / Other Opportunities

Q&A

Topics

Slide
12

Load Testing Tools

Dozens of tools available


Commercial and Open Source


Most are geared towards web
-
based testing


Licensing for commercial software is often per “virtual user”

Our choice: Mercury Interactive LoadRunner


Virtual User Generator (Scripting)


Controller (Execution)


Monitors (Windows, Unix, Database, etc.)


Analysis


Reporting

Slide
13

Virtual User Generator (VuGen)

Script creation


Scripts are “recorded” by the tool automatically


Scripts can be modified by hand if necessary (written in C)


Utilize CSV files for variables

Slide
14

Controller

Controller allows multiple ‘users’ to execute a script


Virtual users can make choices according to variables for sequencing
or randomization


Multiple scripts can be run simultaneously by multiple users

Scenario design


Choose script(s) & sequence


Number of users


Duration


Monitors

Scenario execution


“Watch” the vuser session


Detailed error log per vuser (http responses / error codes)


Live scenario statistics and resource monitoring

Slide
15

Controller Design

Scenario Setup


Choose script


Define users


Create schedule

Slide
16

Controller Execution

Real
-
Time Info


Running users


Statistics


Pass / fail ratio


Errors


System monitors

Slide
17

About Renal Care Group

Load Testing Methodology

Load Testing Tools

Monitoring & Analysis

Our Scenario

Execution

Lessons Learned / Other Opportunities

Q&A

Topics

Slide
18

System Monitoring

Native Windows 2000 performance counters on
WebIntelligence cluster servers


Memory and Processor Usage


Network Traffic


Saved to a CSV file and imported into LoadRunner after test
completed.


Synchronized counter time values with script execution times

WebSphere monitoring via XML/SOAP data stream


JVM Resources


Database connections


Session creates / destroys


Requires installation of IBM’s Performance Servlet Application.


Automated correlation by LoadRunner

Slide
19

Analysis

Charts & Graphs


Transaction times


Transaction counts


Resource usage


Throughput


Vuser activity

Interactive


Overlay graphs


Add / remove
measures


Adjust time range

Slide
20

Reporting

Auto
-
generated

Multiple Formats


HTML pages


Word documents

Customizable


Templates


Style Sheets

Integration


Crystal Reports

Slide
21

About Renal Care Group

Load Testing Methodology

Load Testing Tools

Monitoring & Analysis

Our Scenario

Execution

Lessons Learned / Other Opportunities

Q&A

Topics

Slide
22

Our Scenario

Purpose


Upgrading BusinessObjects Web Intelligence from 6.1b to 6.5.1

Business Goal


Demonstrate stability at current load

Success Definition


Sustained 100 report refreshes per hour per BusinessObjects Web
Intelligence server for 12 hours at under 1.5% error rate


This definition was calculated by analyzing audit data from our
production system and adding 20% to our peak hourly usage


Per server calculation used because we had fewer BusinessObjects
Web Intelligence servers in our test environment than we did in
production

Slide
23

Application Architecture

Custom Application


“InfoCentral”


Written in Java, using BusinessObjects Web Intelligence SDK
(Developer Suite)


1 IBM HTTP Server


2 IBM WebSphere Servers

BusinessObjects Web Intelligence 6.5.1


Cluster Manager plus 2 Secondary Nodes


Windows 2000 Servers

Oracle 9i


Solaris

Slide
24

Scripting

We used an existing script


Written by a contractor to test the initial release of “InfoCentral”


Script has been modified with each release of our application

Includes 13 high
-
usage reports


Reports represent a majority of our system use


40+ possible prompts


Wide variety of queries and objects

Prompt values randomly selected from CSV files


Results in reports of various sizes, just like in production

No “Think Time” means fewer virtual users


Users will execute transactions as quickly as possible

Slide
25

About Renal Care Group

Load Testing Methodology

Load Testing Tools

Monitoring & Analysis

Our Scenario

Execution

Lessons Learned / Other Opportunities

Q&A

Topics

Slide
26

Scenario Execution
-

Validation

10 users executing one script for 15 minutes


Found errors relating to CORBA services not running
-

services were
restarted and environment was validated


Encountered login problems related to user accounts created for
testing
-

corrected account permissions


Considered success when 0 errors occurred during test (13
iterations)

Slide
27

Scenario Execution


Ramp Up

15 users 1 hour


Encountered hardware issues on one of the WebSphere servers


Removed server and continued

15 users 4 hours


Revealed issues with an executable that was not closing out
successfully on the BusinessObjects Web Intelligence servers


Known issue


Business Objects had a hot fix for this, but there was
a conflict with another hot fix we were using


We requested that Business Objects combine the hot fixes


Solution provided within the week

Slide
28

Scenario Execution


Full Test

16 Hour Overnight test (Rounds 1 and 2)


All transactions failed after 1am


Traced the issue to a backup process running on WebSphere /
Solaris servers


Temporarily disabled backup process in test environment


(Later) Modified backup process to bring up services more gracefully


Slide
29

Overnight Test Results (Errors)

All transactions failed after 1 AM

Slide
30

Scenario Execution


Success

16 Hour Overnight test (Final)


150+ Reports refreshed per hour per BusinessObjects Web
Intelligence server (50% above our stated goal)


0% Error rate

Go
-
Live Weekend


Directed LoadRunner at production system to validate upgrade


Initial test revealed a problem with one report


4
-
hour test was successful with no errors

Slide
31

About Renal Care Group

Load Testing Methodology

Load Testing Tools

Monitoring & Analysis

Our Scenario

Execution

Lessons Learned / Other Opportunities

Q&A

Topics

Slide
32

Lessons Learned

Load test after environment is stable


Scripts need to be repeated for each system modification after testing
begins


Ideally, load testing should be performed last

Know server maintenance routine


Backups can impact performance / affect services

Re
-
use test script for validation


Small scale load testing on production systems after upgrade is
complete validates the upgrade process

Slide
33

Other Opportunities

Coding for performance improvements

Failover


BusinessObjects Web Intelligence cluster nodes


WebSphere nodes

Environmental settings


BusinessObjects Web Intelligence settings (process timeouts, etc.)


JVM tweaks (memory allocation, etc.)

Application ceiling for users / report processing


Valuable for future expansion

Compare our custom application to standard InfoView


Measure performance gains / losses in our custom application


Requires a second script to be written to exercise BusinessObjects
InfoView

Slide
34

About Renal Care Group

Load Testing Methodology

Load Testing Tools

Monitoring & Analysis

Our Scenario

Execution

Lessons Learned / Other Opportunities

Q&A

Topics

Slide
35

Q&A

Questions

Contact Information

Kris Kelso

Senior Systems Engineer

Renal Care Group, Inc.

kkelso@renalcaregroup.com




Jerrold Swafford

Systems Engineer II

Renal Care Group, Inc.

jswafford@renalcaregroup.com