Performance Test Plan

businessunknownInternet and Web Development

Nov 12, 2013 (4 years and 1 month ago)

172 views

Performance Test Plan

UMass Online


Proprietary
and Confidential Information of Blackboard Inc.



Copyright

㈰ㄱ

WORLDWIDE HEADQUARTE
RS

INTERNATIONAL HEADQU
ARTERS

BLACKBOARD INC.

BLACKBOARD INTERNATI
ONAL B.V.

650 Massachusetts Avenue NW

Sixth Floor

Washington, DC 20001
-
3796

Dam 27 2nd Floor

1012 JS Amsterdam

The Netherlands

+1 800 424 9299 toll free US & Canada


+1 202 463 4860 telephone

+31 20 5206884 (NL) telephone

+1 202 463 4863 facsimile

+31 20 5206885 (NL) facsimile

www.blackboard.com

www.blackboard.com


Copyright © 1997
-
2011. Blackboard, the Blackboard logo, BbWorld, Blackboard Learn, Blackboard Transact,
Blackboard Connect, the Blackboard Outcomes System, Behind the
Blackboard, and Connect
-
ED are trademarks
or registered trademarks of Blackboard Inc. or its subsidiaries in the United States and other countries. U.S.
Patent Numbers: 6,988,138; 7,493,396; 6,816,878.

Other

product and company names mentioned herein may b
e the trademarks of their respective owners.

No part of the contents of this manual may be reproduced or transmitted in any form or by any means without
the written permission of the publisher, Blackboard Inc.







Proprietary and Confidential Information

of Blackboard Inc

Page
ii




TABLE OF CONTENTS

TABLE OF CONTENTS

................................
................................
................................
.............

ii

Document History

................................
................................
................................
................

iii

Document Metadata

................................
................................
................................
............

iii

Introduction

................................
................................
................................
..........................

1

Performance Test
ing Goals

................................
................................
................................
....

1

Participants and Contact Information

................................
................................
....................

1

Data Model

................................
................................
................................
...........................

2

Structure

................................
................................
................................
.......................

2

Deployment
................................
................................
................................
...................

2

Behavior Model

................................
................................
................................
.....................

2

Stock Use Cases

................................
................................
................................
.............

2

Volumetric

................................
................................
................................
.....................

4

Testing Environmen
t

................................
................................
................................
.............

6

Testing Platform

................................
................................
................................
............

6

Target Hardware

................................
................................
................................
...........

6

Test Plan

................................
................................
................................
...............................

6

Entry and Exit Criteria

................................
................................
................................
...

6

(a)

Entrance Criteria

................................
................................
.........................

6

(b)

Exit Criteria

................................
................................
................................
..

7

Success Crit
eria

................................
................................
................................
.............

7

Communication Plan

................................
................................
................................
.....

7

Testing Flowchart

................................
................................
................................
..........

8

Proposed Tes
t Schedule

................................
................................
................................

8

Testing Definitions
................................
................................
................................
.................

9

Test Details

................................
................................
................................
..................

10

(c)

Smoke Test

................................
................................
................................

10

(d)

Unit Test

................................
................................
................................
....

10

(e)

Calibration

................................
................................
................................
.

10

(f)

Capacity Baseline


Single Application Node

................................
................

11

(g)

Capacity Baseline


Full Application Tier

................................
..................

11

(h)

Capacity Baseline


Load Balancer to Full Application Tier

......................

11

(i)

End User Experience Baseline
-

Direct

................................
..........................

12

(j)

End User Experience Baseline
-

LB

................................
................................

12

(k)

Iterative Tuning


Single Application Node

................................
..............

12

(l)

Iterative Tuning


Full
Application Tier

................................
.........................

12

(m)

Failure
-

Bb VMware ESX server

................................
................................

13

(n)

Failure


Single DB Node on Oracle RAC

................................
...................

13

(o)

Failure


Single Node of
the LB

................................
................................
.

13

(p)

Failure


Single Node of the Firewall

................................
........................

13

(q)

Long Term Stability

................................
................................
...................

14

(r)

Stress Test

................................
................................
................................
.....

14






Proprietary and Confidential Information

of Blackboard Inc

Page
iii




Document History

R
EVISION

D
ATE

A
UTHOR

C
HANGE

1.0

Feb 29, 2012

Mary Nguyen

Initial Document


Document Metadata

D
OCUMENT
N
AME

Performance
Audit
Test Plan

A
SSOCIATED
R
ELEASE
/P
LATFORM

Blackboard Learning System R9.1 SP
8

C
ONTACT
I
NFORMATION

MyMary.Nguyen@blackboard.com

C
LIENT


Online






businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
1

of
14



Introduction

This document details the planned Performance Engineering activities that will be performed
UMass
’s

Blackboard
Learn
R9.1

SP8

environment. These activities will be used to confirm that
the Blackboard environment delivers an acceptable level of performance, capacity, and
stability.

Performance Testing Goals

The following goals were established during the planning session and prio
ritized by
UMass

and
Blackboard:

1.

Determine an appropriate number of application servers in the application tier
. Validity
of this will be determined by being
able to:

a.

Service estimated peak load of traffic



defined in volumetric section

b.

Withstand a
single backend s
erver failure on the VMWare ESX (i.e. 1 of 5 going
down).

2.

Determine system capacity
, identify or clarify source(s) of
possible
system bottleneck for
future environment planning.

3.

Identify end user experience response times at estimated p
eak load. Time permitting,
improve where possible.

4.

Verify

major
system redundancy
points for failover functionality.

5.

Verify system is stable over an extended 24hr period of time to obtain a picture of how
system performs with all normal daily maintenance
and jobs tasks running.

6.

Identify end user experience response times
at higher than

estimated
peak load to
better understand
impact of higher traffic volume.

7.

Test the impact
, TBD
.



Participants and Contact Information

Blackboard Consulting

Name

Role

E
-
mail

Phone

Joshua Suess

Project Manager

Joshua.Suess@blackboard.com


1.202.463.4860
x2547

Tim Livers

Architect

Tim.Livers@blackboard.com

781
-
715
-
0440

Mary Nguyen

Consultant

MyMary.Nguyen@blackboard.com

202.459.4628



UMass

Name

Title/Comments

Email

Phone

John Pruente

Project Management

JPruente@umassp.edu

603.315.9735

Stefanie
Henderson

Business Group
-

Lead

shenderson@umassonline.net




businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
2

of
14



Tim Lambert

Business Group

tlambert@umassonline.net


Ravi Mekala

App Config

rmekala@umassp.edu

508.277.1379

Ramesh Jillella

App Config

RJillella@umassp.edu

508.612.9173

Tom Tenzer

Database

tTenzer@umassp.edu

774.455.7828

Rick Cote

Architecture

rcote@umassp.edu

508.340.8933

Greg Ellis

Networking

gellis@umassp.edu

508.735.9789


Data Model

The
data model is a set of users, courses and enrollment patterns that is loaded into the system
for use by the load generation framework. Each of these components follows well
-
known
naming conventions and have pre
-
defined layout that facilitate the scripting
of use cases.

Structure

The data model for will be based on the standard Blackboard test data model used in Product
Development and by Blackboard Consulting for performance engineering.

The standard data model is composed of 5 course templates of various s
izes (from extra small
to extra large) that are then copied using Snapshot into 500 courses. 240 instructors and 7000
students are then created and enrolled in these test courses following enrollment patterns
consistent with various section sizes.


Deployment

The test data model will be created as follows:

1.

Load the 5 standard course templates using batch import/export.

2.

Use the standard snapshot files to create the 500 courses from the templates.

3.

Run the standard user and enrollment files.

4.

Validate th
e data model via GUI.

5.

Smok
e test in Grinder.

Behavior Model

The behavior model describes what actions are to be performed on the target system. The
model for
UMass

will be based on the standard Blackboard Consulting model listed below. This
model focuses o
n the critical use cases, typically those that are likely to occur at least 10 times
within the duration of a normal test under normal load.

Stock Use Cases

The following use cases are part of the standard
performance audit

framework.

Use Case

Action

Notes



businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
3

of
14



Login

Login Page

Open the login page or the portal, if direct portal
entry is enabled.

Submit Login

Submit the credentials, randomly selected for
the pool of available users.

Portal

Open the default portal frameset

Logout

Logout

Log out of the
system

Course
Navigation

Courses Tab

Open the courses tab.

Open Course

Randomly select and open one course the
current user is enrolled into.

Random navigation to:


Assignments


Course Information


Course Documents


Course Links


Course Communication


Tools

Random navigation through one of the course
content areas.

Gradebook

Open Courses

Open the courses tab.

Open Course

Randomly select and open one course the
current user is enrolled into.

Open Tools

Open Tools area

Open

Student Gradebook

Open the main gradebook view

Assessments

Open Courses

Open the courses tab.

Open Course

Randomly select and open one course the
current user is enrolled into.

Open Assessments

Open test lists

Open Assessment

Random select and
start a test, confirming to
take again and providing the password if
needed.

Submit Assessment

Complete and submit assessment.

Review Assessment

Review assessment.

Course
Content

Open Courses

Open the courses tab.

Open Course

Randomly select and
open one course the
current user is enrolled into.

Open Documents

Open course document content area.

Open Subfolder

Randomly open any one subfolder.

Open Document

Randomly download any one document on the
page.

Discussion
Boards

Open Courses

Open
the courses tab.

Open Course

Randomly select and open one course the
current user is enrolled into.

Open Forums

Open discussion boards page.

Open Forum

Randomly select a discussion board and open it.

Open Forum Message

Randomly select a discussion board message


businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
4

of
14



and open it.

Open Forum Reply

Open the reply form for the current message.

Submit Forum Reply

Submit reply.

Image Traffic

Images

Image traffic.

The image to be downloaded is selected
randomly from a weighted
list of the top 50
images (from access logs).

Grading

Courses Tab

Open courses tab.

Open Course

Randomly select a course that the current user is
teaching.

Control Panel

Open the control panel.

Instructor Gradebook

Navigate to the instructor’s view

of the
gradebook.

Open Grade

Randomly open a grade.

Submit Grade

Submit a modification to the grade.

Wikis

Open Courses

Open the courses tab.

Open Course

Randomly select and open one course the
current user is enrolled into.

Open Wikis

Open
discussion board’s page.

Open Wiki

Randomly select a discussion board and open it.

Open Create Page

Open the reply form for the current message.

Submit Page

Submit reply.

Blogs

Open Courses

Open the courses tab.

Open Course

Randomly select and
open one course the
current user is enrolled into.

Open Blogs

Open discussion board’s page.

Open Blog Entry

Randomly select a discussion board and open it.

Open Create Blog Entry

Open the reply form for the current message.

Submit Blog Entry

Submit

reply.


Volumetric

This aspect of modeling focuses on how much traffic should be generated on the target system
to reflect the predicted load. Below is the aggregate hit data from the access logs on
lmprdweb01
, 1 of 24 active nodes
in the current
Vista 4

production environment. The logs span
Dec 28, 2011

to
Feb 22, 2012
.

Initial analysis of the log files show that the peak hourly usage occurred
on
January 30th

at
7
-
8pm EST

with
51,558

hits per the access logs. Based on these observed peak loads, we will

generate enough load for our baseline tests to simulate a target load of
1,237,392

hits/
hr
(
observed) +
25
% increased growth =
1,546,740

hits/hr

(~1.55 Million hits/hr)




businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
5

of
14




Graph 1: Weekly Hits and Page Request Dec 28, 2011


Feb 22, 2012





Graph 2:

Daily Hits and Page Request Jan 29 to Feb 4, 2012





Graph 3: Hourly Hits and Pages

on
January 30
th



businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
6

of
14




Testing Environment

Testing Platform

Web
-
traffic load testing will be performed using the Blackboard Consulting custom load
generation framework built

on top of Grinder, an open
-
source, highly scalable and
customizable Java load testing framework. In addition to the standard instrumentation that
provides system
-
level statistics on the application and database servers, the load testing
software will coll
ect the following metrics:



Response time



Failure rate



Throughput in transactions per second

.

Target Hardware

All testing will be perfor
med against the Blackboard 9.1 SP8

staging

environment. The table
below summarizes the planned configuration.

Component

Hardware Specification

Load balancer

Cisco ACE
-

Dual

Load Generation Servers

3


V楲iu慬aS敲癥牳Ⱐsp散e



2x processors



24GB


Application Servers

Application Tier, 15 virtual servers, specs:



2x processors



8GB RAM

Content


Isolon NAS

Database

Oracle RAC2 nodes
, Physical Servers, specs:

o

2x QC Xeon

o

72GB RAM

Test Plan

Entry and Exit Criteria

(a)

Entrance Criteria

The performance tests may only start after the following conditions are met:



businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
7

of
14





All key stakeholders must sign off the Test Plan.



The
application has passed functional testing.



Human and physical resources have been assigned and are available for the duration of
the tests.



All hardware and software environments must be in place, and free for performance
test and no other load applied.

(b)

Exit Criteria

The exit criteria detailed below must be met:



All high priority errors from performance test must be jointly reviewed, fixed and re
-
tested within the time of the SOW.



The Customer must sign off on any unresolved medium or low priority perfor
mance
problems once the risk of these problems has been documented.



Final report documentation from Blackboard with results, analysis, outstanding risks,
system administration knowledge transfer and recommendations.



All key stakeholders must review the tes
t results.

Success Criteria

The success criteria detailed below must be met:



Blackboard environment delivers an acceptable level of performance and stability for
release to Production.

1.


An acceptable level of performance and stability

means that r
esponse
time must
be within acceptable range of the b
aseline response times.

2.

Completion of agreed to use cases as outlined in the test plan
.


Communication

Plan



Daily
-

Core Technical Team

o

Attendees



Required
: John Pruente, Ravi Mekala, Ramesh Jillella, Tom Tenzer



Optional
: Greg Ellis, Rick Cote, Tim Livers, Josh Suess

o

Conference call @
9:30 EST



Starting
Feb 27

,2012



Dial
-
In Number

508.856.8222


Conference Code:
1810



Daily “Real Time” Communication

o

E
-
mail Communication

o

Phone Communication




businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
8

of
14



Testing Flowchart

The figure below summarizes the projected execution of this test plan. The details of each test
are provided in section
0
. The high level executi
on plan is as follows:



Batch creation of courses from master courses.



Various
unit testing,
smoke
-
testing and calibration activities to validate and refine the
load testing environment prior to collecting final data.



A baseline against the standard Blackboard
9.1 SP8

environment without
customizations, building blocks

or integration points enabled
.



According to the timeboxing methodology, these different objectives will be evaluated
in order, until either time or wor
k runs out.

Proposed Test Schedule

Bb Consulting


UMass
: Performance Testing Schedule

Week

Monday

Tuesday

Wednesday

Thursday

Friday

Weekend

1


2.20
-
2.26


Client
Interviews
(Onsite)


Client
Interviews
(Onsite)


Client
Interviews
(Onsite)





2


2.27
-
3.04

Load Data
Model/ Log
File Analysis

Load Data
Model/ Log File
Analysis

Scripting /
Testing


System Audit



Scripting /
Testing

System Audit


Scripting /
Testing



3


3.05
-
3.11

Testing

Testing

Testing

Testing

Testing


4


3.12
-
3.18

Testing

Testing

Testing

Testing

Testing


5


3.19
-
3.25

Testing

Testing

Report
Generation

Report
Generation

Report
Generation


6


3.26
-
4.01

Report
Generation

Report
Generation

Report
Generation

Report
Generation


Report
Delivery



7


4.02
-
4.08

Bb Internal
Report
Review

Bb Internal
Report Review

Bb Internal
Report Review

Bb Internal
Report Review

Bb Internal
Report Review



8


Bb Internal
Report
Bb Internal
Report Review

Issue Report






businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
9

of
14



4.09
-
4.15

Review



Table
0
-
1
:
Proposed Testing Schedule


Testing Definitions


The following table defines the types of testing referred to in this document.


Test Type

Description

Smoke

Identify major configuration or performance
issues.

Unit

Verify individual use cases are
functioning as
designed.

Calibration

Determine load generator settings required to
generate the desired load level and use case
distribution.

Baseline

Measurement of performance characteristics
to be used as a comparison against future
testing.

Capacity

Measurement of the peak capacity a
configuration can handle while remaining
within acceptable performance ranges.

Iterative

Multiple tests are executed with specific
environment or settings changes implemented
to determine the overall effect (pos
itive or
negative) on response times and system load
levels.

Stress

Gain an understanding of how the system is
capable of responding under extended periods
of load greater than projected peak.

Impact

Determine the impact of specific business
processes an
d/or failures within an
environment.

Stability

Determine the system ability to provide
consistent response times and system
performance characteristics over an extended
period of time.




businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
10

of
14




Test Details

This section contains the details for the test that will be performed on the environment. For the
order and priority of test please refer to section 2.

(c)

Smoke Test

Name

Smoke test

Objective

Unit test the deployed software, scripts and data model. Screen
for
obvious performance and/or functional defects that would invalidate
the rest of the testing work.

This phase is iterative and will most likely involve multiple tests.

Target

Single server

Access Method

Direct

Volume of traffic

single student; single

instructor; 20 students + 1 instructor

Duration

10 minutes each

Priority Level

Required


(d)

Unit Test

Name

Unit test

Objective

Unit test the deployed software, scripts and data model. Screen for
obvious performance and/or functional defects that would
invalidate
the rest of the testing work.

This phase is iterative and will most likely involve multiple tests.

Target

Single server

Access Method

Direct

Volume of traffic

single student; single instructor; 1 students + 1 instructor

Duration

10 minutes
each

Priority Level

Required


(e)

Calibration

Test Name

Calibration

Objective

Adjust the various load parameters to meet the target use
-
case
distribution.

Target

Single server

Access Method

Direct

Volume of traffic

20 students, 1 instructor

Duration

20

minutes



businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
11

of
14



Priority Level

Required


The number of variables that comes into play to generate transactions (overlapping script click
paths, think times, load generator and virtual user allocation, etc.) makes it difficult to
accurately predict the final dis
tribution. The test parameters are to be determined empirically
through a series of short load tests that will be run and the logs on the target server will be
matched against the target use case distribution to confirm and iteratively adjust the virtual
user count and distribution across scripts.


(f)

Capacity Baseline


Single Application Node

Name

Capacity
Baseline


Single Application Node

Objective

Assess the capacity of a single node in terms of throughput in
hits/
hour

Target

Single Application Node

Access Method

Direct

Volume of traffic

Peak load

Duration

30 minutes

Priority Level

1


(g)

Capacity Baseline


Full Application Tier

Name

Capacity
Baseline


Full Application Tier

Objective

Assess the capacity of the full application tier in terms of
throughput
in hits/hour

Target

Full Application Tier

Access Method

Direct

Volume of traffic

Peak load

Duration

60 minutes

Priority Level

1

(h)

Capacity Baseline


Load Balancer to Full Application Tier

Name

Capacity Load Test

Objective

Assess the
capacity of the full application tier through the LB in terms
of throughput in hits/hour

Target

Full Application Tier

Access Method

Load Balancer

Volume of traffic

Peak load

Duration

60 minutes

Priority Level

1




businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
12

of
14



(i)

End User Experience Baseline

-

Direct

Name

End User Experience Baseline
-

Direct

Objective

Assess the end
-
user experience (in terms of response times)

Target

Full Cluster

Access Method

Direct

Volume of traffic

Peak load

Duration

30
-

60 minutes

Priority Level

1


(j)

End User Experience
Baseline
-

LB

Name

End User Experience Baseline
-

LB

Objective

Assess the end
-
user experience (in terms of response times)

Target

Full Cluster

Access Method

LB

Volume of traffic

Peak load

Duration

30
-

60 minutes

Priority Level

1


(k)

Iterative Tuning



Single Application Node

Name

Iterative Tuning


Single Application Node

Objective

Assess the end
-
user experience (in terms of response times)

Specific system change TBD

Target

Single Application Node

Access Method

Direct

or LB

Volume of traffic

Peak load

Duration

30
-

60

minutes

Priority Level

2


(l)

Iterative Tuning


Full Application Tier

Name

Iterative Tuning


Full Application Tier

Objective

Assess the end
-
user experience (in terms of response times)

Specific system change TBD

Target

Full
Application Tier

Access Method

LB

Volume of traffic

Peak load

Duration

30
-

60 minutes

Priority Level

2



businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
13

of
14




(m)

Failure
-

Bb VMware ESX server

Name

Failure


Bb VMware ESX server

Objective

Determine the impact of the failure on end user and verify
functionality

Target

Full Cluster

Access Method

LB

Volume of traffic

Peak load

Duration

60 minutes

Priority Level

3

(n)

Failure


Single DB Node on Oracle RAC

Name

Failure


S
ingle DB Node on Oracle RAC

Objective

Determine the impact of the failure on

end user and recovery and
verify functionality

Target

Full Cluster

Access Method

LB

Volume of traffic

Peak load

Duration

60 minutes

Priority Level

3

(o)

Failure


Single N
ode of the LB

Name

Failure


Single N
ode of the LB

Objective

Determine the
impact of the failure on end user and recovery and
verify functionality

Target

Full Cluster

Access Method

LB

Volume of traffic

Peak load

Duration

60 minutes

Priority Level

3

(p)

Failure


Single N
ode of the Firewall

Name

Failure


Single
N
ode of the
Firewall

Objective

Determine the impact of the failure on end user and recovery and
verify functionality

Target

Full Cluster

Access Method

LB

Volume of traffic

Peak load

Duration

60 minutes



businessunknown_b76324f4
-
02c9
-
4394
-
9301
-
0416c5a2f945.docx


Proprietary and Confidential Information

of Blackboard
Inc


Page
14

of
14



Priority Level

3

(q)

Long Term Stability

Name

Long Term
Stability

Objective

Assess the systems stability with all standard maintenance and job
scheduled daily.

Target

Full Cluster

Access Method

LB

Volume of traffic

Peak load

Duration

24hrs

Priority Level

1

(r)

Stress Test

Name

Stress Test

Objective

Observe

end user experience with 2x peak load

Target

Full Cluster

Access Method

LB

Volume of traffic

Peak load

Duration

1hr

Priority Level

4