Security&CMMI

triangledriprockInternet and Web Development

Aug 7, 2012 (5 years and 1 month ago)

534 views

FedEx Institute of Technology

University of Memphis

Testing Tools

1

FedEx Institute of Technology

University of Memphis

Outline


Computer
-
Aided Testing


Mercury Tools


Tour of Testing Tools

2

FedEx Institute of Technology

University of Memphis

Computer
-
Aided Testing


Software tools that aid testing
software tools


Advantage:


Increased software testing productivity


Disadvantage:


Errors may be introduced or obscured
by the testing software tools

3

FedEx Institute of Technology

University of Memphis

Who watches the watchmen?


Heisenberg (of uncertainty principle
fame):
The more closely you look at
one thing, the less closely you can
see something else.


Observers alter the measurements of
things they are observing by the act of
observing alone

4

FedEx Institute of Technology

University of Memphis

Testing Software with Other
Software


The act of testing alters that which is
tested


Tested software can fail in ways that
are not the result of defects in the
target but rather defects in the testing
tool


Heisenbug
: a bug that disappears or
alters its characteristics when
researched

5

FedEx Institute of Technology

University of Memphis

Manual vs. Automated Testing


Some believe that test automation is
too expensive
relative to
its value


therefore, it should be used sparingly


Automated testing requires automated test oracles


an oracle is a mechanism or principle by which a problem in
the software can be recognized


Automated testing tools are suitable for


load testing software


regression testing


or in checking for intermittent errors in software


the success of automated software testing depends on
complete and comprehensive test planning

6

FedEx Institute of Technology

University of Memphis

Automated testing


Benefits of automation and tools


Speed


Efficiency


Accuracy and precision


Resource reduction


Simulation and emulation


Relentlessness


7

FedEx Institute of Technology

University of Memphis

Automated testing


Software test automation




Automating the process of combining, starting,
running testing tools with little or no human
intervention


run test cases


look for bugs


analyze what is observed


log the results



8

FedEx Institute of Technology

University of Memphis

Types of Test Automation


Record and re
-
run a test


Record keyboard and mouse actions and play back for later test


Run a test with many test cases and store
results


Assemble stored code components for
integrated tests


Store and manage requirements


Test case repository


Defect management

9

FedEx Institute of Technology

University of Memphis

Commonly Automated Tests


Functional regression


Volume/load testing


Test with normal load


Performance testing


Tuning with normal load


Stress testing


To breaking point


Vulnerability testing

10

FedEx Institute of Technology

University of Memphis

The Mercury Suite


Quality
Center


Quality Center


Web
-
based


Modules:

1.
Test Director

2.
Quick Test Professional

3.
WinRunner

4.
Business Process Testing

5.
Mercury Service Test

6.
Dashboard


Statistics on results

11

FedEx Institute of Technology

University of Memphis

1. Test Director


Requirements management module


4. Business Process Test (BPT)


Assembles stored components for application tests


For example:


Login component


Select a contact component


Service type component


Print label component


Logout component


Designed for non
-
programmers


Makes calls to Oracle or Excel to retrieve test cases and run
them through the application

12

FedEx Institute of Technology

University of Memphis

1. Test Director


Test Plan


Repository of test cases


Test Lab


Runs cases and stores results


Defect Management

13

FedEx Institute of Technology

University of Memphis

2. The Mercury Suite
-

Quick
Test Professional (QTP)


Functional test automation


Regression test automation


Records and plays a test run through an
application


Requires a GUI


Uses the concepts of Keyword
-
driven testing


Simplifies test creation and maintenance

14

FedEx Institute of Technology

University of Memphis

Keyword
-
driven Testing


Keywords such as "login" or "verify" identify
test components for low
-
level actions such
as login and balance checks


Reuse of test components means faster
test script creation


Allegis has reduced its time to production
by about half according to Eric Amo
-
Gottfried, the team lead for Allegis' WebQA
group

15

FedEx Institute of Technology

University of Memphis

2. Quick Test Professional (QTP)


Custom Harness


Runs many test cases through an application
holding parameters constant


Enables test writer to use scripting languages
such as VBScript to include logic in an
automated test


Can make calls to Oracle or Excel to retrieve test
data

16

FedEx Institute of Technology

University of Memphis

3. WinRunner


Functional and regression testing prior to
deployment


Captures, verifies, and replays user interactions
automatically


Benefits:


Reduced testing time by automating repetitive
tasks


Optimized testing efforts by covering diverse
environments with a single testing tool


Maximized return on investment through
modifying and reusing test scripts as the
application evolves

17

FedEx Institute of Technology

University of Memphis

4. Business Process Testing
(BPT)


web
-
based test design solution


subject matter experts can build, data
-
drive, and execute manual and
automated tests without any
programming knowledge


18

FedEx Institute of Technology

University of Memphis

4. Business Process Testing
(BPT)


Start test case design early in the software
development lifecycle


Automate maintenance of test cases and
test plan documentation


Unify distributed tests with test asset reuse


Detect and diagnose application problems
before you go live

19

FedEx Institute of Technology

University of Memphis

5. Service Test


functional and performance tests for
services


works with Load Runner (next)


test service
-
oriented architecture (SOA)
services



20

FedEx Institute of Technology

University of Memphis

5. Service Test


Reduce QA cycle times by automating functional
regression and performance testing of services that
lack a GUI


Simulate J2EE, AXIS, and .NET client environments to
test services interoperability


Simulate production environments through server stub
simulation and asynchronous testing


Create and track server calls to ensure successful
asynchronous performance testing


Deliver service emulation capabilities that allow testers
to get started early, before services are actually built

21

FedEx Institute of Technology

University of Memphis

Load Runner


Load Runner


Volume/load testing


Performance testing


Stress testing


Benefits


Obtain an accurate picture of end
-
to
-
end system
performance


Verify that new or upgraded applications meet
specified performance requirements


Identify and eliminate performance bottlenecks
during the development lifecycle

22

FedEx Institute of Technology

University of Memphis

Tour of Testing Tools


Tool Suites


Defect Management


Functional Testing


Test Drivers (facilitate functional testing
through a non
-
graphical user interface )


GUI Test Drivers


Unit Test Tools


Integration Testing


Load and Performance


Coverage Analysis

23

FedEx Institute of Technology

University of Memphis

Tools


Test Design Tools


tools can automatically specify (NON
-
complete) test cases


test generation tool do not include an "oracle“


they generate the test input, but they do not generate the
expected results to compare against


Test Implementation


assist with testing at runtime (e.g. memory leak checkers)


Test case Management


help organize and execute groups of test cases, also called
test suites


Static Analysis


analyze programs without running them (static metrics
tools)


Miscellaneous

24

FedEx Institute of Technology

University of Memphis

Tool Suites


QA Center (Compuware)


Automated functional and regression
testing


risk
-
based test management


data management for improved quality
and reliability



Automated performance testing and
monitoring for scalability and
responsiveness

25

FedEx Institute of Technology

University of Memphis

Tool Suites


TestWorks (Software Research, Inc. )


automates and streamlines your
software testing process


Regression Testing


Test Suite Management


Test Coverage support


Windows
and

UNIX Platforms

26

FedEx Institute of Technology

University of Memphis

Canoo


Canoo is a web application testing suite


Canoo is not restricted to only testing Java
applications


Uses other Open Source packages (JUnit,
JTidy)


Fairly easy to use

27

FedEx Institute of Technology

University of Memphis

Defect Management Tools


a repository/database of defect
descriptions


filing new defects


monitoring defects


changing the status of a defect


generate reports at project or across
multiple projects

28

FedEx Institute of Technology

University of Memphis

Defect Management Tools


Web
-
based


Standalone

29

FedEx Institute of Technology

University of Memphis

Web
-
based Defect
Management Tools


Defectr


A free online defect tracking and project management tool
developed using IBM Lotus Domino and Dojo Ajax
framework


Bugzilla


highly configurable open source defect tracking system
developed originally for the Mozilla project


DevTrack (TechExcel)


defect
-

and project
-
tracking tool for software development
teams


Intuitive and powerful, provides workflow and process
automation features, robust searching and reporting, and
comprehensive point
-
and
-
click customization


Agility (AgileEdge)


web
-
based interface, fully customizable field lists, workflow
engine, and email notifications

30

FedEx Institute of Technology

University of Memphis

Standalone Defect
Management Tools


Defect Manager (Tiera Software)


Manages defects and enhancements through the complete entire
life cycle of product development through field deployment


Fast BugTrack (Alcea)


Defect Tracking / Issue Tracking
-

Change Management Software
(work flow/process flow)


QuickBugs (Excel Software)


reporting, tracking and managing bugs, issues, changes and new
features involved in product development


extreme ease
-
of
-
use and flexibility


a shared XML repository accessible to multiple users, multiple
projects with assigned responsibilities, configurable access and
privileges for users on each project


GNATS (GNU)


Freeware defect tracking software

31

FedEx Institute of Technology

University of Memphis

Functional Testing


From a user perspective


Does the software do what the user
expects?


Functional testing through a user
interface such as a native GUI or web
interface


GUI Test Drivers

32

FedEx Institute of Technology

University of Memphis

GUI testing tools


Cross
-
Platform Tools


Web Test Tools


Java GUI Test Tools


Windows GUI Test Tools


Linux GUI Test Tools

33

FedEx Institute of Technology

University of Memphis

Cross
-
Platform Tools


Squish (froglogic)


cross platform automated GUI testing
framework


allows to create tests for GUI and Web
applications by recording or manually writing
test scripts


automated GUI testing tool for Windows,
Linux/Unix, Mac OS X, embedded Linux for
Qt, Web/HTML, Java/SWT, Java/Swing

34

FedEx Institute of Technology

University of Memphis

Web Test Tools


COM/DOM drivers for IE


access the Document Object Model (DOM) via IE’s COM interface


Samie


Perl:
http://samie.sourceforge.net


Jiffie


Java:
http://jiffie.sourceforge.net


Pamie


Python:
http://pamie.sourceforge.net


IEUnit


JavaScript:
http://ieunit.sourceforge.net


IETest


C#:
http://ietest.sourceforge.net


Protocol Drivers (HttpUnit


Java; jWebUnit


Java)


Simulate a web browser


Tests execute directly against the server


Selenium


tests run directly in a browser


runs in IE, Mozilla and Firefox on Windows, Linux, and
Macintosh


Test your application to see if it works correctly on different
browsers and operating systems


35

FedEx Institute of Technology

University of Memphis

Windows/Java/Linux GUI Testing


Windows .Net GUI Testing


Build Quick and Easy UI Test Automation Suites with Visual Studio
.NET
, James McCaffrey


http://msdn.microsoft.com/msdnmag/issues/03/01/UITestAutomation/default.a
spx


“Automating Tests for .NET Applications in C#,” Elisabeth
Hendrickson


http://qualitytree.com/autotest/dotnetgui.htm


Java GUI Testing


Abbot:
http://abbot.sourceforge.net/


Testing Java with Java


Marathon:
http://marathonman.sourceforge.net/



Testing Java with Python


Linux GUI Testing


Dogtail
-

An open source GUI test framework written in
Python

36

FedEx Institute of Technology

University of Memphis

Unit Testing


usually
exercises all the methods in
public interface

of a class


testing of a unit of a code



everything
that could possibly break


common sense verification that the
code unite behaves as expected


with this verification the public
interface gains meaning

37

FedEx Institute of Technology

University of Memphis

Unit Testing


part of the cycle of everyday coding


writing tests before coding


test as a guide to assist in
implementation


38

FedEx Institute of Technology

University of Memphis

Unit Testing


tests grouped into
test suites

run
multiple times per day


all tests should always pass


results in high quality system


leads to clean architecture



39

FedEx Institute of Technology

University of Memphis

Integration Testing


Unit tests

are supposed to be


isolated


as independent

as possible


Integration testing


all the code cooperates


discrepancies between expected behaviour
and real behaviour are detected

40

FedEx Institute of Technology

University of Memphis

JUnit


a regression testing framework


implements unit tests in Java


developed by Erich Gamma and Kent
Beck


JUnit is Open Source

41

FedEx Institute of Technology

University of Memphis

Cactus


Testing
Web applications

with
multiple
tiers

is difficult


More complex testing tools are needed


Cactus

is such a tool
extending

JUnit

to
support
testing server
-
side code


42

FedEx Institute of Technology

University of Memphis

Cactus


simple test framework for unit testing server
-
side java code (Servlets, EJBs, Tag Libs,
Filters, ...).


the goal is to lower the cost of writing tests for
server
-
side code


it uses JUnit and extends it

43

FedEx Institute of Technology

University of Memphis

Acceptance Testing: HttpUnit


verify for the customer that the system is
complete


for example, a web site is not done until
it can log in users, display products, and
allow online ordering


44

FedEx Institute of Technology

University of Memphis

Acceptance Testing: HttpUnit


acceptance tests are
less dependent upon
specific implementation


there is
no universal acceptance testing
tool

to be used for arbitrary applications


HttpUnit is
open source software testing
framework


used to perform testing of web sites without
the need for a web browser

45

FedEx Institute of Technology

University of Memphis

HttpUnit


Automates website testing including


form submission


JavaScript


basic http authentication


Cookies


automatic page redirection

46

FedEx Institute of Technology

University of Memphis

Cactus vs. HttpUnit


both
test Web application

components


Cactus is more
unit
-
oriented


exercises the
behaviour of classes and
methods


HttpUnit allows
to test full use cases


E.g. a login use case which is comprised
of several requests/responses

47

FedEx Institute of Technology

University of Memphis

Unit versus Functional
Testing


If a unit test crosses class boundaries, it might
be a functional test


If a unit test is becoming very complicated, it
might be a functional test


If a unit test is fragile (that is, it is a valid test
but it has to change continually to handle
different user permutations), it might be a
functional test


If a unit test is harder to write than the code it is
testing, it might be a functional test

48

FedEx Institute of Technology

University of Memphis

Performance Testing


Performance: Number of simultaneous
users, size of web pages returned...


“the user experience”


It is the end users experience that are key
to the application performance


Locate the bottlenecks as early as
possible


re
-
factor and re
-
architect

49

FedEx Institute of Technology

University of Memphis

JUnitPerf


JUnitPerf

does
unit performance testing


it
decorates

existing JUnit tests so that
they fail if their
running times

exceed
expectations


supports re
-
factoring by verifying that
performance
-
critical code remains within
expected boundaries

50

FedEx Institute of Technology

University of Memphis

JMeter


test performance both on static and dynamic
resources


It can be used to simulate a heavy load on a
server, network or object to test its strength or
to analyze overall performance under different
loads


E.g. the Web server will maintain a three
-
second response time to requests with a 150
users simultaneous load

51

FedEx Institute of Technology

University of Memphis

JMeter


Can load and performance test HTTP and
FTP servers as well as arbitrary database
queries (via JDBC)


Complete portability (100% pure Java
desktop application)


Full Swing support

52

FedEx Institute of Technology

University of Memphis

Coverage Analysis


Finding areas of a program not exercised
by a set of test cases


Creating additional test cases to increase
coverage


Determining a quantitative measure of
code coverage, which is an indirect
measure of quality


Identifying redundant test cases that do
not increase coverage

53

FedEx Institute of Technology

University of Memphis

Coverage Analysis


Can measure by statement, block/path,
branches, conditions, or function/procedure


Requires the coverage tool to be compiled
with the application software


Creates a summary file of the coverage
analysis

54

FedEx Institute of Technology

University of Memphis

Coverage Analysis


Strategy: first look for failures that are
easily found by minimal testing


Invoke at least one function in 90% of the
source files (or classes)


Invoke 90% of the functions


Attain 90% condition/decision coverage in
each function


Attain 100% condition/decision coverage

55

FedEx Institute of Technology

University of Memphis

Coverage Analysis


Condition/decision coverage is
the best
general
-
purpose measure for C, C++, and
Java


Setting an intermediate goal of 100%
coverage (of any type) can impede testing
productivity


Before releasing, strive for 80%
-
90% or more
coverage of statements, branches, or
conditions


Avoid setting a goal lower than 80%

56

FedEx Institute of Technology

University of Memphis

Coverage Goal for Release


each project must choose a minimum percent
coverage for release criteria based on


available testing resources and


the importance of preventing post
-
release failures


safety
-
critical software should have a high goal


set a higher coverage goal for unit testing than
for system testing


because a failure in lower
-
level code may affect
multiple high
-
level callers

57

FedEx Institute of Technology

University of Memphis

Coverage Analysis Tools


Bullseye Coverage


Cantata++


CodeTEST


LOGISCOPE


Panorama C/C++


Rational PureCoverage


TCAT C/C++


GCT

58

FedEx Institute of Technology

University of Memphis

Cobertura


a free Java tool that calculates the
percentage of code accessed by tests


can be used to identify which parts of
your Java program are lacking test
coverage


It is based on jcoverage (measures the
percentage of code that has been
tested)

59

FedEx Institute of Technology

University of Memphis

Summary


Computer
-
Aided Testing


Mercury Tools


Tour of Testing Tools


Tool Suites


Defect Management


Functional Testing


GUI Test Drivers


Unit Test Tools


Integration Testing


Load and Performance


Coverage Analysis

60

FedEx Institute of Technology

University of Memphis

Beware of bugs in the above
code; I have only proved it
correct, not tried it.




Donald Knuth

61

FedEx Institute of Technology

University of Memphis

References and Further
Reading


http://www
-
128.ibm.com/developerworks/library/j
-
test.html



http://www
-
128.ibm.com/developerworks/websphere/library/techarticles/0303_bho
gal/bhogal.html


http://www.aptest.com/resources.html



http://www.software
-
pointers.com/en
-
defecttracking
-
tools.html



http://www.aptest.com/bugtrack.html



http://www.testingfaqs.org/


http://en.wikipedia.org/wiki/Special:Search?search=testing+tools


http://www.bullseye.com/coverage.html

62

FedEx Institute of Technology

University of Memphis

References and Further
Reading


JUnit


http://junit.org


Canoo


http://webtest.canoo.com


HttpUnit
-

http://httpunit.sourceforge.net


Bugzilla


http://www.bugzilla.org


Ant, Cactus, JMeter
-

http://jakarta.apache.org


Vladimir Sazonov: Automated Testing and Continuous
Integration


R.Hightower & N.Lesiecki:
Java Tools for Extreme
Programming,

Wiley, 2002

Some slides were used or adapted from these references.

63