Test Suites, Test Automation and Regression Testing

fanaticalpumaMechanics

Nov 5, 2013 (3 years and 9 months ago)

91 views

Test Suites, Test Automation and
Regression Testing

1

Test Suites

2

Test Suite


A
set of tests
either used to test specific
behavior

in code
or to execute a
specific type of testing
.


A good test suite contains documentation with:


Details about the purpose of the test suite


Any necessary prerequisites for running the test suite


Information about expected system requirements


Description about the tests included in the suite

3

System Testing Test Suites


GUI Testing


Usability Testing


Performance Testing


Security Testing


Stress Testing



Regression Testing


Load Testing


Scalability Testing


Accessibility Testing


Build Verification/Smoke
Testing

4

Other Reasons


Test cases can be bundled into test suites based on:


Requirements priorities


Business priorities


Functionality


User types


Automatically selected based on:


Code churn


Metrics from version control


Coverage metrics




5

Test Automation

6

Test Automation


Using software to control the process of testing software


Sometimes known as
test scripts


Often used to execute one or more
test suites


Two classes of test automation


Code
-
driven testing


GUI
-
driven testing


Can be used to automate existing testing or add new
testing that may be difficult to perform manually

7

Code
-
driven Testing


Writing code to test code


Ex: Unit tests


Benefits:


Often integrated into the development environment


Cheap, efficient and repeatable


Can provide coverage metrics


Drawbacks:


Must know how to write code to test code

8

Example:
Pex


Pex

is a tool from Microsoft Research that does
automated white
-
box testing with the goal of high
statement coverage.


Pex

aims to find interesting input
-
output values for your
methods.


These can be saved as a test suite for later use.


Alternatively, you can attempt to guess at the
implementation of a method based on input
-
output
values. See
http://www.pexforfun.com

9

GUI
-
driven Testing


Automating user inputs to repeatedly test a program


Keyboard strokes, mouse movements and clicks are
recorded to a log


Log is played back to test the application


Benefits:


Little to no programming knowledge needed


Drawbacks:


Changes to GUI require a new recording of the test


Irrelevant activities may be recorded as well

10

Example:
AutoIt


AutoIt

is a freeware automation language for Windows


It allows you to simulate a user’s interactions (keyboard,
mouse, windows) but also give you programmers tools
such as calling functions in a Windows DLL


Neat example of
AutoIt

solving Minesweeper:


https://www.youtube.com/watch?v=
QMntng1RX_M


11

Attributes of a Good Automated Test


Simple


Self
-
checking


Robust


Sufficient


Necessary


Understandable



Efficient


Specific


Independent


Maintainable


Traceable

12

Commonly Automated Testing


Functional testing


Regression testing


Exception/Negative testing


Stress testing


Performance testing


Load testing

13

Types of Automated Testing


Data
-
driven testing


Modularity
-
driven testing


Keyword
-
driven testing


Hybrid testing



Simply combining test approaches


Model
-
based testing


Code
-
driven testing


Using code to test code (ex: reflection and unit tests)

14

Data
-
driven Testing


U
sing a
combination of data and tests
to verify
functionality of a program.


Data contains:


Set of inputs


Corresponding outputs


Tests are often unit or functionality tests, but can also be
GUI
-
driven.

15

Modularity
-
driven Testing


Test cases are build in “levels” based on integration


Example:


Develop unit tests first


Develop tests that aggregate unit tests to test functions


Develop tests that aggregate function tests to test classes or
interactions between classes


And so on…


A bottom
-
up approach to testing that aligns well with
some integration testing techniques

16

Keyword
-
driven Testing


Method separates the documentation of test cases from
the implementation.


Also known as table
-
driven testing or action
-
word
-
based
testing.


Uses a table




Test is automated



Keyword

First

Name

Last
Name

City

Zip

Add User

John

Doe

Corvallis

97339

17

Model
-
based Testing


Deriving tests from an explicit or abstract model of the
system under test.


Multiple approaches:


Finite state machines


Theorem proving


Satisfyability

proving


Model checking


See
http://en.wikipedia.org/wiki/Model
-
based_testing

for
a shallow overview of model
-
based testing

18

Pros of Test Automation


Reliability


Repeatability


Comprehensiveness


Increased quality (hopefully)


Speed over manual testing


Cost reduction (hopefully)

19

Cons to Test Automation


Not everyone has the skills necessary to write “good”
tests.


Debugging automated tests is expensive.


Test maintenance is expensive (Especially in GUI
-
driven
tests).


Maintenance of data for tests is expensive.

20

Regression Testing

21

Candidates for System Testing


GUI Testing


Usability Testing


Performance Testing


Security Testing


Stress Testing



Regression Testing


Load Testing


Scalability Testing


Accessibility Testing


Build Verification/Smoke
Testing

22

Regression Testing


Testing designed to find
regressions

(new bugs) in existing
functional

and
non
-
functional
areas of code after a
change
.



Often consisting of both
unit tests

and
functional tests
.


Usually
automated

due to frequency of code changes.


The primary goal is to achieve
adequate coverage
, using
minimal time
.


23

Purpose


Requirements verification


Detect side
-
effects


Introduction
of new
bugs


Uncovering of undetected old bugs


Changes to quality
attributes


Decrease in performance


Decrease in usability



etc




24

361 Flashback: Quality Attributes


Reliability


Efficiency


Integrity


Usability


Maintainability


Testability


Flexibility


Portability


Reusability


Interoperability

25

Selecting Test Cases


High
-
risk areas


Frequently defective areas


Frequently used functionality


Key requirements


Recently changed code


Boundary cases


Integration test suite

26

Considerations


Adequate coverage


Minimum Test Selection


Process considerations


Schedule


Requirements


Hardware/People availability


Cost


Software brittleness


Generally err on the side of caution when selecting which
tests cases to include.



27

Key Challenges


As tests are added to the regression test suite, over time, the
entire test suite can no longer be run.


How many lines of code are used to test one line of code?


Adequate coverage in minimum time is difficult to achieve.


Determining when to run regression tests
.


Dependence on developers to write the correct tests for
each discovered bug.


Ex: May address symptoms, not the actual bug.


Regression testing is repetitive and slow.

28

Process Considerations


No changes to
code

or
data

during regression testing.


Aim to
r
un the tests that would expose the
most serious
defects first.


Include all tests that were written to test recently fixed
bugs.


After test,
merge results and store
for future use


Organization is key


Label, label, label!


29

Some Approaches


Regression test after final code check
-
in


Complete regression testing


Continuous regression testing


Regression smoke testing


Restart
-
able regression testing


“One
-
hour” regression testing


30

“One
-
hour” Regression Testing


What would you do to prove to a customer that your
software is ready to use in only one hour?


Focus is on finding the biggest problems quickly.


Choices depend on:


M
aturity of the product


M
aturity of development/testing tools


Type of product


T
he customer


Perform the “one
-
hour” regression test prior to your
complete regression test suite.


31

“One
-
hour” Regression Testing


What to test:


Customer
-
focused tests


What is the customer going to do with your product?



Complex and difficult tests


What features best exercise the code?



“Big picture” tests


What is the expected level of performance?



Expected failure tests


Where has the software had problems in the past?

32

Takeaways


A
test suite
is a collection of tests that are grouped
together in some logical manner.


Test automation
is the process of making tests execute
automatically or the automatic generation of tests


Regression testing
is testing code after a change to verify
that code is correct and uncover any side effects.

33