Efficient Techniques for Software Testing

buninnateΛογισμικό & κατασκευή λογ/κού

18 Νοε 2013 (πριν από 3 χρόνια και 6 μήνες)

61 εμφανίσεις

Efficient Techniques for Software
Testing

Jeff Lei

CSE@UTA

April 13, 2007

Outline


Software Testing


Concurrency Testing


Testing Concurrent Programs


Testing Synchronization Components


T
-
Way Interaction Testing


Other Work


Model Checking, Protocol Validation, Pervasive
Computing


Software Engineering


Software has become
pervasive

in modern
society


Directly contributes to quality of life


Malfunctions cost billions of dollars every year,
and have severe consequences in a safe
-
critical
environment


All about building
better
software in
better

ways, especially for large
-
scale development


Requirements, design, coding,
testing
,
maintenance, configuration, documentation,
deployment, and etc.

Software Testing


A
dynamic

approach to detecting software
faults


Alternatively,
static analysis

can be performed,
which is however often intractable


Involves
sampling

the input space,
running

the test object, and
observing

the runtime
behavior


Perhaps the most widely used approach in
practice


Labor intensive, and often consumes more than
50% of development cost

Research Problems


How to detect as many
faults

as possible
spend with as little
effort

as possible?


How to generate a good set of test inputs?


How to perform actual test executions?


How to evaluate test outcomes?

Outline


Software Testing


Concurrency Testing


Testing Concurrent Programs


Testing Synchronization Components


T
-
Way Interaction Testing


Other Work


Model Checking, Protocol Validation


Concurrent Programs


Consists of multiple threads of execution that
proceed in parallel


multithreaded programs, multi
-
process (or
distributed) programs


Advantages:


Better resource utilization, increased computation
efficiency, and providing simplified solutions for
many problem domains


Notoriously difficult to test due to their non
-
deterministic behavior

What Is Unique


Non
-
determinism


Multiple executions with the same data input might
display different behaviors


Synchronization


How to ensure that concurrent events are
exercised in an expected order?


Communication


How to exchange information between different
threads?

Concurrency Testing Strategies


Non
-
deterministic testing

-

execute the same
program with the same input for multiple times:


easy, but inefficient, and some errors cannot be detected


Deterministic testing

-

select a set of test sequences
and then force them to be exercised:


can detect more subtle errors, but requires additional effort
for test sequence selection and runtime control


State exploration



explore the state space in a
systematic manner


suffers the well
-
known state explosion problem


state representation is difficult for programs written in a full
-
fledged programming language

Reachability Testing


Combines
non
-
deterministic

and
deterministic

testing


Generating test sequences
dynamically
, without
constructing any static model


Dealing with
partial orders

directly, and thus no
redundant interleavings!


Can also be considered as a
state exploration

technique


Exercising every possible synchronization
sequence of a program with a given input

The Framework

1.
Execute a program P with a given input
non
-
deterministically

to collect a trace Q

2.
Identify the race conditions in Q and
compute its
race variants

3.
For each variant, conduct a
prefix
-
based

test
run to collect a new trace Q’

4.
Repeat 2, 3, and 4 for each newly collected
trace Q’.

Example

Main Results


Developed three reachability testing
algorithms


Exhaustive RT, Combinatorial RT, and Random
RT


Built a reachability testing tool called RichTest


Portable implementation, applicable to most
common synchronization constructs


Conducted an empirical evaluation


Performed significantly better than VeriSoft (from
Bell Labs)


Publications: STVR07a, TSE06, ISSRE05

Dinning Philosophers

Outline


Software Testing


Concurrency Testing


Testing Concurrent Programs


Testing Synchronization Components


T
-
Way Interaction Testing


Other Work


Model Checking, Protocol Validation


Synchronization Components


Synchronization logic is often encapsulated
into one or more components


Synchronization components need to be
tested separately


Obtain necessary confidence prior to integration


helps to reduce complexity by incremental testing


A different problem than testing complete
programs


The ways in which a component could be used is
typically unknown

Java Monitor


A high
-
level synchronization construct that
supports data encapsulation and information
hiding


A Java class that defines one or more
synchronized

methods


Mutual exclusion

automatically enforced by
the Java runtime


Conditional synchronization

implemented by
the programmer, using
wait

and
notify

operations


Java Monitor Semantics


Consists of three components: an
entry

queue, a
critical section
, a
waiting

queue


A thread must enter the CS before it executes
a
synchronized

method; it has to wait in the
entry

queue if the CS is not empty


The
wait

and
notify

operations can only be
executed inside the CS


wait: puts the current thread in the waiting queue


notify: awakens one thread in the waiting queue

Example

class

BoundedBuffer

{


private

int

fullSlots=0;
private

int

capacity = 0;


private

int
[] buffer = null;
private

int

in = 0, out = 0;


public

BoundedBuffer
(
int

bufferCapacity) {

1. capacity = bufferCapacity;

2. buffer =
new

int
[capacity];


}


public

synchronized

void

deposit (
int

value) {

3.
while

(fullSlots == capacity) {

4.
try

{ wait(); }
catch

(
InterruptedException

ex) {}



}

5. buffer[in] = value;

6. in = (in + 1) % capacity;

7.
if

(fullSlots++ == 0) {




8. notifyAll();


}






}


public

synchronized

int

withdraw () {

9.
int

value = 0;

10.
while

(fullSlots == 0) {

11.
try

{ wait(); }
catch

(
InterruptedException

ex) {}


}

12. value = buffer[out];

13. out = (out + 1) % capacity;

14.
if

(fullSlots
--

== capacity) {

15. notifyAll();


}

16.
return

value;


}


}

Graphic View

Critical Section

Entry Queue

W

Condition Queue

D

Assume that the buffer is initially empty.

Research Questions


What is unique to testing Java monitors?


Used to synchronize, and thus typically accessed
by, multiple threads simultaneously


Can existing work on OO testing apply?


OO testing methods typically assume a single
-
threaded test driver


How to effectively test Java monitors?


Technical Challenges


To simulate possible scenarios in which a
Java monitor can be accessed, multiple
threads need to be created


Some faults can only be detected by a certain
minimum number of threads. But how many?


Multiple threads can display non
-
deterministic
behavior. How to describe and control thread
behavior?

A State Exploration
-
Based
Approach


Explores the state space of a monitor as an
open

system


Threads introduced on
-
the
-
fly, and as needed, to
simulate race conditions


State abstraction used to ensure termination and
to control state explosion


Thread behavior is controlled at finer granularity
than method calls


Main Results


Developed an algorithm that implements the
state exploration
-
based approach


Built a prototype tool, called
MonitorExplorer
,
and conducted an empirical evaluation


Empirical results suggest that this approach can
be very effective


Ongoing work: Extend to general monitors,
and general synchronization components


Publication: ISSRE06

Outline


Software Testing


Concurrency Testing


Testing Concurrent Programs


Testing Synchronization Components


T
-
Way Interaction Testing


Other Work


Model Checking, Protocol Validation, Pervasive
Computing


Combinatorial Testing


Creates tests by combining different
parameter values


The input space of a test object is represented as
a set of parameters and their values


Advantages: light specification, test input
generation can be fully automated, requires
no access to source code


How to deal with the combinatorial explosion
problem?


T
-
Way Interaction Testing


Motivation: Not every parameter contributes
to every fault


Many faults can be exposed by interactions
involving a few parameters


Given any
t

parameters, every combination of
values of these parameters be covered by at
least one test


Allows faults triggered by at most t parameters to
be detected


Makes an excellent trade
-
off between test
effort and test coverage

T
-
Way Testing
-

Example

P1 P2 P3


0 0 0

0 0 1

0 1 0

0 1 1

1 0 0

1 0 1

1 1 0

1 1 1

P1 P2 P3


0 0 0

0 1 1

1 0 1

1 1 0

Three parameters, each with values 0 and 1

2
-
way testing

exhaustive testing

State of the Art


Greedy construction
:
Each step tries to cover
as many combinations as possible


Involves explicit enumeration of all possible
combinations


Algebraic Construction
:
test sets are
constructed using pre
-
defined rules


Most approaches focus on 2
-
way (or
pairwise) testing

Strategy In
-
Parameter
-
Order


Input: A set of parameters and values;
Output: a
t
-
way

test set


Construct a
t
-
way

test set for the first
t
parameters



Extend the test set to cover each of the
remaining parameters one by one


Horizontal growth

-

extends each existing test by
adding one value for the new parameter


Vertical growth



adds new tests, if needed, to
make the test set complete

Example

(a)

P1 P2 P3

0 0 0

0 0 1

0 1 0

0 1 1

1 0 0

1 0 1

1 1 0

1 1 1

(b)

P1 P2 P3 P4

0 0 0 0

0 0 1 1

0 1 0 2

0 1 1 0

1 0 0 1

1 0 1 2

1 1 0 0

1 1 1 1

P1 P2 P3 P4

0 0 0 0

0 0 1 1

0 1 0 2

0 1 1 0

1 0 0 1

1 0 1 2

1 1 0 0

1 1 1 1

1 0 1 0

0 1 0 1

0 0 1 2

1 1 0 2

* 0 0 2

* 1 1 2

(c)

Horizontal growth

Vertical growth



Four parameters: P1, P2, P3, and P4



P1, P2, and P3 have 2 values



P4 has 3 values

3
-
way test set

Main Results


Developed two algorithms, IPOG and IPOG
-
D, for general t
-
way testing


Has the lowest algorithmic complexity among the
existing algorithms


Built a t
-
way testing tool called FireEye, and
conducted an empirical evaluation


Performed significantly better than existing tools,
e.g., ITCH, TConfig, Jenny, TestVector


Publications: TSE02, ECBS07, STVR07b

TCAS Example

t
-
way

FireEye

ITCH

Jenny

TConfig

TVG

Size

Time

Size

Time

Size

Time

Size

Time

Size

Time

2

100

0.8

120

0.73

108

0.001

108

>1 hour

101

2.75

3

400

0.36

2388

1020

413

0.71

472

>12 hour

9158

3.07

4

1361

3.05

1484

5400

1536

3.54

1476

>21 hour

64696

127

5

4219

18.41

NA

>1 day

4580

43.54

NA

>1 day

313056

1549

6

10919

65.03

NA

>1 day

11625

470

NA

>1 day

1070048

12600

TCAS:
Seven 2
-
value parameters, two 3
-
value
parameters, one 4
-
value parameter, two 10
-
value
parameters


Outline


Software Testing


Concurrency Testing


Testing Concurrent Programs


Testing Synchronization Components


T
-
Way Interaction Testing


Other Work


Model Checking, Protocol Validation, Pervasive
Computing


Other Work


Model checking: How to deal with the state explosion
problem?


True concurrency model vs interleaving
-
based concurrency
model


Publication: FM05


Protocol validation: A more specific problem than
model checking


Allows multiple threads to proceed simulatenously during
state exploration


Publication: CJ06


Pervasive computing


Context
-
awareness: How to derive high level events from a
sequence of low level events?


Power
-
constraints: How to reduce the power consumption
on small mobile devices?

References


[STVR07a] Y. Lei, R. Carver, R. Kacker, and D. Kung. A Combinatorial Testing Strategy for
Concurrent Programs. Accepted for publication in
Software Testing, Verification, and
Reliability
, 2007.


[STVR07b] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence. Two deterministic
strategies for multi
-
way software testing.
Software Testing, Verification, and Reliability
,
pending review.


[ECBS07] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence. A general strategy for multi
-
way software testing. Int’l Conf. on Engineering of Computer
-
Based Systems, 2007.


[ISSRE06] Yu Lei, R. H. Carver, D. Kung, V. Gupta, M. Hernandez. A State Exploration
-
Based Approach to Testing Java Monitors. Int’l Symposium Software Reliability Engineering,
256
-
265, 2006.


[CJ06] Q. Ye, Y. Lei, and D. Kung. A Blocking
-
Based Approach to Protocol Validation, The
Computer Journal, 49:541
-
553, 2006.



[TSE06] Y. Lei and R. Carver, Reachability testing of concurrent programs,
IEEE
Transactions On Software Engineering
, 32(6):382
-
403, 2006.


[FM05] Y. Lei and P. Iyer. An Approach to Unfolding Asynchronous Communication
Protocols,
Proc. 13th Intl. Symp. on Formal Methods
, pp. 334
-
349, 2005.


[TSE02] K. C. Tai and Y. Lei, "A test generation strategy for pairwise testing",
IEEE
Transactions on Software Engineering
, Vol. 28, No. 1, pp. 109
-
111., Jan. 2002.