Systematic Software Testing Using Test Abstractions

aniseedsplashSoftware and s/w Development

Aug 15, 2012 (5 years and 1 month ago)

395 views

Systematic Software Testing

Using Test Abstractions

Darko Marinov

RIO Summer School

Rio Cuarto, Argentina

February 2011

Examples of Structurally Complex Data

<library>


<book year=2009>


<title>T1</title>


<author>A1</author>


</book>


<book year=2010>


<title>T2</title>


<author>A2</author>


</book>

</library>

/library/book[@year<2010]/title

XML document

web sites

s1

s2

s4

s3

s5

0

1

3

2

red
-
black tree

Java program

class

A {


int

f
;

}


class

B
extends

A {


void

m() {


super
.
f

= 0;


}

}

Running Example: Inheritance Graphs

Java

program

interface
I {


void
m();

}


interface

J {


void

m();

}


class
A

implements
I {


void
m() {}

}


class

B
extends
A


implements
I,J {


void

m() {}

}

.

I

J

A

B

Inheritance

graph

Properties

1. DAG

2. ValidJava

Testing Setup






Examples of code under test


Compiler or refactoring engines, input: programs


Abstract data type, input/output: data structure


Web traversal code, input: web topology


XML processing code, input/output: XML document

code

test

generation

test

oracle

0

1

3

2

0

3

2

0

3

2

0

3

pass

fail

inputs

outputs

Test Generation

tool

code

0

1

3

2

0

3

2

inputs

Fully automatic

Manual

0

1

3

2

0

3

2

inputs

tool

Semi automated

input set

description

0

1

3

2

0

3

2

inputs

Test Abstractions


Each
test abstraction

describes a set of
(structurally complex) test inputs


Example: inheritance graphs with up to N nodes


Workflow


Tester manually writes test abstractions


Tool automatically generates test inputs


Benefits


No need to manually write large number of test inputs


Useful for test generation, maintenance, and reuse

Key Questions for Test Abstractions

tool

test

abstraction

0

1

3

2

0

3

2

inputs

1. Which test inputs

to generate?

2. What language to use
for test abstractions?

3. How to generate test inputs
from abstractions?

4. How to check test outputs?

5. How to map failures to faults?

Which Inputs to Generate?


Bounded
-
exhaustive testing


Generate
all inputs

within given (small) bounds


Rationale: small
-
scope hypothesis
[Jackson & Damon ISSTA’96]


Found bugs in both academia and industry


Java compilers, model checkers…
[Gligoric et al. ICSE’10]


Refactoring engines in
Eclipse/NetBeans

[Daniel et al. FSE’07]


Web traversal code from
Google

[Misailovic et al. FSE’07]


XPath compiler at
Microsoft

[Stobie ENTCS’05]


Fault
-
tree analyzer for
NASA

[Sullivan et al. ISSTA’04]


Constraint solver, network protocol
[Khurshid & Marinov J
-
ASE’04]

How to Describe/Generate Inputs?


Filtering test abstractions
[SOFTMC’01, ASE’01, FME’02, ISSTA’02,
OOPSLA’02, SAT’03, MIT
-
TR’03, J
-
ASE’04, SAT’05, LDTA’06, ALLOY’06, ICSE
-
DEMO’07, STEP’07,
FSE’07, ISSTA’08, ICST’09, CSTVA’10]

search

input
s

filters

bounds


Testers describe
what

properties
test inputs satisfy


Tool:
search


Generating test abstractions
[FSE’07, FASE’09]

execute

input
s

generators

bounds


Testers describe
how

to generate
test inputs


Tool:
execution

Previously Explored Separately

filters

generators

Inheritance

graph

DAG

X



ValidJava



X


Other

Java

properties

property 3



X







property N

X




Some properties easier/harder as filters/generators


Key challenge for combining: search vs. execution

UDITA: Combined Both [ICSE’10]

filters

generators

Inheritance

graph

DAG

X



ValidJava



X


Other

Java

properties

property 3



X







property N

X




Extends Java with non
-
deterministic choices


Found bugs in Eclipse/NetBeans/javac/JPF/UDITA

Outline


Introduction


Example


Filtering Test Abstractions


Generating Test Abstractions


UDITA: Combined Filtering and Generating


Evaluation


Conclusions

Example: Inheritance Graphs

class
IG

{


Node[ ] nodes;


int size;


static class Node {


Node[ ] supertypes;


boolean isClass;


}

}

Properties


DAG


Nodes in the graph should
have no direct cycle


ValidJava


Each class has at most one
superclass


All supertypes of interfaces
are interfaces

Example Valid Inheritance Graphs

I

J

A

B

G2

C

G1

G3

A

C

B

G4

C

D

J

I

J

A

B

C

G5

Example Invalid Inheritance Graphs

I

A

B

G2

C

G1

G3

A

C

B

G4

C

J

I

J

A

B

C

G5

J

cycle

D

class
implements
class

class
extends two
classes

interface
extends
class

class
extends
interface

(In)valid Inputs


Valid inputs =
desired

inputs


Inputs on which tester wants to test code


Code may produce a regular output or an exception


E.g. for Java compiler


Interesting (il)legal Java programs


No precondition, input can be any text


E.g. for abstract data types


Encapsulated data structure representations


Inputs need to satisfy invariants


Invalid inputs = undesired inputs

Outline


Introduction


Example


Filtering Test Abstractions


Generating Test Abstractions


UDITA: Combined Filtering and Generating


Evaluation


Conclusions

Filtering Test Abstractions


Tool requires:


Filters: encode
what

properties inputs satisfy


Bounds


Tool provides:


All inputs within bounds that satisfy the properties

search

input
s

filters

bounds

Language for Filters


Each filter takes an input that can be valid or invalid


Returns a boolean indicating validity


Experience from academia and industry showed
that using a new language makes adoption hard


Write filters in standard
implementation language

(Java, C#, C++…)


Advantages


Familiar language


Existing development tools


Filters may be already present in code


Challenge: generate inputs from imperative filters

Example Filter: DAG Property

boolean
isDAG
(IG ig) {


Set<Node> visited = new HashSet<Node>();


Set<Node> path = new HashSet<Node>();


if (ig.nodes == null || ig.size != ig.nodes.length)


return false;


for (Node n : ig.nodes)


if (!visited.contains(n))


if (!isAcyclic(n, path, visited)) return false;


return true;

}

boolean isAcyclic(Node node, Set<Node> path,


Set<Node> visited) {


if (path.contains(node)) return false;


path.add(node); visited.add(node);


for (int i = 0; i < supertypes.length; i++) {


Node s = supertypes[i];


for (int j = 0; j < i; j++)


if (s == supertypes[j]) return false;


if (!isAcyclic(s, path, visited)) return false;


}


path.remove(node);


return true;

}

Input Space


All possible object graphs with an
IG

root

(obeying type declarations)


Natural input spaces relatively easy to enumerate


Sparse: # valid test inputs << # all object graphs

0

1

3

2

0

3

2

0

3

0

3

0

3

0

3

0

3

0

3

0

3

0

3

0

3

0

3

Bounded
-
Exhaustive Generation


Finite (small) bounds for input space


Number of objects


Values of fields



Generate
all

valid inputs
within

given
bounds


Eliminates systematic bias


Finds all bugs detectable within bounds



Avoid equivalent (isomorphic) inputs


Reduces the number of inputs


Preserves capability to find all bugs

Example Bounds


Specify number of objects for each class


1 object for IG: { IG
0
}


3 objects for Node: { N
0
, N
1
, N
2

}


Specify set of values for each field


For size: { 0, 1, 2, 3 }


For supertypes[i]: { null, N
0
, N
1
, N
2

}


For isClass: { false, true }


Previously written with special library


In UDITA:
non
-
deterministic calls

for initialization

Bounds Encoded in UDITA

IG initialize(int N) {


IG ig = new IG(); ig.size = N;


ObjectPool<Node>

pool = new ObjectPool
<Node>(N);


ig.nodes = new Node[N];


for (int i = 0; i < N; i++) ig.nodes[i] =
pool.getNew
();


for (Node n : nodes) {


int num =
getInt
(0, N − 1);


n.supertypes = new Node[num];


for (int j = 0; j < num; j++)


n.supertypes[j] =
pool.getAny
();


n.isClass =
getBoolean
(); }

return ig; }


Java with a few extensions for non
-
determinism

static void mainFilt(int N) {


IG ig = initialize(N);


assume
(isDAG(ig));


assume
(isValidJava(ig));


println(ig); }

IG
0

N
0

N
1

N
2

size

s[0]

s[1]

isC

s[1]

s[0]

s[1]

isC

s[0]

isC

N
1

N
1

null

null

null

null

3

2

3

1

Example Input Space


1 IG object, 3 Node objects: total 14 fields

(“nodes” and array length not shown)

null

N
0

N
1

N
2

null

N
0

N
1

N
2

false

true

null

N
0

N
1

N
2

null

N
0

N
1

N
2

false

true

null

N
0

N
1

N
2

null

N
0

N
1

N
2

false

true


4 * (4 * 4 * 2 * 3)
3

> 2
19

inputs, but < 2
5

valid

0


1


2


3

IG
0

N
0

N
1

N
2

size

s[0]

s[1]

isC

s[1]

s[0]

s[1]

isC

s[0]

isC

Input Generation: Search


Na
ï
ve “solution”


Enumerate entire (finite) input space, run filter on
each input, and generate input if filter returns true


Infeasible for sparse input spaces (#valid << #total)


Must reason about behavior of filters


Our previous work proposed a solution


Dynamically monitors execution of filters


Prunes

large parts of input space for each execution


Avoids isomorphic inputs

Outline


Introduction


Example


Filtering Test Abstractions


Generating Test Abstractions


UDITA: Combined Filtering and Generating


Evaluation


Conclusions

Generating Test Abstractions


Tool provides:


Generators: encode
how

to generate inputs


Bounds


Tool requires:


All inputs within bounds (that satisfy the properties)

execute

input
s

generators

bounds

Example Generator: DAG Property

void mainGen(int N) {


IG ig = initialize(N);


generateDAG
(ig);


generateValidJava
(ig);


println(ig); }

void
generateDAG
(IG ig) {


for (int i = 0; i < ig.nodes.length; i++) {


int num =
getInt
(0, i);


ig.nodes[i].supertypes = new
Node[num];


for (int j = 0, k = −1; j < num; j++) {


k =
getInt
(k + 1, i − (num − j));


ig.nodes[i].supertypes[j] = ig.nodes[k];


}


}

}

N0

N1

N2

Outline


Introduction


Example


Filtering Test Abstractions


Generating Test Abstractions


UDITA: Combined Filtering and Generating


Evaluation


Conclusions

Filtering vs. Generating: DAG Property

boolean
isDAG
(IG ig) {


Set<Node> visited = new HashSet<Node>();


Set<Node> path = new HashSet<Node>();


if (ig.nodes == null || ig.size != ig.nodes.length)


return false;


for (Node n : ig.nodes)


if (!visited.contains(n))


if (!isAcyclic(n, path, visited)) return false;


return true;

}

boolean isAcyclic(Node node, Set<Node> path,


Set<Node> visited) {


if (path.contains(node)) return false;


path.add(node); visited.add(node);


for (int i = 0; i < supertypes.length; i++) {


Node s = supertypes[i];


for (int j = 0; j < i; j++)


if (s == supertypes[j]) return false;


if (!isAcyclic(s, path, visited)) return false;


}


path.remove(node);


return true;

}

void
generateDAG
(IG ig) {


for (int i = 0; i < ig.nodes.length; i++) {


int num =
getInt
(0, i);


ig.nodes[i].supertypes = new Node[num];


for (int j = 0, k = −1; j < num; j++) {


k =
getInt
(k + 1, i − (num − j));


ig.nodes[i].supertypes[j] = ig.nodes[k];


}


}

}


Filtering arguably harder
than generating for DAG


20+ vs. 10 LOC


However, the opposite
for ValidJava property

UDITA Requirement: Allow Mixing

filters

generators

DAG

X



ValidJava



X

UDITA

input
s

generators

bounds

filters

Other Requirements for UDITA


Language for test abstractions


Ease of use
: naturally encode properties


Expressiveness
: encode a wide range of properties


Compositionality
: from small to large test abstractions


Familiarity
: build on a popular language


Algorithms and tools for
efficient generation

of
concrete tests from test abstractions


Key challenge: search (filtering) and execution
(generating) are different paradigms

UDITA Solution


Based on a popular language (Java) extended with
non
-
deterministic choices


Base language allows writing filters for search/filtering
and generators for execution/generating


Non
-
deterministic choices for bounds and enumeration


Non
-
determinism for primitive values:
getInt/Boolean


New language abstraction for objects:
ObjectPool


class ObjectPool<T> {


public ObjectPool<T>(int size, boolean includeNull) { ... }


public T
getAny
() { ... }


public T
getNew
() { ... } }

UDITA Implementation


Implemented in Java PathFinder (JPF)


Explicit state model checker from NASA


JVM with backtracking capabilities


Has non
-
deterministic call: Verify.getInt(int lo, int hi)


Default/eager generation too slow


Efficient generation by
delayed choice execution


Publicly available JPF extension

http://babelfish.arc.nasa.gov/trac/jpf/wiki/projects/jpf
-
delayed


Documentation on UDITA web page

http://mir.cs.illinois.edu/udita

Delayed Choice Execution


Postpone choice of values until needed







Lightweight symbolic execution


Avoids problems with full blown symbolic execution


Even combined symbolic and concrete execution has
problems, especially for complex object graphs

Eager


int x =
getInt
(0, N);



y = x;



… = y; // non
-
copy

Delayed


int x =
Susp
(0, N);



y = x;



force
(y);

… = y; // non
-
copy

non
-
determinism

Object Pools


Eager implementation of getNew/getAny by
reduction to (eager) getInt


Simply making getInt delayed does not work because
getNew is stateful


We developed a novel algorithm for delayed
execution of object pools


Gives equivalent results as eager implementation


Polynomial
-
time algorithm

to check satisfiability of
getNew/getAny constraints (disequality from a set)


Previous work on symbolic execution typically encodes into
constraints whose satisfiability is NP
-
hard (disjunctions)

Outline


Introduction


Example


Filtering Test Abstractions


Generating Test Abstractions


UDITA: Combined Filtering and Generating


Evaluation


Conclusions

Evaluation


UDITA


Compared Delayed and Eager execution


Compared with a previous Generating approach


Compared with Pex (white
-
box)


Case studies on testing with test abstractions


Some results with filtering test abstractions


Some results with generating test abstractions


Tested refactoring engines in Eclipse and NetBeans,
Java compilers, JPF, and UDITA

Eager vs. Delayed

.

Generating vs. UDITA: LOC

Generating vs. UDITA: Time

UDITA vs. Pex


Compared UDITA with Pex


State
-
of
-
the
-
art symbolic execution

engine from MSR


Uses a state
-
of
-
the
-
art constraint solver


Comparison on a set of data structures


White
-
box testing: tool monitors code under test


Finding seeded bugs


Result: object pools help Pex to find bugs


Summer internship to include object size in Pex

Some Case Studies with Filtering


Filtering test abstractions used at Microsoft


Enabled finding numerous bugs in tested apps


XML tools


XPath compiler (10 code bugs, test
-
suite augmentation)


Serialization (3 code bugs, changing spec)


Web
-
service protocols


WS
-
Policy (13 code bugs, 6 problems in informal spec)


WS
-
Routing (1 code bug, 20 problems in informal spec)


Others


SSLStream


MSN Authentication




Some Case Studies with Generating


Generating test abstractions used to test Eclipse
and NetBeans refactoring engines [FSE’07]


Eight refactorings: target field, method, or class


Wrote about 50 generators


Reported
47 bugs


21 in Eclipse: 20 confirmed by developers


26 in NetBeans: 17 confirmed, 3 fixed, 5 duplicates,

1 won't fix


Found more but did not report duplicate or fixed


Parts of that work included in NetBeans

Typical Testing Scenario


Create a model of test inputs (e.g., Java ASTs)


Write filters/generators for valid inputs


UDITA generates valid inputs


Translate from model to actual inputs (pretty
-
print)


Run on two code bases, compare outputs

pass

fail

=?

one

code

another

code

<library>


<book year=2001>


<title>T1</title>


<author>A1</author>


</book>


<book year=2002>


<title>T2</title>


<author>A2</author>


</book>


<book year=2003>


<title>T3</title>


<author>A3</author>


</book>

</library>

/library/book[@year<2003]/titl


<title>T1</title>

<title>T2</title>



<title>T1</title>

<title>T2</title>


UDITA

bounds

filters/generators

pretty

printer

Some More Bugs Found


Eclipse vs. NetBeans refactoring engines


2 bugs in
Eclipse

and 2 bugs in
NetBeans


Sun javac compiler vs. Eclipse Java compiler


2 reports (still unconfirmed) for
Sun javac


Java PathFinder vs. JVM


6 bugs in
JPF

(plus 5 more in an older version)


UDITA Delayed vs. UDITA Eager


Applying tool on (parts of) itself


1 bug in
UDITA
(patched since then)

Example Bug

Generated program

import
java.util.List;

class
A
implements
B, D {


public
List m() {


List l =
null
;


A a =
null
;


l.add(a.m());


return

l; } }


interface

D {


public
List m();

}


interface
B

extends
C {


public
List m();

}


interface

c {


public
List m();

}

Incorrect refactoring

import
java.util.List;

class
A
implements
B, D {


public
List<List> m() {


List<List<List>>

l =
null
;


A a =
null
;


l.add(a.m());


return

l; } }


interface

D {


public
List<List> m();

}


interface
B

extends
C {


public
List<List> m();

}


interface

c {


public
List<List> m();

}

Outline


Introduction


Example


Filtering Test Abstractions


Generating Test Abstractions


UDITA: Combined Filtering and Generating


Evaluation


Conclusions

Summary


Testing is important for software reliability


Necessary step before (even after) deployment


Structurally complex data


Increasingly used in modern systems


Important challenge for software testing


Test abstractions


Proposed model for complex test data


Adopted in industry


Used to find bugs in several real
-
world applications

Benefits & Costs of Test Abstractions


Benefits


Test abstractions can
find bugs

in real code (e.g., at
Microsoft or for Eclipse/NetBeans/javac/JPF/UDITA)


Arguably
better than manual

testing


Costs

1. Human time for

writing test abstractions

(filters or


generators)
[UDITA, ICSE’10]

2. Machine time for generating/executing/checking tests


(human time
waiting for the tool
)
[FASE’09]

3. Human time for
inspecting failing tests
[FASE’09]


Collaborators from U. of Illinois and other schools


Brett Daniel


Danny Dig


Kely Garcia


Vilas Jagannath


Yun Young Lee


Funding


NSF CCF
-
0746856, CNS
-
0615372, CNS
-
0613665


Microsoft gift

Credits for Generating TA and UDITA


Milos Gligoric (Belgrade
-
>Illinois)


Tihomir Gvero (Belgrade
-
>EPFL)


Sarfraz Khurshid (UT Austin)


Viktor Kuncak (EPFL)

Test Abstractions


Test abstractions


Proposed model for structurally complex test data


Adopted in industry


Used to find bugs in several real
-
world applications
(e.g., at Microsoft, Eclipse/NetBeans/javac/JPF/UDITA)


UDITA
: latest approach for test abstractions


Easier to write test abstractions


Novel algorithm for faster generation


Publicly available JPF extension

http://mir.cs.illinois.edu/udita