Model Based Test Generation

translatoryazooInternet and Web Development

Nov 12, 2013 (3 years and 6 months ago)

69 views

Advanced Modeling


Model Based Test Generation

and

A
bstract
s
tate
m
achine
L
anguage

Keith Stobie

Test Architect

XML Web Services

Microsoft Corporation



2

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


What’s a Model?

A model:

Is an
abstraction

or simplified
representation

of the system
from a particular perspective

Supports investigation, discovery,
explanation, prediction, or
construction

May be expressed as a
description, table, graphical
diagram, or quantitative
mathematical model

Is
not

necessarily
comprehensive

From: PNSQC 2002 “
Modeling: A Picture's Worth 1000 Words”
Copyright (c) 2002, Quality Tree Software, Inc.

3

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


Models in Everyday Life

Inspired by :“Modeling: A Picture's
Worth 1000 Words” Copyright (c)
2002, Quality Tree Software, Inc.

4

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


Examples of Models


Flow Charts


Data Flow Diagrams


Entity
-

Relationship Diagrams


State Diagrams


Deployment Diagrams


Class Diagrams


Use Cases


Activity Diagrams


State Transition Tables

Inspired by :“Modeling: A Picture's
Worth 1000 Words” Copyright (c)
2002, Quality Tree Software, Inc.

5

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


Model Based Test Generation

NOT

just model based input generators
(pair
-
wise, random data, grammars, etc.)

NOT

modeling tool
s (e.g. UML)

set of

Test Cases




sequence of stimuli



expected responses

Formal
Model

Test
Generation

Directives

Generator

6

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


Modeling

Modeling

Validation

Refinement

Verification

Specification

AsmL

Implementation

C/C++/C#

Product Idea

/ Specification

Are you building
the product
right
?

Are you
building the
right

product?

What
product
are you
building?

7

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


Benefits
(GOTCHA)

Testing pilot amounted to
10 person months
,
including the
GOTCHA

learning curve.

This time is
less than

the time used to
originally

test the subsystem (
12 person
months
).

revealed 2 documentation changes defects

15 of the 18 defects found

by the original
function test effort would also have been found
by this pilot test effort
.”


From:
Using a model
-
based test generator to
test for standard conformance

8

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


Benefits
(TestMaster)

“In a typical application of this approach, test
engineer
productivity has increased by a
factor of five to ten

over conventional manual
approaches (Figure 5)”.


From:
Model Based Testing

9

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved




FSM: Start State & action

Start state (Entry Point)

FilesShown: 0

FileSelect: NotSelected

ExitPoint: False

Invert_File
Selection
From: Intelligent Test Automation
by Harry Robinson

Software Testing & Quality Engineering (STQE)
magazine.
Sept/Oct 2000

Action
10

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


other

States

State
Invert_File
Selection
start

Create_File
FilesShown: 0
FileSelected: NotSelected
ExitPoint: True
End State (Exit Point)
FilesShown: 1
FileSelected: Selected;
ExitPoint: False
Delete_File
11

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


Simple FSM Model

12

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


File
-
in
-
a
-
directory Model

exit
Invert_File
Selection
Invert_File
Selection
Invert_File
Selection
Delete_
File
Delete_File
Create_File
FilesOpened 0
FileSelected: NotSelected
ExitPoint: False
FilesOpened: 1
FileSelected: Selected
ExitPoint: False
FilesOpened 1
FileSelected: NotSelected;
ExitPoint: False
FilesOpened 0
FileSelected: NotSelected
ExitPoint: True
start
Start State (Entry Point)
State
Action
End State (Exit Point)
13

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


Create

Invert Selection

Invert Selection

Delete

Walk Every State
-
Change

14

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


File
-
in
-
a
-
directory Model

exit
Invert_File
Selection
Invert_File
Selection
Invert_File
Selection
Delete_
File
Delete_File
Create_File
FilesOpened 0
FileSelected: NotSelected
ExitPoint: False
FilesOpened: 1
FileSelected: Selected
ExitPoint: False
FilesOpened 1
FileSelected: NotSelected;
ExitPoint: False
FilesOpened 0
FileSelected: NotSelected
ExitPoint: True
start
Start State (Entry Point)
State
Action
End State (Exit Point)
C2

C1

C3

C4

15

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


A: Invert Selection

B: Create; Delete

C: Create; Invert Selection; Invert Selection; Delete

Test Every Walk < 5 Steps

16

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


Test Model Toolkit

Source: Testing Internet Explorer:


From Underdog To Best
-
In
-
Class Browser

17

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


Test Model Toolkit
-

Impact

Used by over 20 teams at MS

Time to automate fell by as much as 88%


BizTalk
-

Took 1 week to generate a set of test cases
that took 8 weeks by hand

Code coverage increased by as much as 50%


One subteam in Shell


In 2 weeks increased code
coverage from 20% to 75%, while the number of test
cases increased from ~75 to ~2000.

Many spec and implementation bugs are
uncovered in the course of model creation


Source: Testing Internet Explorer:


From Underdog To Best
-
In
-
Class Browser

18

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


The AsmL
modeling

toolkit

Abstract state machine Language is a
powerful modeling language


AsmL provides


Authoring via MS Word and XML


Simulation and interoperability via COM and .NET


Automatic test oracle


Semi
-
automatic test case generation


Integration with Microsoft’s test platform


19

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


AsmL: Getting Started, ‘sort’

example

var

A
as

Seq
of

Integer = []

Swap()

choose

i
in

indices(A),



j
in

indices(A)


where
i < j
and
A(i) > A(j)



A(i) := A(j)



A(j) := A(i)


sort()

step
until fixpoint



swap()

A = [2,3,1]

A = [1,2,3]

A = [2,
1
,
3
]

Non
-
determinism

A = [
1
,3,
2
]

Parallelism

20

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


AsmL: Predefined Data Types

Sets



menu =
{
"
ham
","
cheese
", "
blt
"
}




Sequences

random

=

[
3,

6,

7,

1,

5
]

//
as

Seq

of

Integer

Maps


directory
as Map of
String
to
Integer =


{
“police”
-
>

112 , “info”
-
>

118
}

Tuples

nameAndAge

=
(
“John”,39
)

//
(
String
,
Integer)



21

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


The cool part: Asml’s high
-
level
-
ness

comprehension

to pick 1 of the oldest filters in a
Most Recently Used (MRU) cache.


class
CacheFilter


var
lastAccessed

as
DateTime

var
MruSet

as Set of
CacheFilter

= {}



let

victim =
any

i

| i
in

MruSet
where

(
forall

j

in

MruSet
holds



//
j access time newer or equal i



(Compare(
j.lastAccessed,





i.lastAccessed
) >= 0)
)


22

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


XML & XPath

<DocumentRoot>


<ParentElement Attrib1=1 attrib2=2 >



<childElementA/>



<childElementB/>


</ParentElement>

</DocumentRoot>

Child Axis

Attribute Axis

XPath:

/DocumentRoot/ParentElement/
childElementB

/DocumentRoot/ParentElement/@attrib2

23

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


XPath Model


axis nodes

We interpret an axis as producing either

1) all of the children of a given node or

2) all of the attribute children of a given node.

EvaluateAxis (a
as

Axis, c
as

Context)
as

Seq
of

Node


let

x = ContextNode(c)


match

a


Child :
return

NodeChildren(x)


Axis.Attribute :
return

NodeAttributes(x)


_ : error "
EvaluateAxis: internal error
"

a=Child


Context=ParentNode

Returns:

childElementA

childElementB

24

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


XPath model


node test

An axis path (for example, "child::para") has a two
-
step evaluation.

The axis ("child") is used to generate a list of
matching nodes,

which is subsequently filtered to include only those
nodes that pass the
given node test

("para").






ap
as

AxisPath:


let

nodes = EvaluateAxis(ap.axis, c)


return



[nodes(i
-

1) | i
in

[1..Length(nodes)]


where

PassesNodeTest(ap.test,







Context(nodes, i))]

25

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


Using models for test

Model

Implementation

Test Oracle

Test Cases

Are run by

Provides

actual results for

Pass

No pass

Provides expected

results for

Generates

User

Info

26

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


Tools & Techniques

Tool&Language

Test Technique

TMT

FSM

ASML

ASM

State Walk

Test Case Gen

Queries

e.g. all Nodes

all Edges

Test Properties


=> FSM


=> state walk

Simulation

Test Oracle

Dynamic
Traversal

Model Manager

Model Coverage
via random Test
Case Gen

currently
manual


random paths

ASML test case
generation with
feedback

27

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


AsmL
-
based test harness

AsmL

model

Test

harness

Implementation

under test

Side
-
by
-
side runs

Discrepancies
automatically
flagged

Any

client

28

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


AsmL test case generation
-

1

Solves two problems

How to find “interesting” parameter
values and combinations

How to find “interesting” call sequences


Approach

Use information derived from the model
to optimize the number of test cases

29

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


AsmL test case generation
-

2

Divide state space according to state predicates


System “covers” the state predicates


Driven by input generator (parameter selection) or


Driven by enabledness (call sequences)


Tester provides inputs, gives interesting state
predicates, directs the search

30

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


AsmL

Model
M


The dual role of ASM models

Test cases

Guided

generation

(Param

Select

with

Feedback)

Reduce

FSM

tool

IUT

Call next action

Get state

AsmL

Model
M

Testing harness

Is
S

valid according

to
M
?

Fail
: witness

Pass

S

view

view

view

FSM

31

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


AsmL test case generation
-

3

Generate only as many test cases as needed to
have full coverage of the specification


Generator

Model

Input

Coverage criteria


Branch: Every “branch” of the spec must be
taken


Modified condition: independent atomic
conditions


Off
-
by
-
One: off
-
by
-
one errors


Hyper
state coverage: Every
abstract

state
must be visited

Feedback

32

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


Model Branch Coverage
undecided, 5
Generator
covered, 104
dead, 36
manually
covered, 9
XPath Model coverage

via random Param Test Generation

Model Statistics:

1,003,638 tests generated
(115Mb file)

Only 37 tests are needed
to cover all 113
(reachable 104+9)
branches!

33

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


AsmL Test (asmlt)

asmlT can be used to:

1.
Find interesting sequences of method calls


finds the shortest route in a generated FSM that uses every
edge to traverse the states

2.
Find interesting parameters for each method call

Access Driven Filtering
(ADF)


you write a predicate (boolean function) to decide if a given
parameter is interesting

3.
Perform a conformance test

Compare Model with Implementation

34

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


References

Model
-
Based Test Generation Tools
, Alan Hartman,

AGEDIS CONSORTIUM

http://www.agedis.de/documents/ModelBasedTestGenerationTools_cs.pdf



Finite State Model
-
Based Testing on a Shoestring
, Harry Robinson

http://
www.model
-
based
-
testing.org
/shoestring.htm



Using a model
-
based test generator to test for standard conformance


E. Farchi, et al IBM SYSTEMS JOURNAL, VOL 41, NO 1, 2002

http://researchweb.watson.ibm.com/journal/sj/411/farchi.pdf


Model Based Testing
,
L. Apfelbaum, et al, Software Quality Week
Conference in May, 1997.

http://
www.model
-
based
-
testing.org
/sqw97.pdf



Modeling for Testers: A Picture’s Worth 1000 Words
,

Elisabeth Hendrickson, Quality Tree Software, Inc.

http://www.raddue.com/pnsqc/Pnsqc02.pdf




Good reference for
existing tools

35

SASQAG 16Jan03

AsmL Modelling

©

2003 Microsoft Corporation, All Rights Reserved


More References

Abstract State Machine Language
, Microsoft Research

http://research.microsoft.com/fse/asml/



The AsmL Test Generator tool
, Microsoft Research

http://
research.microsoft.com/fse
/AsmL/doc/AsmLTester.html



Testing Internet Explorer:


From Underdog To Best
-
In
-
Class Browser

Jason Taylor, Microsoft Corporation at
StarWest 2002


Intelligent Test Automation
,
Harry Robinson

Software Testing & Quality Engineering (STQE) magazine
Sept/Oct 2000

http://www.model
-
based
-
testing.org/robinson.pdf


Confessions of a Modeling Bigot: Parts
III
-
IV

Michael Corning

, ASPToday Dec02, Jan03

http://www.asptoday.com/content.asp?id=2043


© 2003 Microsoft Corporation. All rights reserved.

This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.