Choosing the right open source test automation tool – an easy ...

subduedlockMechanics

Nov 5, 2013 (3 years and 7 months ago)

56 views


Choosing the right open
source test automation
tool – an easy approach

Alexander Vukovic
2
Agenda
l

Motivation
l

Team
choosing
a
tool

l

An easy
approach

l

Criteria
&
Preselection

l

The
candidates

l

Recommendations

21.11.12
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH
The
team

needs

to
do
manual

regression

testing
, but
that‘s

l

boring

l

long

lasting

l

expensive
l

inefficient

l

not
repeatable

l

not
reproduceable

So stick on
techniques

like

session

based

testing

for

manual

testing
,
but
automate
your
regression

testing
.
21.11.12
3
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH

Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection

○ The
candidates


Recommendations

Which

level

to

automate

and

how

much
?

Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection

○ The
candidates


Recommendations

[
Crisping
, Gregory],
thoughts
on that
here
:
http://blog.goneopen.com/2010/08/test-automation-pyramid-review/


21.11.12
4
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH
versus
A
tool

solves

problems
, or...







creates

new
:-)
21.11.12
5
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH

Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection

○ The
candidates


Recommendations

GUI-
Testing

is

critical

for
agile
teams

l

It

is

hard
...
l

to

learn
,
create
,
maintain

l

to

choose

the

right

test

cases

l

to

avoid

the

cone

l

to

avoid

waste

l

to

avoid

technical

testing

debt

21.11.12
6
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH
And

everything

begins

with

the

tool

you

choose
!

Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection

○ The
candidates


Recommendations

Agile Teams
need
a
practical

solution

l

Available
fast
l

Steep

learning

curve

l

Sustainable

l

As
low

cost

as

possible

21.11.12
7
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH

Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection

○ The
candidates


Recommendations

Open Source Software...
l

Found
fast
l

Downloaded

immediatly

l

No

salesmen

necessary

l

Well

documented

(
sometimes

J
)

l

Lower
Total
Cost

of

Ownership
21.11.12
8
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH

Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection

○ The
candidates


Recommendations

But
no
light
without

shadows
...
l

How

big

is

the

community

using

this

tool
?
l

How

sustainable

is

future

development

of

the

tool
?
l

Is

there

commercial

support

available
?
l

Is

there

any

danger

of
a
closed-source

conversion
?
l

Are
there

trainings

available
?
l

Whats

the

available

timebox

to

choose
a
tool
?
l

Who
coaches

the

team
?
21.11.12
9
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH

Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection

○ The
candidates


Recommendations

An easy
approach

to
find
the

right

tool

l

Do
it
in
sprint
0
l

Do
it

timeboxed
(
approx
.
1
day
)
l

Choose

systematically

l

Based
on
common

and

context

criterias

l

Look
at

the

whole
open
source

market

21.11.12
10
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation


Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection

○ The
candidates


Recommendations

Tool
radar
open
source

test

automation

l

Search „
test

automation

SourceForge
:
approx
. 550
projects

OpenSourceTesting.org
: 122
projects

WhichTestingTool.com
: 39
projects

l

Search „open
source

test

automation

-

Google: 3.710.000 Hits
-

Wiki: 16
tools

l

How

to

choose

the

right

one
?
21.11.12
11
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation


Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection

○ The
candidates


Recommendations

An easy
approach

for

how

to

fill

the

timebox
...
1. Define context
2. Define
criterias

3.
Preselection

4. Shortlist & detailed
eval

5. Practical feasibility
6.
Hitlist

7. Team decision
21.11.12
12
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation


Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection

○ The
candidates


Recommendations

Step
1:
Define

Context

In
our
sample,
the

most

common

context

l

Two
or
three

tier

architecture

l

Web
application

l

Volatile
browsers

l

Dynamic
content

-

Ajax-
toolkits

-

Flash
-

HTML5
21.11.12
13
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach



Criteria
&
Preselection

○ The
candidates


Recommendations

Step
2:
Define

Criterias

Three
sample
projects
...
21.11.12
14
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach



Criteria
&
Preselection

○ The
candidates


Recommendations

Step
2:
Define

Criterias

Project-
specific

criterias
(sample)
l

Common


Handling &
usability



Reporting


Documentation


Community
support



Commercial
support



License

models



Necessary

skills

l

Test
automation



Architecture



Datadriven-testing



Capturing



Script
language



Pre-defined

functions



Script-IDE
21.11.12
15
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach



Criteria
&
Preselection

○ The
candidates


Recommendations

Step
3:
Preselection

Three

potentially

good

solutions

for

our

projects
:
Candidates

shortlist
...
Selenium
Sikuli

http://watin.sourceforge.net/
http://seleniumhq.org/
http://www.sikuli.org/
21.11.12
16
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach



Criteria
&
Preselection

○ The
candidates


Recommendations

Step
4: Evaluation
of

preselection
in
detail

WatiN
– Common
criterias

21.11.12
17
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection



The
candidates


Recommendations

Step
4: Evaluation
of

preselection
in
detail

WatiN
– Project
criterias

21.11.12
18
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection



The
candidates


Recommendations

19
Step
5:
Practical

Feasibility

Demo
WatiN

21.11.12
Choose the right open source test automation tool - an easy approach. (c) 2012 SEQIS Software
Testing GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection



The
candidates


Recommendations

Step
4: Evaluation
of

preselection
in
detail

Selenium
-
Architecture

Selenium IDE
Selenium

Webdriver
21.11.12
20
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection



The
candidates


Recommendations

Selenium
RC
(
deprecated
)
Step
4: Evaluation
of

preselection
in
detail

Selenium
– Common
Criterias

21.11.12
21
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection



The
candidates


Recommendations

Step
4: Evaluation
of

preselection
in
detail

Selenium
– Project
criterias

21.11.12
22
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection



The
candidates


Recommendations

Step
5:
Practical

feasibility

Demo
Selenium

21.11.12
23
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection



The
candidates


Recommendations

Step
4: Evaluation
of

preselection
in
detail

Sikuli
-
Architecture

Image & Text Recognition








Any
OS or
Browser
21.11.12
24
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection



The
candidates


Recommendations

Step
4: Evaluation
of

preselection
in
detail

Sikuli
– Common
Criterias

21.11.12
25
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection



The
candidates


Recommendations

Step
4: Evaluation
of

preselection
in
detail

Sikuli
– Project
criterias

21.11.12
26
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection



The
candidates


Recommendations

Step
5:
Practical

feasibility

Sikuli

21.11.12
27
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection



The
candidates


Recommendations

Step
6:
Hitlist

Which

tool

for

which

project
?
21.11.12
28
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection



The
candidates


Recommendations

10
things

to

remember
:-)
(1)

Open Source Tools
create

costs
(TCO)
(2)


Define

the

criterias

for
your
project

(3)

Look
at

common

test

automation

criterias

(4)

Do a
tool

evaluation

timebox

before

you

start

to

automate

(5)

Test
the

practical

feasibility

with

the

most

complex

controls
in your
application

21.11.12
29
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection

○ The
candidates



Recommendations

10
things

to

remember
:-)
(6)

Find a compromise between automation-ability and
effort
(7)

Look at the open source environment (Community,
license, sustainability, maintenance costs)
(8)

Use coaching to steepen the learning curve
(9)

Look at experiences of other users
(10)

Choose the right open source tool for your context
to avoid waste and debt
21.11.12
30
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection

○ The
candidates



Recommendations

Looking

forward

to
your
questions

J

21.11.12
31
Choose

the

right
open
source

test

automation

tool
- an easy
approach
. (c) 2012 SEQIS Software
Testing
GmbH



Motivation
○ Team
chooses

tool

○ An easy
approach


Criteria
&
Preselection

○ The
candidates


Recommendations


Choosing the right open
source test automation
tool – an easy approach

Alexander Vukovic