Test Automation. Let's talk business.

saucecopywriterInternet and Web Development

Feb 2, 2013 (4 years and 5 months ago)

97 views

Test Automation


Let’s talk business


Igor Gershovich

Connected Testing, Inc.

www.connectedtesting.com

igershovich@connectedtesting.com

Overview


Test Automation Myths


Return on Investment (ROI)


Frameworks

Test Automation Myths

Myth #1
-

Test Automation is
simple, that every tester can do it


This myth is promoted by the tool sales people.
They are trying to promote the following test
automation process:


Record the script


Enhance the script by adding functions and data
driving


Run the scripts


Report results


Under the influence of this myth QA manager
can proudly report: All our testers are developing
test automation.

Reality
-

Test automation is a
software development task


Automation should be designed, developed and
tested


You need to have some kind of a
programming background
to implement test
automation. Test Automation is not as complex
as C++/C#/Java development.


Test automation standards should be developed


Automated test components are assets that
should be treated like application source code


Myth #2


Commercial test tools
are expensive


Under the influence of this myth some
companies, especially the small ones:


Try to develop their own test automation tools


Use scripting languages like Perl and Ruby


Use shareware test tools


Do not consider test automation at all

Reality


Commercial tools are
cheap


Per seat license for most expensive automation tool is
$8K


This tool will be used for 5 years.


Maintenance/Support fees are 20% of tool cost or
$1,800 per year


The cost of this tool is $8K/5+$1,800 = $3,100 per year


The automation developer cost with overhead is $100K
per year


The cost of this tool is just 3% of the person who uses it,
but productivity gain can be very significant

Commercial Tool Benefits:


Customer support. Many of the open source tools come and go with
little to no support


Most commercial tools are constantly being updated as technologies
change


Most commercial tools usually have more functionality (QTP can test
various GUI applications: Web, .Net, Java, VB, C/C++,
PowerBuilder, etc. vs. WATIR


Web only)


Commercial tools usually have a large community of users, which
translates into better availability of qualified resources


Commercial tools require less advanced programming


More test automation frameworks are available for commercial tools


Commercial tools are integrated with Test Management tools which
makes reporting and execution much simpler.

Absolutely Unscientific research:


Employer demand by tool expertise

(May, 2008)


Dice search results across US (30 days)

Tool

Search string

Matches

QTP

Qtp OR quicktest OR "quick test“

613

Functional
Tester

"rational robot" OR "functional
tester”

118

SilkTest

silktest OR "silk test”

86

TestComplete

testcomplete OR "test complete”

26

TestPartner

testpartner OR "test partner“

19

VSTE

VSTE AND test

45

WATIR

watir

26

Selenium

Selenium

108

Demand per tool
HP QTP
60%
Open Source
WATIR
2%
Microsoft
VSTE
4%
AutomatedQA
TestComplete
2%
Compuware
TestPartner
2%
Borland
SilkTest
8%
IBM
Functional
Tester
12%
Open Source
Selenium
10%
QTP
Functional Tester
SilkTest
TestComplete
TestPartner
VSTE
WATIR
Selenium
Excerpt from IDC Report:


“Worldwide Automated
Software Quality 2007
-
2011
Forecast and 2006 Vendor
Shares…”

Return On Investment (ROI)

Classic ROI Calculation

ROI = BENEFIT/COST


Automation Cost = Price Of HW + Price of SW + Development Cost +
Maintenance Cost + Execution Cost


Manual Testing Cost = Development Cost + Maintenance Cost +
Execution Cost


ROI = (Manual Testing Cost
-

Automation Cost)/Automation Cost


Looks right, isn’t it?

Problems with Classic ROI Calculation


You can’t compare Automated Testing
and Manual Testing. There are not the
same and they provide different
information about the AUT.


You can’t compare cost of multiple
execution of automated tests vs. manual
tests. You would never dream of
executing that many test cases manually.

Automation Real ROI


ROI value
IS NOT

the value of Automation
vs. Cost of executing these tests manually


Automation ROI value
IS

the benefit of
this type of testing, and it can be:


Reducing Time to Market


Increased Test Efficiency (Productivity)


Increased Test Effectiveness


Reduced Time to Market


Can get a greater market share


Makes people available to work on
other projects


Higher margins, if no competitive
products are currently available



Productivity and Effectiveness


More testing gets done faster, increasing the odds of
finding defects


Defects found early have better chances to be fixed


Manual Testers can concentrate on clever ways to
finding defects, instead of typing test inputs and verify
output.


About 7% of bug fixes create new bugs, sometimes in
already tested parts of the system. With automation you
can rerun tests for those modules. This almost never
happened when testing done manually.

ROI summary


Each project requires different types of
automation
-

there is often no easy
formula available to calculate ROI


Performing ROI calculation can help to
determine upfront what type of
automation, what level of skills, what tools
will be required.

Automation Frameworks

Test Automation Framework


A
Test Automation Framework

is a set
of assumptions, concepts and tools that
provide support for Automated Software
Testing.


Correctly implemented Test Automation
Framework can further improve ROI by
reducing the development and
maintenance costs.

Types of Test Automation
Frameworks


Modular


Data
-
Driven


Keyword

Driven


Model
-
Based

Modular framework


The Modular framework is the natural
progression from Record
-
and
-
Playback


The modular framework seeks to minimize this
repetition of code by grouping similar actions
into “modules” (e.g.: login)


Test Data is in a script


This is what tool sales people are promoting.

Data
-
Driven/Keyword Driven


These frameworks are similar in that the
data is separated from the test script



The script is just a "driver" or delivery
mechanism for the data.


The difference:


In keyword
-
driven testing, the navigation data
and test data are contained in the data source


In data
-
driven testing, only test data is
contained in the data source.

Model
-
Based testing


Model
-
based testing

is software testing
in which test cases are derived in whole or
in part from a
model

that describes some
(usually functional) aspects of the system.


Model
-
based testing for complex software
systems is still an evolving field.





We’ll be concentrating on
Keyword
-
Driven

testing since it
is a most beneficial framework for
large
-
scale test automation.


Advantages of Keyword
-
Driven
Test automation


This Framework addresses the
most common problem

with test automation:


Automation Engineers do not have domain knowledge
and the End Users (Subject Matter Experts/Test
Engineers) usually do not have automation expertise.


When properly implemented and maintained, it presents
a superior ROI because each business event is
designed, automated and maintained as a discrete
entity.


Keywords can then be used to design test cases, but the
design and automation overhead for the keyword has
already been paid.

Advantages of Keyword
-
Driven
Test automation


Reduced the cost and time spent maintaining
and updating tests


The modular structure of keyword
-
driven testing
means that new tests can easily be created from
pre
-
existing modules


The test team is capable of entirely automating
tests, even without programming knowledge


Can be easily modified to use with different test
tool


Reusability across different projects

Classic Keyword
-
Driven
Example

Object

Action

Data

Textfield (username)

Enter Text

<username>

Recent Success with Keyword
-
Driven
approach


Estimate using Modular Framework: 1 year


Estimate with Keyword
-
Driven Framework: 6 months


Project was completed in
4 months by 1 person
, 2 month
ahead of schedule.

This was the first project for the automation developer
utilizing the Keyword
-
Driven Framework.


170 components were developed (1 component per Web
page).

109 Regression Test cases were automated.

Example of Test Data for Keyword
-
Driven test
automation developed by Connected Testing Inc.
for TestComplete and QTP

Resources


“Everything you want to know about Test Automation…” by Brian Le
Suer


“Manager’s Guide to GUI Test Automation” by Yury Makedonov


“The ROI of Test Automation” by Michael Kelly


“What is my ROI?” by
m
Verify Corporation


SQA Forums


“An Overview of Test Automation Frameworks” by Nathaniel
Ritmeyer


“The benefits of keyword
-
based software test automation” by David
W. Johnson


Wikipedia

Igor Gershovich

Company: Connected Testing, Inc.

E
-
Mail:
igershovich@connectedtesting.com

Phone: 720
-
933
-
9395

Website:
www.connectedtesting.com