ItsAutomation_NotAutomagic_v4 - Meetup

knowledgeextrasmallStorage

Dec 11, 2013 (3 years and 7 months ago)

201 views

…or proving Einstein was right about insanity

Presented by

Jim Hazen

Introduction


Jim Hazen


Veteran of the Software Testing Trenches


Experience in software testing, both commercial
and consulting work.


Working with automation and tools since 1989.


Implemented Automation Tools on DOS, OS/2,
Windows and Web platforms.


Experience with tools; including HP/Mercury,
IBM/Rational, and other ‘scripting’ languages.

Sage Words to Remember


“Insanity: doing the same thing over and over again
and expecting different results.”


Albert Einstein

Agenda


What is ‘Test Automation’


A Brief History


Different types of test automation, tools and uses


Automation Myths


How to correct them


Resources & Skills needed


Setting expectations for Management & Others


Summary


Q&A


What
is Test Automation?

It’s a Big Topic to Discuss

Bigger Than We Think!

Is It This?

Or More Like This?


Who Builds the Automation
?

Some Are Seen This Way

Most Look Like This Instead

What ‘Test Automation’ Really Is


“Automation” definition


Webster’s


“the technique of making an apparatus, a
process, or a system operate automatically”


In Software Testing we use ‘tools’ to drive our processes


People only think of tools that
execute

a test as
‘Automation
’ .


Other
tools can be used for Test Design, Test Data
Generation, Defect Tracking, Test Management, etc
.


“Automation” is using a tool during any part of the
process of testing.

Who Test
Automator’s

Really Are

A Brief History


Test “Execution” tools have been around a long time!

Mainframes (pre
Mid
-
80’s)

PC DOS
(Mid 80’s


90’s)

PC GUI &
Web (1990’s


present)

Mobile
(2008


present)


Mainframe


Home grown ‘macro’ tools
1
, Hyperion


PC DOS


AutoTester
,
SuperKey

(macro tool)


PC GUI


Automated Test Facility (ATF), Rational Robot,
QA Run,
Winrunner
, etc.


PC Web


QTP,
SilkTest
, Selenium,
TestComplete
, etc.


Mobile


MonkeyTalk
,
Robotium
, etc.

Other Types of Test Automation Tools


Other Test Execution Tools


xUnit

(
JUnit

&
NUnit
), SOAPUI,
FitNesse
, etc.


Test Design &
Test Data Generation


Hexawise
,
AllPairs
, Tosca Test Suite, etc.


Test Execution Management


HP ALM, MS Team Foundation Server,
TestLink
, etc.


Test Environment Management


VMWare

vCloud

Director, etc.


Defect Tracking & Reporting


Bugzilla
, JIRA, etc.


Automation Myths & Misconceptions

Old Myths


You
can automate 100% of your tests


Automation will “cure” all problems with testing


Automation tools are usable by anyone


Record/Playback is all you need


Automation allows Management to cut test staff


One tool is all you need


Automation will speed up
testing process


* Oldies, but goodies!!


New
Myths


You
don’t have to know how to
program


Opensource

tools are “free



Commercial tools are “expensive”


GUI
level automation is a waste of
time


“Codeless” and “
Scriptless
” tools


Commercial tools can’t support agile


Developers only need to focus on Unit Tests

How to Correct Them


100% Automation


Certain tests shouldn’t or can’t be
‘automated’. 80/20 Rule; automate 20% of
functionality used 80% of time.


Automation solves all testing problems


it helps, but
it creates new problems too.


Automation can be done by anyone


Not everyone has
the ‘mindset’, it is both learned and inherent (logic).


Record/Playback is all you need


Proven time & time
again to be false, and to fail.

How to Correct Them cont.


One tool is all you need


There isn’t a one
-
size fits all.


Automation speeds up testing process


The “Illusion
of Speed” problem. Both execution and creation.


You don’t need to know programming


Anyone can
Automate problem.


Opensource

tools are free


Costs due to ramp up time
(money) and skilled staff to do it (more expensive).
2


Commercial = Expensive


Not really in grand scheme
of things.

2

How to Correct Them cont.


GUI level automation is a waste of time


Not really

How to Correct Them cont.


Codeless/
Scriptless

tools


Just a different UI
. You’re
still writing a ‘test’ which is a set of ‘instructions’
(code).


Commercial tools don’t do Agile


Yes they do, it’s how
you use the tool that counts.


Developers only create Unit Tests


Other tools require
someone to ‘build’ the fixtures/code to ‘talk’ to the
SUT.

Why Failures Happen

15%

11%

17%

20%

37%

Failure Reason

Success
Tool Incompatibility
Insufficient Budget
Lack of Experience
Lack of Time
Source: IDT Survey
3

85% Failure Rate! Too High!!

Einstein was right!


We keep repeating
history, why?


Always a ‘new’ group of management and testers.


Companies always looking to do testing faster, cheaper
and with less resources.


Vendors and others who sell “
Automagic
”.


Instant gratification mindset. Automation takes time.


Not learning from past mistakes.


Software not designed with “testability” in mind.


Testers not involved
early with decisions
, not included
by management.

Resources for Automation


Dedicated Staff


Skilled & experienced staff


Dedicated Equipment/Environments


Reduce risk of competition of resource or teams
stepping on each other


Dedicated Budget


Automation isn’t cheap, plan appropriately


Dedicated Management Support


Can’t have any of the other 3 without it

Skills for Test Automation


It’s a different view and purpose for the script code.


Creating ‘software’ to ‘test’ software; i.e.,
PROGRAMMING!


Need basic understanding of programming methods.


Understand different types of technology layers


GUI, API, System, Web, Services, Database, etc.


You need to know how to program in different
languages when needed.


C#, Java, VBScript, Jscript, XML, SQL, etc.


Test Automation Skills Cont.


Need to understand how to ‘hook’ the software under
test and interact with it, and at what level.


API, or Object, or Service, or Protocol, etc.


Understand how tool ‘interacts with’ the software
under test; what functions/methods are to use.


GUI Object ‘methods’ (functions)


Ex:
SwfWindow
(“
LoginForm
").
SwfEdit
(“ID”).SET “Jim”


Need to understand how to “pick apart” the software
and “drive it” programmatically.

Axioms to remember


SMART


Standalone


Test/Script able to run by itself, minimize dependencies


Modular/Maintainable


Scripts and code need to be modular in design and easy to
maintain


Appropriate


Build according to need for project, don’t try to automate
100%


Reusable


Build for reuse; keep script code compact and focused.


Testable & Traceable


Make it easy to debug your code, and be able to link together.

Axioms to remember cont.


SEARCH
4,5


Setup


Test/Script needs to check for pre
-
test conditions and flag if
needed.


Execute


The logic and data needed to run the test itself


Analyze


Validations, Error Handling, etc.


Report


Progress, Error, Summation of run information (
Logfile
)


Cleanup


Test/Script needs to ‘cleanup’ (temp files, reset data, etc.)


Help (or Home)


Setting Realistic Expectations


Management & Other Groups


100
%
automation
is an unreachable
goal.
6,7



Automation is its own form of software development.



Ramp
-
up
time will be
required, longer than you think.



Benefits realized over time, no
immediate payback for
investment.

Expectations cont.


Management & Other Groups


No tool supports all systems & technologies out of the box.



Not all testers can write scripts. Specialized resources are a
must and need to be dedicated to the work.



Nothing is “Free”, there are always costs involved.



Don’t believe the marketing hype! Don’t buy the snake oil.
8


There is no Magic!

Otherwise, the result

Summary


What is ‘Test Automation



A lot more than you think!


History


Lots of tools over the last 30 plus years.


Different types of test
automation
tools


Not just for
execution of a test only.


Automation
Myths


Old and New;
lots of
failures due
to them.


How to
fix the problems


Be realistic, dispel the myths.


Resources and Skills needed for test
automation


Again, a lot more than you think. It is Programming!


Setting expectations for Management &
Others


Without proper expectations you are painted into a
corner.

Remember… It’s automation,

Not
Automagic
!

Q&A

Contact Info


Jim Hazen


Company: Freelance Contractor



Home email:
calkelpdiver@gmail.com


LinkedIn profile:
http://
www.linkedin.com/profile/view?id=2835000&trk=tab_pro



References

1)
“Experiences
of Test Automation: Case Studies of Software Test
Automation”,
Chapter 5, Graham &
Fewster
,
Addison
-
Wesley, 2012

2)
“Test Automation
-

Let's Talk Business”, Igor
Gershovich
, Connected Testing,
2006.

3)

Implementing Automated Software Testing”,
IDT Survey,
Elfriede

Dustin et
al., Addison
-
Wesley, 2009.

4)
“How We Test Software At Microsoft”, page
187, Alan
Page, Microsoft Press,
2009.

5)
“How to Automate Testing: The Big Picture“, Keith
Stobie

& Mark Bergman,
1992.

6)
“Seven Steps to Test Automation Success”, Bret
Pettichord
,
http://www.io.com/~wazmo/papers/seven_steps.html
, 2001
.

7)
“Success with Test Automation”, Bret
Pettichord
,
http://www.io.com/~wazmo/succpap.htm
, 2001
.

8)
“Test Automation Snake Oil”, James Bach, 1996




Thank you for attending this session.


Please fill out an evaluation form.