Automation Culture - Agile Austin

possehastyMechanics

Nov 5, 2013 (3 years and 9 months ago)

101 views

Automation Culture:

Essential to Agile Success

Agile Austin

Geoff Meyer,
geoffrey_meyer@dell.com

October 2013


Last updated: September 29, 2013

Session Objectives


Challenges to expect when adopting test
automation during your Agile transition


Practical solutions to
w
hat & when to automate


Real
-
world examples − from a
large

organization
perspective


Challenges automating in a HW
-
dependent
environment


For organizational
l
eaders driving test strategy



A Tools Discussion


Focused on how to develop Test Automation

IS

IS

NOT

Introductions

3

Geoff Meyer


Dell Inc, 1998


present


Test Architect


Agile
T
est & Automation Strategy


Agile
Steering
Committee Co
-
Chair


Global Projects


15 Scrum
teams
(2 regions)


11
Scrum teams
(4
regions)


9 scrum teams (2 regions)


7 Scrum teams (4 regions)


NCR Corp. 1984


1998


SW developer, Project Lead, SW Manager


B.S. Computer Science, San Diego State University


Masters Engineering Management
-

NTU

4

5

Agenda


Why
Automation is essential to Agile


The Dell
Context
and Agile @ Dell


Challenges
and Common Pitfalls


The Automation Landscape


Foundations of an Automation Culture


Care
and
Feeding
of the Automation
Culture

The Need
,


The Challenge,


The Pitfalls

6

Why is Automation So Important in Agile?


Near
-
term


Ensures
that you don’t break what you just
built


Provides safety net for developers & rapid feedback to new changes


Continuous Integration and use of Build Verification Test (BVT)


Long
-
term


Maximizes velocity of
Scrum team (increased value creation)


Creates
capacity for Exploratory
and ad
-
hoc Testing


Enables activities that can’t be done cost
-
effectively by humans


And if you
don’t…

Quality is at risk from an unmanageable
regression
suite

Differences with Automation in Waterfall?

In Waterfall…


Automated tests
are
derived


from the backlog of
completed features


In Agile, Automation can be incorporated in the requirement


Testing
and automation is
performed
after

Development
is
“complete”


Focus
is
on
first
-
time discovery

of defects and
optimizing
your
test coverage


v
s. Agile…
where automation development is
ongoing

and provides
immediate

feedback throughout
development

8

9

The

Context @ Dell

http://amazngwallpapers.blogspot.com

“Apply Test Automation in
the context
of your
organizational automation
needs”


---

Bob Galen, Agile Coach

Dell Enterprise Solutions Group

Global Design and Development using Agile Scrum

10

Noida Design

Center

Bangalore

Design Center

Austin

Design Center

Silicon Valley

Design Center

Dell Enterprise Solutions

Software Products


Server Systems Management


Converged Infrastructure Systems Management


Private Cloud Systems Management


Console Plug
-
ins (i.e. for SCCM, vCenter…)

11

Common Product Characteristics:


Large hardware support test matrix


Software is installed in the Data Center


Enterprise update cycles ~6 months


Products must function even as underlying HW, FW,
BIOS, and Drivers are upgraded
-

SUSTAINING

Agile @ Dell Roles/Responsibilities


12

Product
Owner


Product Owner Proxy

~1 per Scrum

Scrum Master

1 per
Scrum

Development

4
-
5 per Scrum

Test

3
:1

Tech Pubs

1 per 3 scrum teams

UI

1 per Scrum

Scrum teams


Small teams


Co
-
located


Distributed Projects


Teams formed
from functional
silos

Agile @

Dell with Adaptations


13








Hardening




















1

2

3

N
-
1

N

1

2

3

N
-
1

N

1

2

3

N
-
1

N



Release

Exit

Feature

Complete

Code Freeze

Sprints

Release Plan

Define

Plan

Develop

Launch

OLP



Software System Test

Extended Sprint Test

Stability

Pitfalls encountered at


Development didn’t
historically automate unit
tests


Build
teams were staffed
with
non
-
Build practitioners


Minimal guidance to Test beyond “
Go forth and automate


14


Black Box Test Mentality


Architecture(s) not optimized
for Automatability


Automation
was
interpreted
by many as
“Automate the UI”


Insufficient
SW engineering
background across Test
teams

Which Resulted In…


Over
-
emphasis on UI automation


Test Automation
not keeping up within the sprint


Feature Devotion
1


Multiple
automation tools & licenses


Test scripts not designed for
re
-
use

15

1


“A
nasty condition where people start valuing ticking off features more than tracking the real outcome of
the project
.”
---

Martin Fowler

16

The Automation
Landscape

http://amazngwallpapers.blogspot.com

Opportunities for Automation


It’s not just about automating test cases



Can also provide efficiencies to:


Test Preparation, Setup and Configuration


Non
-
Functional Testing (“
ility
”)


Longevity, Scale and Performance Characterization


Compatibility Testing (Solution and Device)

17

Test Preparation


Bare
-
metal Deployment


Setup
&
Configuration


OS Provisioning and
Configuration


Test
case staging


Environment
Cleanup/Baseline


Virtualization
-
based test
environments

18

Test Content
Development

Whole team ownership of QA & Automation


UI automation on Customer
Usage
workflows

19

CAUTION: Application Architecture can be an enabler or inhibitor


Unit Test Automation


In
-
Sprint, Automated
Acceptance Tests


Web
-
services
(or CLI) Test
Automation for Functional
Testing


Automated CI (UT, Build,
BVT)


2

-

Agile
Testing: A Practical Guide for Testers and Agile teams

2

Application Architecture Matters


Does it facilitate testing at the API/Services level?


Does the Business/Error Logic reside below the services
level?


Does the UI architecture support test automation other
than record
-
playback?


For Large organizations:


Standardize UI architectures across the product portfolio


HTML5
, Flex, Silverlight…


Standardize Service
architectures across the product portfolio


SOAP
,
REST, CLI, API

20

Prioritize and Identify what NOT to Automate

What NOT to Automate


Everything


GUI


E
xcept for High
-
value customer usage flows after
the UI is
stable


Tests
that would only find low severity
bugs


Where the product is unchanging


21

Prioritize
H
igh ROI

1.
BVT Candidate Acceptance Test


Test of core functionality that executes in a short duration

2.
Core Functionality to be run in Nightly Regression

3.
Sustaining Test Candidate Test


Functional test that verifies the application can withstand subsystem
changes


Non
-
Functional Testing


Performance
Characterization


Longevity


Stress


Scale


Concurrency

22

Often times analysis of
these areas are simply
too cost
-
prohibitive to
be done manually

3
-

http
://lisacrispin.com/wordpress/2011/11/08/using
-
the
-
agile
-
testing
-
quadrants/


3

23

Foundations of
an Automation
Culture

Images.yahoo.com

Keys To A Culture Transition


Development vs. Test


A different “School” of
Test
4


Evolve from Functional Responsibility


to Whole Product Ownership

24

4



Scott Barber “
Approaches to Software Testing: An Introduction



Establishing an Automation Culture



Identify


Near
and long
-
term
automation focus areas


Inventory
the
culture
and skillset of
organization


Establish


Tooling and Infrastructure
Standardization


Your
‘community’


Develop


Workforce
transition
plan


Organize


Embedded vs
. Specialized Automation team


Re
-
align project staffing


Operationalize

25

Identify

Focus Areas for Automation

26

4

-

http
://lisacrispin.com/wordpress/2011/11/08/using
-
the
-
agile
-
testing
-
quadrants/


4

Establish


Tooling and
Infrastructure


Tooling Standards and Frameworks


Team to
develop/manage home
-
grown tools


Lead tool evaluations to meet new architectures &
technologies






Community: Internal and External


27

Multiple automation Tools/FW’s may be
needed to automate at layers of testing:


Unit level


M
iddle tier


UI
-
driven

Develop & Organize


Adjust
Hiring Practices


Re
-
align Project staffing


Embedded
Automation vs.
Specialized Team


Training


New
-
hire


Ongoing


Confidential

28

Operationalize


Whole team commitment in words and actions


Include Test Automation in Acceptance Criteria


Unit


Acceptance


Continuous Integration and BVT


Establish Metrics, Governance & Rewards


29

Automation @ PG Enterprise Solutions Group
How did we go about it?

30

Process/Project

Culture/Organization

In
-
Sprint
Automation

Established ALT

2011

2014

UI Automation


“Automate
-
First”

Service
-
level
automation

(CLI & Web
-
services)

IVT

Networking
& Servers

Automated BVT

IVT

Device

Accelerate
Automation
Skillset

Maximize
Utilization

Automation
Architects

Setup &
Config

Internal
Training
Programs

2012

TDD &
Automated
UT

Non
-
Functional

Scale, Longevity

2010

Automation
Day

2013

Automation
Content
Libraries

Care and
Feeding of the
Automation
Culture

31

Maintaining the Automation Culture


Operationalize
Automation


Include Test Automation in the Acceptance Criteria


Encourage Community Practice and Participation


External
-

Industry User Groups & Conferences


Internal
-

Brown
-
bag sessions, Showcases, Mini
-
conference


For
Large organizations:


Overcome skillset deficiencies by adjusting staffing strategy


Automation
Leadership T
eam, Automation Architect(s)


Continually monitor the alignment of
Architecture, Dev
and
Test


Develop
reward
systems

32


Recognizing a Successful Automation Culture


Automation
is
a
shared
responsibility


Automatability is a key
architectural
consideration


Continuous Integration & BVT


Teams are staffed to include
an
automation skillset


Automation
is operationalized

Resources


Agile Manifesto

-

http://agilemanifesto.org/


Articles/Blogs:


http://support.smartbear.com/articles/testcomplete/automated
-
testing
-
agile
-
environment
/

-

SmartBear


http://lisacrispin.com/wordpress/2011/11/08/using
-
the
-
agile
-
testing
-
quadrants
/

-

Lisa Crispin


http://
www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=177
93&tth=DYN&tt=siteemail&iDyn=2



Rajini Padmanaban


Beyond
Agile Testing:
http
://
www.utest.com/int
-
v1/a/beyond
-
agile
-
testing

-

uTest


Testing and Checking
Refined
:

http
://
www.satisfice.com/blog/archives/856

-

James Bach
and Michael Bolton


Books:


Scaling
Software Agility: Best Practices for Large
Enterprises



Dean Leffingwell


How Google Tests Software


James Whittaker, Jason Arbon, Jeff Carollo


Agile Testing: A Practical Guide for Testers and Agile teams


Lisa Crispin, Janet Gregory


http://
www.dorothygraham.co.uk/automationExperiences/index.html

-

Dorothy Graham


Presentations:


Approaches to Software Testing: An Introduction



Scott Barber


Agile Testing: Challenges Beyond
the Easy Contexts


Bob Galen

34

Questions?

35