Practical Approaches to

heavyweightuttermostMechanics

Nov 5, 2013 (3 years and 9 months ago)

75 views

IBM Software Group

Practical Approaches to

End
-
to
-
End Automation with

STAF and STAX




IBM Software Group

Agenda


Introduction


STAF Services


STAX Service


Automation Tasks


Extending the Infrastructure

IBM Software Group

Introduction


STAF (Software Testing Automation Framework) is a
multi
-
platform,

multi
-
language framework
designed around the idea of reusable components,
called
services

(which provide functionality such as process invocation,
resource management, logging, and monitoring)



STAF helps you increase the
efficiency
,
productivity
, and
quality

of your testing
by improving your level of automation and reuse in your individual testcases
as well as your overall test environment



STAF removes the tedium of building an
automation infrastructure
, thus
enabling you to focus on building your automation solution

IBM Software Group

STAF Services


A STAF service is a
reusable component

which provides a focused set of
functionality (such as logging)


STAF's services can be leveraged to provide end
-
to
-
end automation of your
test environment. Here are some examples.


Event

Provides a publish/subscribe notification system

EventManager

Allows you to call STAF services when a specified Event occurs

Cron

Calls into STAF services at a specified time interval

Process

Allows you to start, stop ,and query processes

Queue

Provides a network
-
enabled IPC mechanism for STAF programs

Monitor

Allows a testcase to publish its current running execution status for others to
read

Log

Provides a full
-
featured logging facility

Semaphore

Provides network
-
enabled named event and mutex semaphores

FileSystem

Allows you to get and copy files across the network, list directories, and
delete files

HTTP

Allows you to quickly and easily make HTTP requests

ResourcePool

Allows you to manage exclusive access to pools of elements, e.g. UserIDs
or Software Licenses

Email

Allows you to send email messages

IBM Software Group

Introduction


STAX is an
execution engine

which helps you thoroughly automate your entire
test environment

ƒ
System Setup

ƒ
Product Installation

ƒ
Testcase Distribution

ƒ
Testcase Execution/Monitoring

ƒ
Results Analysis

ƒ
Notification



STAX builds on top of three existing technologies,
STAF
,
XML
, and
Python
, to
place great automation power in the hands of testers



STAX provides a powerful
GUI monitoring application

which allows you to
interact with and monitor the progress of your jobs

IBM Software Group

STAX Service


The STAX Service is what drives all of your testcase automation, from
beginning to end. Your end
-
to
-
end testcase automation may be comprised of
a single STAX job, or multiple STAX jobs.


Features provided by the STAX Service

ƒ
Support for
sequential
and
parallel

execution

ƒ
User
-
defined granularity of execution control

ƒ
Support for nested
testcases

ƒ
Ability to
control
amount of execution time

ƒ
Ability to import modules at run time

ƒ
Support for existing Python and Java modules/packages

ƒ
XML
-
based

ƒ
Includes common utility modules

ƒ
Ability to
extend

the STAX XML language


Features provided by the STAX Monitor GUI

ƒ
Ability to view and interact with your entire test environment from any number of
remote machines

ƒ
Dynamic

updating of the GUI to reflect the current state of the job

ƒ
Ability to
extend

the STAX Monitor GUI

IBM Software Group

Automation Tasks

Execution

Synchronization

Monitoring

Resource Management

Automation Completion

Testcase Output Analysis

Results Notification

Testcase Cleanup

Testcase Execution

System Setup

Automation Startup

IBM Software Group

Automation Tasks
-

Automation Startup


Automation startup refers to the stage where your end
-
to
-
end automation
begins and is usually the result of a prior piece of work, such as a product
build successfully completing or by the passage of a designated time interval

ƒ
In the first case, the
Event

and
EventManager

services can be used to initiate
STAF/STAX automation


STAF
-
enabled testcases can register for a certain Event, and take appropriate
action when the Event (such as a product build completion) occurs


Instead of using STAF
-
enabled testcases to listen for these events, you can use
the EventManager service to execute an arbitrary STAF command when the
Event occurs


In most scenarios, the STAF command is the submission of a STAX Job which
will kick off the target automation

ƒ
In the second case, the
Cron

service can be used to execute an arbitrary STAF
command at a designated time interval (for example, every hour, every day at 12
midnight, every 15th day of month, etc.)


In most cases, the STAF command is the submission of a STAX Job which will
kick off the target automation

IBM Software Group

Automation Tasks
-

System Setup


During the setup phase, a baseline OS can be established by using
Ghost/Drive images and/or
VMWare

images. STAF's Process Service allows
you to boot these images.



Prior to testcase execution, STAF's
FileSystem

and
Process

services can be
used to install the product under test and any pre
-
requisite software



In most cases, testcase files will need to be extracted from a server or a code
repository. STAF's Process service allows testcases to be
extracted

from
code repositories, and STAF's FileSystem service allows testcase files to be
transferred

from a server machine to test machines.

IBM Software Group

Automation Tasks
-

Testcase Execution


Testcases are typically files (such as binary executables, Java class files, shell
scripts, Perl/Python/Tcl/Rexx scripts etc.) or system commands that will be
executed

on a test machine


STAF and STAX allow you to start any executable file/script/command, just as
if you were executing that file or command while logged into a terminal on the
machine


Testcases are executed in the STAF environment by the
Process

service

ƒ
The Process service can execute testcases that are completely unaware that they
are being executed by STAF/STAX, or they can
leverage
STAF services in order to
take advantage of additional automation capabilities

ƒ
In your STAX job, the <process> element is used to submit a START command to
the Process service


To automate
GUI

testing, you would need to use tools such as SilkTest,
Rational Robot, WinRunner, etc.

ƒ
Once you have the GUI testing automated, you would then use STAF/STAX to kick
off the GUI automation tools to run the tests

ƒ
After the tests complete, STAF/STAX can
analyze

the output to determine if the
tests were successful, and take appropriate action

IBM Software Group

Automation Tasks
-

Testcase Execution
-

Monitoring


It is desirable to have the means to easily
determine

the status of testcases
as they are executing


Testcases can leverage the Monitor service in order to
publish

their current
execution status

ƒ
As an example, a testcase that loops through a set of tests 1000 times could
publish the current loop number as its latest status

ƒ
By using the Monitor service, you could then, from a central machine in the STAF
environment,
query

the current status of all of your testcases

ƒ
Furthermore, when using the STAX Monitor, all of your testcases' current status will
be
dynamically

updated in the STAX Monitor

ƒ
The STAX Monitor allows you to remotely view all of your executing testcases in a
single view


Throughout your STAX job, you can define
blocks
, which represent a portion
of your job over which you would like to have
runtime control

ƒ
The STAX Monitor allows you to hold, release, and terminate any blocks that are
defined in your job

IBM Software Group

Automation Tasks
-

Testcase Execution
-

Synchronization


Testcases may need to synchronize at certain points before proceeding with
additional execution. STAF and STAX allow testcases, even when on remote
machines, to synchronize via the
Semaphore

service.


The STAX service allows you to synchronize the completion of multiple
activities via the
<parallel>

and
<paralleliterate>

elements


The STAX service also allows you to place time constraints on portions of
your automation jobs via the
<timer>

element

ƒ
Allows cleanup of "infinite" tasks after the time limit is reached

ƒ
Allows detection and cleanup of "hung" tasks

ƒ
The STAX engine handles all cleanup activities and gives you direct control over
the outcome of an expired timer

IBM Software Group

Automation Tasks
-

Testcase Execution
-

Resources


Testcases can have pools of
resources

that can be
shared

among a number
of other testcases. Resources are typically a set of machines, printers, or
other hardware devices to which you want to
manage access
. Resources can
also be "logical" resources, such as software licenses or userids.

ƒ
One example would be if part of your test effort includes a certain software package
for which you only have 5 software licenses, but you have 20 testcases that may, at
some point during their execution, need to
obtain

one of these software licenses.
The ResourcePool service could be used to set up a resource pool with 5 software
licenses, from which the 20 testcases would request
acquisition

of one of the
resources before continuing with their execution.

ƒ
Another example would be if you had 3 userids that your testcases could use to log
into a system. You could again use the ResourcePool service to control access to
the 3 userids.

ƒ
A third example would be if you had a machine pool of 10 machines on which your
testcases can execute. You could leverage the ResourcePool service to control
access to the machines.

IBM Software Group

Automation Tasks
-

Testcase Output Analysis


Testcases often write output to log files or to the standard output stream


When using STAF and STAX, you can have any number of files and/or the
entire standard error/output stream
returned

when the testcase completes


Every STAF command and process returns a return code which can be used
to determine the automation flow


The files and/or standard error/output stream can then be
parsed

through, or
otherwise analyzed in STAX, and appropriate action can then be taken
depending on the content of the files and/or standard error/output


Throughout your STAX job, you can define
<testcase>

elements and
corresponding
<tcstatus>

elements to record testcase passes and failures


The FileSystem service can be used to check for the presence/absence of
files as part of the output analysis


The FSExt service can be used to search files for particular strings as part of
the output analysis


Within your STAX job, you can use Python’s Regular Expression search
capabilities to analyze testcase output

IBM Software Group

Automation Tasks
-

Testcase Cleanup, Results Notification


Testcase Cleanup

ƒ
After your testcases have completed execution, STAF's
FileSystem

and
Process

services allow you to cleanup any temporary output and/or the testcases
themselves

ƒ
The FileSystem service can also transfer any output or log files that have been
created by the testcases to a central server



Results Notification

ƒ
After your automation has completed, you can use STAF's
Email

service to
distribute the automation results

ƒ
STAF's
HTTP

Service allows you to post forms with the automation results on the
web

IBM Software Group

End
-
to
-
End Automation with STAF and STAX

Event, EventManager, Cron

FileSystem, Process

Process, Monitor, Log, Variable,

Queue, ResourcePool, Semaphore

Log, Process


Process, FileSystem


Email, HTTP

S

T

A

X


J

o

b

<function>

<parallel>

<sequence>

<process>

<stafcmd>

<testcase>

<timer>

<block>

<loop>

<message>

<log>

<import>

<job>

<iterate>

<paralleliterate>

<hold>

<release>

<terminate>


Automation Completion

Results Notification

Testcase Cleanup

Testcase Output Analysis

Testcase Execution

System Setup

Automation Startup

IBM Software Group

End
-
to
-
End Automation with STAF and STAX

Test systems in your lab

STAX Service Machine

STAX Monitor Machine
-

this could be
your office or home system


Automation Startup


Event


EventManager


Cron

Log

Process

Monitor

FileSystem

ResPool

Http

Email

Variable

Queue

Semaphore

IBM Software Group

End
-
to
-
End Automation with STAF and STAX

Extract source code

STAX Service Machine

Cron service starts build
job at 1am every day

WIN32

AIX

Linux

OS/400

z/OS

Transfer code to build machines

Build product on each
platform

Transfer product
installation packages to
common network
location

Transfer build results to
project web site

Transfer product binary
files to ISMP machine

Build ISMP packages

Generate "Build Available" event

IBM Software Group

End
-
to
-
End Automation with STAF and STAX

Download product
installation packages from
common network location to
SVT machines

STAX Service Machine

EventManager service
receives a notification for
the "Build Available"
event, and kicks off the
SVT job

WIN32

Solaris

HP
-
UX

OS/400

z/OS

Install product on SVT
machines

Post SVT results to
project website

Run SVT testcases

If SVT fails, send a page to
project manager

Acquire SVT test machines
from the machine pool

IBM Software Group

Extending the Infrastructure


STAF

ƒ
You can
extend

the STAF infrastructure by writing you own custom STAF Services
in Java, C++, or REXX

ƒ
These services can then be
dynamically plugged into

the STAF infrastructure



STAX

ƒ
Extensions

to the STAX service can be written which define new elements that can
be used in a STAX xml file

ƒ
Extensions to the STAX Monitor can be written which define
plug
-
in views

which
can be displayed via the STAX Monitor

IBM Software Group

End of Presentation