Use Free Automated GUI Test Tools

possehastyMechanics

Nov 5, 2013 (3 years and 10 months ago)

193 views

Derek Roberts

Director / Consultant

Scapa Technologies

Use
F
ree
A
utomated
GUI
Test
T
ools

© 2013 WWRUG Canada Inc. All Rights Reserved

2

Introduction

-
My background

-
Questions
to the
audience; Anyone tried to automate Remedy via GUI
automation?

Objectives of this talk

Live
Demo, single user (without Scapa)

Difference Between GUI and API
automation

Live
demonstration of the Scapa 3 Step process
;


Agenda

© 2013 WWRUG Canada Inc. All Rights Reserved

3

Objectives and Results

Objectives

-
To understand the advantages of GUI automation
vs

API automation

-
To understand the complexities of using GUI automation

-
How to use GUI automation to for functional testing and SLA monitoring
tests.

-
How to use GUI automation to uncover bottleneck, scalability, capacity and
performance characteristics vi integration with Scapa Test and Performance
Platform (Scapa TPP).

Results

-
A simple automated GUI script; login, navigation of the Remedy Menu, query
and logout transaction.

-
Live demonstration of running a the GUI automation script against the
Remedy Mid Tier Interface

-
Live demonstration of running a GUI automation script
the Remedy Mid Tier
Interface, for more than one user, using Scapa TPP to control the test.


© 2013 WWRUG Canada Inc. All Rights Reserved

4

Objectives and Results

Skills developed

-
A understanding on how to running a Scapa “Capture, Process, Go” test

-
A basic understanding in the interpretation of results from Scapa TPP


Via real life examples

-
A new appreciation of the performance testing process.

-
A basic understanding on how you can implement performance testing to
resolve existing issues and prevent performance issues.

© 2013 WWRUG Canada Inc. All Rights Reserved

5

Full title:

-
Use free automated GUI test tools to capture and replay user interactions
with the Remedy web mid
-
tier, to automate common tasks (macro style);
Use automated GUI scripts for end user experience monitoring, SALs,
functional testing and load/stress testing.

Introduction

© 2013 WWRUG Canada Inc. All Rights Reserved

6

Full description:

-
A Scapa Technologies consultant will give a live demonstration
-

how to use
free automated GUI test tools to capture and programmatically replay user
interactions with the Remedy web mid
-
tier, to automate common tasks
(macro style).

-
The
attendees will be shown how to extend the capabilities of captured
automated GUI scripts by using them with Scapa TPP for load/stress testing,
end user experience monitoring (often from different geographical location),
performance tuning, SLA compliance, early warning alert system when
performance deteriorates, functional and regression testing in addition to
testing desktop and application delivery platforms that Remedy might be
deployed on, such as Citrix
XenApp
/
XenDekstop
, VMware View and
Microsoft RDS

Introduction

© 2013 WWRUG Canada Inc. All Rights Reserved

7

Derek Roberts


his consultancy role in Scapa

-
Senior consultant and director at Scapa Technologies

-
Over
ten years’ experience in the testing
field

-
I
mostly find myself testing systems and applications deployed via Microsoft
Terminal Server, Microsoft Remote Desktop Server, Citrix
XenApp
/
XenDesktop

and VMware View.


I do this by automating user interactions with the real application GUI (for
any application, including Remedy mid
-
tier
). I will be giving a talk on how
to automate the Remedy mid
-
tier via the GUI in another talk.

-
I
also undertake Remedy testing via the user tool and Remedy mid
-
tier
through their appropriate protocols
.

Introduction (Cont.)

© 2013 WWRUG Canada Inc. All Rights Reserved

8


Before I the demonstration

-
Questions to the audience


anyone in the room who has tried to automate
the Remedy mid
-
tier?


If so, what tools did you use?


Did you find it difficult or easy?

Introduction (Cont.)

© 2013 WWRUG Canada Inc. All Rights Reserved

9

Live Demo Time

-
A
pre
-
prepare GUI script


Running a single user activity with an
AutoIt

script

Live Demo

9

© 2013 WWRUG Canada Inc. All Rights Reserved

10

Benefits of GUI Automation

You can automate common user interactions

-
If you find yourself, performing repetitive interactions, automation can be a
time saver and free you to perform other activities.

You can run the GUI script via Scapa TPP for load, performance,
capacity, reliability and benchmark testing.

-
This is useful to obtain the final data rendering times in the web browser.

You can run SAL monitoring style tests.

-
Schedule tests to run throughout the working day, to gain insight in to the
performance of the system from the end users perspective.

-
You can set breach criteria and send alerts if something goes wrong with the
application or if end user experience metrics are above or below a certain
value.


© 2013 WWRUG Canada Inc. All Rights Reserved

11

Benefits of GUI
Automation (Cont.)

GUI tests can be run along side Scapa TPP’s Remedy API (headless)
tests

-
E.g., use Scapa’s Remedy API driven tests to place the bulk of the load on to
the system and have a handful of automated GUI users (also controlled by
Scapa TPP) running at the same time.


With this technique you can uncover if an issue is related to the final
rendering of the presentation layer (the web page) at the clients desktop
or if its back end related.


If you are delivering a users desktop via a thin client or virtual desktop
architecture, then you can uncover if issues are related to this layer or the
Remedy layer.

If something goes wrong, you can automatically take a screenshot
via the script.

© 2013 WWRUG Canada Inc. All Rights Reserved

12

Difference Between GUI and API automation

GUI Automation

-
Drives the actual application GUI, the same way as a real user would

API automation

-
Drives the application at the protocol layer and is headless


you cannot see
any interaction with the GUI

© 2013 WWRUG Canada Inc. All Rights Reserved

13

Live Demo Time

-
Running a pre
-
prepare script (one user) via Scapa TPP

-
Running a pre
-
prepare script
(many user
) via Scapa
TPP
(cant do)


Live Demo

13

© 2013 WWRUG Canada Inc. All Rights Reserved

14

Benefits of API Automation

API Automation tests can scale up with limited hardware
requirements.


You can run 800
-
1000, or
more users (
depending on number of test cases,
speed of execution of test scenarios
e.t.c
.,) users per
load injector.


Test cycles can be short as you can remove
user think time
from a test


Automatic
handling of API return codes from HTTP
API and the Remedy
back channel
.

© 2013 WWRUG Canada Inc. All Rights Reserved

15

Benefits of API Automation

Simplicity


No programing
required (unique to Scapa)

-
Other API driven solutions involved coding

-
With Scapa, automated
test
creation,
including
the handling
of data
dependencies and custom parameterisation
is undertaken without
programing


No in
depth AR
System API knowledge is required

As such, test cases can be built in minutes


There are more benefits, but I will leave it at that for the moment.

© 2013 WWRUG Canada Inc. All Rights Reserved

16

Disadvantages of GUI
vs

API Driven
T
ests

Basically, the opposite of the API driven tests.


More hardware to inject load


Without Flash, you can run between 60
-
180 users per load
injector
(depending
hardware).


With Flash, you get less.


No automatic handling of API return codes from HTTP or Remedy back
channel.

Scripting required

-
A

basic knowledge of programming constructs


if then, loops, variables
e.t.c
.,

© 2013 WWRUG Canada Inc. All Rights Reserved

17

Disadvantages of GUI
vs

API Driven
Tests (
Cont
.)

It takes longer to create an automated GUI script

-
This is in comparison to Scapa Remedy API driven tests.

-
Competitor's API solution's can actually take longer than GUI automation as
you need to understand the Remedy protocol and backchannel servlet calls,
in addition to the HTTP protocol.


Only Scapa TPP fully automates the API
test creation,
including
the
handling
of data dependencies and custom parameterisation without
programing or
in depth
AR System or application knowledge.


© 2013 WWRUG Canada Inc. All Rights Reserved

18

Benefits of GUI Automation Over API Automation

GUI Automation is less abstract than API (headless) test.


You can see the automated user/s interacting with the GUI


You can write head up displays (HUD) to assist in the analysis if
something goes wrong


This is very useful if you make the script take screenshots when errors
occur

You can use GUI tests for automated functional testing

-
More control over execution and standardization of reporting

-
You can schedule runs to take places

-
Removed the ‘boredom a real user would have when running functional
regression tests.

© 2013 WWRUG Canada Inc. All Rights Reserved

19

Benefits of GUI Automation Over API Automation (Cont.)

You can use GUI test for on going monitoring tests for sampling
the experience a user would have on the production system.

-
Useful for SALs

© 2013 WWRUG Canada Inc. All Rights Reserved

20

Benefits
of GUI
Automation Over API Automation (Cont.)

If you deliver user desktops via thin client or VDI you can test this
layer in addition to the Remedy layers

-
You can test the performance, capacity and reliability of the desktop delivery
mechanism. E.g., desktops and thus a users web browser may be delivered
via
via


Microsoft Remote Desktop Server (RDS)


Microsoft
Terminal Server (TS)


Citrix
XenApp


Citrix
XenDesktop


Vmware

View


© 2013 WWRUG Canada Inc. All Rights Reserved

21

© 2012 WWRUG Canada Inc. All Rights Reserved

Full screen slide example

Scapa TPP is running in the top left. Many users running a GUI automation script

within MS Remote Desktops sessions

© 2013 WWRUG Canada Inc. All Rights Reserved

22

© 2012 WWRUG Canada Inc. All Rights Reserved

Full screen slide example

Scapa TPP interface

© 2013 WWRUG Canada Inc. All Rights Reserved

23

© 2012 WWRUG Canada Inc. All Rights Reserved

Full screen slide example

© 2013 WWRUG Canada Inc. All Rights Reserved

24

List of GUI Automation Tools

Free Tools

-
Winbatch

-
Autohotkey

-
AutoIt

-

I’ll use this one for the demonstrations

Low cost tool

-
Wintask

© 2013 WWRUG Canada Inc. All Rights Reserved

25

Live Demo Time. The how to script part of the demo

-

Very quick demo of
wintask

to show you the recording tool

-
Very
quick demo of
AutoIT

to
show you the recording
tool


Drawbacks of
AutoIT

recorder and other free tools

-
Most free tools lack recorders

-
Those that do records mouse co
-
ordinates


not underlying objects

Live Demo

25

© 2013 WWRUG Canada Inc. All Rights Reserved

26

Drawbacks of
AutoIT

recorder and other free tools

-
Most free tools lack recorder tools

-
Those that do, record mouse co
-
ordinates and mouse clicks, not underlying
objects


If elements in the web page move, then this will be problematic for
successful automated replay of the script

Drawbacks of
AutoIT

Recorder
and
Other Free Tools

26

© 2013 WWRUG Canada Inc. All Rights Reserved

27

A well written script must contain robust scripting features;

-
Error detection

-
Error handling


(
is it
a fatal
/
non
-
fatal error? Should the script take
a screenshot
on error
e.t.c
.,).

-
Multi outcome handling for a given action.


Sometimes the
interaction
with the GUI is not linear.

-
Modular script code


Create an external
library of reusable
components that can be reused and
shared across scripts.


E.g
, write the Remedy login and logout process once. The you can call
then via a function.

»
remedyWebLogin
(
url,username,password
)

»
remedyWebLogout
()

Extending the Script
-

Robust Scripting

27

© 2013 WWRUG Canada Inc. All Rights Reserved

28

A well written script must contain robust scripting features
(Cont.);

-
Ability
to clean
-
up
and re
-
set the
stage
, if a fatal error occurs


Trap the error, take a screenshot
and re
-
start the script


On restart, logout and close the application so that the script can start
from a known entry point.


-
Note: When running the GUI script via Scapa TPP, the restarting of the Script
is automated.


This enables you to sustain pressure on a system, even when the
underlying application has fatal error issues.


This simplifies the triage efforts as you can go hunting or change
configurations on the fly whilst the load is maintained

Extending the Script
-

Robust
Scripting (
Cont
.)

28

© 2013 WWRUG Canada Inc. All Rights Reserved

29

Templatizing

is the key to fast scripting and robustness

Generic templates


e.g., If fatal error occurs, then you can call a


fatalError
(message) function

»
This function may take a message, send a message to Scapa’s
centralized log, take a screenshot and store it in a central location
before exiting. You only have to write this once, then call it from
other scripts at the appropriate time.

Remedy Specific Templates

-
remedyWebLogin
(
url,username,password
)

-
remedyWebClickMenu
(“Incident Management”)

-
remedyWebLogout
()

These functions can be stored centrally and accessed from other scripts

The Importance of Script Templates


29

© 2013 WWRUG Canada Inc. All Rights Reserved

30

Add Scapa specific function to the GUI Script.

-
You can call additional Scapa specific functions from the GUI script. This
enables Scapa to control the automated GUI script, pass back messages to
the Scapa log, pass back timer metrics, send data to the script at run time,
e.t.c
., Some of the
funcitons

available are;


Func

ScapaTimerStart
($name)


Func

ScapaTimerStop
($name)


Func

ScapaTimerAbort
($name)


Func

ScapaTimerPause
($name)


Func

ScapaMsgWarning
($
pstrMsg
)


Func

ScapaMsgError
($
pstrMsg
)


Func

ScapaMsgInfo
($
pstrMsg
)

The Importance of Script
Templates (Cont.)

30

© 2013 WWRUG Canada Inc. All Rights Reserved

31

Add Scapa specific function to the GUI Script.

-
Controlling an automated GUI script from Scapa TPP via the Scapa script
extensions.


Func

ScapaSleep
($
millisecs
)


Func

ScapaGetLocalString
($sample, $group, $choice,
ByRef

$result,
$
rtSize
)


Func

ScapaStillRunning
()


Func

ScapaStopTest
($
pstrMsg
,$
pintVal
)


Func

ScapaStopScript
($
pstrMsg
, $
pintVal
)


Func

ScapaRendezvous
()

The Importance of Script Templates (Cont.)

31

© 2013 WWRUG Canada Inc. All Rights Reserved

32

Once you have templates, the requirement for a Script recorder
reduces

-
The fact that
AutoIT

does not have a useful recorder is no big deal as the GUI
script becomes a list of user interactions. E.g.,




The Importance of Script Templates (Cont.)


32

Hud
(@
UserName
$ +
“about to login to Remedy”)

scapaTimerStart
(“Login Time”)

remedyWebLogin
(
url,username,password
)

scapaTimerStop
(“
Login Time
”)

Hud
(@
UserName
$ + “now logged in to Remedy”)

scapaSleep
(1000)


Hud
(@
UserName
$ +
“about to open Incident
Management form”)

scapaTimerStart
(“Open IM Time”)


remedyWebClickMenu
(“Incident Management
”)

scapaTimerStop
(“Open IM Time”)

Hud
(@
UserName
$ + “now
in Incident Management form”)

scapaSleep
(1000)


© 2013 WWRUG Canada Inc. All Rights Reserved

33

You can run
l
oad, performance, capacity and reliability tests

-
Run many automated GUI users at the same time

-
Centralized results and log repository in a SQL database

Centralized repository of data, supplied to the script at run time


E.g., store the username/passwords in Scapa


not in the GUI script.

Graphical
V
iew of Timers



displayed live and post test within the Scapa GUI

Alerting on breach criteria

-
Useful for SLA monitoring tests

Benefits of Running via Scapa TPP

33

© 2013 WWRUG Canada Inc. All Rights Reserved

34

Add timers to the script.

-
E.g., time how long it takes for the login screen

-
Time how long it takes to login

-
Time to query
e.t.c
.,

Benefits of Running via Scapa
TPP (Cont.)

34

© 2013 WWRUG Canada Inc. All Rights Reserved

35

© 2012 WWRUG Canada Inc. All Rights Reserved

Full screen slide example

© 2013 WWRUG Canada Inc. All Rights Reserved

36

© 2012 WWRUG Canada Inc. All Rights Reserved

Full screen slide example

© 2013 WWRUG Canada Inc. All Rights Reserved

37





















Scalability and Performance
C
haracteristics

© 2013 WWRUG Canada Inc. All Rights Reserved

38





















Scalability and Performance Characteristics (cont.)

© 2013 WWRUG Canada Inc. All Rights Reserved

39




















Real Life Results



Mixed Remedy GUI and Remedy API Test

Edge Of Capacity (Pointing at the green line) @ Approximately
425 + 40 = 465TPH (Combined the API+GUI TPH).

Sub
-
Sequence Timers Shown here in light blue/green.

© 2013 WWRUG Canada Inc. All Rights Reserved

40




















Real Life Results


Wan Optimizer Comparison

Edge of capacity can be seen here. The system scaled in proportion to load up to this point. The amount of work the system
can

perform
(green line) does not increase after this point


even though were request it to do so, by the introduction of additional concur
rent users (blue
line). As the system cannot perform any more work than it is currently handling. The login request are therefore queued


the l
ogin time
increases (black line)


© 2013 WWRUG Canada Inc. All Rights Reserved

41




















Real Life Results


Memory Leak

© 2013 WWRUG Canada Inc. All Rights Reserved

42




















Real Life Results


Poor Performance and Scalability

Testing and scalability signature before the fix.
This system had serious issues.

© 2013 WWRUG Canada Inc. All Rights Reserved

43




















Real Life Results


Good Scalability

Good result


system
now scales
in proportion to load.

High
transactional throughput achieved at the edge of approximately
2.9

Almost
6 times more
capacity

AND consistency in end user experience.

© 2013 WWRUG Canada Inc. All Rights Reserved

44




















Real Life Results
-

Serialization

© 2013 WWRUG Canada Inc. All Rights Reserved

45




















Real Life Results
-

Serialization

© 2013 WWRUG Canada Inc. All Rights Reserved

46

Lets take a look at a
templetized

script written in
AutoIT

Lets run that script via Scapa with multiple concurrent users

Lets force a failure and see the
fatalErrorHandle

and post recovery
clean up in action.

Lets run a Scapa report on the test results

Lets run a query against the Scapa SQL
datastore

Lets control Scapa TPP via an
AutoIt

script

-
Scapa TPP has its own API which other application can call to control and run
tests.


Live Demo


46

© 2013 WWRUG Canada Inc. All Rights Reserved

47

1
. All in
One Solution


Multiple Entry point in the Architecture.

-
The
only tool that provides complete Remedy testing in a single product
enabling unrestricted and unrivalled visibility of system performance and
end user
experience
.


Mid Tier


AR
System
API


Web Services


GUI

10 Reasons to choose Scapa (cont.)

© 2013 WWRUG Canada Inc. All Rights Reserved

48


GUI automation of common activities can free you up for other
activities

Tempting is critical in reducing the time taken to create robust
GUI scripts

GUI automation is less abstract than testing via the Remedy APIs

GUI testing is necessary for functional and functional regression
testing

-
Run pre and post change tests and discover differences in functionality and
or performance and scalability

Running functional and performance tests will lower
costs
by
trapping and avoiding issues early

Conclusion

© 2013 WWRUG Canada Inc. All Rights Reserved

49

Scapa TPP is an all
in One Solution


Multiple Entry point in the
Architecture.

-
The only tool that provides complete Remedy testing in a single product
enabling unrestricted and unrivalled visibility of system performance and
end user experience.


Mid Tier


AR System API


Web Services


GUI


Test can be run in mixed mode.

-
E.g., running GUI, mid tier (http and Remedy backchannel) and AR System
API at the same time.

Conclusion

© 2013 WWRUG Canada Inc. All Rights Reserved

50

When
run via Scapa TTP, you can run many concurrent
GUI users
and many concurrent API users, to
identify
problematic areas
within the Remedy stack, thus

-
improving
the quality of the system during development to reduce release
cycles, speed time
-
to
-
production and minimize costly rework once the
system is deployed
.

Can reduce
risk by quantifying performance, scalability, capacity
and
reliability of the application and hardware stack

-
Scapa TTP
can reduce
risks associated with the performance,
scalability
,
capacity and
reliability. Issues are uncovered
early so that
remedial
work
can be planned if required.

Repeatability

-
Manually repeating the same activity within a timeframe is difficult, if not
impossible. Repeatability is required for benchmarking and regression
testing, therefor automation key.


Conclusion

© 2013 WWRUG Canada Inc. All Rights Reserved

51


Testing can maximize
profit, return on investment and improve
competitiveness

-
Scapa TTP maximize business benefits by
enabling you to increase the
responsiveness of applications, optimizing the system through tuning and
removal of bottlenecks, thus enhancing user productivity
.

Testing can assure
the quality of
Remedy
based
solutions

-
Scapa TTP
provides
the information required for a
confident and
smooth
production rollout
.

Automate test to run periodically on the production system to
maintain on going
continuity and Service
Levels.

-
Performance
baselines are used to
maintain on going
performance and
availability of the system in production.

Conclusion

© 2013 WWRUG Canada Inc. All Rights Reserved

52

Your thoughts (the audience)

-
Did you find this talk useful?

-
Can you see benefits of running GUI automation script on in your own
environments?


Question/Discussions

© 2013 WWRUG Canada Inc. All Rights Reserved

53

Contact information

-
Stop by the Scapa stand for a chat with Derek and/or
Armen

-
Phone Derek on +44 (0)7740 368459

-
http://uk.linkedin.com/pub/derek
-
roberts/3/b58/396/


-
Visit
www.scapatech.com

-
UK +44 (0)1382 220163

-
USA +1 530 853 4004

-
Ireland +353 1 442 8419


Wrap
-
up