Our automated solution at a glance By: Justin Hollander

subduedjourneyΛογισμικό & κατασκευή λογ/κού

28 Οκτ 2013 (πριν από 4 χρόνια και 8 μήνες)

64 εμφανίσεις

Our automated solution at a glance

By: Justin Hollander

About the Presenter

Employed at Squirrel Systems since August 2007 as a QA

Volunteered one year in Guatemala to help build an
online system for the Central American district. Trained
staff and volunteers to use the systems. Prepared
manuals and other documentation

BCIT Alumni and currently attending BCIT

About the Presentation

Benefits of Automation

Description of automation
needs at Squirrel Systems

Design goals vs. cost

Previous attempts at
solving the automation /
autoretest problem

The current design of our
autoretest solution

Problems we overcame

Future Considerations

Questions and Answers

Our Automation Needs

Intended to reduce overhead of manual certification

To allow other testers to do more functional /
exploratory testing on other parts of the system that
cannot be tested in an automated fashion

Consistency in testing. Day in and day out.

Previous Attempts

Attempt 1: Batch processing, relied on playback
utilities, with an Access Database(DB) to manually
track successes and failures of the running scripts

Attempt 2: Improved batch processing(more readable
scripts), relied on playback utilities, without the
Access DB to track the successes and failures of the
running scripts

Problems with Attempt 1

Script information and state were manually tracked

Batch scripting is not as flexible or powerful as a programmatic
solution to get the options you need for a comprehensive automated

Results were not archived and therefore could not be linked

Output was in a raw text only format and very cumbersome to process

Too many false positives as comparison was blind, and not intelligent,
nearly impossible to trust results

Problems with Attempt 2

Same as attempt 1, except now, without the Access DB,
there was no way to track the running scripts

Solution was scaled back due to resource constraints
and the project had a hard time getting traction and

Ultimately, the project was suspended

What we Learned

Automation/autoretest is more complex than we thought

There is no “easy” solution

Externalizing our test input data while using our playback utilities to
retest the system is impossible

At a minimum; output data must be analyzed better to eliminate false

Batch scripting isn’t working

A more dedicated approach is required

Autoretest, TAKE 3!

We decided to try this again

Decision to scrap previous work and start again from
design phase

Requirements were well understood, however, the
right skill
set was missing until now

High level Design vs. Cost

Design goals

Cost Considerations

To use existing playback tools within a
framework to retest recorded ‘tracks’

Use intelligent comparison algorithms
to rule out false positives

Output results in a human friendly
format in order to ensure a quicker
analysis of the potential problems

Externalize the test input data from
the system

usable parts to extend framework
to all parts of POS system

Limited to 40 man
hours a
week(Basically me


Development team on a tight schedule
and have limited resources for this

No money for off
shelf solution

Management wanted a solution rather
quickly(6 months)

Money already invested; 2 previous
attempts in solving this in last 5 years
have failed

The Current Design

Client/Server model

Written in Java with easy to use administrative GUI

Externalized data as much as possible

Used intelligent output comparison algorithms

One DB configuration supports one or more test scripts

Supports XML processing

Comparison output formatted in HTML

Automatically detect if software that is being tested is the current build. Will update if

The Current Design cont..

Test outputs and comparison results archived to achieve the possibility
of linking of historical data to present results

Script information stored in a relational SQL DB

Script information linked to GUI using custom written SQL driver

Script states linked to scripts via GUI and they are not static

Client has a head
less mode supported by multi
threaded timer

Extensible due to its polymorphic nature and can easily support
additional forms/levels of testing

Screenshot of GUI

Screenshot of formatted Output

Screenshot of XML Utilization

Code Sample

DB Compare

private static void
, Vector<String>


if (
//table contents equal??





//everything not equal probable bug or code change in squirrel


if (
() ==
//just mismatches ??


for (int i = 0; i < vB.size(); ++i)


if (


, "VALID");

, "VALID");
//the only ones not tagged are the mismatches and the can get filtered out while still preserving the row number




else if(
() >



else if(
() <
, this should be rare


// that should leave us with any differences for the HTML Manager to process further.


Problems Solved with this

All DB false positives have vanished

We are able to use our scripts longer, as it is easier to trust
output with limited false positives

Output is easy on the eyes with HTML colour coded output

Playback utilities have been made more robust

System is much more configurable than before

Considerations for the Future

Remote Web Administration using JSP and Java

Make better use of RMI technology and turn this into a
distributed system from a client/server model

Completely externalized data

Build robots to do the maintenance on the system

Questions and Answers