Automated Software Testing using Open Source Testing Tools

farrightSoftware and s/w Development

Aug 15, 2012 (5 years and 1 month ago)

380 views



Automated Software Testing using Open Source
Testing Tools

By Elfriede Dustin



http://www.idtus.com

2

Agenda



Who is IDT


State of Software Testing


What is Automated Test and Re
-
test (ATRT)


Approach to ATRT


Advantage of open source


ATRT and open source


Our IDE and
solution


ATRT Challenges





http://www.idtus.com

3

Who is IDT?


IDT specializes in the design, development, and
implementation of Automated Software Testing and
Re
-
Test (ATRT) and Quality Assurance (QA)
solutions


Deliver turn key automated test suite


Automated test strategy and identification of highest payoff areas to
apply automation for your project


Selection of best automation tools for your project


Using your existing test cases or develop test cases for you


Test results documented in requirements traceability matrix


Training and pilot project implementation with companies




http://www.idtus.com

4

Who is IDT?


IDT specializes in the design, development, and
implementation of Automated Software Testing and
Re
-
Test (ATRT) and Quality Assurance (QA)
solutions


Deliver turn key automated test suite


Automated test strategy and identification of highest payoff areas to
apply automation for your project


Selection of best automation tools for your project


Using your existing test cases or develop test cases for you


Test results documented in requirements traceability matrix


Training and pilot project implementation with companies





We are currently hiring Java Developers


Plus we’d like to hire 2 or 3 grad students to
work for us part time ( i.e. 20 hours week )



must be US citizen

http://www.idtus.com

5

State of Software Testing
-

Why ATRT?

?

http://www.idtus.com

6

State of Software Testing
-

Why ATRT?

Utilize Automated Testing Strategies and
Technology to Improve Productivity and Quality

Reduction in
Test Days

IDT Provides Automated Software Testing Solutions

Increased
Test
Coverage

Manual vs. Automated

Spiral / Release 1

Spiral / Release 2

Spiral / Release 3

Spiral / Release 4

Size & Complexity of SW Baseline

Test
Days

Test
Days

Test
Days

Test
Days

50% or More of Overall Development Cost is
Typically Spent on Testing

http://www.idtus.com

7

What is ATRT?


Our definition of ATRT is:



Application and implementation of software
technology throughout the entire Software
Testing Life Cycle (STL) and QA lifecycle;
with the goal to improve STL efficiencies and
effectiveness


http://www.idtus.com

8

ATRT Spans the Software Development

Life Cycle


Visual Modeling


Modeling and test case generation

Development Tools

Components

Use
cases

Desig
n

Build

Assembl
e

Round
-
Trip Engineering

Test

http://www.idtus.com

9

ATRT Spans the Software Development

Life Cycle


Requirements




Organizes, tracks, & controls requirements


Requirements Traceability Matrix (RTM)

Development Tools

Components

Visual
Modeling

Requirements Management and

Process Automation


http://www.idtus.com

10

Automated Software Testing

ATRT Spans the Software Development

Life Cycle


Automated Test Tools

Development Tools

Components

Visual Modeling

Execution

Management

Development

Requirements Management and

Process Automation





Automates test cases using vendor
-
provided,


open
-
source tools or in
-
house development

Automated Testing Tools

http://www.idtus.com

11

ATRT Spans the Software Development

Life Cycle


Integrated Suite of Tools

Additionally:


Middleware


Infrastructure


Defect Tracking


Configuration
Management


Memory Leak
Detectors


Performance
Testing Tools


Documentation
Tools


others

Development Tools

Components

Visual Modeling

Automated Testing Tools

Requirements Management and

Process Automation


Software Configuration Management

Defect Tracking

Middleware

Infrastructur
e

http://www.idtus.com

12

Types of Software Test
Life
-
cycle Support Tools





Which tools are you familiar with?

http://www.idtus.com

13

Types of Automated Test
Tools

Life
-
Cycle

Phase

Type of Tool

Tool Description

Business

Analysis

Phase

Business Modeling

Records definitions of user needs and automates rapid
construction of

flexible, graphical, client
-
server applications

Configuration

Management

Baselines important data repositories

Defect Tracking

Manages system life
-
cycle defects

Technical Review

Management

Facilitates communication, while automating the
technical review/inspection process

Documentation

Generators

Automate document generation

Requirements

Definition

Phase

Requirements

Management

Manages and organizes requirements; allows for test
procedure design and test progress reporting

Requirements

Verifiers

Verify syntax, semantics, and testability

Use Case

Generators

Create use cases

http://www.idtus.com

14

Types of Tools (Cont’d)

Programming
Phase

Memory Leak and
Runtime Error
Detection

Detects runtime errors and memory leaks

Source Code Testing

Verifies maintainability, portability, complexity, and standards compliance

Static and Dynamic
Analyzers

Depict quality and structure of code

Life
-
Cycle

Phase

Type of Tool

Tool Description

Analysis and
Design Phase

Database Design

Develops second generation enterprise client
-
server systems

Structure Charts,
Flowcharts, and
Sequence Diagrams

Manage processes

Test Procedure
Generators

Generate test procedures from requirements, design, or data and object
models

Syntax Checkers/
Debuggers

Perform syntax checking and have debugging capability; usually available
with built
-
in programming language compiler

Metrics Tools

Code (Test) Coverage
Analyzers or Code
Instrumentors

Identify untested code and support dynamic testing

Usability Measurements

Provide usability testing as conducted in usability labs

http://www.idtus.com

15

Types of Tools (Cont’d)

Life
-
Cycle

Phase

Type of Tool

Tool Description

Other Testing

Life
-
Cycle

Support Tools

Data Extraction Tool

Extract Data from various formats into various formats

Test Data Generators

Generate test data

File Compare Utilities

Find discrepancies between files that should be identical in content

Simulation

Simulates application to measure for scalability, among other tasks

Test Management

Tests management

Network Testing

Monitors, measures, tests, and diagnoses performance across the
entire network

GUI Testing
(Capture/Playback)

Conducts automated GUI tests; capture/playback tools record user
interactions with online systems so they may be replayed
automatically

Load/Performance Testing

Conducts load/performance and stress testing

Security Testing

Performs security testing and vulnerability scanning at the
application or network level; plus debuggers will allow to check for
security coding errors (source code checkers)

http://www.idtus.com

16

Deployment Approach



Adapted ATLM

Collect/Create:



Requirements



Test Cases



Test Procedures



Expected Results



Interface Specifications



System/Component

Configuration Description

Identify or Develop

& Pilot



Test Manager



Test Scripts



Data Comparators



Performance Test Tools



GUI Record/Playback



Storing/Retrieving Results


Populate Test Manager
with Test Cases and
Requirements







Develop Test Scripts
from Test Procedures








Test Results:



Pass/Fail by Test Case



Pass/Fail Summary



Test Time Required

http://www.idtus.com

17

Why Open Source

?

http://www.idtus.com

18

Why Open Source?


Advantage of Open Source:


no licensing issues; i.e. license cost; maintenance;
etc.


easily modifiable


adaptable


lightweight


flexibility
-

not tied to one vendor


http://www.idtus.com

19

ATRTs IDE
-

Eclipse

Eclipse


this site lists all the plugins


http://www.eclipse
-
plugins.info/eclipse/plugins.jsp?category=SCM&pager.offset=0&firstItem=1


Eclipse/TPTP


Open Source
-

Developed by IBM


Eclipse/Test and Performance Tools Platform (TPTP)
offers a common extensible framework for the following
functions:


Requirements Management
http://sourceforge.net/projects/osrmt/



Configuration Management
-

Subversion


Testing Tools (http://www.nabble.com/Eclipse
-
TPTP
---
Testing
-
Tools
-
f2262.html)


Profiling: default Java applications, but can be extended to other
apps

http://www.idtus.com

20

ATRTs IDE
-

Eclipse

Eclipse/Test and Performance Tools Platform (TPTP)
offers a common extensible framework for the
following functions (cont):


Static Analysis: default for C++ and Java, but can be
extended to other apps


Application monitoring and log analysis


Build tool


ant


Xml input and output
-

using an open source Java
package
http://jakarta.apache.org/ecs/

, which is a Java
library that has an API for constructing and outputting
XML.



Test Manager


STAF/STAX



http://www.idtus.com

21

Current Test Tool Implementation


Example and Challenges

Vendor
-
provided (Capture/Playback) Tool:


Automated test tools mimic actions of the test engineer.

During testing, the engineer uses the keyboard and mouse to perform
some type of test or action.

Testing tool captures all keystrokes and subsequent results, which are
baselined in an automated test script.

During test playback, scripts compare latest outputs with previous
baseline.

Testing tools have built
-
in, reusable test functions.

Most test tools provide for non
-
intrusive testing; i.e., they interact with
the “application
-
under
-
test” as if the test tool was not involved.

http://www.idtus.com

22

Current Testing Tools


Challenges
(cont)


In
-
House Software Development required:


Capture/Playback tools generate hard
-
coded values; test
scripts are not reusable, nor do they implement software
development best practices right out of the box; scripts need
to be modified.

Capture/Playback tools don’t necessarily provide all testing
features required; code enhancements are often required to
meet testing needs.

Capture/Playback tools are not necessarily compatible with
system engineering environment, and software testing scripts
need to be developed in
-
house.

Developed testing scripts are not cross
-
vendor compatible,
even if the same scripting language is used


http://www.idtus.com

23

Linux Capture/Playback Tools

IDT Findings

Replay Xcessory

http://www.scl.com/products/ics/motif/testing/replay/

www.ics.com

QF
-
Test

http://www.qfs.de/en/qftestJUI/index.html

Froglogic

http://www.froglogic.com/pg?id=Google&category=squishjava

Redstone Software

Eggplant

KD Executor

http://www.kdab.net/?page=products&sub=kdexecutor

Jameleon

http://sourceforge.net/projects/jameleon/

ShUnit2

for

shell

scripts

testing

GNU Linux Desktop Project

http://ldtp.freedesktop.org/wiki/Home

Cantana

http://www.ipl.com/pdf/p0003.uk.pdf

crontab

Part

of

Unix



crontab, cruisecontrol, www.adminschoice.com/docs/crontab.htm

Dogtail

http://people.redhat.com/zcerza/dogtail/about.html

http://www.cyberciti.biz/tips/linux
-
automated
-
gui
-
testing
-
with
-
python
-
based
-
dogtail
-
tool.html

Expect

http://expect.nist.gov/

http://www.idtus.com

24

Automated Test and Re
-
Test
(ATRT)


ATRT includes a common framework comprised of
re
-
usable software services


Open Source Software Test Automation Framework
(STAF)



Provides automated test procedures, test inputs, and
test results for SW components


Suitable for use by 3
rd

parties not associated with
development of the component

http://www.idtus.com

25


STAX is an automation system


With STAX you can automate and monitor your entire test environment

ƒ
System/product setup

ƒ
Testcase distribution

ƒ
Testcase execution

ƒ
Testcase results collection and analysis


STAX consists of

ƒ
A programming language designed for automation


Programs written in this language are called jobs

ƒ
An execution engine (similar to an interpreter for other programming/scripting
languages) which manages the execution and runtime behavior of the job

ƒ
A GUI application, called the STAX Monitor, which provides a dynamically
updated view of your executing jobs

ƒ
A tool (STAXDoc) that is used to generate documentation for your STAX jobs


What is STAX?

http://www.idtus.com

26



* Windows 95



* Windows 98 (and 98 SE)



* Windows Millenium Edition



* Windows NT Version 4.0



* Windows 2000



* Windows XP



* Windows Server 2003 (IA32, IA64, AMD64)



* Windows Vista (IA32, AMD64)



* Linux (Intel32/64, AMD64, PPC32/64)



* Linux on zSeries (31
-
bit, 64
-
bit)



* AIX Version 4.3.3.0 or higher (32
-
bit, 64
-
bit)



* Solaris (Sparc) 2.6 and higher



* Solaris AMD Opteron 64
-
bit



* HP
-
UX 11.00 and higher (PA
-
RISC, IA64 32
-
bit and 64
-
bit)



* OS/400 V5R2 or higher



* z/OS UNIX V1.4 and higher

STAF can run on:

http://www.idtus.com

27

STAF Internal Services





DIAG
Provides diagnostics services
Internal ("DIAG")


DELAY
Provides a means to sleep a specified amount of time
Internal ("DELAY")


ECHO
Echos back a supplied message
Internal ("ECHO")


FILE SYSTEM
Allows you to get and copy files across the network
Internal ("FS")


HANDLE
Provides information about existing STAF handles
Internal ("HANDLE")


HELP
Provides Help on STAF error codes
Internal ("HELP")


MISC
Handles miscellaneous commands such as displaying the version of STAF


that is currently running
Internal ("MISC")


PING
Provides a simple is
-
alive message
Internal ("PING")


PROCESS
Allows you to start, stop, and query processes
Internal ("PROCESS")


QUEUE
Provides a network
-
enabled IPC mechanism for STAF Programs
Internal ("QUEUE")


SEMAPHORE
Provides network
-
enabled named event and mutex semaphores
Internal ("SEM")


SERVICE
Allows you to list services available on a machine and to examine the


Requests that have been submitted on a machine
Internal ("SERVICE")


SHUTDOWN
Provides a means to shutdown STAF and register for shutdown


Notifications
Internal ("SHUTDOWN")


TRACE
Provides tracing information for STAF services
Internal ("TRACE")


TRUST
Interfaces with STAF's security
Internal ("TRUST")


VARIABLE
Provides a method for maintaining configuration and runtime data


(variables)
Internal ("VAR")



http://www.idtus.com

28

STAF External Services

The executable code for external STAF services resides outside of STAFProc, for
example in a Java jar file, a C++ DLL file, or a Rexx script file.



CRON Calls into STAF services at a specified time interval
External (Java)


EMAIL Allows you to send email messages
External (Java)


EVENT Provides a publish/subscribe notification system
External (Java)


EVENTMANAGER Allows you to call STAF services when a specified Event
occurs
External (Java)


HTTP Allows you to make HTTP requests which can be grouped together in a
session
External (Java)


LOG Provides a full
-
featured logging facility
External (C++)


MONITOR Allows a testcase to publish its current running execution status for
others to read
External (C++)


RESOURCE POOL Allows you to manage exclusive access to pools of elements,
e.g. VM UserIDs or Software Licenses
External (C++)


STAX Provides an XML
-
based execution engine
External (Java)


ZIP Provides a means to zip/unzip/list/delete PKZip/WinZip compatible archives
External (C++)

http://www.idtus.com

29


STAF

ƒ
STAF provides the infrastructure on which STAX builds. The full power of STAF
and its services is exposed for use within STAX jobs.


XML

ƒ
The STAX programming language is based on XML. This provides built
-
in
structure to your jobs, as well as providing a set of existing tools for use in
constructing your jobs, such as XML (aware) editors and XSLT.


Python

ƒ
The STAX programming language builds on Python to provide a rich and
accessible data model

ƒ
STAX's Python integration also allows you to access the wealth of existing
Python libraries


Java

ƒ
The STAX programming language allows you to access existing Java
classes/libraries, providing another source for reuse

Technologies used in STAX

http://www.idtus.com

30

Automation Tasks

Execution

Synchronization

Monitoring

Resource Management

Automation Completion

Testcase Output Analysis

Results Notification

Testcase Cleanup

Testcase Execution

System Setup

Automation Startup

http://www.idtus.com

31

End
-
to
-
End Automation with STAF and
STAX

Event, EventManager, Cron

FS, Process

Process, Monitor, Log, Variable,

Queue, ResPool

Log, Process


Process


Email, HTTP

S

T

A

X


J

o

b

<function>

<parallel>

<sequence>

<process>

<stafcmd>

<testcase>

<timer>

<block>

<loop>

<message>

<log>

<import>

<job>

<iterate>

<paralleliterate>

<hold>

<release>

<terminate>


Automation Completion

Results Notification

Testcase Cleanup

Testcase Output Analysis

Testcase Execution

System Setup

Automation Startup

http://www.idtus.com

32

Key ATRT Architecture
Requirements



Support applications running on multiple computers



Support applications developed in different languages



Support applications running on different types of OS’s



Support applications which have GUI and those which do not (for

example Interface testing)



Support applications which use different types of network


protocols such as TCP/IP, DDS, etc



Support integration of multiple commercial testing tools from


different vendors ( allowing as new or better products emerge

in the market they can be utilized )



Support testing w/o having to install ATRT on the same


computers as the application under test and be able to be

distributed across computers

http://www.idtus.com

33


Display Application

Display Application

Analysis & Display

Application


Simulation of Application

Processing Messages


Simulation of Application

Processing Messages

System Under Test

Test Manager

RTM

Bugzilla

ATRT

Eggplant

R/P

VNC Robot

R/P

STAF/STAX

Prototype


Proof of Concept

Windows

Based

Linux Based

http://www.idtus.com

34




Able to successfully execute ATRT for 48 hours without operator
intervention




During the tests:




VNCRobot
-

Approximately 25,000 operator key strokes were
replayed



STAF/STAX
-

More than 1,000,000 messages processed and
verified by ATRT


ATRT Prototyping Effort

http://www.idtus.com

35

Additional Requirement


Pristine test environment required


automated
testing tool cannot be installed on Application
-
Under
-
Test PC




Eggplant


allows for remote capture/playback via
VNCserver and KVM switch



VNCRobot

http://www.idtus.com

36

Automated Software
Testing


Software testing paradox


What is it?



http://www.idtus.com

37

Effective Automated
Software Testing

Current State:

Developing software in order to test software

Automate the test support development: automated IDL
to code generation; test data generation (using FireEye
from NIST); compare utilities; log file analyzers;

Our Automated Software Testing Goal:


Minimize Human interaction


Possibly get on par with Hardware Automated Testing?

http://www.idtus.com

38

Hardware Testing: current
state


Many automated test generators for
manufacturing/ logic devices, circuit testing,
etc., available i.e. SEI/ATG, uMaster, etc.


Automated testing without human interaction
is the norm for hardware testing

http://www.idtus.com

39

Hardware vs. Software

AT

How is hardware testing different?


Software Challenge:


Changing requirements


GUI interfaces


Code access/availability


Defects uncovered earlier in the software
development lifecycle are cheaper to fix


Solution flexibility required


problem
dependent




http://www.idtus.com

40

Proposed solutions


Focus on Reusable Components


Test Data / Expected Results


Interfaces


Middleware

Assuming code and models are available:


Model based testing; automated test case generators based on models


Eclipse


effective ATRT IDE


Probes


see Aprobe


Code instrumentation


Building self
-
testable components


Using standard language MOF to Text to generate automate test cases

GUI Testing tool ideas

New tools
-

other


Working with NIST to produce additional tools



http://www.idtus.com

41

GUI Testing


Jemmy
-

java library to test UI



http://jemmy.netbeans.org/



http://abbot.sourceforge.net/doc/overview.shtml



Perl modules



run via VNCServer?

http://www.idtus.com

42

Questions

Email
edustin@idtus.com

with any follow
-
up questions

http://www.idtus.com

43

IDT

We are hiring…….

Please come see me now if interested or send

email
edustin@idtus.com

with any follow
-
up