Software Testing

aniseedsplashSoftware and s/w Development

Aug 15, 2012 (4 years and 10 months ago)

282 views

cs498dm

Software Testing

Darko Marinov

January 22, 2009

Course Overview


Introduction to software testing


Systematic, organized approaches to testing


Based on models and coverage criteria


Testing is not (only) about finding “bugs”


Improve your testing (and development) skills


Teaching staff


Insructor: Darko <marinov@cs.uiuc.edu>


TA: Vilas <vbangal2@cs.uiuc.edu>

Administrative Info


Deliverables


No exams: no final, no midterm


Five problem sets (5*15%) and a project (25%)


Project: proposal, two reports, hopefully bug reports


Project is on testing a piece of refactoring engine


Undergrads must be registered for 3 hours


For more, you can do an independent study


If interested in research on testing, contact me


Juniors: ask for permission if you didn’t

“Assignments”


Did you receive emails on the list

cs498dm@cs.uiuc.edu


Wiki permissions should be set now

https://agora.cs.uiuc.edu/display/cs498dmsp09/Home


Did you try to find the textbook?


Did you try out Eclipse/NetBeans?


Course will use Java


If you’re not familiar, please review some intro

“Prizes”


Potential categories


The “buggiest” bug found in the course


Hardest to find, most important, realistic case study


Most bugs found or tests generated


Not the best measures of testing effort



Example bug from Matt


Armadillo Aerospace X Prize lunar lender crash
http://www.youtube.com/watch?v=W01a3xj
-
9Xs

This Lecture: Introduction (
Cont’d
)


Why look for bugs?


What are bugs?


Where they come from?


How to detect them?



Topics that will be covered in the course


Related topics that will not be covered

“Bugs” in IEEE 610.12
-
1990


Fault


Incorrect lines of code


Error


Faults cause incorrect (unobserved) state


Failure


Errors cause incorrect (observed) behavior



Not used consistently in literature!

Correctness and Quality


Common (partial) properties


Segfaults, uncaught exceptions


Resource leaks


Data races, deadlocks


Statistics based



Specific properties


Requirements


Specification

Traditional Waterfall Model

Requirements

Analysis

Design

Checking

Implementation

Unit Testing

Integration

System Testing

Maintenance

Regression Testing

We will look at general
techniques, applicable in
several phases of testing

Phases (1)


Requirements


Specify what the software should do


Analysis: eliminate/reduce ambiguities,
inconsistencies, and incompleteness


Design


Specify how the software should work


Split software into modules, write specifications


Checking: check conformance to requirements,
using for example
conformance testing

Phases (2)


Implementation


Specify how the modules work


Unit testing
: test each module in isolation


Integration


Specify how the modules interact


Integration testing
: test module interactions


System testing
: test the entire system


Maintenance


Evolve software as requirements change


Regression testing
: test changes

Testing Effort


Reported to be >50% of development cost
[e.g., Beizer 1990]



Microsoft: 75% time spent testing


50% testers who spend all time testing


50% developers who spend half time testing



When to Test


The later a bug is found, the higher the cost


Orders of magnitude increase in later phases


Also the smaller chance of a proper fix



Old saying: test often, test early


New methodology: test
-
driven development

(write tests even before writing code)

Software is Complex


Malleable


Intangible


Abstract


Solves complex problems


Interacts with other software and hardware


Not continuous

Software Still Buggy


Folklore: 1
-
10 (residual) faults per 1000
nbnc lines of code (after testing)



Consensus: total correctness impossible

to achieve for complex software


Risk
-
driven finding/elimination of faults


Focus on specific correctness properties

Approaches to Detecting Bugs


Software testing


Model checking


(Static) program analysis




Software Testing


Dynamic approach


Run code for some inputs, check outputs


Checks correctness for some executions



Main questions


Test
-
suite adequacy (coverage criteria)


Test
-
input generation


Test oracles

Other Testing Questions


Selection


Minimization


Prioritization


Augmentation


Evaluation


Fault Characterization





Testing is not (only) about finding faults!

Current Status


Testing remains the most widely used
approach to finding bugs


Validation: are we building the right system?


Verification: are we building the system right?


Testing is gaining importance with test
-
first
development and increased reliability needs


A lot of research on testing (part of mine too)


This course is not about research

“Schools” of Software Testing


Bret Pettichord described four schools


Analytic (a branch of CS/Mathematics)


Factory (a managed process)


Quality (a branch of quality assurance)


Context
-
Driven (a branch of development)



This course focuses on artifacts, not process


Do you want a guest speaker from industry?

Topics Related to “Finding Bugs”


How to “eliminate bugs” (localize faults)?


Debugging


How to “prevent bugs”?


Programming language design


Software development processes


How to “show absence of bugs”?


Theorem proving


Model checking, program analysis

Testing Topics to Cover


Test coverage and adequacy criteria


Graph, logic, input domains, syntax
-
based


Test
-
input generation


Test oracles


Model
-
based testing


Testing software with structural inputs


Test automation


Testing in your domain of interest?

Summary of the Introduction


Eliminate bugs to save lives and money


“Bugs” may mean faults, errors, failures


Several approaches for detection: software
testing, model checking, static analysis…


Software testing is the most widely used
approach for validation and verification


We will cover systematic approaches to testing,
based on coverage criteria for various models


Testing is not (only) about revealing faults

Next Lecture


Tuesday, January 27, 2pm, 1302 SC


Example Interactive Testing Session


It should be fun


We can learn from mistakes as we go


We will learn some testing terminology


Assignments


Sign up on Wiki

https://agora.cs.uiuc.edu/display/cs498dmsp09/Home


Try out Eclipse and/or NetBeans