Notes on Java Unit Testing

paltryboarpigΛογισμικό & κατασκευή λογ/κού

3 Νοε 2013 (πριν από 3 χρόνια και 9 μήνες)

114 εμφανίσεις

Page
1

of
23

Notes on
Java Unit Testing

Created 07/04/06

Updated 08/11/06
,
Updated 11/05/06
,
Updated 11/26
/06
,
Updated 12/22
/06
,
Updated 12/26/
06
,

Updated 12/29/06

Updated 03/06/07
, Updated 12/01/07
, Updated 01/31/08
, Updated 03/30/08
, Updated 12/12/08
, Updated 03/12/0
9

Updated 03/26/09
,
Updated 03/27/09,
Updated 05/09
/09
, Updated 09/19/09
, Updated 10/27/09
, Updated 02/15/10

Updated 09/30/10
, Updated 10/30/10
, Updated 01/22/11
, Updated 04/17/11
, Updated 05/12/11
, Updated 02/07/12

Introduction

During 2006, we began to st
udy approaches to unit testing that provided better results than our typical test cases and
testing procedures. Basically, unit testing can be a support facility of code checks that can be carried out by the
build and runtime environment, being done by de
velopers before being handled over to QA.

U
sing unit testing
actually helps you write code faster while increasing code quality.


There are a number of support tools and frameworks for performing unit testing with a very good facility being
provided in Ja
va, and supported in Eclipse.
The most well
-
known one is JUnit, for which the main developers are
Kent Beck, Erich Gamma
, who are well
-
known from their “Design Patterns” work. JUnit
gives us a means to carry
out automated testing of the lower
-
level parts

of an application
or software platform
without requiring larger,
external tools such as Mercury Interactive.

A similar test framework is TestNG.


This document includes: notes on effect
ive testing

plans and policies
, notes on
the design

of JUnit, notes o
n

usage of
J
Unit in Eclipse,
notes on the design of TestNG,
notes

on usage of TestNG in Eclipse,
and notes on other testing
tools and frameworks (JMeter, etc.).

These are
drawn from a number of open
-
source web pages and guides.


Unit testing is a critical

part of the “Agile” development process, as it allows for continuous testing of the
continuous builds.

There is a very useful quote from one of the books used in these notes:


You don’t test a bridge by driving a single car over it right down the middle
on a clear, calm day. Yet
many programmers approach testing that same way


one pass right down the middle and they call it
“tested”. Pragmatic programmers can do better than that!


In these not
es, we have used two

example game program
s as models
fo
r try
ing out the ideas. One

game is
“MasterMind”, and the implementation is based on one published in “Basic Computer Games
,


by David Ahl,

in
1978.

The other is a text adventure game with battles and character attributes, written by Brandon Risberg during
20
05 to 2006.


We taught this material in the Computer Science course at Menlo School during 2006, using JUnit 3.8.3.

Since
then, the document has been updated to focus on JUnit 4.x.

Resources

https://www.strongspace.com/jbrains/public/JUnit4OnOnePage.pdf

This is a one
-
page reference to JUnit 4, update
to date as of early 2011.


“JUnit

in Action, Second Edition” by Petar Tachiev, Felipe Leme, Vincent Massol, and Gary Gr
egory.
M
anning
Press,
July

2010
,
504
pages. List price $49.95
, Am
azon price $31.49, used from $28.00
. Rated 4.5

stars on
Amazon.com. The

second edition covers JUnit 4.5
. The first edition is from 2003, and is relatively old.


“Pragma
tic Unit Testing, in Java w
ith J
Unit”, by Andrew Hunt and Dave Thomas. The Pragmatic Bookshelf, 2003
-
2004, 153 pages.
List price
$29.95, Amazon price $21.33, used from $18.64. This is part of the “Pragmatic
Programmer” series,
which is a series of

publications on programming skil
ls and approaches that first came out in
the late 1990’s. Each book gets excellent review comments, as they tend to be concise a
nd practical
.

This book got
very high
marks, and is still a common reference years later.


“Next Generation Java Testing:

Tes
tNG and Advanced Concepts”, by
Cédric Beust

and
Hani Suleiman
. Addison
-
Wesley
Professional, October

2007, 512 pages. List price $49.99,
Amazon

price, $40.96, used from $32.93. Rated
Page
2

of
23

4 stars on Amazon.com.


This book describes using TestNG along with som
e advanced TestNG concepts and goes
on to show how to use the framework to test out JEE projects.

There are also chapters showing the developer how
to integrate Test
NG with other frameworks (like S
pring, DBUnit, jwebunit etc) which is useful as this is pe
rhaps the
only place where JUnit is better than TestNG
.


“Test Driven: TDD and Acce
ptance TDD for Java Developers”

by
Lasse Koskela.

Manning Press, October 2007,
470

pages.
List price $44.99, Amazon price $29.69, used from $1
6.0
0. Rated 4 stars on Amazo
n.com.
Relatively
current
, and c
onsidered to be the best current book on test
-
driven design and development. C
overs the development
methodologies associated with testing, with less focus on the technology itself.


The text is
very well written and
engagi
ng.


Also surveys the range of tools available.


“Eclipse Distilled” by David Carlson.


Addison
-
Wesley, March 2005, 280 pages.
List p
rice $3
4.99
, Amazon price
$2
2.30
, used price $5
.94
.
Rated 4 stars on Amazon.com.
While this is primarily
a
user’s guide
to Eclipse, it has a
chapter on using JUnit and documents the JUnit test runner that is provided with Eclipse.

The book is written for an
early 3.x Eclipse release, so some screens are out of date.

Releases

The JUnit web site is
www.junit.org
. The project is maintained on SourceForge.


The current version of J
Unit is 4.10
, which
came out in
October 2011
. This is a point release in the JUnit 4.x series,
whic
h was initially released in February
2006.

One of the most c
ommonly
-
used releases is 4.8.3, which was current
in 2010.


However, JUnit 4
.x

is quite different from its predecessors, which

went up to
release
3.8.x, because JUnit 4

uses

Java 1.5’s
annotation
facilities to organize the test cases.

In this document,
mo
st

examples use the

JUnit 4.x
conventions and but some still
follow the earlier ones.


Starting with Eclipse 3.2.x, there has been support for JUnit 4.x, an
d Eclipse 3.6
.
1

i
s
shipped with JUnit version
4.8.1
.


S
ummary of chan
ges in recent versions of JUnit
:



4.8


contains

pre
liminary support for Categories, which expand on the category concept in version 4.5.



4.7


contains

a new mechanism for changing the behavior and interpretation of test methods.

This feature
is called Rules.



4.6


contains
a n
ew exper
imental Core, `MaxCore` which remembers the results of previous test runs in
order to run new tests out of order. Adds an assert to compare arrays of doubles. Minor bug fixes and doc
updates.



4.5


adds facility for each test method and test class can be

annotated as belonging to a
category
:

The
runner can than be configured to run all tests within a category.



4.4


adds a new family of assertions, called “assertThat”. Adds a facility to collect tests and run them
under “Theories”.


In this document, we

assume use of JUnit
4.3.x
, since that is the version within Eclipse

during 2007
-
2008
. Hence,
we have never used the concepts of ‘
theories
’, ‘categories’, ‘assertT
hat’, or the runner core.


For TestNG, the
web site

is

http://testng.org/doc/index.html
.


The
current
version

is 6.0.1

released in late July

2010.

The 6.x series initially came out
in March 2011
. The 5.x
series
initially came out in July 2006.


There is a
plug
-
in

for running TestNG in Eclipse av
ailable for use in the “Software Update” feature of Eclipse at
http://beust.com/eclipse
.

Page
3

of
23

Effective Testing

Testing is not always easy to carry out.
It can be hard to look at a method or class and try to come up wit
h all the
ways that it might fail; to anticipate all of the bugs that might be lurking in there.


The “Pragmatic Unit Testing”
book suggests six specific areas to think about and check test coverage
, which
can be
remembered using the acronym “Right

BICEP”
:




Right


Are the results right?



B


Are all the boundary conditions correct?



I


Can you check inverse relationships?



C


Can you cross check results using other means?



E


Can you force error conditions to happen?



P


Are performance characteristics wit
hin bounds?

Are the Results Right?

The first and most obvious area to test is simply to see if the expected results are right, in other words to validate the
results. You need to be able to answer the key question: If the code ran correctly, how would we

know? If it is not
clear what the correct results are, you must check with users and potentially refine the specifications.


For sets of tests with large amounts of test data, you should consider putting the test values and/or results into a
separate dat
a file that the test loads and uses. The test data file might be XML or might be a simple text file.

Are all the Boundary
Conditions

Correct?

Identifying boundary conditions is one of the most valuable parts of unit testing, because this is where most bug

are
present


at the edges.
Many bugs in code occur around “boundary conditions”, that is, under conditions where the
code’s behavior may be different from the normal, day
-
to
-
day routine.


To help you think of tests for boundary conditions,

the “Pragmati
c Unit Testing” book suggests seven specific areas
to think about and check test coverage, which can be remembered using the acronym

CORRECT:




Conformance



does the value conform to expected format?

Examples here include input strings, and their
values.



Ordering


is the set o
f

values order
ed

or unordered as appropriate
? For example, when testing a sort or
compare, you must test with items in order, unordered, and reverse ordered.



Range


is the value within reasonable minimum and maximum values?

Once t
he values are processed,
check against specification
-
domain or business
-
domain knowledge, such as checking that angles in degrees
are from 0 to 359. Also includes checks for invariants.



Reference


does the code reference anything external that isn’t unde
r direct control of the code itself?

For
example, a method in a web application to display a customer’s account might require that the customer is
first logged on.



Existence


does the value exist?

Check to handle blanks, nulls, and zeros. Also, if ther
e is an id which is
referring to some other object, make sure that the code correctly handles id’s which refer to older objects.



Cardinality


are there exactly enough values?

Make sure you have handled cases of 0, 1, and n arguments.



Time (absolute and r
elative)


is everything happening in order? At the right time?

Are the events
happening in the amount of time allowed from the specification?

What about time zone issues?

Check Inverse Relationships

Some methods can be checked by checking their inverse

relationship
. Many math functions can be validated this
way. The classic example is square root


check that squaring the square root produces the original value again
(within a tolerance).


Another example is in data loaders and dumpers. Test a load s
equence followed by a dump of the loaded data and
check the original result is provided again. Or, you could organize this in reverse: start with a known data set, then
perform a dump, then upload the dumped file, and check that what you have matches wha
t you started with.

Page
4

of
23

Cross Check

Probably the most useful of all the checks other than the “correct value” check: Check the same number as found by
two different methods. For instance, in a library book
-
tracking system, the number of copies of a book that

are
checked out plus the number of copies sitting on the shelves should always equal the total number of copies in the
inventory.

An another example, in the text adventure game, if an item was taken from a room and placed into a
player’s inventory, the r
oom’s item count must now be one less, and the inventory item count must be one greater.


In many reporting systems, you can add up the values along different dimensions. For instance, the total demand
should be same if is carried out by summing across th
e product groups or by summing across customer groups.


Another classic example was the Space Shuttle software system: in which two different companies developed the
same software functions and the results were compared. If they matched, both systems were

considered to have
passed such tests.

For
ce Error Conditions to H
appen

In the real world, error
s

happen.
Simple examples are ones of incorrect inputs. Likewise, the input handlers must
be designed to produce clear indications that inputs were incorrect
(typically this means throwing an exception).


More complex examples of errors which must be considered and mimicked include

running out of disk space,
having network dropouts, etc.


Here is a list of issues

to consider,

again

based on the “Pragmatic Uni
t Testing” book:



Running out of memory



Running out of disk space



Issues with wall
-
clock time



Network availability and errors



System load



Limited color palette



Very high or very video resolution


In order to get these error conditions to
occur

under control
led conditions, tests can be constructed using mock
objects. These
are objects that have classes that are interchangeable with classes with as servlets, HTTP request and
responses, JDBC connections and result sets, etc. This approach is covered below.

Pe
rformance
C
haracteristics

One area that might prove beneficial to examine is performance characteristics


not performance itself, but trends
as input sizes grow, as problems become more complex, and so on.


We would like to have a quick regression test fo
r this. Accordingly, create some rough tests that check filters for
larger query results, for instance. For instance,


p
ublic void testFilter() {



Timer timer = new Timer();




// First, try a small list

URLFilte
r filter = new URLFilter(smallL
ist);



t
i
mer.start();

f
ilter.process();

t
imer.end();

assertTrue(timer.elapsedTime() < 1.0);


//
Next
, try a
large

list

URLFilter filter = new URLFilter(large
L
ist);



t
imer.start();

f
ilter.process();

t
imer.end();

as
sertTrue(timer.elapsedTime() < 2
.0);


//
Next
, try
a
huge

list

Page
5

of
23

URLFilter filter = new URLFilter(hugeList);



t
imer.start();

f
ilter.process();

t
imer.end();

as
sertTrue(timer.elapsedTime() < 4
.0);

}


In this case, smallList, largeList, and hugeList are
data lists that are
set up in the text fixture.

Using Jav
a’s Asserts

No discussion of Java Unit testing would be complete without a discussion of the assert system that is built into Java
1.4 and later. This allows you to use statements such as:




assert userDAO != null;


These will be validated at runtime

if assertion checking is turned on, and ignored with near
-
zero performance hit if
not. An assertion that fails will throw an AssertionException.


Java asserts are used exactly like JUnit asserts, though they are not as flexible. All of the guidelines
above still
apply, and they can kept in your code.


To activate assertions, use the

ea flag in the Java arguments. There is also a way to turn on assertions for specific
trees of packages.

Using JUnit
and TestNG
for Java Unit Testing

Here are some notes
from a primer

on JUnit written by Mike Clark. These also apply equally well to TestNG:




JUnit
and TestNG
tests allow you to write code faster while increasing quality.


When you write tests using JUnit, you'll spend less time debugging, and you'll have c
onfidence that
changes to your code actually work. This confidence allows you to get more aggressive about refactoring
code and adding new features.


Without tests, it's easy to become paranoid about refactoring or adding new features because you don't
kn
ow what might break as a result. With a comprehensive test suite, you can quickly run the tests after
changing the code and gain confidence that your changes didn't break anything. If a bug is detected while
running tests, the source code is fresh in your
mind, so the bug is easily found. Tests written in JUnit help
you write code at an extreme pace and spot defects quickly.




JUnit
and TestNG are elegantly simple frameworks.


Writing tests should be simple
-

that's the point! If writing tests is too comple
x or takes too much time,
there's no incentive to start writing tests in the first place. With JUnit, you can quickly write tests that
exercise your code and incrementally add tests as the software grows.


Once you've written some tests, you want to run t
hem quickly and frequently without disrupting the
creative design and development process. With JUnit, running tests is as easy and fast as running a
compiler on your code. In fact, you should run your tests every time you run the compiler. The compiler
te
sts the syntax of the code and the tests validate the integrity of the code.




JUnit
and TestNG
tests check their own results and provide immediate feedback.


Testing is no fun if you have to manually compare the expected and actual result of tests, and i
t slows you
down. JUnit tests can be run automatically and they check their own results. When you run tests, you get
simple and immediate visual feedback as to whether the tests passed or failed. There's no need to manually
comb through a report of test re
sults.


Page
6

of
23



JUnit
and TestNG
tests can be composed into a hierarchy of test suites.



JUnit tests can be organized into test suites containing test cases and even other test suites. The composite
behavior of JUnit tests allows you to assemble collections of t
ests and automatically regression test the
entire test suite in one fell swoop. You can also run the tests for any layer within the test suite hierarchy.

This is an area where TestNG is even more powerful than JUnit, as it allows you to define dependencie
s
between the test cases.




Writing JUnit

and TestNG

tests is inexpensive.



Using the JUnit testing framework, you can write tests cheaply and enjoy the convenience offered by the
testing framework. Writing a test is as simple as writing a method that exer
cises the code to be tested and
defining the expected result. The framework provides the context for running the test automatically and as
part of a collection of other tests. This small investment in testing will continue to pay you back in time and
quali
ty.




JUnit

and TestNG

tests increase the stability of software.


The fewer tests you write, the less stable your code becomes. Tests validate the stability of the software and
instill confidence that changes haven't caused a ripple
-
effect through the sof
tware. The tests form the glue
of the structural integrity of the software.




JUnit
and TestNG
tests are developer tests.


JUnit tests are highly localized tests written to improve a developer's productivity and code quality. Unlike
functional tests, whic
h treat the system as a black box and ensure that the software works as a whole, unit
tests are written to test the fundamental building blocks of the system from the inside out.


Developer
s write and own the JUnit tests. When a development iteration is c
omplete, the tests are promoted
as part and parcel of the delivered product as a way of communicating, "Here's my deliverable and the tests
which validate it."




JUnit
and TestNG
tests are written in Java.


Testing Java software using Java tests forms a s
eamless bond between the test and the code under test. The
tests become an extension to the overall software and code can be refactored from the tests into the
software under test. The Java compiler helps the testing process by performing static syntax che
cking of the
unit tests and ensuring that the software interface contracts are being obeyed.

Organization of JUnit

JUnit
3.8.x
is designed around two key design patterns: the
Command

pattern and the
Composite

pattern.


A
TestCase

is a command object.

An
y class that contains test methods should subclass the
TestCase

class. A
TestCase

can define any number of public
testXXX()

methods.

When you want to check the expected and
actual test results, you invoke a variation of the
assert()

method.


TestCase

sub
classes that contain multiple
testXXX()

methods can use the
setUp()

and
tearDown()

methods to initialize and release any common objects under test, referred to as the test fixture.

Each test runs in the
context of its own fixture, calling
setUp()

before a
nd
tearDown()

after each test method to ensure there can be
no side effects among test runs.


TestCase

instances can be composed into
TestSuite

hierarchies that automatically invoke all the
testXXX()

methods defined in each
TestCase

instance.

A
TestSuite

is a composite of other tests, either
TestCase

instances or other
TestSuite

instances.

The composite behavior exhibited by the
TestSuite

allows you to
assemble test suites of test suites of tests, to an arbitrary depth, and run all the tests automaticall
y and uniformly to
yield a single pass or fail status.

Page
7

of
23


The UML class diagram is shown below:



The
re are other parts of JUnit

which deal with test run
ners. Basically, there are classes for running of tests with a
text
-
based output, and for running of t
ests with a Swing GUI
-
based output. These are covered only briefly here,
because we tend to use the test
-
runners that are built into Eclipse.


In JUnit 4.x, there are no TestCase and TestSuite objects that you
are required to inherit

from

(they are still
provided
for backward compatibility). I
nstead you decorate POJO classes with annotations to indicate which methods are
part of the test.

Basic Coding Issues

In 3.8.x,

import junit.framework.*. Then create a series of classes extending TestCase.


Within t
hat class, create methods that start with “test”.

Here is an example:


public class Position
TestCase

extends TestCase {


protected Game game;


protected Board board;




public PositionTestCase
(String name) {



super(name);


}



public void setUp() {



gam
e = new Game(1, 6, 4);





board = new Board(game);






}




public void testCreateFromIndex
() {



Position p1 = new Position(board, 0);






assertEquals(
0, p1.getPegAt(0)
);



assertEquals(
0, p1.getPegAt(1)
);



assertEquals(
0, p1.getPegAt(2)
);



assertEq
uals(
0, p1.getPegAt(3)
);

}

}




The testing framework will locate these methods using reflection, and run them
.
Specifically (for JUnit 3)

it will run
all method
s whose name begins with “test”, and (for JUnit 4) it will run methods marked with the annotat
ion @Test.

Page
8

of
23


Within the t
est methods, use checks such as:



assertTrue(
<expr>)



assertFalse(
<expr>)



assertEqual
s
(
<expected>, <actual>
)



this one is used very often. If the ‘expected’ and ‘actual’ are strings
and they differ, JUnit is smart enough to report t
he specific substring that is different.



assertNull(<object>)



assertNotNull(<object>)



assertSame(<object>,<object>)


asserts that two arguments refer to the same object.



assertNotSame(<object>,<object>)


asserts that two arguments do not refer to the sam
e object.



fail()



this can be used to mark a test case as having failed, and would be reached through a specific
branch sequence, for instance.


There are also variations on each of the “asserts” which takes an assertion name as a first argument, then has

the
remaining arguments the same. This provides an extra level of detail and reporting on the test run outputs.


Here are some suggestions provided by Brad Barclay
:
Limit your use of
the
binary assertion operators assertTrue
and

assertFalse to boolean v
ariables. This is especially true for

anything involving equality comparisons, where
assertEquals is a

better option. The reason for this is

that it will improve the error messages
.
Without using
assertEquals, you get a diagnostic such as:






[Assertion failed]
-

this expression must be true



Contrast this to a re
-
written version using assertEquals:





expected:<[NEW YORK]> but was:<[LOS ANGELES
]>


Get to know the CoreMatchers class

(org.hamcrest.CoreMatchers). This

provides a series of matcher

methods for
use with Assert's assertThat method. For example,

to test that two strings are not equal, you can write:





Assert.assertThat(string1, CoreMatchers.not(string2));



In cases where you expect
to use the CoreMatchers frequently,

you can statically import its methods, to write more
declarative

assertion logic for those complex cases that are too hard to

handle with Assert and ComparableAssert.


If the code should throw an exception, use the follo
wing approach:


public void test2() {



try {




Posit
ion p2 = new Position(board, 0
);






fail("should never get here");



}



catch (RuntimeException e) {




assertTrue(true);



}


}


If there is initialization code, place it into the setUp() method.

F
or instance in the text adventure test classes, these
methods created a module and loaded its rooms, items, etc.


The
teardown() method can be used to reverse the setUp() method
, such as closing database connections.


I
n JUnit 4
, import org.junit.Test and
org.junit.Assert. T
ests are identified by an
@Test

annotation, as shown here:


import org.junit.Test;

import junit.framework.TestCase;


public class AdditionTest extends TestCase {



private int x = 1;

Page
9

of
23


private int y = 1;




@Test public void testAddi
tion() {


int z = x + y;


assertEquals(2, z);


}


}


The
TestCase

class still works, but you're no longer required to extend it.


As long as you annotate test methods
with
@Test
, you can put your test methods in any class at all.

However, you'll n
eed to import the
junit.Assert

class to access the various assert methods, as shown here:


import org.junit.Assert;


public class AdditionTest {



private int x = 1;


private int y = 1;




@Test public void addition() {


int z = x + y;


Assert.as
sertEquals(2, z);


}


}


You also can use the new static import feature in JDK 5 to make this just as simple as the old version:


import static org.junit.Assert.assertEquals;


public class AdditionTest {



private int x = 1;


private int y = 1;




@
Test public void addition() {


int z = x + y;


assertEquals(2, z);


}


}


This approach makes testing protected methods much easier because the test case class can now extend the class that
contains the protected methods.


JUnit 4 also introduces
a new feature that really has no equivalent in JUnit 3: class
-
scoped
setUp()

and
tearDown()

methods.

Any method annotated
@BeforeClass

will run exactly once before the test methods in
that class run, and any method annotated with
@AfterClass

will run exac
tly once after all the tests in the class
have been run.


For instance, suppose each test in the class uses a database connection, a network connection, a very large data
structure, or some other resource that's expensive to initialize or dispose of.

Rat
her than re
-
creating it before each
and every test, you can create it once and tear it down once.

This approach will make some test cases run a lot
faster.

As an example, when
we

test error
-
handling in code that calls into third
-
party libraries,
we

often

like to
redirect
System.err

before the tests begin so the output is not cluttered with expected error messages.

Then
we

restore it after the tests end like so:


// This class tests a lot of error conditions, which

// Xalan annoyingly logs to System.err.
This hides System.err

// before each test and restores it after each test.

private PrintStream systemErr;



@BeforeClass

protected void redirectStderr() {


systemErr = System.err; // Hold on to the original value


System.setErr(new PrintStream(n
ew ByteArrayOutputStream()));

}



@AfterClass

protected void tearDown() {

Page
10

of
23


// restore the original value


System.setErr(systemErr);

}


There's no need to do this before and after each and every test.

Do be careful with this feature, however.

I
t has the
potential to violate the independence of the tests and introduce unexpected coupling.

If one test somehow changes
an object that is initialized by
@BeforeClass
, it may affect the outcome of other tests.

It can introduce order
dependence into th
e test suite and hide bugs.

As with any optimization, only implement this after profiling and
benchmarking have
proven you have a real problem.


Exception testing is one of the biggest improvements in JUnit 4.

The old style of exception testing is to wra
p a
try

block around the code that throws the exception, then add a
fail()

statement at the end of the
try

block.

For
example, this method tests that division by zero throws an
ArithmeticException
:


public void testDivisionByZero() {




try {



int n = 2 / 0;


fail("Divided by zero!");


}


catch (ArithmeticException success) {


assertNotNull(success.getMessage());


}



}


Not only is this method ugly, but it tends to trip up code
-
coverage tools because whether the te
sts pass or fail, some
code isn't executed.

In JUnit 4, you can now write the code that throws the exception and use an annotation to
declare that the exception is expected:


@Test(expected=ArithmeticException.class)


public void divideByZero() {


i
nt n = 2 / 0;

}


If the exception isn't thrown (or a different exception is thrown), the test will fail.

However, you'll still need to use
the old
try
-
catch

style if you want to test the exception's detail message or other properties
.

Summary of Annotati
ons

(as of JUnit 4.x)

Annotation

Description

@Test public void method()

Annotation @Test identifies that this method is a test method.

@Before public void method()

Will perform the method() before each test. This method can
prepare the test environment,

e.g. read input data, initialize the
class)

@After public void method()

Test method must start with test

@BeforeClass public void method()

Will perform the method before the start of all tests. This can be
used to perform time intensive activities fo
r example be used to
connect to a database

@AfterClass public void method()

Will perform the method after all tests have finished. This can be
used to perform clean
-
up activities for example be used to
disconnect to a database

@Ignore

Will ignore the
test method, e.g. useful if the underlying code has
been changed and the test has not yet been adapted.

@Test(expected=IllegalArgumentException.class)

Tests if the method throws the named exception

@Test(timeout=100)

Fail
s if the method takes longer t
ha
n 100 milliseconds


The purpose of the
@Rule

annotation is to mark public fields of a test class.

These fields

must

be of type
MethodRule
, or an implementing class.

Such
MethodRule
s behave similar to a AOP aspects, of course without
use of any AOP lib
rary and specialized for Tests.

They can execute code before, after or instead of a test method.


Example use cases listed in the
release
notes

include:

Page
11

of
23



Notification on tests



Setting up or tearing down resources, especially when they are used in multiple test classes



Special checks performed after every test, possibly causing a test to fail.



Making information about the test available insid
e the test

Designing for Testability

To

design a class to be effectively

tested, it should have well
-
defined state and API. For instance, even if the
application program doesn’t require a particular method, but the test is more effective if present, add i
t.


For example, in the MasterMind game program, the Board class had an addPosition method, but didn’t require a
method to access the positions since they were only used by the print() method. This made the
position

management
invisible to the unit test
s.


Design classes so that they can be initialized without further dependencies.

For instance, in the text adventure
program, the module class initially had a hard
-
coded filename to load for the module definition. We changed this to
be an import argument
, in order to use the module class for the real game some of the time, and to load in the
definitions of a test module for the test cases. The resulting initializer looked like:



p
rotected Module module;


protected ModuleLoader ml;



p
ublic void setup()
{



ml = new ModuleLoader();



module =
ml.load
(“test
case
data.xml”);


}


Another example was that in the MasterMind game, the standard constructor for Game carries out a user interaction
to determine the number of rounds, colors, and pegs. We create
d

a sp
ecial constructor which takes all of these as
argument
s

to use in the test cases (and then separately tested the UI
-
based editing of information using a non
-
JUnit
test case).


A further example of providing feedback to the test cases in addition to (or in
place of) output on the user interface
appears in the commands of the text adventure. A command such as “take” applied to an invalid item will inform
the user. It also should inform the caller, as shown in this example, where a boolean result is return
ed

to indicate the
status of the command.


public boolean perform(Module module, String arg) {



AbstractCharacter player = module.getPlayer();



Room curRoom = player.getCurRoom();



AbstractGameObject ago = curRoom.getGameObject(arg);




if (ago != null && ago instanceof AbstractItem) {




curRoom.removeGameObject(ago);




player.addToInventory((AbstractItem) ago);




System.out.println("Taken.");




return true;



}



else if (ago != null)




System.out.println("You can't take that!");



else




System.out.println("I don't see that here.");






return false;


}


Page
12

of
23

Another principle is to d
esign classes so that they can
be
tested without random behavior being carried out in the
class. For instance, in MasterMind, there is a “makeGuess” r
outine that shouldn’t always return the first of a list of
possible guesses, or the user will perceive the system as behaving i
n too machine
-
like a fashion.
However, the test
perspective is that the class should behave in a
very

regular basis. To solve t
his, we wrote the makeGuess to take a
seed value for the random behavior


the test cases can pass in a c
onstant seed, and the actual usage

can pass in a
random seed.

Test Suites

A test suite is a group of related JUnit tests. To use this, create an insta
nce of TestSuite, and a
dd the classes to it, as
shown
:


public class AllTests {


public static Test suite() {



TestSuite suite =

n
ew TestSuite(“name”);




s
uite.addTestSuite(ItemTest.class);



s
uite.addTestSuite(BoardTest.class);



r
eturn suite;


}

}


Whe
n this class is run, it will carry

out all test cases for the classes in sequence.

There is a facility in Eclipse that
will update a TestSuite to include runs all of the currently
-
present test cases.


There is a corresponding facility in JUnit 4.x, based
on annotations.

Test Fixtures

What if you have two or more tests that operate on the same or similar sets of objects?


Tests need to run against the background of a known set of objects. This set of objects is called a test fixture. When
you are writing t
ests you will often find that you spend more time writing the code to set up the fixture than you do
in actually testing values.


To some extent, you can make writing the fixture code easier by paying careful attention to the constructors you
write.

Howe
ver, a much bigger savings comes from sharing fixture code.

Often, you will be able to use the same
fixture for several different tests.

Each case will send slightly different messages or parameters to the fixture and
will check for different results.


When you have a common fixture, here is what you do:


1.

Add a field for each part of the fixture

2.

Define a setUp method, and i
nitialize the variables in that method

(marked as @Before in JUnit 4.x)

3.

Define a teardown method if needed,
to release any permane
nt resources you allocated
in setUp (marked as

@After in JUnit 4.x)


For example, to wr
ite several test cases that need

to work with different combinations of 12 Swiss Francs, 14 Swiss
Francs, and 28 US Dollars, first create a fixture:


public class Money
Test
extends TestCase
{




private Money f12CHF;




private Money f14CHF;




private Money f28USD;






public void setUp() {




f12CHF = new Money(12, "CHF");




f14CHF = new Money(14, "CHF");




f28USD = new Money(28, "USD");




}

}

Page
13

of
23


Once you have the fixture in place, you can add

as many Test Cases as you'd like

quickly, and you can add many
test methods quickly.

Custom Asserts

You can define additional assert methods, perhaps ones that operate on specific classes within th
e test cases, or carry
out specific comparisons. Each of the assert methods must call one of the existing assert methods underneath.


For instance, in the MasterMind game, we wrote a custom assert that compares a position to a given string pattern:


publi
c void assertMatch(String pattern, Position position) {



String positionPattern = position.getKeysString();






assertEquals(pattern.toUpperCase(), positionPattern);




}


Note that this must be used inside the MasterMindTestCase text fixture, because bo
ard must be found from the test
case rather than being passed into the assert.

Organizing Test Cases in Code

The last step is to decide where the tests will live within our development environment.


Here's the recommended way to organize tests:


1.

Create t
est cases in the same package as the code under test. For example, the
com.mydotcom.ecommerce

package would contain all the application
-
level classes as well as the test
cases for those components.

2.

To avoid combining application and testing code in your s
ource directories, create a mirrored directory
structure aligned with the package structure that contains the test code.

3.

For each Java package in your application, define a
TestSuite

class that contains all the tests for
validating the code in the package
.

4.

Define similar
TestSuite

classes that create higher
-
level and lower
-
level test suites in the other
packages (and sub
-
packages) of the application.

5.

Make sure your build process includes the compilation of all tests. This helps to ensure that your tests
are
always up
-
to
-
date with the latest code and keeps the tests fresh.


By creating a
TestSuite

in each Java package, at various levels of packaging, you can run a
TestSuite

at any
level of abstraction.

For example, you can define a
com.mydotcom.AllTests

that runs all the tests in the
system and a
com.mydotcom.ecommerce.EcommerceTestSuite

that runs only those tests validating the
e
-
commerce components.


The testing hierarchy can extend to an arbitrary depth.

Depending on the level of abstraction you're d
eveloping at in
the system, you can run an appropriate test. Just pick a layer in the system and test it!


Here's an example test hierarchy:


AllTests (Top
-
level Test Suite)


SmokeTestSuite (Structural Integrity Tests)


EcommerceTestSuite



ShoppingCartTestCase


CreditCardTestSuite


AuthorizationTestCase


CaptureTestCase


VoidTestCase


UtilityTestSuite


MoneyTestCase

Page
14

of
23


DatabaseTestSuite



ConnectionTestCase


TransactionTestCase


LoadTestSuite (Performance and Scalability Tests)


DatabaseTestSuite


ConnectionPoolTestCase


ThreadPoolTestCase

Running JUnit Tests (outside of Eclipse)

JUnit provides both a

textual and a graphical user interface

in the test
-
runners
.

Both user interfaces indicate how
many tests were run, any errors or failures, and a simple completion status.


The simplicity of the user interfaces is
the key to running tests quickly. You sho
uld be able to run your tests and know the test status with a glance, much
like you do with a compiler.


To run our test case using the textual user interface, use:

java

ea
junit.textui.TestRunner
<nameOfTestClass>



The “
-
ea” flag is not part of JUnit,

but is part of built
-
in Java asserts. It is a best practice to run these as well.


The textual user interface displays "OK" if all the tests passed and failure messages if any of the tests failed.


To run the test case using the graphical user interface
, use:

java

ea
junit.swingui.TestRunner
<nameOfTestClass>



The graphical user interface displays a Swing window with a green progress bar if all the tests passed or a red
progress bar if any of the tests failed.

Running JUnit

At Menlo School, we use Ec
lipse for nearly all Java development. Eclipse has excellent integration of JUnit testing
facilities at the code development and test running phases.


For instance, to create a new test case, select the package into which the test should be place, right c
lick, and select
the JUnit Test Case Wizard. This brings up the following dialog box:


Page
15

of
23



Enter the test case name, and the rest can be defaulted (unless you need the setup or teardown methods, in which
case clicking these check boxes will give you a stub

version of these methods in the generated test case class).


If you specify the class which is under test, on the “Next” dialog box of the wizard, Eclipse will offer a list of
methods in the class which it will generate test method stubs. This can save a

good bit of time for a large class. You
still have to fill in the bodies, however.


For assistance in generating, the “assert” calls in the code, using the Eclipse “Code Assist” facility, reached by
pressing control
-
space. It will provide the signature
of all the assert methods that are available.


Before starting to run any tests, turn on the Eclipse flag that will add the

ae flag to all generated JUnit launch
configurations. This is located on the Window | Preferences… menu item’s dialog and looks li
ke:


Page
16

of
23



To create a custom test configuration, use the “Run As…” dialog box for the JUnit Test target. This allows you to
set up Java command line arguments, for instance.


To create a JUnit Test Suite, use the Test Suite Wizard. This isn’t on the list f
or quick access, so find it by picking
“Other > Java > JUnit > JUnit Test Suite”.


When a test case or test suite is run, the JUnit view will appear, as shown below:

Page
17

of
23



The rectangle near the top is red or green as a traffic light to indicate success or fa
ilure of the tests. This gives you a
quick check.


Unsuccessful tests fall into two categories:



Failures: Anticipated problems checked with JUnit assertions. For example,
assertEquals(a, b)

fails when the expected and actual result are not equal



Errors.
Unanticipated problems like a
NullPointerException

or
ArrayIndexOutOfBoundsException
.


The most useful thing about this view that we’ve seen is that there is a button to re
-
run the test, after you have fixed
the code. It is the green arrow
-
to
-
the
-
right,
near the top.

Organization of TestNG

TestNG is a similar
unit
test framework. It gained credibility when it supported Java 5 annotations prior to any
released version of JUnit, and it has a number of advanced features, including:




JDK 5 Annotations (JDK 1
.4 is also supported with JavaDoc annotations).



Flexible test configuration.



Support for data
-
driven testing (with @DataProvider).

Page
18

of
23



Support for multiple instances of the same test class (with @Factory)



Support for parameters.



Allows distribution of tests on

slave machines.



Powerful execution model (no more TestSuite).



Supported by a variety of tools and plug
-
ins (such as
Eclipse
,
IDEA
,
Maven
).



Embeds
BeanShell

for further flexibility.



Default
JDK

functions for runtime and logging (no dependencies).



Dependent methods for application server testing.


The concepts used in TestNG
are as follows:




A suite is repre
sented by one XML file.

It can contain one or more tests and is defined by the
<suite>

tag.



A test is represented by
<test>

and can contain one or more TestNG classes.



A TestNG class is a Java class that contains at least one TestNG annotation.

It is r
epresented by the
<class>

tag and can contain one or more test methods.



A test method is a Java method annotated by
@Test

in your source.


A TestNG test can be configured by
@BeforeXXX

and
@AfterXXX

annotations which allows to perform some
Java logic bef
ore and after a certain point, these points being either of the items listed above.


Here is a very simple test:


package example1;


import org.testng.annotations.*;


public class SimpleTest {




@BeforeClass



public void setUp() {





// code that will b
e invoked when this test is instantiated



}




@Test(groups = { "fast" })



public void aFastTest() {





System.out.println("Fast test");



}




@Test(groups = { "slow" })



public void aSlowTest() {





System.out.println("Slow test");



}

}

Annotations

in TestNG

Configuration information for a TestNG class:



@BeforeSuite:
The annotated method will be run before all tests in this suite have run.

@AfterSuite:
The annotated method will be run after all tests in this suite have run.

@BeforeTest
: The anno
tated method will be run before the test.

@AfterTest
: The annotated method will be run after the test.

@BeforeGroups
: The list of groups that this configuration method will run before. This method is guaranteed to
run shortly before the first test method

that belongs to any of these groups is invoked.

@AfterGroups
: The list of groups that this configuration method will run after. This method is guaranteed to run
shortly after the last test method that belongs to any of these groups is invoked.

@BeforeCl
ass
: The annotated method will be run before the first test method in the current class is invoked.

@AfterClass
: The annotated method will be run after all the test methods in the current class have been run.

Page
19

of
23

@BeforeMethod
: The annotated method will be r
un before each test method.

@AfterMethod
: The annotated method will be run after each test method.



Writing TestNG tests

First of all, import org.testng.annotations.*.


Then place the @Test annotation on each test. Use the @BeforeTest or @BeforeClass a
nnotation to mark any test
setup code.


The
available set of

assertions include:


static

void

assertEquals
(boolean

actual, boolean

expected)




Asserts that two booleans are equal.


static

void

assertEquals
(boolean

actual, boolean

expected, java.lang.String

message)



Asserts that two booleans are equal.


static

void

assertEquals
(byte[]

actual, byte[]

expected)



Asserts that two arrays
contain the same elements in the same order.


static

void

assertEquals
(byte[]

actual, byte[]

expected, java.lang.String

m
essage)



Asserts that two arrays contain the same elements in the same order.


static

void

assertEquals
(byte

actual, byte

expected)



Asserts that two bytes are equal.


static

void

assertEquals
(byte

actual, byte

expected, java.lang.String

message)



Asserts
that two bytes are equal.


Page
20

of
23

static

void

assertEquals
(char

actual, char

expected)



Asserts that two chars are equal.


static

void

assertEquals
(char

actual, char

expected, java.lang.String

message)



Asserts that two chars are equal.


static

void

assertEquals
(java.util.Collection

actual, java.util.Collection

expected)



Asserts that two collections contain the same elements in th
e same order.


static

void

assertEquals
(java.util.Collection

actual, java.util.Collection

expected,
j
ava.lang.String

message)



Asserts that two collections contain the same elements in the same order.


static

void

assertEquals
(dou
ble

actual, double

expected, double

delta)



Asserts that two doubles are equal concerning a delta.


static

void

assertEquals
(double

actual, double

expected, double

delta,
java.lang.String

message)



Asserts that two doubles are equal concerning a delta.


static

void

assertEquals
(float

actual, float

expected, float

delta)



Asserts that two floats are equal concerning a delta.


static

void

assertEquals
(float

actual, float

expected, float

delta,
java.lang.String

message)



Asserts that two floats are equal concerning a delta.


static

void

assertEquals
(int

actual, int

expected)



Asserts that two ints are equal.


static

void

assertEquals
(int

actual, int

expected, java.lang.String

message)



Asserts that two ints are equal.


static

void

assertEquals
(long

actual, lo
ng

expected)



Asserts that two longs are equal.


static

void

assertEquals
(long

actual, long

expected, java.lang.String

mess
age)



Asserts that two longs are equal.


static

void

assertEquals
(java.lang.Object[]

actual, java.lang.Obje
ct[]

expected)



Asserts that two arrays contain the same elements in the same order.


static

void

assertEquals
(java.lang.Object[]

actual, java.lang.Object[]

expected,
java.lang.String

message)



Asserts that two arrays contain the same elements in the same order.


static

void

assertEquals
(java.lang.Object

actual, java.lang.Object

expected)



Asserts that two objects are equal.


static

void

assertEquals
(java.lang.Object

actual, java.lang.Object

expected,
java.lang.String

message)



Asserts that two objects are equal.


static

void

assertEquals
(short

actual, short

expected)



Asserts that two shorts are equal.


static

void

assertEquals
(short

actual, short

expected, java.lang.String

message)



Asserts that two shorts are equal.


static

void

assertEquals
(java.lang.String

actual, java.lang.String

expected)



Asserts that two Strings are equal.


Page
21

of
23

static

void

assertEquals
(java.lang.String

actual, java.lang.String

expected,
java.lang.String

message)



Asserts that two Strings are equal.


static

void

assertEqualsNoOrder
(java.lang.Object[]

actual, java.lang.Object[]

expected)



Asserts that two arrays contain the same elements in no
particular order.


static

void

assertEqualsNoOrder
(java.lang.Object[]

actual, java.lang.Obj
ect[]

expected,
java.lang.String

message)



Asserts that two arrays contain the same elements in no particular order.


static

void

assertFalse
(boolean

condition)



Asserts that a condition is false.


static

void

assertFalse
(boolean

condition, java.lang.String

message)



Asserts that a condition is false.


static

void

assertNotNull
(java.lang.Object

object)



Asserts that an object isn't null.


static

void

assertNotNull
(java.lang.Object

object, java.lang.String

message)



Asserts that an object isn't null.


static

void

assertNotSame
(java.lang.Object

actual, java.lang.Object

expected)



Asserts that two objects do not refer to the same object.



static

void

assertNotSame
(java.lang.Object

actual, java.lang.Object

expected,
java.lang.String

message)




Asserts that two objects do not refer to the same objects.


static

void

assertNull
(java.lang.Object

object)



Asserts that an object
is null.


static

void

assertNull
(java.lang.Object

object, java.lang.String

message)



Asserts that an object is null.


sta
tic

void

assertSame
(java.lang.Object

actual, java.lang.Object

expected)



Asserts that two objects refer to the same object
.


static

void

assertSame
(java.lang.Object

actual, java.lang.Object

expected,
java.lang.String

message)




Asserts that two objects refer to the same object.


static

void

assertTrue
(boolean

condition)



Asserts that a condition is true.


static

void

assertTrue
(boolean

condition, java.lang.String

message)



Asserts that a condition is true.


static

void

fail
()



Fails a test with no message.


static

void

fail
(java.lang.String

message)



F
ails a test with the given message.


static

void

fail
(java.lang.String

message, java.lang.Throwable

realCause)



Fails a test

with the given message and wrapping the original exception.

Running TestNG

You can invoke TestNG in several different ways:



With a
testng.xml

file

Page
22

of
23



With ant



From the command line

Before you can run the tests,
however, you must configure TestNG using a special XML file, conventionally named
testng.xml.

The syntax for this file is very simpl
e, and is presented below
.

This file begins by defining a test suite,
My test suite,

composed of a unique test,
First test
,

that is made by the
StringUtilsTest

class.


<!DOCTYPE suite SYST
EM "http://
testng
.org
/testng
-
1.0.dtd" >

<suite name="My test suite">


<test name="First test">


<classes>


<class name="tests.StringUtilsTest" />


</classes>


</test>

</suite>


If this sample testng.xml file doesn't seem very useful (there's only one test class), the great news is that it is actually
the
only file you need to write

to define your test suites.

This is an improvement over JUnit, where

the definition of
your sui
tes was probably spread over several files:

JUnit's
TestSuite
s, property files, and, obviously, Ant build
files.

With TestNG, all the data you need is gathered in the testng.xml file.

No more
TestSuite
s, and a thinner
build file.


Additional informatio
n about the dependencies of tests or test suites can also be expressed in this file.


There is a plug
-
in that is a runner for TestNG. It is available at
http://code.google.com/p/testng
-
eclipse/
.

Us
ing Mock Objects

As mentioned above, trying set up a complex and complete test fixture that involves database connections, UI code,
or similar can be difficult. A better approach is to use
mock objects
.


Mock objects are like “stand
-
ins” during a movie sc
ene shooting sequence. What we are going to do in unit testing
is similar of lighting doubles in the movie: we will use a cheap stand
-
in that kind of close to the real thing, at least
superficially, but that will be easier to work for our purposes.


The
code under test can be
isolated from the actual

supporting objects, since it only refers to
t
he

object by its
interface.
There are a number of situations where mock objects can help us:




The real object has non
-
deterministic behavior



The real object is di
fficult to set up



The real object has behavior that is hard to trigger



The real object is low



The real object has (or is) a user interface



The test needs to ask the real object how it is used



The real object does not yet exist


There is a large library of
mock objects available at
www.mockobjects.com
.

Other Testing Facilities

This document has focused on JUnit

and TestNG
, but it is also important to review:




JMeter



this provides testing for web
-
based gui’s and w
eb applications
. See
http://jakarta.apache.org/jmeter/
.



M
ercury Interactive
’s product line
, which include WinRunner and LoadRunner
. The company was

bought
by Hewlett
-
Packard

in 2006
.

Page
23

of
23



Perhaps other unit te
st products that are part of software engineering suites such as
Rational
, Seapine, or
others.