Need for White Box Testing

jetmorebrisketSoftware and s/w Development

Aug 15, 2012 (5 years and 11 months ago)


White Box Testing


The need


Code review

Unit tests and automation

cppUnit Concepts

Memory leaks





Best Practices



Cyclomatic Complexity

Need for White Box Testing

Better to test anything from outside
in as well as inside

Most of the functional and performance issues arise due to bad coding

Ultimately the code matters

hence test the code left right and center

Black box tests are used to check cause and effect without getting into

White box tests do the same by getting into the internals of every program

Every developer is by default a white box tester

Medical field requires general physicians and surgeons

Certain problems can be identified and solved only by surgeons

A person may look healthy but internally the person may have high bp and
sugar; they are not visible externally

So far, the importance of white box testing is not upto the mark

Product companies definitely need white box testing

If your application must scale to a very large extent, white box tests are

Black Box Cannot Test These

It takes long time to check memory leaks

Any exceptions that happened but unnoticed or unhandled

Which portion of if condition is evaluated

What path the program took to achieve the end result

For every request what is the memory and cpu time taken

Is there any dead and unused code

Is there any extra code that is not needed

What are the potential breaking points in code

Is the code compliant to best practices

In large business apps, the data paths will be too large to test; it will
require senior business analysts; if done thru white box mode,
normal programmers can bring out great bugs

Diagnostics tools

IBM Rational Purifyplus


Parasoft tools

HP Mercury Diagnostics


.Net profiler




All tools work on instrumentation and trace options

Some tools help in analyzing system memory space as well

Usually the system cpu executions are privileged, and hence we
may not get access to those

Business Benefits

Ensures that the core building blocks are intact, when
adding new features and modules

Go to market with confidence on product stability

Provides greater visibility in terms of technical and
internal compliance of the product

Vulnerability testing always needs a greater amount of
white box testing

Resource utilization reduction is key for marketing.
Products that use less memory and bandwidth are the
most welcome ones

Keeps business leaders informed on the underlying
strengths and weaknesses of the application platform

Understand Design

Honestly speaking, how many of us use proper design before

Can the same happen to real estate industry without a proper floor
plan or for pharma industry without proper process and formula for

Design has 4 parts

UI, database, Interface and Logic design

UI design is key aspect when it comes to usability and compliance

Database design ensures scalability and storage

Interface design is key for plug and play during integration

Logic is important for the business flow of the application

Logic design can be a flow chart or text algorithm or pseudo code

Most of the modelers do have pictorial representation of design

Ideally the code must reflect the translation of pseudo code

Traceability of design elements to code is one of the key areas we
need to look into

A change in design may impact several parts of code

Code Review

Most of the times, this is ignored or not done to its true

Code review must have design and code side by side

Code review can save upto 30% of testing cost at a later
point in time

Usual problems happen due to cut and paste of code
where it is not required

When there is a transition from one developer to
another, code goes thru challenges

Quick fixes on big projects cause enormous amount of
turbulence on production systems

Ideally senior people must review code

Spend at least 10% of coding time on code review

Code Review Checklist

Just try these 10 critical points in daily life

No hard coding

Ensure loop termination conditions

No object creation inside loops

Close every object that you open

Give comments to every code block

Use database connection only when you need

Remove unused variables and code portions

Follow a consistent naming convention

Try to reduce overloading methods very often

Check SQL or file status soon after every disk

Unit Testing

The smallest piece of code that can be independently
run and tested is a unit

Unit can be a page or a function or a method within a
class or a whole program itself

It is the developer’s responsibility to unit test the code

Unit test must focus on data type of every parameter

Focus on data format of every parameter

Focus on boundary values of every parameter

For any given code, what deliverable do we provide as
part of unit testing?

Most of the times it is just the trust on the developer

Developer must spend at least 30% of coding time on
unit testing alone

How to draw boundaries

What is the line that demarcates unit and
other tests?

One expected output out of a function

No chaining

When to stop unit tests?

When a set of parameter combinations are
tested, stop the same

Boundaries and equivalence partitions must
be tested before exit

Unit Testing Checklist

Are the naming conventions followed?

Are the comments done properly?

Is there any hard coding?

Test case for custom exceptions.

Test case for system exceptions.

Test case for “body” of an if condition.

Test case for “body” of an else condition.

Test case for every loop termination.

Test case for every recursion termination.

Test case for pointers release (for memory leaks).

Test case for every procedure entry and exit.

Test case for every parameter validation for procedures.

Test case for resource release (Closing DB connections,
releasing objects etc.)

Unit Test Matrix

A single test must make a call to a function with some
parameters combination and expect return value(s)

If a function has got multiple parameters that are of
different types, then combinations of all parameters are
possible and it will become a Cartesian product table

As the first step, a matrix needs to be prepared with all
such combinations

Then the matrix must be optimized to reflect the
practicality of the inputs

Possibilities of return paths must also be documented

The samples may go in 100s but not to worry on

Since automated unit tests run extremely fast without
human inputs, 1000s of tests can be executed in minutes

Automate Unit Tests

It is not practically possible to give all combinations of the inputs to
programs and run

To reduce time, we need to write a program, that calls the program
under test and pass parameters

Ideally the test program must be in the same language as the
program being tested

This will help in maintaining the homogeneity of the data type of
parameters passed

Since developers know the programming language, it is easy for
them to write the test program as well

These are called test suites or test scripts

The investment on the unit test automation is one time as long as
changes are minimal

The person who automates test for a program must know the
program well

The core of unit automation

Class A

Method m1(p1, p2)

Method m2(p3, p4)

TestClass TA

Testcase tm1

ta = new Class A

rc = ta.m1(4, 5)

If (rc == exp_result)




// destroy ta

Level 1 Automation

Crude method

quick solution for short
term projects

Write wrapper callers

Hard code parameters

Cut and paste test code with no hesitation

Spend little time on test creation

Test code can be throw
away soon

Simple logs, more manual result analysis

Level 2 Automation

Parameters to tests must be separated
from test code

You can use csv or xml or any format for
test data

Useful to run same test on larger test data

When a mixture of customers use the
product, the data set also will change

this will be helpful in that case

Use your own test logs with more details

Level 3 Automation

Use test tools such as cppUnit or jUnit etc.

Build auto recovery options if tests fail in

Ensure every test has start

Use this on long term test engagements

When tests are to be reusable, this is the
best option

If you know cppUnit, the concepts are
exactly the same in nUnit, jUnit etc.

Stubs and Drivers

When the program under test needs some pre
built data or state, we need to mimic the same

This is a short
cut approach, but it serves the

Stub is the one that mimics a called function

Driver is the one that mimics a caller function

When we use stubs and drivers, it is a must that
we ensure that the data created by them are
destroyed soon after the test

Testing APIs

APIs do exchange of data from one piece to another

APIs are usually consumed by many independent

APIs in general are meant to provide a window of
information to 3


One change in API will affect all consumers at once

Most of the current day apps provide APIs

We can send tweets from our application using APIs

We can send emails from Amazon SES thru APIs

API calls must be assumed to be context neutral

Some APIs do require authentication mechanism

APIs fall under external interfaces

Wrapper Tools for APIs

A set of business critical APIs are now being
used in all EAI scenarios

FIX, SWIFT, News feeds, HL7, HIPAA, Part 11
standards etc

A variety of commercial tools are available in
each category

When messages are sent, these tools validate
the structures of XML or EDI against the

Also, there are different versions of these APIs
exist like html 5 etc.


Write test wrappers in cpp for cpp programs

If you are familiar with cpp programming, you may not
require cppUnit !

There are many versions of cppUnit itself as it is open

many people modified it for their convenience

You can write your own test library and framework

A set of pre
built classes and methods are already made
available to minimize test coding time

Testcase, Testsuite, Assert are the important aspects in

Testcase is the basic block

every single test

Testsuite contains multiple tests ( a batch)

Assert is the actual verification point where actual result
is compared with expected result



known set of objects for test

setUp is the place where we create testcases

tearDown is the place where we remove all
created tests

Commonly used asserts







White box tools and profiler tools use one single concept called

Source code is compiled and built as binary

These tools understand the format of these binary files

function entry points are made, how they exit, where stack is
maintained etc.

Before profiling, they instrument the binary

inject their own log
codes in the binary locations

During the execution, the instrumented code is run and every detail
is logged using these injected code (such as entered function x,
executed loop y, exited function y, destroyed object o etc.)

Using these logs, the tools provide internal details

This is very similar to sending a probe inside our body and doing a
scan of that probe


When profilers run along with the instrumented
programs, it adds load to the machine

This is a necessary overhead we need to take

The performance overheads are negligible when
compared to actual object size

The statistics that are collected out of the
programs will vary from time to time; but if the
variance is beyond 5%, then the tool has real

Also, do not run 2 competing tools on the same
machine. The effects are unpredictable

Memory Leaks

When a memory pointer is declared and
initialized and it is not released, it is a leak

Even in managed code sections, the leaks are
there for a time till the garbage collector acts

Leaked memory locations are useless and

One program leaking 10 bytes of memory per

can bring down the system in a day when
1000s of users use the same for 100s of

This is very important when we deal with device
drivers and embedded systems

Memory Profiling

This happens at runtime

Profiling is the basic block of performance

Concept of instrumentation and tracing happens

Call graph will tell what is the path the program
touched in terms of functions

Statistical profilers talk about memory and cpu

Memory consumed at that instance by variables
and functions

Memory released at any given point of time

Memory Quantification

What is the consumption of memory and
cpu is quantified

Usual quantification happens in basic unit
of measurement

UOM can be KB or machine cycles

How many cpu cycles are spent in this
method when this is running

How many cpus cycles are spent to
initialize the class

Memory Usage

















Buffer Overflow

When arrays or memory blocks are declared and we try
to access beyond those boundaries it is called overflow

When program variable space is packed, the overflows
can cause damage to program counters or other stack

When stack is compromised, the result is program abort

Dynamic arrays are also vulnerable to overflow when
people do not keep track of the max allocations

Buffer overflow can result on network buffers also when
the response buffer is underestimated

Protection of buffer overflow is usually made by
providing padding space to every memory space

this needs to be done against memory optimization

Object Issues

Objects require space and the methods need
cpu cycles

When objects are declared and not destroyed,
they also cause leaks

Unused objects are vulnerable areas for attack

Closure and nullifying objects are essential for
optimized use of memory

Creating a series of objects in a multi user
environment can cause spiral problems

wise Quantification

Every function has variables and statements

Parameters and variables take space from
symbol table as well as stack

Statements do take cpu cycles based on how
many times they get executed

When one function calls another, there is a huge
amount of context switch happening at heap and
program registry levels

How many times these context switches happen
will also determine the complexity of the

Test case for coverage

A path is defined as a route thru which a specific
functionality is achieved by calling a series of steps

If programs flow from top to bottom, it is easy

The more conditions, the more complex it is

Coverage must ultimately make up to 100% of the tested

Purecoverage will help in achieving this

Covering statements based on design is the best method

But when design is not in sync with program, we need to
apply simple logic to hit every statement that is present
in a program

Coverage Output





myproc (p1, p2)



if (p1 < p2)



call proc1();



if (p1 > p2)



call proc2();



if (p1 == p2)



call proc3();


Coverage for conditions

a < b && c > d

In the above c>d is evaluated only when a < b comes to true

a < b || c > d

In the above c>d is evaluated only when a < b comes to false

Though part of condition is not evaluated by compiler as
part of optimization, they may be later used in the

Conditions need not be derived based on business case

Most of the time, the conditions are based on parameter
values of the functions to determine the path to be taken

Coverage for blocks

This depends on how many conditions we have in the

Write one test for every relational operation of the

Definitely test db, IO and memory exceptions

Write one condition for every catch block

In case the exception cannot be easily triggered, have
conditional compiled statements to raise such exceptions

Most of the time, people ignore exceptions thinking that it
will be taken care

remember, it will be caught, but how
to treat the exception is up to us

Coverage goals

Coverage must have 100% at the end of all tests
for every function

There must be 1 test for every relational
operation in conditions

There must be 1 test for every loop termination

There must be 1 test for every exception

There must be 1 test for every test exit point (if
method has multiple return points)

There must be one test for successful sql

There must be one test for unsuccessful sql

Metrics Collection

Average number of unit test cases per page/component/any specific
unit in the project

Total number of unit test cases

Total number of unit test cases failed in test pass

Total number of unit test cases failed in test pass

Total number of unit test cases failed in test pass

Total number of memory leaks found

Percentage code coverage as per the tool statistics

Execution time for each test run

Total test execution time for all units

Ratio of # of defects found in unit testing to # of defects found in
code review

Defect distribution across units (asp related, db related, parameter
related etc.)

Metrics Analysis

Trend analysis helps in seeing whether the error
rate moves up or down

See pattern of failures and find out the corrective

Metrics vary a lot from developer to developer;
hence find out error rate pattern across

Analyze failure pattern and time of failure

It is not easy to catch all errors in one round of
testing; hence do more rounds; more often

Configuration Management

Since we develop test code, we need to check in
these code sections to VSS or subversion or

Maintain a uniform x.y version code to all test
cases and suites

Check in the test data along with test code as

Have one single person who can oversee all the
CM operations

Never allow people to have local copies of test

Best Practices

Keep adding more tests every day

More data preparation is the key

Test often

even twice a day

Never ignore warnings from IDEs

Run unit tests on a clean test bed

Review unit test code as well

Run regression unit tests daily

Run unit tests from other developer’s machine

Best Practices

Run unit tests by a different developer

Make unit test code simple

Never try to have if then else in a test code

Do not have loops in a test code

Run critical tests first

Run all unit tests in a single session; never


Do not test entire functionality in one test

One test must satisfy just one condition

Never underestimate any parameter

When skype or sony programs can come down
just by 1 bug, our program too can come down

If certain validations are performed in other
layers, do not do in unit tests

Never run tests on a test machine without clean


Never miss the null value test for any

Never miss any negative value tests for
database connection or query tests

Never miss trying to access an object or
record that does not exist, because these
are the places where exceptions get

Do not miss to pass a wrong xml structure
to web services


White box testers must be developers

Hence companies use them to do core development
work and ignore writing white box tests

White box tests need tools and sometimes they are a
little expensive; hence companies avoid this route

Main challenge comes in not documenting design. Since
design is the basis, and if that is not present or
incomplete, white box tests do not yield good results

Project planning never considers the time for white box
testing; especially in service companies, this is out of

When changes are made to the programs being tested,
the changes are not documented well and hence unit
tests go out of gear


When same developer runs the unit tests, there
is a chance to ignore issues

so have
developer A testing the code of developer B

Initial unit test automation code takes some time;
hence do not expect the ROI on day 1

Results can be seen only after one or 2 quarters.
Many management teams do not have patience
to wait for such a long time

People try to implement logic in different layers
that they are designated for; instead of coding
some logic in Java, they code in stored
procedure and call that from java

this makes
profiling and coverage difficult

Cyclomatic Complexity

This is to arrive at the testing complexity of code

Nodes, edges are the primary factors

Higher the complexity, higher the number of tests we need

This is based on graph theory

Many physical problems can be solved by graph theory

e.g. traveling salesman theory

M = E

N + 2P


number of edges (lines)


number of nodes (rounds)

P is the connected components (start and destination points)

The more if conditions and branches, the nodes and edges increase

Call graph of all methods will show the independent connected components

Business Flow Tests

When we do white box tests, we test individual

Hence it comes to unit tests

Divide flow into multiple components

Test them one by one

Aim at one transaction at a time

Club white box tools with functional test tool

Since business transactions need database
setup, ensure you have test stubs that pre
populate records in tables

How to speed

Let a senior create a sample test for every

Let a senior prepare unit test matrix

For a few tests, senior and junior
developers must do pair test automation

Then junior takes care of rest of the unit
tests for automation

Senior reviews the unit test code

Lead runs all tests every day

Get ready to take blame

Unit tests are code; hence they may also have
bugs. Developers may start blaming the tests

Tests may be redundant if not properly reviewed

Sequence of unit tests must be carefully done.
Always use bottom up approach on unit tests

test the smallest one first

Run unit tests soon after build is released, and
before BVT starts by the black box team

Never Give Up

Teams give up white box tests after a quarter or so

This is because, they do not see visible returns


these things fail is a feel
good factor on
testing; at the same time, these things work is more
important as well

Most of the times, great systems are brought down due
to missing white box tests and not black box functionality

It may take years to find out, one great bug

but it is
essential for big time products

If you are in product company, be patient and white box
will pay back