Need for White Box Testing

jetmorebrisketSoftware and s/w Development

Aug 15, 2012 (5 years and 6 days ago)

403 views

White Box Testing


Agenda


The need


Benefits


Code review


Unit tests and automation


cppUnit Concepts


Memory leaks


Quantification


Coverage


APIs


Challenges


Best Practices


Don’ts


Metrics


Cyclomatic Complexity


Need for White Box Testing


Better to test anything from outside
-
in as well as inside
-
out


Most of the functional and performance issues arise due to bad coding


Ultimately the code matters


hence test the code left right and center


Black box tests are used to check cause and effect without getting into
internals


White box tests do the same by getting into the internals of every program


Every developer is by default a white box tester


Medical field requires general physicians and surgeons


Certain problems can be identified and solved only by surgeons


A person may look healthy but internally the person may have high bp and
sugar; they are not visible externally


So far, the importance of white box testing is not upto the mark


Product companies definitely need white box testing


If your application must scale to a very large extent, white box tests are
inevitable

Black Box Cannot Test These


It takes long time to check memory leaks


Any exceptions that happened but unnoticed or unhandled


Which portion of if condition is evaluated


What path the program took to achieve the end result


For every request what is the memory and cpu time taken


Is there any dead and unused code


Is there any extra code that is not needed


What are the potential breaking points in code


Is the code compliant to best practices


In large business apps, the data paths will be too large to test; it will
require senior business analysts; if done thru white box mode,
normal programmers can bring out great bugs

Diagnostics tools


IBM Rational Purifyplus


JProbe


Parasoft tools


HP Mercury Diagnostics


Dynatrace


.Net profiler


AppPerfect


Netbeans


AQTime


All tools work on instrumentation and trace options


Some tools help in analyzing system memory space as well


Usually the system cpu executions are privileged, and hence we
may not get access to those


Business Benefits


Ensures that the core building blocks are intact, when
adding new features and modules


Go to market with confidence on product stability


Provides greater visibility in terms of technical and
internal compliance of the product


Vulnerability testing always needs a greater amount of
white box testing


Resource utilization reduction is key for marketing.
Products that use less memory and bandwidth are the
most welcome ones


Keeps business leaders informed on the underlying
strengths and weaknesses of the application platform

Understand Design


Honestly speaking, how many of us use proper design before
coding?


Can the same happen to real estate industry without a proper floor
plan or for pharma industry without proper process and formula for
drugs?


Design has 4 parts


UI, database, Interface and Logic design


UI design is key aspect when it comes to usability and compliance


Database design ensures scalability and storage


Interface design is key for plug and play during integration


Logic is important for the business flow of the application


Logic design can be a flow chart or text algorithm or pseudo code


Most of the modelers do have pictorial representation of design


Ideally the code must reflect the translation of pseudo code


Traceability of design elements to code is one of the key areas we
need to look into


A change in design may impact several parts of code

Code Review


Most of the times, this is ignored or not done to its true
spirit


Code review must have design and code side by side


Code review can save upto 30% of testing cost at a later
point in time


Usual problems happen due to cut and paste of code
where it is not required


When there is a transition from one developer to
another, code goes thru challenges


Quick fixes on big projects cause enormous amount of
turbulence on production systems


Ideally senior people must review code


Spend at least 10% of coding time on code review

Code Review Checklist


Just try these 10 critical points in daily life

1.
No hard coding

2.
Ensure loop termination conditions

3.
No object creation inside loops

4.
Close every object that you open

5.
Give comments to every code block

6.
Use database connection only when you need

7.
Remove unused variables and code portions

8.
Follow a consistent naming convention

9.
Try to reduce overloading methods very often

10.
Check SQL or file status soon after every disk
operation


Unit Testing


The smallest piece of code that can be independently
run and tested is a unit


Unit can be a page or a function or a method within a
class or a whole program itself


It is the developer’s responsibility to unit test the code


Unit test must focus on data type of every parameter


Focus on data format of every parameter


Focus on boundary values of every parameter


For any given code, what deliverable do we provide as
part of unit testing?


Most of the times it is just the trust on the developer


Developer must spend at least 30% of coding time on
unit testing alone

How to draw boundaries


What is the line that demarcates unit and
other tests?


One expected output out of a function


No chaining


When to stop unit tests?


When a set of parameter combinations are
tested, stop the same


Boundaries and equivalence partitions must
be tested before exit




Unit Testing Checklist


Are the naming conventions followed?


Are the comments done properly?


Is there any hard coding?


Test case for custom exceptions.


Test case for system exceptions.


Test case for “body” of an if condition.


Test case for “body” of an else condition.


Test case for every loop termination.


Test case for every recursion termination.


Test case for pointers release (for memory leaks).


Test case for every procedure entry and exit.


Test case for every parameter validation for procedures.


Test case for resource release (Closing DB connections,
releasing objects etc.)


Unit Test Matrix


A single test must make a call to a function with some
parameters combination and expect return value(s)


If a function has got multiple parameters that are of
different types, then combinations of all parameters are
possible and it will become a Cartesian product table


As the first step, a matrix needs to be prepared with all
such combinations


Then the matrix must be optimized to reflect the
practicality of the inputs


Possibilities of return paths must also be documented


The samples may go in 100s but not to worry on


Since automated unit tests run extremely fast without
human inputs, 1000s of tests can be executed in minutes

Automate Unit Tests


It is not practically possible to give all combinations of the inputs to
programs and run


To reduce time, we need to write a program, that calls the program
under test and pass parameters


Ideally the test program must be in the same language as the
program being tested


This will help in maintaining the homogeneity of the data type of
parameters passed


Since developers know the programming language, it is easy for
them to write the test program as well


These are called test suites or test scripts


The investment on the unit test automation is one time as long as
changes are minimal


The person who automates test for a program must know the
program well

The core of unit automation


Class A


Method m1(p1, p2)


Method m2(p3, p4)



TestClass TA


Testcase tm1


ta = new Class A


rc = ta.m1(4, 5)


If (rc == exp_result)


Pass


Else


Fail


// destroy ta



Level 1 Automation


Crude method


quick solution for short
term projects


Write wrapper callers


Hard code parameters


Cut and paste test code with no hesitation


Spend little time on test creation


Test code can be throw
-
away soon


Simple logs, more manual result analysis
required



Level 2 Automation


Parameters to tests must be separated
from test code


You can use csv or xml or any format for
test data


Useful to run same test on larger test data
sets


When a mixture of customers use the
product, the data set also will change
largely


this will be helpful in that case


Use your own test logs with more details

Level 3 Automation


Use test tools such as cppUnit or jUnit etc.


Build auto recovery options if tests fail in
between


Ensure every test has start
-
run
-
cleanup
structure


Use this on long term test engagements


When tests are to be reusable, this is the
best option


If you know cppUnit, the concepts are
exactly the same in nUnit, jUnit etc.


Stubs and Drivers


When the program under test needs some pre
-
built data or state, we need to mimic the same


This is a short
-
cut approach, but it serves the
purpose


Stub is the one that mimics a called function


Driver is the one that mimics a caller function


When we use stubs and drivers, it is a must that
we ensure that the data created by them are
destroyed soon after the test

Testing APIs


APIs do exchange of data from one piece to another


APIs are usually consumed by many independent
programs


APIs in general are meant to provide a window of
information to 3
rd

parties


One change in API will affect all consumers at once


Most of the current day apps provide APIs


We can send tweets from our application using APIs


We can send emails from Amazon SES thru APIs


API calls must be assumed to be context neutral


Some APIs do require authentication mechanism


APIs fall under external interfaces

Wrapper Tools for APIs


A set of business critical APIs are now being
used in all EAI scenarios


FIX, SWIFT, News feeds, HL7, HIPAA, Part 11
standards etc


A variety of commercial tools are available in
each category


When messages are sent, these tools validate
the structures of XML or EDI against the
templates


Also, there are different versions of these APIs
exist like html 5 etc.

cppUnit


Write test wrappers in cpp for cpp programs


If you are familiar with cpp programming, you may not
require cppUnit !


There are many versions of cppUnit itself as it is open
source


many people modified it for their convenience


You can write your own test library and framework


A set of pre
-
built classes and methods are already made
available to minimize test coding time


Testcase, Testsuite, Assert are the important aspects in
cppUnit


Testcase is the basic block


every single test


Testsuite contains multiple tests ( a batch)


Assert is the actual verification point where actual result
is compared with expected result


cppUnit


Fixture


known set of objects for test


setUp is the place where we create testcases


tearDown is the place where we remove all
created tests


Commonly used asserts


Assert


Assert_message


Assert_equal


Assert_throw


Assert_equal_message



Instrumentation


White box tools and profiler tools use one single concept called
Instrumentation


Source code is compiled and built as binary


These tools understand the format of these binary files


where
function entry points are made, how they exit, where stack is
maintained etc.


Before profiling, they instrument the binary


inject their own log
codes in the binary locations


During the execution, the instrumented code is run and every detail
is logged using these injected code (such as entered function x,
executed loop y, exited function y, destroyed object o etc.)


Using these logs, the tools provide internal details


This is very similar to sending a probe inside our body and doing a
scan of that probe

Overheads


When profilers run along with the instrumented
programs, it adds load to the machine


This is a necessary overhead we need to take


The performance overheads are negligible when
compared to actual object size


The statistics that are collected out of the
programs will vary from time to time; but if the
variance is beyond 5%, then the tool has real
issues


Also, do not run 2 competing tools on the same
machine. The effects are unpredictable

Memory Leaks


When a memory pointer is declared and
initialized and it is not released, it is a leak


Even in managed code sections, the leaks are
there for a time till the garbage collector acts


Leaked memory locations are useless and
vulnerable


One program leaking 10 bytes of memory per
call


can bring down the system in a day when
1000s of users use the same for 100s of
transaction


This is very important when we deal with device
drivers and embedded systems



Memory Profiling


This happens at runtime


Profiling is the basic block of performance
engineering


Concept of instrumentation and tracing happens


Call graph will tell what is the path the program
touched in terms of functions


Statistical profilers talk about memory and cpu


Memory consumed at that instance by variables
and functions


Memory released at any given point of time

Memory Quantification


What is the consumption of memory and
cpu is quantified


Usual quantification happens in basic unit
of measurement


UOM can be KB or machine cycles


How many cpu cycles are spent in this
method when this is running


How many cpus cycles are spent to
initialize the class

Memory Usage

Method

Kbytes

logon

138

do_registration

1607

view_bill

254

pay_bill

2897

dispute_bill

4876

view_summary

2346

get_call_details

1345

logon
1%
do_registration
12%
view_bill
2%
pay_bill
22%
dispute_bill
36%
view_summary
17%
get_call_details
10%
Buffer Overflow


When arrays or memory blocks are declared and we try
to access beyond those boundaries it is called overflow


When program variable space is packed, the overflows
can cause damage to program counters or other stack
areas


When stack is compromised, the result is program abort


Dynamic arrays are also vulnerable to overflow when
people do not keep track of the max allocations


Buffer overflow can result on network buffers also when
the response buffer is underestimated


Protection of buffer overflow is usually made by
providing padding space to every memory space


but
this needs to be done against memory optimization

Object Issues


Objects require space and the methods need
cpu cycles


When objects are declared and not destroyed,
they also cause leaks


Unused objects are vulnerable areas for attack


Closure and nullifying objects are essential for
optimized use of memory


Creating a series of objects in a multi user
environment can cause spiral problems

Function
-
wise Quantification


Every function has variables and statements


Parameters and variables take space from
symbol table as well as stack


Statements do take cpu cycles based on how
many times they get executed


When one function calls another, there is a huge
amount of context switch happening at heap and
program registry levels


How many times these context switches happen
will also determine the complexity of the
programs

Test case for coverage


A path is defined as a route thru which a specific
functionality is achieved by calling a series of steps


If programs flow from top to bottom, it is easy


The more conditions, the more complex it is


Coverage must ultimately make up to 100% of the tested
function


Purecoverage will help in achieving this


Covering statements based on design is the best method


But when design is not in sync with program, we need to
apply simple logic to hit every statement that is present
in a program

Coverage Output

Line

Code

Time

47

myproc (p1, p2)

7181

48

if (p1 < p2)

2

49

call proc1();

388

50

if (p1 > p2)

0

51

call proc2();

0

52

if (p1 == p2)

2

53

call proc3();

6789

Coverage for conditions


a < b && c > d


In the above c>d is evaluated only when a < b comes to true



a < b || c > d


In the above c>d is evaluated only when a < b comes to false



Though part of condition is not evaluated by compiler as
part of optimization, they may be later used in the
program


Conditions need not be derived based on business case


Most of the time, the conditions are based on parameter
values of the functions to determine the path to be taken


Coverage for blocks


This depends on how many conditions we have in the
code


Write one test for every relational operation of the
condition


Definitely test db, IO and memory exceptions


Write one condition for every catch block


In case the exception cannot be easily triggered, have
conditional compiled statements to raise such exceptions


Most of the time, people ignore exceptions thinking that it
will be taken care


remember, it will be caught, but how
to treat the exception is up to us

Coverage goals


Coverage must have 100% at the end of all tests
for every function


There must be 1 test for every relational
operation in conditions


There must be 1 test for every loop termination


There must be 1 test for every exception


There must be 1 test for every test exit point (if
method has multiple return points)


There must be one test for successful sql
operation


There must be one test for unsuccessful sql
operation

Metrics Collection


Average number of unit test cases per page/component/any specific
unit in the project


Total number of unit test cases


Total number of unit test cases failed in test pass
-
1


Total number of unit test cases failed in test pass
-
2


Total number of unit test cases failed in test pass
-
n


Total number of memory leaks found


Percentage code coverage as per the tool statistics


Execution time for each test run


Total test execution time for all units


Ratio of # of defects found in unit testing to # of defects found in
code review


Defect distribution across units (asp related, db related, parameter
related etc.)

Metrics Analysis


Trend analysis helps in seeing whether the error
rate moves up or down


See pattern of failures and find out the corrective
action


Metrics vary a lot from developer to developer;
hence find out error rate pattern across
developers


Analyze failure pattern and time of failure


It is not easy to catch all errors in one round of
testing; hence do more rounds; more often


Configuration Management


Since we develop test code, we need to check in
these code sections to VSS or subversion or
PVCS


Maintain a uniform x.y version code to all test
cases and suites


Check in the test data along with test code as
well


Have one single person who can oversee all the
CM operations


Never allow people to have local copies of test
code

Best Practices


Keep adding more tests every day


More data preparation is the key


Test often
-

even twice a day


Never ignore warnings from IDEs


Run unit tests on a clean test bed


Review unit test code as well


Run regression unit tests daily


Run unit tests from other developer’s machine

Best Practices


Run unit tests by a different developer


Make unit test code simple


Never try to have if then else in a test code


Do not have loops in a test code


Run critical tests first


Run all unit tests in a single session; never
stagger


Don’ts


Do not test entire functionality in one test


One test must satisfy just one condition


Never underestimate any parameter
combination


When skype or sony programs can come down
just by 1 bug, our program too can come down


If certain validations are performed in other
layers, do not do in unit tests


Never run tests on a test machine without clean
up

Don’ts


Never miss the null value test for any
parameter


Never miss any negative value tests for
database connection or query tests


Never miss trying to access an object or
record that does not exist, because these
are the places where exceptions get
triggered


Do not miss to pass a wrong xml structure
to web services

Challenges


White box testers must be developers


Hence companies use them to do core development
work and ignore writing white box tests


White box tests need tools and sometimes they are a
little expensive; hence companies avoid this route


Main challenge comes in not documenting design. Since
design is the basis, and if that is not present or
incomplete, white box tests do not yield good results


Project planning never considers the time for white box
testing; especially in service companies, this is out of
scope!


When changes are made to the programs being tested,
the changes are not documented well and hence unit
tests go out of gear

Challenges


When same developer runs the unit tests, there
is a chance to ignore issues


so have
developer A testing the code of developer B


Initial unit test automation code takes some time;
hence do not expect the ROI on day 1


Results can be seen only after one or 2 quarters.
Many management teams do not have patience
to wait for such a long time


People try to implement logic in different layers
that they are designated for; instead of coding
some logic in Java, they code in stored
procedure and call that from java


this makes
profiling and coverage difficult

Cyclomatic Complexity


This is to arrive at the testing complexity of code


Nodes, edges are the primary factors


Higher the complexity, higher the number of tests we need


This is based on graph theory


Many physical problems can be solved by graph theory


e.g. traveling salesman theory



M = E
-

N + 2P


E
-

number of edges (lines)


N
-

number of nodes (rounds)


P is the connected components (start and destination points)



The more if conditions and branches, the nodes and edges increase



Call graph of all methods will show the independent connected components

Business Flow Tests


When we do white box tests, we test individual
components


Hence it comes to unit tests


Divide flow into multiple components


Test them one by one


Aim at one transaction at a time


Club white box tools with functional test tool


Since business transactions need database
setup, ensure you have test stubs that pre
-
populate records in tables

How to speed
-
up


Let a senior create a sample test for every
function


Let a senior prepare unit test matrix


For a few tests, senior and junior
developers must do pair test automation


Then junior takes care of rest of the unit
tests for automation


Senior reviews the unit test code


Lead runs all tests every day

Get ready to take blame


Unit tests are code; hence they may also have
bugs. Developers may start blaming the tests
themselves


Tests may be redundant if not properly reviewed


Sequence of unit tests must be carefully done.
Always use bottom up approach on unit tests


test the smallest one first


Run unit tests soon after build is released, and
before BVT starts by the black box team

Never Give Up


Teams give up white box tests after a quarter or so


This is because, they do not see visible returns


Remember


these things fail is a feel
-
good factor on
testing; at the same time, these things work is more
important as well


Most of the times, great systems are brought down due
to missing white box tests and not black box functionality


It may take years to find out, one great bug


but it is
essential for big time products


If you are in product company, be patient and white box
will pay back