# Traps in Test Estimation

Mechanics

Nov 5, 2013 (4 years and 6 months ago)

87 views

Traps in Test Estimation

Shrini Kulkarni

Principal Consultant
-

Testing

iGATE Global Solutions

Bangalore, India

shrinik@igate.com

http://shrinik.blogspot.com

What is a Trap and what is an
Estimation?

Trap:
noun

-

Something (often
something deceptively attractive)
that catches you unaware or place
of confining or embarrassing
situation

Estimation: An approximate
calculation of quantity or degree or
worth of some activity or an object
of value

Test Estimation: An Act or process
of estimation applied to a testing

Consider this … A typical situation

Test Size

Test Effort

Schedule

Key Question

How many testers for how long?

Inputs

Requirements

Test cases

Use cases

User guide

Expert Opinions

Is there any Scientific Method for Test estimation?

A List of Commonly Used Techniques …

Guestimation

Wild Ass Guess

rule of thumb

A percentage of development effort

Risk Based methods

Determine what to test and how much

Use of historical Data + Industry Standards

Work
-
break down Structure

WBS method

Use of Dev Size estimates

Use case, function points, COCOMO

Test point Analysis method

Delphi

Wideband Delphi Method with WBS

Consensus opinion in a team of expert estimators

But what all is included in “Product” and

in the phrase “test” ?

How long it takes to you
test this Product ?

It all starts with this Motherhood Question …

Getting Trapped?

How will you know you will be able to complete Testing as per the

estimate ?
We don't. Why pretend we do?
-

Matthew Heusser

Trap # 1
-

Model of Testing

There's a slippery slope between asking for good faith estimates ("Knowing what

you know now, when do you think you can deliver?") and predicting the future.

-

Mathew Heusser

A simplified model of Testing is assumed.

Testing is an open ended search for problems

decision to

stop testing is often decided by the stakeholders depending

upon the parameters that are out side the purview of testing

Actual Testing model is complex hence estimates tend to go

wrong

Testing is assumed to be complete when planned test cases

are executed

Dependencies on environment, availability of Stable build to

test and test data are typically ignored or down played

Trap # 2

Testers do not decide the
ship date

Project Manager or the customer picks the date

Why rob him the

opportunity to do so?

Testers provide service to PM to decide about state of the product

Testers do not have the ability to add or remove the features

Testers do not have the ability (in most of the cases) to extend the

Ship date beyond what is planned by the PM

This creates bounded time space for testers to operate on

Thus all you have is to test till PM says stop

Will estimates matter

then?

Trap # 3
-

Unknown Future

We’re too optimistic, with short memories that mask the painful overruns
from previous projects

Karl E. Wiegers

Things change during the course of the project in unimaginable ways

Each test cycle presents its own challenges and issues

There might be be too many or too less bugs discovered in any cycle

There might be developer delays that eats up test time

There is might be pressure to skip tests and constantly change test sets

Development might lag behind bug find rate

There might be requirements changes hence development delays

There might be failures in Build verification tests

There might be significant number of “non reproducible defects”

Trap # 4
-

Too many Variables

Giving an Estimate would mean a commitment

which you are not

that is a trap

Test scope

that keeps changing depending upon available time

for testing

Set of features available in a given build

Time available for testing for any given cycle

Test environment availability

Stability of build for testing

Unknown number of bugs discovered any given cycle

Variable bug find and fix rate

Additional cycles of Testing for every new bug or regression bug

Trap # 5

Match that MAGIC number

Role Play …

Unlike other forms of estimation, I can make the estimate totally
predictable: I'll stop when the allotted time is up

Michael Bolton

Michael Bolton’s Recipe for Estimation …

Testing is an open ended search for problems

we would never say “we are

Done”

instead we temporarily stop and pass on the information to the

Stakeholder and wait for next opportunity to start ….

When do we stop then ? Several ways ….

Please see Next slide …

Heuristics for “Stopping Testing”

If we've modeled the product in lots of different ways, and then tested in accordance

with those models, we can say that we've done enough testing and stop.

If we're satisfied that we have addressed the list of compelling questions that we set out

to answer (along with the other questions that we realized are important along the way),

then we can say that we've done enough testing and stop.

If the product is so horribly broken that it has to be send in for major rework, then we can

say that we've done enough testing and stop.

If management decides that it has sufficient confidence to ship, and ships, then we can

say that we've done enough testing and stop.

If management decides that it must ship the product, even though confidence in its quality

is less than what we'd like it to be, we can say that we've done enough testing and stop.

If we're testing on behalf of some one who is trying to decide whether the software is

acceptable, and they say it is, we can say that we've done enough testing and stop.

If we're testing on behalf of someone who is trying to decide whether the software

is acceptable, and they say it isn't, until we get another version, we can say that we've

done enough testing and stop.

So … what is the solution?

I hear you, I hear you…”So Mr. Big Man, what’s

your solution?” ….. the truth is

I don’t have one.

I don’t think that’s a failure, but a realization

I’m not the lone ranger, and the estimation

problem may be a bugbear, but it’s not a

werewolf.

Source : http://blog.geeksmithology.com/2006/11/14/estimating
-
is
-
bunk

But I do have some suggestions …

what date?

Take overall testing as cycle by cycle

Propose an Adaptive and iterative approach

Resist the temptation to commit for ONE date

Say

you will test until ship date and until told not to

test…

Hell, there are no rules here

we’re trying to accomplish something.
-

Thomas A. Edison

And …

Questions?

Thank you

http://www.igate.com

http://shrinik.blogspot.com

shrinik@igate.com

shrinik@gmail.com

For further discussions:

Shrini Kulkarni

Principal Consultant
-

Testing

iGATE Global Solutions Ltd

Bangalore

References

Rapid Test Estimation

Michael Bolton’s blog post

http://www.developsense.com/2007/01/test
-
project
-
estimation
-
rapid
-
way.html