Traps in Test Estimation

fanaticalpumaMechanics

Nov 5, 2013 (3 years and 11 months ago)

74 views

Traps in Test Estimation







Shrini Kulkarni

Principal Consultant
-

Testing

iGATE Global Solutions

Bangalore, India

shrinik@igate.com

http://shrinik.blogspot.com


What is a Trap and what is an
Estimation?


Trap:
noun

-

Something (often
something deceptively attractive)
that catches you unaware or place
of confining or embarrassing
situation


Estimation: An approximate
calculation of quantity or degree or
worth of some activity or an object
of value



Test Estimation: An Act or process
of estimation applied to a testing
task


Consider this … A typical situation

Test Size

Test Effort

Schedule

Key Question


How many testers for how long?

Inputs





Requirements



Test cases



Use cases



User guide



Expert Opinions


Is there any Scientific Method for Test estimation?

A List of Commonly Used Techniques …



Guestimation


Wild Ass Guess


rule of thumb




A percentage of development effort




Risk Based methods


Determine what to test and how much




Use of historical Data + Industry Standards




Work
-
break down Structure


WBS method




Use of Dev Size estimates


Use case, function points, COCOMO




Test point Analysis method




Delphi


Wideband Delphi Method with WBS


Consensus opinion in a team of expert estimators


But what all is included in “Product” and

in the phrase “test” ?

How long it takes to you
test this Product ?

It all starts with this Motherhood Question …

Getting Trapped?

How will you know you will be able to complete Testing as per the

estimate ?
We don't. Why pretend we do?
-

Matthew Heusser


Trap # 1
-

Model of Testing

There's a slippery slope between asking for good faith estimates ("Knowing what

you know now, when do you think you can deliver?") and predicting the future.



-

Mathew Heusser



A simplified model of Testing is assumed.




Testing is an open ended search for problems


decision to


stop testing is often decided by the stakeholders depending


upon the parameters that are out side the purview of testing




Actual Testing model is complex hence estimates tend to go


wrong




Testing is assumed to be complete when planned test cases


are executed




Dependencies on environment, availability of Stable build to


test and test data are typically ignored or down played

Trap # 2


Testers do not decide the
ship date

Project Manager or the customer picks the date


Why rob him the

opportunity to do so?



Testers provide service to PM to decide about state of the product




Testers do not have the ability to add or remove the features




Testers do not have the ability (in most of the cases) to extend the


Ship date beyond what is planned by the PM




This creates bounded time space for testers to operate on


Thus all you have is to test till PM says stop


Will estimates matter

then?



Trap # 3
-

Unknown Future

We’re too optimistic, with short memories that mask the painful overruns
from previous projects


Karl E. Wiegers



Things change during the course of the project in unimaginable ways




Each test cycle presents its own challenges and issues



There might be be too many or too less bugs discovered in any cycle



There might be developer delays that eats up test time



There is might be pressure to skip tests and constantly change test sets



Development might lag behind bug find rate



There might be requirements changes hence development delays



There might be failures in Build verification tests



There might be significant number of “non reproducible defects”


Trap # 4
-

Too many Variables

Giving an Estimate would mean a commitment


which you are not

sure about fulfilling


that is a trap



Test scope


that keeps changing depending upon available time


for testing



Set of features available in a given build



Time available for testing for any given cycle



Test environment availability



Stability of build for testing



Unknown number of bugs discovered any given cycle



Variable bug find and fix rate



Additional cycles of Testing for every new bug or regression bug

Trap # 5


Match that MAGIC number

Role Play …

Unlike other forms of estimation, I can make the estimate totally
predictable: I'll stop when the allotted time is up


Michael Bolton

Michael Bolton’s Recipe for Estimation …

Testing is an open ended search for problems


we would never say “we are

Done”


instead we temporarily stop and pass on the information to the

Stakeholder and wait for next opportunity to start ….


When do we stop then ? Several ways ….


Please see Next slide …







Heuristics for “Stopping Testing”



If we've modeled the product in lots of different ways, and then tested in accordance


with those models, we can say that we've done enough testing and stop.





If we're satisfied that we have addressed the list of compelling questions that we set out


to answer (along with the other questions that we realized are important along the way),


then we can say that we've done enough testing and stop.




If the product is so horribly broken that it has to be send in for major rework, then we can


say that we've done enough testing and stop.




If management decides that it has sufficient confidence to ship, and ships, then we can


say that we've done enough testing and stop.




If management decides that it must ship the product, even though confidence in its quality


is less than what we'd like it to be, we can say that we've done enough testing and stop.




If we're testing on behalf of some one who is trying to decide whether the software is


acceptable, and they say it is, we can say that we've done enough testing and stop.





If we're testing on behalf of someone who is trying to decide whether the software


is acceptable, and they say it isn't, until we get another version, we can say that we've


done enough testing and stop.


So … what is the solution?






I hear you, I hear you…”So Mr. Big Man, what’s

your solution?” ….. the truth is


I don’t have one.



I don’t think that’s a failure, but a realization




I’m not the lone ranger, and the estimation

problem may be a bugbear, but it’s not a

werewolf.


Source : http://blog.geeksmithology.com/2006/11/14/estimating
-
is
-
bunk


But I do have some suggestions …




Ask what is in your sponsor's mind


what date?



Take overall testing as cycle by cycle



Propose an Adaptive and iterative approach



Resist the temptation to commit for ONE date



Say


you will test until ship date and until told not to


test…

Hell, there are no rules here


we’re trying to accomplish something.
-

Thomas A. Edison


And …

Questions?

Thank you






http://www.igate.com

http://shrinik.blogspot.com


shrinik@igate.com

shrinik@gmail.com








For further discussions:

Shrini Kulkarni

Principal Consultant
-

Testing

iGATE Global Solutions Ltd

Bangalore

References


Rapid Test Estimation


Michael Bolton’s blog post




http://www.developsense.com/2007/01/test
-
project
-
estimation
-
rapid
-
way.html