The closeness of agreement between independent
results with the same method on identical test material
but under different lab’s, apparatus, conditions,
operators and/or different intervals of time.
Repeatability is a measure of the variability of a
single response within a single laboratory.
Reproducibility measures the variability of same
response across different laboratories.
Replication means that other researchers in other
settings with different samples attempt to reproduce
the research as closely as possible.
Reliability is the stability of a set of observations
generated by an indicator under fixed set of
conditions, regardless who collects the observations
or when or where they are collected
To investigate the extent to which key dimensions of
software projects and products are reproduced.
What dimensions to measure reproducibility?
What is the best way to measure them?
What values indicate low, medium, high
Coefficient of Variation:
Reproducibility = 1/
81 Norwegian and international software
consultation companies: the specification
of requirements , Java programming and minimum
size of the company
35 companies provided bids
4 companies were selected to develop individual
Contractor Related costs
Actual Lead time
A set of software providers that similar project and
If a random set of providers produce same results.
If dissimilar providers produce similar results.
If different types of providers produce different
results that can be predicted from differences among
Methodological : The measurement framework of
entire cycle of a project and multifaceted
Scientific: Better understanding of Reproducibility
in software productions that should provide the
basis for SE methods, tools and studies.
Practical: Descriptions of detailed scenarios of what
can be expected in a software contracting context
and how to use software bids to anticipate project
How accurately the values of the variables
How different values are aggregated to
measure the dimensions of software
How well those dimensions represent the
constructs that they suspend to
Variation in skills
No previous Business or personal Relationships
Higher Reproducibility for Usability
9 Companies developed same system to investigate
the three development platforms
cost based model
Analogy based Estimation
use the cost model of
completed similar projects
More companies and investigate different contexts
of software production.
Maintenance Phase should be that would complete
the picture of the entire life cycle.
Outsource the participation in studies to
professionals in countries with lower costs.
Deep and detailed understanding of the relationships
Investigate only isolated aspects of software
production ,that does not capture how software
works in real practice.
Effort was put into the conceptual definitions and
of fundamental SE terms.