Benefits that Deserve Further Study - Environmental Sciences Division

boingcadgeΜηχανική

18 Νοε 2013 (πριν από 4 χρόνια και 1 μήνα)

199 εμφανίσεις

DRAFT









IDEAS ON A FRAMEWORK AND METHODS FOR
ESTIMATING THE BENEFITS OF GOVERNMENT
-
SPONSORED ENERGY R&D



Russell Lee, Oak Ridge National Laboratory*



With contributions from:


James L. Wolf, Independent Consultant

Mary Beth Zimmerman, Office Energy Effic
iency and Renewable Energy, DOE

Jay Braitsch, Office of Fossil Energy, DOE

Trevor L. Cook, Office of Nuclear Energy, Science and Technology, DOE

Robert Vallario, Office of Science, DOE

Jeanne Powell, Advanced Technology Program, NIST

Robert Pearce, Office
of Aerospace Technology, NASA

Robert C. Ricci, Volpe Transportation Center, DOT






February 27, 2002


Prepared as pre
-
conference reading material for the conference on


"Estimating the Benefits of Government
-
Sponsored Energy R&D,"


March 4 and 5, 2002, A
rlington, Virginia, USA.


__________________


* Oak Ridge National Laboratory is managed by UT
-
Battelle, LLC for the U.S. Department of Energy
under Contract No. DE
-
AC05
-
00OR22725. The views expressed in various sections of this paper are
solely those of
their respective authors; and do not necessarily represent those of Oak Ridge National
Laboratory, UT
-
Battelle, the U.S. Department of Energy, the National Institute of Standards and
Technology, the National Aeronautical and Space Administration, or the U.
S. Department of
Transportation.
DRAFT


i



TABLE OF CONTENTS


Page


SUMMARY


1.

PURPOSE


2.

CONTEXT AND MOTIVATION FOR THE CONFERENCE


3.

METHODS USED BY DOE ENERGY RESOURCES AND SCIENCE OFFICES TO
ESTIMATE OR ASSESS THE BENEFITS OF THEIR R&D PROGRAMS


3.1.

OVERVIEW OF MET
HODS USED BY THE OFFICE OF ENERGY
EFFICIENCY AND RENEWABLE ENERGY FOR GPRA


3.2.

MEASURING FUTURE BENEFITS FOR FOSSIL ENERGY RESEARCH
AND DEVELOPMENT




3.2.1.

Fossil Fuel Conversion



3.2.2.

Fossil Energy Resources



3.2.3.

Future Directions



3.3.

OVERVIEW OF METH
ODS USED BY THE OFFI
CE OF SCIENCE


4.

METHODS USED BY OTHER FEDERAL AGENCIES TO MEASURE THE
OUTCOME
-
RELATED PERFORMANCE OF THEIR PROGRAMS



4.1.

MEASURING THE BENEFITS OF THE ADVANCED TECHNOLOGY
PROGRAM




4.1.1.

ATP's Mission and Operations

4.1.2.

ATP's Multi
-
Co
mponent Assessment Program

4.1.3.

ATP's Evaluation Results




4.2.

FEDERAL RAILROAD ADMINISTRATION'S (FRA'S) R&D PROJECT
EVALUATION AND INVESTMENT ANALYSIS



5.

NATIONAL RESEARCH COUNCIL’S METHODOLOGY FOR DEFINING THE
BENEFITS OF ENERGY R&D



5.1.

METHODOLOGICAL
FRAMEWORK



5.2.

DERIVATION OF COLUMNS FOR THE BENEFITS MATRIX

DRAFT


ii


TABLE OF CONTENTS (Cont’d)


Page




5.3.

MAJOR ISSUES CONSIDERED


ALL NOT NECESSARILY RESOLVED




5.3.1.

Establishing a Baseline for Realized or Projected Benefits



5.3.2.

Effect of Governmen
t Program



5.3.3.

Nature of Benefits


6.

BUILDING ON THE NATIONAL RESEARCH COUNCIL’S STUDY


IDEAS ON
FRAMEWORK AND METHODS FOR ESTIMATING THE BENEFITS OF
GOVERNMENT R&D



6.1.

PROSPECTIVE ANALYSIS OF BENEFITS




6.1.1.

Definition of Expected Prospective
Benefits



6.1.2.

Projected Baseline



6.1.3.

Government Impact




6.2.

OPTION VALUE





6.2.1.

Option Concept as Described in the NRC Study



6.2.2.

Applying Real Options Theory to R&D Investments



6.3.

KNOWLEDGE VALUE





6.3.1.

Assessment of Knowledge
Benefits



6.3.2.

Spillovers




6.4.

SECURITY, ECONOMIC, AND ENVIRONMENTAL BENEFITS




6.4.1.

Security Benefits



6.4.2.

Economic Benefits



6.4.3.

Environmental Benefits


7.

CONSIDERATIONS IN IMPLEMENTING A FRAMEWORK AND METHODS



7.1.

DATA AND DATA MANAG
EMENT CONSIDERATIONS



7.2.

PROCESS CONSIDERATIONS



7.3.

ASSESSING A PORTFOLIO OF R&D PROGRAMS AS OPPOSED TO JUST
INDIVIDUAL PROJECTS


DRAFT


iii


TABLE OF CONTENTS (Cont’d)


Page




7.4.

ASSESSING MULTI
-
YEAR PROJECTS OR PROGRAMS AS OPPOSED TO
AN INDIVIDUAL YEAR’S BU
DGET REQUEST FOR A PROJECT


8.

CONFERENCE FORMAT



8.1.

PLENARY SESSIONS



8.2.

WORKSHOP A: PROSPECTIVE BENEFITS


CROSS
-
CUTTING ISSUES


8.2.1.

Developing a Baseline, Absent a Technology Developed in Part




by a Government Program



8.2.2.

Impact of the Governme
nt Program



8.3.

WORKSHOP B: OPTION VALUE



8.4.

WORKSHOP C: KNOWLEDGE VALUE



8.5.

WORKSHOP D: SECURITY, ECONOMIC, AND ENVIRONMENTAL
BENEFITS


DRAFT


iv


IDEAS ON A FRAMEWORK AND ASSOCIATED METHODS
FOR ESTIMATING THE BENEFITS OF GOVERNMENT
-
SPONSORED ENERGY R&D



S
UMMARY


This white paper presents a starting point for discussions at the conference on "Estimating the
Benefits of Government
-
Sponsored Energy R&D." The specific objectives of this conference are
to identify major streams of thought on:


(a)

a useful methodol
ogical framework for identifying the benefits of government
-
sponsored
energy R&D; and


(b)

practical approaches for developing improvements to current methods of est
i
mating the
benefits of energy resource and science R&D, which might be used to enhance the
per
formance
-
based management of these programs under the Government
Performance and Results Act of 1993 (GPRA).


The contents of this paper should
not

be interpreted as recommendations to any government
agency. Rather, the ideas are intended to stimulate disc
ussion at the conference.


Background and Motivation for the Conference


The motivation for this conference is the priority that the energy resources and science offices in
the U.S. Department of Energy (DOE) are giving to measuring and assessing the perf
ormance
of their R&D programs. Under GPRA, DOE and other federal agencies are required to report
annually on their programs' plans and performance. Furthermore, President George W. Bush
and the U.S. Congress are placing great emphasis on performance
-
based
assessments in the
funding of federal programs.


Estimates of program benefits are one part of the performance requirements established by
GPRA and implemented throughout the federal government. GPRA addresses both expected
performance and retrospective ev
aluation at all levels of government activity


project, program,
office, agency

and in all time frames

monthly and quarterly execution, annual and multi
-
year
milestones, final outputs, and resulting market impacts or benefits. This conference focuses on
w
ays of improving our ability to estimate the ultimate benefits that result from the outputs of
energy R&D programs. The conference does not directly address other GPRA requirements.


Estimating the benefits of R&D programs is challenging because of the ver
y long time frames
required for basic and applied research, the process of introducing the resulting product into the
market, and the lifetime of investments made in the product. In some cases, over half a decade
might be required to examine the full impa
cts of a line of research. In addition, the results of a
research portfolio are not simply the sums of the results of its individual parts. Because benefits
are often the result of multiple areas of effort, this conference focuses on benefits at the
progra
m level and above, rather than on performance related to individual research projects.


The level of benefits depends both on the success of the research effort itself (outputs) and on
changes in the various “external factors”


market or policy condition
s


that affect the market
DRAFT


v


penetration and market shares of these new technologies. Another challenge is the
implementation of a methodological framework in a sufficiently transparent way that the relative
contribution of the R&D, and of the external assu
mptions, on projected benefits is clear, and that
the different reasons why projected benefits are (or are not) realized can be documented.


A government agency's research portfolio can affect literally thousands of products throughout
the economy over sev
eral decades. It is essential to carefully consider the value of this
research, both retrospectively and prospectively. It is also important to undertake such studies
of the benefits of R&D
--

for the purposes of GPRA and program planning
--

in ways that a
re
practical and cost
-
effective.


Benefits Framework Developed by National Research Council Study


There are many ways of addressing the stated objectives of the conference. However, the
recent study done by the National Research Council's (NRC) Committee

on Benefits of DOE
R&D on Energy Efficiency and Fossil Energy provides an important context for this conference.
1

The NRC study developed and implemented a methodological framework to estimate the
retrospective benefits of individual energy efficiency and

fossil energy R&D programs in DOE.


Thus, the framework that we suggest as a starting point for discussions at the conference is
adapted

from the one developed in the NRC study. The NRC committee used a matrix with
three rows and three columns to represe
nt the benefits framework it developed. Three basic
types of impacts are listed along one dimension of the matrix (i.e., the rows): economic,
e
n
vironmental, and security. The other dimension of the matrix (i.e., the columns) reflects the
degree of commerci
alization, certainty, and specific use of the benefits. Along this dimension,
the benefits are classified as being realized, options, or knowledge. Each of these categories of
benefits is described below.




Economic Benefits: Measured by the change in the
market value of goods and services
that are produced under “normal” economic conditions resulting from the introduction of a
technology stemming from DOE research. The benefit is measured net of all public and private
costs, and can typically be reflected
in changes in the level of goods and services produced or
in their market prices. The NRC considered the program costs
--

the costs borne by DOE and
private industry in conducting the R&D
--

as well as any incremental costs of the technology
borne by the e
nd
-
user or consumer.




Environmental Benefits: Based on changes in the quality of the environment because of
DOE research. The benefit is typically not measured directly by changes in market prices, but
rather by some measure of the value individuals in soc
iety are willing to place on changes in the
quality of the environment and improved public health. In cases where the research reduces the
costs of complying with environmental regulations, the benefit takes the form of lower
compliance costs rather than i
mproved environmental conditions.




Security Benefits: Measured by changes in the probability or severity of abnormal energy
-
related events that would adversely impact the overall economy, public safety, or the



1

National Research Council’s Committee on Benefits of DOE R&D on Energy Efficiency and Fossil
Energy,
Energy Research at DOE: Was It Worth It?
, Washington, DC: National Academy Press, July
2001. The report was requested by the Appropri
ations Committee of the U.S. House of Representatives.


DRAFT


vi


environment. Traditionally, this was primarily

a concern about volatile oil markets. Recently,
there is increased concern about the reliability and security of the energy supply infrastructure.


The NRC recognized that R&D might lead to benefits even when a technology developed by
that R&D does not en
ter the market immediately or to a significant degree at all. This lack of
commercial use might be due to changes in the forecasted economic or policy conditions, or to
technical shortcomings. To account for this uncertainty, and to reflect the different d
egrees of
technology development in a retrospective assessment, the NRC established three categories:




Realized Benefits: These economic, environmental or security benefits have been realized
by the current time, or are almost certain to be realized in the

near future, as a result of the
research effort leading to the development of commercially
-
attractive technologies. Often, these
results are an acceleration of technologies that would have otherwise been commercially
available at a later date. In other ca
ses, they might represent new technologies or changes in
product attributes that the private sector would not have produced on its own. The NRC
included all lifecycle benefits of the units of the evaluated technologies that were projected to be
installed b
y the year 2005.




Option Benefits: This category covered technologies that are fully developed but for which
existing economic or policy conditions are not likely to be favorable for commercialization. To
be considered an option by the NRC committee, the
technologies needed to be favorable for
commercialization under some credible or plausible circumstances. For example, carbon
sequestration technologies are options whose value might be realized if new regulations or
trading of carbon emission permits take

effect.




Knowledge Benefits: R&D, whether successful or not, typically produces knowledge. The
generation of scientific knowledge is a key part of DOE’s mission. The NRC considered
knowledge benefits to be scientific knowledge and useful technological co
ncepts that have not
yet been incorporated into commercialized results from the R&D program, but that hold promise
for future use or that are useful in unintended applications. These benefits are in addition to
those accounted for in the other areas of rea
lized and option benefits. Knowledge benefits tend
to be the earliest benefits from the R&D process.


Benefits Framework Adapted from the NRC Study


As we turn from the
retrospective

context of the NRC study, to the
prospective

one needed for
program plann
ing and GRPA, we suggest that the NRC benefits framework could be adapted or
expanded in the following ways:




The concept of realized benefits in the retrospective context could be augmented with a
projection of
expected prospective benefits

under the most

likely scenario. This change
would effectively add one column to the benefits matrix to represent these expected
prospective benefits.




The definition of the baseline condition, against which a new technology or new science is
"added" as a result of R&D,
could be expanded from a strictly retrospective context to
include the prospective situation.




The impact of a government R&D program, on technologies developed in part by the private
sector, could be estimated prospectively.

DRAFT


vii





The definition of options cou
ld be expanded to include not only the retrospective definition
(that options are technologies that are fully developed but for which existing economic or
policy conditions are not likely to be favorable for commercialization), but also the
prospective con
text of R&D investments under uncertainty.




Retrospective indicators of the knowledge generated by past R&D could somehow be
expanded to consider the potential knowledge from ongoing and planned science projects.




Methods for estimating some of the securit
y, economic, and environmental impacts of R&D
programs, which the NRC committee did not have an opportunity to develop, could be re
-
analyzed, especially in light of the terrorist events of "September 11."




Data and data management requirements for prospect
ive analyses could be assessed,
including whether the requisite data are available and affordable. Also, consideration could
be given to whether the methods can be implemented without having to rely on unverifiable
assumptions, which would limit the credib
ility of the resulting estimates.




Within a GPRA and program management context, greater attention could be given to
developing methods for program planning and decision making that
use

the estimates of the
benefits of R&D programs, a point made by the NRC

study as well.


This white paper expands on each of points above to provide a basis for discussions at the
conference. Each point is discussed further in the rest of this summary.


Expanded Benefits Framework to Accommodate the Need for Prospective
Analys
is


Pursuant to its charge, the NRC study developed its framework to assess retrospective benefits
on a case
-
study basis. In order to adapt the framework to GPRA purposes, we suggest a
framework that builds on the NRC's 3 x 3 matrix. GPRA requires that per
formance statements
be tracked from pre
-
program planning through post
-
program evaluation. As a result, the benefits
framework is extended to address
prospective

benefits in a way that allows the benefit
estimates to be tracked over time. This requires the
addition of a fourth column in the matrix.
The need to undertake prospective analysis also requires consideration of the ways in which
benefits develop as R&D matures, and considerations of the differences between retrospective
and prospective information.

In the following figure, we have modified the NRC matrix by adding
the "prospective" column. It was not part of the NRC framework because that study considered
only re
t
rospective benefits.



Modification of National Research Council Committee's Framework

For Identifying Benefits of Energy R&D



Realized

Retrospective

Expected
Prospective

Option

Knowledge

Economic





Environmental





Security







DRAFT


viii


Once a framework is established, practical means are required for estimating the benefits
represented by

each cell. Populating each of these twelve cells with estimates is a daunting
task, especially because many of the cells represent cutting
-
edge areas of valuation theory, are
areas of controversy in the literature, or require data that are either unavaila
ble or extremely
costly to collect and maintain. We suggest a framework in which benefits can be described in
ways as closely related to the research efforts as possible (e.g., physical units of energy
produced or saved), with augmentation, where possible,

with transparent valuation into dollars.
We also recognize that benefits that are not directly measurable even in physical units must
sometimes be represented by proxy or index information.


Expected Prospective Benefits


We suggest that the expected pro
spective benefits could be defined as those expected from
future deployment of a technology developed as a result of R&D, under the projected baseline
set of future market and policy conditions, compared to the expected conditions under the same
projected
baseline, without the technology. Thus, estimation of expected prospective benefits
requires a baseline characterization of future energy markets without the government research,
and an estimation of how baseline markets will react to the new or accelerate
d technology,
including its expected market penetration. These expected prospective benefits could be
economic, environmental or security
-
enhancing; and they could be calculated over some pre
-
determined time horizon, such as the next twenty years. Longer t
ime frames might be required
to capture the full benefits of basic research or technologies requiring fundamental changes in
energy infrastructure.


This definition of expected prospective benefits is somewhat analogous to that used by the NRC
committee fo
r retrospective benefits in that the benefits of the technology are net of those of the
next
-
best alternative. In the prospective case, the next
-
best alternative is reflected in the
reference baseline, which includes technical improvement. The question of
defining the
projected baseline is addressed in Section 6.1.2.


The projected market penetration of a new technology is a key parameter that greatly affects the
magnitude of its expected prospective benefits. In this regard, we suggest that it could be
pre
ferable to characterize the attributes of the technology and to use a model to project how
well the technology will do in the market, than to simply assume a certain level of market
penetration after some number of years.


We further suggest that the model

used for this purpose be calibrated, to extent possible, to the
same one used for the projected baseline. Then, the Reference Case run of that model would
be the projected baseline,
without

the technology in question. A run of the same model
augmented wit
h a simple representation of the technical performance and supply (curve) of the
technology in question, could be the case
with

the R&D program. Where a different model is
better suited to analyzing the specific technology than the one used for the baselin
e, then the
other model can be calibrated to the baseline. The difference between the economic,
environmental and security conditions in these two runs of the model would be the expected
prospective benefits of the technology.


We suggest that the results
of these calculations, which provide projections of annual net
benefits, could be reported in both of the following ways:




Total net benefits, in five
-
year intervals over the next twenty years (for example), in real
dollars, with no time discounting,
and

DRAFT


ix





Net present value, in which the estimated future benefits and costs over the operating life of
equipment deployed are discounted.


Projected Baseline


Defining a baseline is an exercise in identifying the next
-
best alternative. Expected
improvements in te
chnology, that can occur regardless of whether a DOE program exists, must
be accounted for.


For the prospective case, we suggest that the "reference case" of a DOE forecasting model
such as the National Energy Modeling System (NEMS) could be considered fo
r defining the
projected baseline. If this model is inadequate for this purpose, then we suggest the use of
alternative models that have better descriptions of the technology or markets under
consideration. Alternative models that could be used include MAR
KAL and consumer choice
models that identify the marginal reference
-
case choices of technologies (i.e., the next
-
best
alternative). If such models are used, then we suggest that they could be calibrated to the more
important assumptions in the NEMS referen
ce case to the extent possible.


In the retrospective case, the issue is one of defining the counterfactual past condition. In either
case, engineering judgement might be required to "back out" the technology in question, which
might implicitly already be
included in the reference case assumptions.



Impact of the Government


The 5
-
Year Rule adopted by the NRC study as a "conservative" means of valuing benefits
(NRC, p. 18) assumes that the private sector would lag DOE by five years, with the same level
of
R&D and the same outcomes, if DOE had not itself invested. We suggest that this rule could
be generalized and that the impact of a government R&D program could be represented as a
change in the timing of the benefits of the technology developed from that p
rogram. We further
suggest that the specific time lag could be determined on a program by program basis. Just as,
for example, supply curves that describe the quantity of energy services available at different
prices are different for different technologie
s, government R&D programs will reduce the time it
takes for technologies to reach market by different amounts of time. For prospective estimates
of benefits, the technology improvements in the reference case implicitly define the time frame
for commercial
ization without the government R&D program.


For technologies that are developed to the extent that they are fairly close to commercialization,
such many of those funded in the past by the National Institute of Standards and Technology's
(NIST's) Advanced
Technology Program (ATP), the median lag was found to be about 3 years.
The lag ranges from about 1 year to an infinite number of years (i.e., the firm would never have
carried out the R&D without support from ATP).


The magnitude of the lag will affect th
e magnitude of the calculated benefits associated with
public R&D programs, and it would be informative to compare the relative values of potential
R&D efforts when different rates of accelerated commercialization are assumed. Thus, we
suggest that lags co
uld be assessed on a program
-
by
-
program basis. If the technologies under
development are on the verge of commercialization, then we suggest that a three
-

to five
-
year
lag could be appropriate. For new programs and R&D initiatives in their early stages, the

lag
DRAFT


x


might be ten years or more. Many programs would have infinite lags, i.e., the private sector
would never undertake the R&D.


Note that whatever lag is used, it represents the impact or contribution of the government
program in prospective estimates of

benefits. Then there would be no need to reduce the
benefits beyond this time
-
lag adjustment. Alternative approaches are discussed in the paper.


Option Benefits


The state of the future projected in the "reference case" is (in theory) the most likely, bu
t
certainly not the only possible future. We suggest expanding the NRC study's concept of option
technologies, which was defined in a retrospective context, to include the prospective situation.



In general, options provide discretionary choices to deal w
ith deviations from planned scenarios.
We suggest that a real options approach could be used for plausible energy scenarios other
than the reference case scenario used for estimating the expected prospective benefits.


In a
prospective

sense, there could b
e option value to R&D on technologies that are not being
developed
primarily

to enter the market under the most likely conditions, but that would provide
economically viable solutions under alternative plausible conditions. These plausible futures are
gene
rally of energy, environmental, security or other policy concern. Physical or intellectual
assets (not necessary fully developed technologies as defined in the NRC study) that might be
deployed in the future (not just under improbable conditions) may also
contain significant option
value. In this sense, options provide insurance in the face of market uncertainties, yet retain the
ability to capture the upside benefits should improbably scenarios be realized.


In a
retrospective

sense, technologies that are

already developed, but that are unlikely to be
commercialized under current or anticipated market conditions, may yet contain option value.
This value is derived from the uncertainty surrounding future market conditions. These
technologies might have bee
n d
e
veloped to enter the market but did not do so because of
changes in conditions
--

the technologies remain available for the market in the future should
conditions change to make them commercially viable. For example, carbon sequestration
technologies
could be options whose value might be realized if new regulations or trading of
carbon emission permits take effect.


Often, a research line is pursued because it has limited, but important, expected benefits
through niche market applications, but would be

expected to have much wider market
applications under some alternative futures, e.g., an oil disruption. While option values can
persist after the research is completed (often referred to as backstop or shelf technologies), the
value of the option will ch
ange as we move further into the future and more certain about
whether the technology application is economically viable.


R&D can be considered as investments in options that provide opportunities to realize benefits,
in the event of alternative future ev
ents. In addition to the option to commercialize, R&D
investments contain a host of other investment timing options. At each stage, as new
information becomes available about the probabilities of different outcomes, choices can be
made regarding continua
tion of the research, abandoning the research, mothballing the
research, etc.

DRAFT


xi



To calculate option values, input data are typically required for:




Amount of the R&D investment, for which the corresponding value of the option is to be
calculated, as well a
s some estimate of the technology outcomes that result from the R&D;




Degree of uncertainty about the costs to provide energy services using the technology in
question, as a function of R&D;




Degree of uncertainty about the costs to provide energy services

using competing
technologies, i.e., degree of uncertainty of the projected baseline prices; and




Degree of technical uncertainty (risk) about the cost of bringing the technology to market.


Developing these input data is a real challenge, especially when
done for the first time. The
uncertainties can be expressed "continuously" using probability density functions, or "discretely"
using a few scenarios. Much of the real options literature uses continuous representations of
uncertainty over the range of rele
vant future market factors. On the other hand, scenario
analysis is frequently more intuitively appealing. It is a well
-
established approach to planning
that allows individuals to assign their own views about the probabilities associated with various
scena
rios.


Knowledge Benefits


We suggest that bibliometric methods could be used to develop indicators of the value of the
knowledge associated with research programs. For retrospective analysis, we suggest that
indicators could be developed that reflect the
linkages between science and innovation, such
as:




Current Impact Index. This index considers how frequently other patents are cited in
patents granted in a current year.




Science linkage. This method counts the number of times a patent cites scientific
papers or
similar research publications. This measure indicates how strongly a patent has relied on
fundamental scientific research.




Technology
-
cycle time. This index is the median age of patents cited on the front page of a
U.S. patent document. This me
asure indicates the speed of innovation in a company or
industry.


Similar analysis could be done with R&D 100 Awards.


For prospective analysis, research might first be done on whether results from retrospective
studies could be used to develop simple mod
els that describe the linkages between the outputs
of science programs and their ultimate outcomes. Then, statistical relationships might be
developed for these linkages, which might in turn be used for prospective assessments.


We suggest that peer
-
review

methods could be adapted and used as the primary method for
assessing the prospective benefits of scientific programs. An example might be the use of
Delphi processes to develop assessments of the prospective knowledge
-
related benefits of
DRAFT


xii


various science
programs. GPRA
-
like projections are generally much different from typical
technical peer reviews, though, so that peer reviews for the purpose of GPRA
-
related
requirements would need to be crafted to address these needs.


Security Benefits


We suggest that

security benefits related to oil use and imports, the vulnerability of energy
infrastructure and systems, and the reliability of electric power supply could be included among
the possible benefits of energy R&D. This is a broader definition than what the
NRC study
addressed. In particular, we suggest that the definition of security benefits could be broadened
to include those that reduce oil dependence and its costs, not only those associated with
abnormal events; and those that improve electric power reli
ability, including that under "normal"
operations.


Oil Security
: Oil security measures are well developed in the literature. They can be expressed
in barrels of reduced oil consumption or reductions in oil intensity (oil used per dollar of GDP).
The energ
y security literature has also estimated values in dollars per barrel reduction in oil
consumption.


Infrastructure and Systems:

We suggest that benefits of energy infrastructure and system
security could be estimated using probabilistic risk analyses to e
stimate the impacts of the
technologies on the following:


(a)


Reduced cost of outage or disruption to users,


(b)

Reduced cost of providing a given, desirable level of security (e.g., cost of security
personnel and equipment),


(c)

Reduced impact on an industry

and on the economy region
-

or nationwide (e.g.,
estimated using economic input
-
output models).


Electric Power Reliability
: We suggest that the benefits of electric power reliability could be
valued by taking into account:


(a)

"Real
-
time energy value"


rep
lacing the measures of energy savings and their financial
value (Btu's and Btu’s times

average

price, respectively), which are currently used, with
a measure of financial savings that accounts for energy savings (and increases) and the
different

electricit
y rates at different times of the day and year;


(b)

“Reduced outage costs”


the financial value of reliability as measured by the estimated
reduction in customers’ outage costs; and


(c)

Reduced costs of providing required level of reliability, including reduced

costs of
providing alternative means of reliability.


Economic Benefits


We suggest that, in addition to the economic benefits associated with energy savings, other
economic benefits could be included:


DRAFT




xiii

(a)

Consumer benefits other than energy savings, such as

increased productivity, which
could be estimated using engineering analysis, survey, and experimental economic
methods; and


(b)

Macroeconomic benefits that will result from spillover effects, and which could be
estimated using econometric methods.


The NRC s
tudy did not address these benefits in the same detail as it did the economic benefits
of energy savings.


Environmental Benefits


We suggest that environmental benefits could be reported based on physical units of reduction,
and valued using $/ton values
for criteria pollutants and for carbon dioxide, as developed in the
extensive literature on these subjects. This literature advocates using estimates of the damages
from the pollutants as measures of their (deleterious) value, as opposed to the costs of
co
ntrolling these emissions. Sensitivity analyses could be done to account for the significant
range of estimates of these values, particularly since the values are sensitive to the size and
geographic distribution of the population exposed to pollutants.


W
e also suggest that a separate index could be used for ecosystem impacts. This index could
be an estimate of the value of increased "services" provided by ecological systems. Values
expressed in $/hectares per year might be used for open
-
ocean, coastal, fo
rest,
grass/rangeland, wetland, lakes/rivers, desert, tundra, and ice/rock biomes. Technology impacts
might then be subjectively evaluated, and expressed in terms of this measure.


Data Considerations


Regardless of which framework and methods are identifi
ed, they are useful only if they can be
implemented using publicly available and affordable data. To the extent possible, data should
be compiled from field
-
verified results. For retrospective analysis, statistical studies can provide
useful information on

technical performance.


We suggest that where subjective estimates of key input parameters are required, that they
could be periodically reviewed through a systematic process with independent experts.


We further suggest that it could be unnecessary, and
sometimes it cannot be justified, to
develop quantitative estimates of all of the benefits. Sometimes, the quality and imprecision of
input data might not justify detailed numerical results
--

order of magnitude estimates might very
well be better in these

cases. In other cases, the very nature of the benefits in terms of their
social or ecological impacts might inherently preclude their being quantitatively estimated. In
such situations, these benefits might be just as, or even more, important than other t
ypes of
benefits that can be quantified. If this were the case, then GPRA reporting and program
planning would have to somehow take this into account.


Portfolios of Programs


We suggest that R&D programs could be evaluated in the context of the overall po
rtfolio of
options they provide, not simply in terms of their expected individual benefits. We suggest that
financial portfolio methods could be used to help construct R&D portfolios among different types
DRAFT




xiv

of R&D that have different timeframes, different ri
sks, different
types
of risk, and different types
of possible benefits.


Some R&D programs would be expected to achieve economic benefits; others would be
targeted more to achieving environmental benefits; still others would have security benefits.
Likewis
e, some programs should likely focus on attaining expected prospective benefits; others
should focus more on attaining option value, given the great uncertainty in all R&D; and still
other programs, especially the science programs, will have as their prima
ry mission the
development of new knowledge.


Yet, in all types of programs, "losers" should be expected in these portfolios and decision rules
could be developed to help decide which programs to continue and which to terminate.




DRAFT




1


IDEAS ON A FRAMEWORK A
ND METHODS FOR
ESTIMATING THE BENEFITS OF GOVERNMENT
-
SPONSORED ENERGY R&D



1.

PURPOSE


This white paper provides a common basis and point of departure for discussions at the
conference on "Estimating the Benefits of Government
-
Sponsored Energy R&D." The i
deas
presented as possible suggestions should not be interpreted as recommendations or preferred
approaches for any government agency. Rather, the ideas are offered as a starting point for
discussions.


The purpose of this conference is to synthesize insig
hts and information from confe
r
ence
participants, which the energy resource and science offices of the U.S. Depar
t
ment of Energy
(DOE) might use to improve their methods for estimating the benefits of their R&D programs.
2


The motivation for this conferenc
e is the priority that these DOE offices are giving to measuring
and assessing the performance of their R&D programs. Under the Government Performance
and Results Act of 1993 (GPRA), DOE and other federal agencies are required to report
annually on their p
rograms' plans and performance. Furthermore, President George W. Bush
and the U.S. Congress are placing great emphasis on performance
-
based assessments and
funding of federal programs.


The specific objectives of this conference are to identify major strea
ms of thought on:


(a)

a useful methodological framework for identifying the benefits of government
-
sponsored
energy R&D; and


(b)

practical approaches for developing improvements to current methods of est
i
mating the
benefits of energy resource and science R&D, wh
ich might be used to enhance the
performance
-
based management of these programs under GPRA.


The next section of this paper discusses more the motivation for the conference. Section 3
summarizes methods which some of the energy resources and science office
s in DOE use to
estimate or assess the benefits of their R&D programs. Section 4 presents methods used by
some of the other, non
-
DOE departments in the Federal government. Section 5 summarizes the
framework and concepts that were developed in an important
study done by the National
Research Council (NRC).
3

Section 6 offers suggestions that build on the NRC framework to
address issues beyond the scope of that study, but that are important for assessing



2

Conference facilitators and rapporteurs will identify convergence, as well as divergence, of ideas on
methods that might be useful to these DOE offices. The purpose of the conference and of the subs
equent
draft report, to be written following the conference, is to identify and to suggest methods to these DOE
offices, but not to provide recommendations or to act in any advisory capacity.


3

National Research Council’s Committee on Benefits of DOE R&D
on Energy Efficiency and Fossil
Energy,
Energy Research at DOE: Was It Worth It?
, Washington, DC: National Academy Press, July
2001. The report was requested by the Appropriations Committee of the U.S. House of Representatives.

DRAFT




2

prospectively the benefits of R&D and for enhancing perf
ormance
-
based management of these
programs. Section 7 discusses important issues related to implementation of such methods.
Section 8 describes the objectives and agenda of each session in the conference.


The NRC framework and the suggestions that build o
n it are certainly not the only useful set of
methods. However, given the context for this conference and the progress made by the NRC
committee in its study, we have decided to "pick up where they left off."



2.

CONTEXT AND MOTIVATION FOR THE CONFERENCE


An important part of the U.S. Department of Energy's (DOE’s) mission is to foster a s
e
cure and
reliable energy system that is environmentally and economically sustainable. Several offices in
DOE play key roles in carrying out this mission. The Office of

Energy Efficiency and Renewable
Energy (EERE) conducts research to develop and deploy clean and efficient energy
technologies. The Office of Fossil Fuel (FE) undertakes r
e
search and development to promote
the efficient and environmentally sound productio
n and use of fossil fuels. The Office of Nuclear
Energy, Science and Technology (NE) advances the application of nuclear technology by
investing in new or innovative o
p
portunities for its expanded use. The Office of Science (SC)
complements the “energy r
esources business line” of these other offices by advancing basic
research and the foundations of science. All of these offices assess the benefits of their
programs to a
s
sist their planning and budget development, to meet the requirements of the
Gover
n
me
nt Performance and Results Act (GPRA), and to respond to requests for information
from the White House, Congress, and others.


These estimates are important means of assessing both the potential future benefits from public
R&D, and the performance and resu
lts of past research efforts. Under the Government
Performance and Results Act of 1993 (GPRA), federal agencies are r
e
quired to report annually
on their plans and performance. In his “management agenda,” President George W. Bush
emphasized that the feder
al government needs to measure the effectiveness of its R&D
investments and he chose "energy resources" as the first area to apply new R&D selection
criteria, including their contributions to public benefits.
4

The National Energy Policy identifies a
numbe
r of such potential benefits, including e
n
ergy security and environmental improvements.
5


Improved methods of estimating the value of energy R&D can increase the effectiv
e
ness of
future investments in it. A recent National Research Council (NRC) study, ci
ted in Section 1,
d
e
veloped an initial framework for evaluating the benefits of DOE’s past energy efficiency and
fossil energy R&D programs. The study's implementation of this framework i
n
cluded
conventional methods used in programmatic and economic analys
is, as well as cutting
-
edge
methods in need of further assessment and development.


This paper adapts the framework that the NRC study developed to define a starting point for
discussions at the conference.





4

Executive Office of the P
resident, Office of Management and Budget,
The President’s Ma
n
agement
Agenda, Fiscal Year 2002
, August 2001.


5

National Energy Policy Development Group,

National Energy Policy
, Washington, DC, May 2001.

DRAFT




3


3.

METHODS USED BY DOE ENERGY RESOURCES AND SC
IENCE
OFFICES TO ESTIMATE OR ASSESS THE BENEFITS OF THEIR R&D
PROGRAMS


3.1 OVERVIEW OF METH
ODS USED BY THE OFFI
CE OF ENERGY
EFFICIENCY AND RENEW
ABLE ENERGY FOR GPRA
6


Each year, the Office of Energy Efficiency and Renewable Energy (EERE) estimates
future
benefits of its portfolio of research and deployment programs. Metrics currently
estimated include economic, environmental, and security elements, and are listed in the
table below. The types of benefits estimated are not complete


for example, reliabili
ty
-
related security benefits are not currently addressed.



The following chart, reproduced from EERE’s FY 2003 Congressional Budget request,
is a summary of the results of EERE’s
FY 2003 GPRA Benefits Reports
. It is an
example of the sort of performance dat
a that GPRA measurements can produce for
decision makers to better evaluate the results of funding EERE’s program and activities.







6

This section was written by Mary Beth Zimmerman,
Office of Energy Efficiency and Renewable Energy,
U.S. Department of Energy.

Energy Metrics

Financial Metrics

Environmental Metrics


Total Primary Energy Displaced
(Trillion Btu)


Energy Cost Savings (Millions of
1999 $)


Carbon Emissions Displaced
(MMTC)

Direct Electri
city Displaced
(Billion kwh)

Non
-
Energy Cost Savings
(Millions of 1999 $)

Other Greenhouse Emissions
Displaced (MMTCe)

Direct Natural Gas Displaced
(Billion Cubic Feet)

Consumer Investment (Millions of
1999 $)

CO Displaced (Metric Tons)

Direct Petroleum
Displaced
(Million Barrels)

EERE Expenditures (Millions of
1999 $)

SO2 Displaced (Metric Tons)

Direct Coal Displaced (Million
Short Tons)

Other Government Expenditures
(Millions of 1999 $)

NOx Displaced (Metric Tons)

Direct Biomass Displaced
(Trillion Bt
u)

Private Sector Expenditures
(Millions of 1999 $)

VOCs Displaced (Metric
Tons)

Direct Energy Displaced from
Feedstocks (Trillion Btu)


PM10 Displaced (Metric
Tons)

Direct Energy Displaced from
Wastes (Trillion Btu)


Other Environmental Benefits
(Metric

Tons)

Other Direct Energy Displaced
(Trillion Btu)



DRAFT




4

EERE initiated the effort to develop these metrics prior to GPRA and the methodology
has evolved significantly since the
n. The annual effort is a multi
-
stage process, initiated
with an annual “call” and guidance identifying key budget, energy, economic, and other
assumptions to be used in developing program
-

and sector
-
level inputs. Each program
uses this guidance to devel
op estimates of likely commercialization dates and initial
market penetration rates of the technologies under development. Program assumptions
are reviewed on a rotating basis by AD Little to help ensure consistency and improve
projections. Interactions am
ong technologies developed (e.g., when two technologies
under development compete for the same market segment) are addressed by
integrating technology results across sectors and the economy as a whole.


The reference case for EIA’s most recent Annual Energ
y Outlook (AEO) is used as the
baseline for analysis, although it is modified to “back out” any program results already
included in the AEO. This approach provides forecasted baseline energy prices and
market sizes, as well as incorporating underlying (no
n
-
program) improvements in
energy technology performance. To help ensure that analysis occurs on a marginal,
rather than average basis, EIA's National Energy Modeling System (NEMS) is run at
lower energy demand levels to determine which fuel sources might
be most effected as
efficiency and renewable technologies make their way into the market.





Office of Energy Efficiency and Renewable Energy

EERE Programs Projected Benefits by Sector through the Year 2020


Total Primary Energy Saved
or Produced

(quadrillion BTUs)


Energy Cost Savings

($ billions)


Carbon Reductions

(million metric tons)




2005


2010


2020


2005


2010


2020


2005


2010


2020


Transportation


(equivalent
barrels of oil
saved, mbpd)

0.03
-
0.04

(0.06
-
0.14)

0.5
-
0.7

(0.3
-
0.5)

2.8
-
4.7

(1.5
-
2.5)

0.8
-
3.9

9.4
-
19.8

31.5
-
61.5

0.7
-
2.3

8.9
-
14.4

54.5
-
92.1


Industry

0.5

1.3
-
1.4

3.4
-
4.3

1.8
-
1.9

5.4
-
5.5

16.6
-
18.0

7.9
-
8.4

23.0
-
24.5

54.6
-
82.7


Buildings

0.3

0.9

1.9
-
2.8

2.2

7.1
-
9.3

17.1
-
29.9

4.7
-
5.1

16.5
-
17.0

32.7
-
51.0


Federal

0.02

0.04

0.06

0.1

0.2

0.3

0.3

0.7

1.1

Power

0.3
-
0.7

1.0
-
2.2

2.0
-
4.9

1.6
-
2.1

4.2
-
4.8

10.6
-
1
5.2

6.5
-
28.5

20.4
-
62.5

36.0
-
122.6



DRAFT




5

3.2 MEASURING FUTURE

BENEFITS FOR FOSSIL
ENERGY RESEARCH
AND DEVELOPMENT
7


Most past Fossil Energy (FE) R&D benefits estimation has focused on two broad program
c
ategories:


1.

Fossil Fuel Conversion


conversion of coal and gas to electricity and other fuels, and

2.

Fossil Energy Resources


domestic oil and gas supply.


The nature of these programs and their benefits are very different, which has led to rather
differen
t approaches for calculating their future benefits.


3.2.1 Fossil Fuel Conversion


FE has been estimating future benefits for many years. The benefits resulting from
deployment of advanced fossil generation and environmental technologies (including
CO
2

se
questration) are quantified in terms of dollars saved due to deployment of less
expensive technologies:




Increased GDP due to the use of lower cost technology;



New direct and indirect domestic jobs;



Reduced emissions of SO
x
, NO
x
, and carbon dioxide; and



Do
llars of exports.


These benefits estimates are based on the the Energy Information Agency (EIA) forecast of
electricity demand growth through 2020, and extrapolation of this forecast beyond 2020 to
estimate the demand for new gas
-
fired and coal
-
fired elec
tric generators in the U.S. through
2050. In addition, several studies (by WEFA, Resource Dynamics Corp, Parsons, and others)
are also used which estimate the export of clean coal technologies, the use of advanced
emissions control technologies, and the
associated macroeconomic effects. In this analysis, no
effort is made to parse the credit for the benefits between industry and government.


With respect to estimating the benefits of deploying advanced fossil generating technologies in
the U.S., the metho
dology assumes that all new fossil plants built after 2015 are high
-
efficiency
Vision 21 plants with sequestration of carbon emissions. The existing inventory of fossil plants
is assumed to retire on a business
-
as
-
usual schedule. Generation from new Visi
on 21 plants is
assumed to be 10% less expensive than an equivalent amount of generation from new
conventional fossil plants. Carbon emissions reductions are estimated by comparing total
emissions from the electric generation sector through 2050 with base

case carbon emissions
assuming new conventional gas and coal plants deployed without sequestration.


3.2.2 Fossil Energy Resources


Estimates of benefits of advanced technology for oil and gas production depend heavily on the
nature of the existing resour
ce. Extensive efforts have been undertaken to measure benefits
using economic and engineering models. FE developed the first oil model in 1984 in
conjunction with the National Petroleum Council. Model development included input from
experts from industr
y, universities, government, and private nonprofit organizations. Over time,



7

This section was written by Jay Braitsch, Office of Fossil Energy, U.S. Department of Energy.

DRAFT




6

this model has evolved into an integrated suite of databases and analytical computer models, all
striving to emulate the actual complexities of producing domestic oil. This eng
ineering and
economic model was joined by another model to mimic domestic gas production. The models
contain actual data from the Nation’s oil and gas wells and fields.


Within appropriate bounds set by other databases, like those from EIA, the two engi
neering
based models analyze the amount of oil and gas on a field by field basis to calculate how much
was, is, or will be produced over a specified time interval. This is the oil and gas world
without

FE R&D.


FE R&D includes a portfolio of oil and natura
l gas projects. For each project FE develops key
parameters such as the probability of success, likelihood of the technology penetrating into the
marketplace, and the size of the applicable oil or gas resource; and analyzes the potential
impacts. Results
for individual projects are then fed into the oil and gas models to estimate
impacts
with

FE R&D. The difference between these two simulations provides estimates of the
benefit of FE R&D.


The models use a consistent methodology, and the results are perio
dically peer reviewed and
projects validated. The primary benefit measured is the additional oil and gas resulting from FE
R&D. This approach has also allowed FE to estimate economic benefits and new direct and
indirect jobs, as well as certain environmen
tal benefits, like emission and effluent impacts.


Currently, staff at the National Energy Technology Laboratory are working to integrate separate
models into a single one to permit more accurate and efficient benefit computations.


3.2.3 Future Directi
ons


Today’s standard decision
-
making tool is Net Present Value (NPV ). NPV calculates the value
of a project by predicting future payout, adjusting for the risk, and subtracting the investment
cost. NPV does not account for the ability of government and

industry leaders to react to new
circumstances. FE has begun to explore the use of Real Options to better quantify the value of
its programs. Real Options is an investment valuation tool that accounts for and credits
flexibility. Given a highly unpredicta
ble future, the Real Options approach assesses the value of
spending money now so that in the future there is better information on which to exercise the
option to deploy, discard, or continue to develop a technology.


3.3 OVERVIEW OF METH
ODS USED BY THE O
FFICE OF SCIENCE
8


The challenge of evaluating science programs has been the subject of widespread attention for
many years, including initial recognition within the Government Performance and Results Act,
and more recently by significant focus of the Nati
onal Academy of Science’s Committee on
Science, Engineering, and Public Policy (COSEPUP), as well as the White House Office of
Management and Budget’s present efforts to develop separate evaluation criteria for basic
research. Problems arise because of t
he inherent difficulties in modeling the non
-
linear
innovation and knowledge diffusion process
---

knowledge that is the primary goal of the
research and the principal output of science. Consequently, the Office of Science draws from a



8

This section was authored by Robert Vallario of the Office of Science, U.S. Depa
rtment of Energy.

DRAFT




7

broad array of met
hods, and is researching and developing many more, to evaluate its basic
research programs.


Across the various groups exploring this issue, some elements appear to emerge as common
themes.




Evaluation is more successful around the process and outputs
of science (sometimes
referred to as process
-
outcomes)



Expert review forms the cornerstone for evaluating the process and outputs of science,
and the more useful dimensions of analysis include:

o

Quality of the science

o

Relevance of science

o

Leadership and/or
Performance of the science



Evaluating the outcomes (and estimating the benefits) of science, while a desirable goal,
cannot completely and effectively account for the complexities of knowledge diffusion
and the broad range of interactions and factors that
contribute to innovation.



Techniques that explore the downstream impacts are often incomplete, and provide only
focused insights into the benefits through various indicators.


Listed below in Table 1 are evaluation techniques that are currently employed

within the Office
of Science (SC) to evaluate, either retrospectively, prospectively, or both, the science programs.
Collectively, these techniques address the basic research and the infrastructure stewardship
role of SC, the latter consisting of the va
st array of scientific user facilities and premier tools of
science, the human capital, and the critical support and sponsorship for core science
capabilities that SC contributes to the nation’s public science enterprise.
Table 1. Current Evaluation Methods Within the Office of Science


•Regular peer review for the university grant programs, laboratory re
search programs, and
facilities.

•Cost, schedule, technical scope, and management reviews (“Lehman Reviews”) of
construction projects, i.e., major scientific facilities

•Occasional Advisory Committee reviews of facilities and programs

•Occasional external
assessments of programs, e.g., by JASONS or NRC

•Metrics and customer survey for facilities

•Historical retrospectives and annual highlights

•Tracking of a very (very!) few selected metrics such as prizes/publications/PI leadership

•Discipline/sub discipli
ne international benchmarking

DRAFT




8

Beyond these more traditional too
ls, the Office of Science has been sponsoring research and
investing staff resources in a variety of promising areas intended to help illuminate the pathways
of knowledge diffusion and, ultimately, improve the evaluation of science outcomes. Although
no
individual technique appears a panacea for the aforementioned difficulties, collectively they
seem to offer promise, providing a powerful, complementary perspective to the continued
mainstay of expert review.
















Additionally, the categorical/
topical structure of one item, Case Study Enhancements, further
reveals methodological areas that SC is exploring, both through continued research, and
through pilot studies based on SC’s recently completed “Science Manager’s Resource Guide to
Case Studies
”. Methodologies include:


•Classic, Interview
-

and Records
-
Based Method

•Expert Review Method

•Historical Tracing Method

•Bibliometrics and Citation Methods

•Content Analysis Methods (Co
-
Word, Data Tomography, etc.)

•Sociometric/Social Network Methods

•U
ser/Participant Survey Method

•Benefit
-
Cost (Cost
-
Benefit) Methods

•Statistical/Econometric Methods

•Key Indicator Method



Table 2. Evaluation Methods Under Consideration and/or
Development by the Office of Science


•Mission Mapping

•Outcome Mapping

•Patent, Bibliometric, Data Mining & Visualization Tools

•Foresighting

•Case Study Enhancement
s (Resource Guide)

•Options Theory

•Network Analysis

•Expanded International Benchmarking

•Innovative S&T Metrics

DRAFT




9

4.

METHODS USED BY OTHER FEDERAL AGENCIES TO MEASURE THE
OUTCOME
-
RELATED PERFORMANCE OF THEIR PROGRAMS


4.1 MEASURING THE BE
NEFI
TS OF THE ADVANCED T
ECHNOLOGY
PROGRAM
9


4.1.1 ATP’s Mission and Operations


The ATP partners with industry to accelerate the development of innovative technologies for
broad national economic benefit. The program’s focus is on co
-
funding collaborative, mu
lti
-
disciplinary technologies and enabling technology platforms that appear likely to be
commercialized, with private sector funding, once the high technical risks are reduced. Industry
-
led projects are selected for funding in rigorous competitions on the
basis of technical and
economic merit. Since 1990, ATP has co
-
funded 581 projects, with 1,250 participants and
another 1,200 subcontractors. More than 60 percent of the projects are led by small
businesses. More 160 different universities and more than
20 national laboratories participate.
ATP has committed over $1.8 billion of funding to these projects, and industry has committed
another $1.75 billion. Although ATP competitions are open to all technologies, to date 35
projects, with $30 million in ATP

funding, are directed at solutions to the nation’s energy
challenges.


4.1.2 ATP’s Multi
-
Component Assessment Program


Evaluation has been a central part of ATP operations from the beginning, as a management tool
to provide feedback to project selection
and program operations and to meet requests from
external sources for ATP program results.


The ATP has developed a multi
-
component evaluation strategy to provide measures of progress
and performance matched to the stage of project evolution:




for the sh
ort
-
term, from the time of project selection and over the course of the ATP
-
funding
period;




for the mid
-
term, as commercial applications are pursued, early products reach the market,
and dissemination of knowledge created in the R&D projects occurs; and



f
or the longer
-
term, as more fully
-
developed technologies diffuse across multiple products
and industries, with related net impacts on formation of new industries, job creation, and
U.S. economic growth.


Current approaches to evaluation include:




statist
ical profiling of applicants, projects, participants, and technologies



progress tracking of all projects and participants (through a business reporting system
and other surveys)



status reports of all completed projects, with a performance rating against AT
P’s mission




9

This section was written by Jeanne Powell, Senior Economist, Economic Assessment Office, Advanced
Technology Program.

DRAFT




10



detailed microeconomic case studies of selected projects, including the following
metrics: public rate of return (return attributable to ATP), private and social rates of
return, net present value, and benefit
-
cost ratio



econometric and other s
tatistical studies of innovation, productivity, and portfolio
impacts, including use of control groups



macroeconomic analysis extending firm and industry level impacts to impacts on GDP
and national employment



development and testing of new assessment mode
ls and techniques.


The ATP’s Business Reporting System, a unique internal database created in 1993, tracks
progress during and after the performance of each project. This system first identifies the
project’s goals and expected commercial advantage, its
strategies for commercialization, and
collaborative activities and experiences of its members, with emphasis on comparing what ATP
funding makes possible with what would likely occur in the absence of ATP funding. Each
project is evaluated in terms of the
effect of ATP on the project’s timing, scale, scope, risk level,
ability to do long
-
term R&D, and the ability to attract private investment dollars. After the project
is finished, the ATP’s Economic Assessment Office follows up with regular surveys and st
udies
of progress in commercialization and knowledge dissemination. For select projects and
programs, detailed benefit
-
cost analyses are performed.


4.1.3 ATP’s Evaluation Results


Recently completed mini case studies of ATP’s first 50 completed projects
provide an evaluation
of the performance of each project using ATP’s Composite Performance Rating System. Each
project is scored on a set of measures of knowledge creation and dissemination and progress
toward commercial goals.


ATP’s Composite Performa
nce Rating System
10

Knowledge Creation and Dissemination
Measures

Technical awards

Collaborations

Patent filings

Publications and presentations

New product/process in market or expected
soon

Commercialization Progress Measures


New product/process in market

or expected
soon

Attraction of capital

Employment gains

Business awards

Outlook


The results from all the measures are used to construct a composite performance score to
indicate the overall project effectiveness against ATP’s mission (measured 2
-
3 years

after the
end of ATP funding). The result is a four
-
star system of ratings, with scores ranging from zero
to four stars. The results of this analysis for the first 50 completed ATP projects find that sixteen
percent of the projects are top
-
rated in term
s of overall project performance, with four stars.
Twenty
-
four percent are in the bottom group of zero or one stars. Sixty percent make up the
middle group.


Not all ATP projects are fully successful. The program’s emphasis on funding high
-
risk,
technol
ogy development that the private sector is unwilling and unable to fund alone dictates
that most projects will fail to accomplish all their goals. Some projects are stopped before



10

The scoring system was developed by Rosalie Ruegg, TIA Consulting.

DRAFT




11

completion of the funding period. Others fail to meet all their technical
goals, or encounter
business difficulties before the technologies are commercialized.


Evaluation studies to date suggest ATP is delivering results: 1) Estimated net benefits (including
projections) from just a few projects exceed the total cost of ATP t
o date; 2) a vast proportion of
these benefits extends beyond the organizations who received ATP funding, in keeping with the
program’s emphasis on generating significant spillover effects; and 3) there is substantial
evidence that ATP has made a differenc
e in the ability of the nation to realize these benefits.


The ATP evaluation studies are available on the website at:
www.atp.nist.gov/eao/eao_pubs.htm.



4.2

FEDERAL RAILROAD ADM
INISTRATION’S (FRA’S
) R&D PROJECT
EVALUATION AND INVES
TMENT
ANALYSIS
11


At the urging of the Transportation Research Board (TRB) Committee for Review of the FRA
R&D Program, FRA has developed a structured process to identify safety research areas and
select specific safety R&D projects for funding. The approach cons
ists of five logical steps that
were applied to the FRA’s safety R&D program. Subsequently, as new information becomes
available about sources of hazards, the logical steps may be followed for specific types of safety
hazards to add to the list of potenti
al safety R&D projects.


Step 1: Review of Rail Industry Historical and Potential Harm


The first step in the FRA safety R&D project development and selection process is a review of
recent rail industry harm data and an assessment of causes of potential sa
fety hazards. In this
context, harm is considered to be the aggregate cost of fatalities, injuries and property damage
due to rail accidents. Historical hazard data is compiled in FRA rail accident databases and
accident investigation reports. Potential

for future safety hazards can be understood by
reviewing rail industry operating trends with expert knowledge of how railroad accidents occur.


The four relevant databases which hold historical rail incident data are the FRA's Rail
Accident/Incident Repor
ting System (RAIRS), Highway
-
Rail Grade Crossing Accident/Incident
Database, Railroad Injury and Illness Summary Database, and the Research and Special
Projects Administration's (RSPA's) Hazardous Materials Incident Database. The information in
these data
bases is very detailed in terms of accident causes. However, these databases,
typically, do not address specific contributing causes that result in railroad accidents or
incidents.


Detailed accident reports from the National Transportation Safety Board (
NTSB) and the FRA
are the most important source of information, compiled by experts, about accident
circumstances that contribute to hazards. Since accident databases and accident reports can
only reflect historical accident causes and circumstances, rail
road industry operational trends



11


Chapter 4, Five Year Strategic Plan for Railro
ad Research, Development, and Demonstrations, FRA,
soon to be released. Provided by Robert C. Ricci of the Volpe Center.

DRAFT




12

must also be taken into account. In this way, an understanding of causes of hazards that are
not reflected in the historical databases can be developed.


Step 2: Conduct Failure Analysis


For a given cause of an accident,
or factor contributing to hazard, fault
-
tree logic is applied to
identify specific items to be addressed by countermeasures. These specific items represent
points along the accident chain
-
of
-
events at which the accident, or subsequent harm, or both,
could

have been prevented. Thus, the fault tree approach permits identification of multiple
causes of accidents, enriching the information gleaned from the accident databases, which
typically assign a single, primary accident cause. Countermeasures are propos
ed with the goal
of breaking the accident or hazard chain
-
of
-
events at the identified points.



These countermeasures are advanced with an understanding of current regulatory and industry
practices for the relevant area of rail operations.


Types of possi
ble countermeasures include:


$

New or revised regulations

$

Industry standards and best practices

$

Equipment and infrastructure improvements

$

Enforcement

$

Education.


Step 3: Survey Government and Industry Countermeasures and R&D Requirements


Once specific countermeasures are identified, FRA R&D will review current and potential
industry and government countermeasures to identify and assess areas of technological
opportunity for

R&D. That is, FRA R&D will identify countermeasures that would be enabled by
R&D. For example, a potential operating rule may need research into the train speed regimes
at which a particular type of train control system affords safe operation.


Step 4:
Develop and Rate Individual Projects


For each countermeasure that may be aided by R&D, one or more R&D project summaries are
developed to describe projects that provide information to enable the countermeasures. The
project summaries are structured descr
iptions of projects that will be used to compare and
select projects during R&D program development. Project summaries address expected
outputs and outcomes, project costs and durations, as well as implementation issues for project
results. Based on the

project summaries, projects are then rated according to objective criteria
for expected contribution to safety and likelihood of success. For a given program area, these
project ratings are plotted in two dimensions (likelihood of success versus contribu
tion to safety)
to provide a high
-
level comparison tool for the project selection process.


DRAFT




13

Step 5: Select Projects and Assign to Program Areas


The last step in the FRA safety R&D program development process entails selecting projects for
each program ar
ea based on the two
-
dimensional plots and project summaries. The goal is to
select the best research opportunities available to obtain the best return on investment possible
from the FRA R&D budget. That is, the most highly rated projects, regardless of
program area,
are selected to develop budget request estimates. Once the budget has been finalized, the
projects are revisited, and funding levels and schedules are adjusted appropriately. The FRA
R&D budget request, for each program area, becomes the su
m of the funding required for each
of the selected projects in the program area.


This process, augmented by Congressional directives as well as by requirements to carry out
research in direct support of rulemakings mandated by law, was used by FRA in the
preparation
of its FY 2002 and FY2003 budget requests and its soon to be released
Five
-
Year RD&D Plan
.
The FRA anticipates that it will continue to use it in the future in the development of its R&D
program.



5.

NATIONAL RESEARCH COUNCIL'S METHODOLOGY FO
R DEFINING
THE BENEFITS OF ENERGY R&D
12


There are many ways of addressing the stated objectives of the conference. However, the
recent study done by the National Research Council (NRC) Committee on Benefits of DOE R&D
on Energy Efficiency and Fossil Energy

provides an important context for the conference. In
legislation approving the FY 2000 energy R&D budget for DOE’s energy efficiency and fossil
energy programs, Congress directed the NRC to conduct an evaluation of the benefits that have
accrued to the na
tion since 1978 from these programs. This charge was
a retrospective

look at
the benefits that had been realized to date. The NRC did not consider any prospective benefits
from the programs.


To undertake its task, the NRC developed and implemented a metho
dological framework to
estimate the retrospective benefits of individual energy efficiency and fossil energy R&D
programs. Thus, the framework we suggest as a starting point for discussions at the conference
is adapted from the one developed in the NRC stu
dy. In this section of the paper, we review this
NRC methodology. In Section 6, we turn to a discussion of "strawperson" ideas for building on
and extending NRC framework because of the need to consider R&D programs prospectively,
for the purposes of GPRA
and program planning.



5.1

METHODOLOGICAL FRAME
WORK


To undertake this task, the NRC first had to develop a methodology or framework to assess
what were the benefits realized. The NRC recognized that there are many different types of
benefits. The NRC be
lieved that the benefits considered foremost must meet DOE’s stated



12

James L. Wolf authored Section 5 and Sections 7.4 and 7.5.


DRAFT




14

mission
-

which the NRC culled by reviewing statures and public mission statements issued by
DOE. The NRC viewed the public benefits as:
13





Economic: Measured by the change in the market v
alue of goods and services that are
produced under “normal” economic conditions resulting from the introduction of a technology
stemming from DOE research. The benefit is measured net of all public and private costs, and
can typically be reflected in marke
t prices of goods or services in the economy. The NRC
considered the costs for the program
-

the costs borne by DOE and private industry in
conducting the R&D
-
, as well as the costs of the technology borne by the end
-
user or consumer
so that the net bene
fits were assessed. In essence, an economic benefit is reducing the cost of
energy services.




Environmental: Based on changes in the quality of the environment because of DOE
research. The benefit is typically not measured directly by changes in market p
rices, but rather
by some measure of the value society is willing to place on changes in the quality of the
environment.




Security: Measured by changes in the probability or severity of abnormal energy
-

related
events that would adversely impact the overa
ll economy, public health and safety, or the
environment. Traditionally, this was an oil focused concern, but recently has expanded to
include the reliability and security of the energy supply infrastructure
-

both electric and gas.


The NRC recognized that

R&D might lead to benefits even when a technology developed by
that R&D does not enter the market immediately or to a significant degree. This lack of
commercial deployment might due to changes in the forecasted economic or policy conditions,
or to techno
logical barriers. To account for this uncertainly, and to reflect the different degrees
of technology development in the snapshot in time the NRC was evaluating the benefits, the
NRC established three categories:




Realized Benefits: These benefits have be
en realized or are almost certain to be realized in
the near future
-

the technology is developed and economic and policy conditions are favorable
for commercialization. The NRC included all lifecycle benefits of the units of the evaluated
technologies tha
t were projected to be installed by the year 2005.




Option Benefits: This category covered technologies that are fully developed but for which
existing economic or policy conditions are not likely to be favorable for commercialization. To
be considered an

option by the NRC, the technologies needed to be favorable for
commercialization under some credible or plausible circumstances.




Knowledge Benefits: R&D, whether successful or not, typically produces knowledge
benefits. This scientific knowledge produced

by R&D is a key component of DOE’s mission.
The NRC considered as knowledge benefits scientific knowledge and useful technological
concepts that have not yet been incorporated into commercialized results from the R&D
program but hold promise for future u
se or are useful in unintended applications. These hold
value over and above that accounted for in the other areas of realized and option benefits.





13

Note that these benefits are those from energy technologies. Other R&D
programs and technologies
might have other types of impacts, e.g., public safety, mobility, and reduced (as well as economic,
environmental, and security) congestion in the transportation sector.

DRAFT




15

The following table reflects the interplay of the stages of technology development and economic
and policy
conditions as the NRC conducted its retrospective review of benefits.