Usage of Decision Analysis Methods

dotardhousesΜηχανική

18 Νοε 2013 (πριν από 3 χρόνια και 11 μήνες)

70 εμφανίσεις

Usage of Decision Analysis Methods
Outside of a Classroom Environment
by Aerospace Researchers

Sharon Monica Jones
NASA Langley

Rafael E. Landaeta, C. Ariel Pinto and Resit Unal
Old Dominion University

James T. Luxhøj
Rutgers University


Hampton Roads Area INCOSE Conference on Decision Analysis and Its
Applications to Systems Engineering, Newport News, VA (November 17
-
18, 2009)



National Aeronautics and Space Administration

www.nasa.gov

National Aeronautics and Space Administration

Jones,, et al. (2009)

2

Outline


Background/Definitions


Data Collection Process


Results


Concluding Remarks

National Aeronautics and Space Administration

Problem Definition


Aerospace technology managers need:


To predict future technologies


To assess progress toward R&D goals


Aerospace technology portfolio decisions are difficult because:


Very little time to acquire background data


Uncertainty factors (e.g., politics, global economy, environment, funding)


Prescriptive decision analysis methods


Have been used for financial portfolio assessment


Value for policy related decisions has been questioned


Jones, et al (2009)

3

National Aeronautics and Space Administration

Ralph Keeney’s Suggestions for Making
Better Decision Makers*


Develop concepts, tools and procedures to help decision makers


“My experience is that many people, including well
-
educated people, have a very difficult
time in structuring their decisions. They can get mixed up about the difference between
fundamental concepts such as alternatives and objectives.”


Use real decisions, not just laboratory problems in decision research


“We have learned a great deal from all the laboratory settings where decision experiments
have been conducted. There have also been some research studies of real decision
problems. I feel there is much more to be gained by having more of this type of research.”


Teach people what they can and will learn and use


“…hundreds and thousands of people have had at least a course that included a substantial
part on decision analysis and very few have probably ever conducted a formal decision
analysis. Once we find out what people can and will learn and use, that should constitute
the basis for much of our teaching of decision analysis.”



*Source: Ralph L. Keeney, “Making Better Decision Makers”,
Decision Analysis
, 1:4 (2004)

Jones,, et al. (2009)

4

National Aeronautics and Space Administration

Decision Analysis Usage in Aerospace
Portfolio Development



Aerospace managers have investigated the use of decision analysis methods for portfolio
investment decisions:


Commercial Aviation Safety Team (CAST)


NASA Aviation Safety Program


Future Aviation Safety Team (FAST)


These technology assessments involved:


Resource commitments (e.g., employee time, travel money, software acquisition, training)


Assumption that decision analysis methods would improve aerospace technology
assessment process



Jones,, et al. (2009)

5

National Aeronautics and Space Administration

Technology Assessment



“ A process for measuring the impact of established or new technologies”

*
Hans Mohr, “Technology Assessment in Theory and Practice”,
Society for Philosophy and Technology
, 4:4 (Summer 1999)


Jones, et al (2009)

6

National Aeronautics and Space Administration

Aerospace Technology Assessment

Three different processes for examining impact of a set of technologies


Technology assessment

Technology forecasting

Technology foresight



Aerospace Technology

Assessment

Jones, et al (2009)

7

National Aeronautics and Space Administration

Conceptual Model *

1. Identify Context &
Understand Objectives

2. Identify Alternatives

Aerospace Technology Assessment

Decision
Analysis
Methods

3. Decompose & Model the
Problem

a. Model of Problem Structure

b. Model of Uncertainty

c. Model of Preferences

4. Choose the Best
Alternative

5. Sensitivity
Analysis

6. Implement the
Chosen Alternative

*Adapted from Robert T. Clemen, Making Hard Decisions, 2
nd

Edition (1995)

Jones, et al (2009)

8

National Aeronautics and Space Administration

Purpose of Study

What is known:

Decision analysis methods
in financial portfolio
assessment

Decision experiments in
laboratory settings


Technology assessment in
medical R&D

What is unknown:

Decision analysis methods
for policy related decisions


Decision analysis methods
in real decision problems


Technology assessment in
aerospace

Jones, et al (2009)

9

National Aeronautics and Space Administration

Study Overview


Population was aerospace researchers with experience in one or more of the following:


Aerospace program/project management


Aerospace technology assessment


Aerospace technology selection


Aerospace R&D portfolio development


Methods that were investigated in the study


Decision trees


Influence diagrams


Criteria aggregation methods


Explicit tradeoff approaches


Participants were questioned about their usage of these methods for aerospace technology
assessment



Jones, et al (2009)

10

National Aeronautics and Space Administration

Decision Trees

11

Buy shares of XYZ
stock
Don’t buy XYZ
stock
Value of XYZ stock
increases
Value of XYZ stock
unchanged
Value of XYZ stock
decreases
Earn money
Total amount
of money
unchanged
Lose money
Total amount
of money
unchanged
= decision node

= chance node

= consequence node

Jones, et al (2009)

National Aeronautics and Space Administration

Influence Diagrams

12

Buy XYZ stock?
Change in amount
of money
Price of XYZ
stock at
future point
decision node

chance node

consequence node

Jones, et al (2009)

National Aeronautics and Space Administration

Criteria Aggregation Methods


Methods in which two sets of aggregated indices are developed and used to evaluate the
alternatives in the decision problem


Methods in the category include:


Analytical Hierarchy Process (AHP)


Weighted Sum Model (WSM)


13

Jones, et al (2009)

*E. Triantaphyllou, Multi
-
Criteria Decision Making Methods: A Comparative Study(1995)

Example of Simple Weighted Sum Model *



n



A*
WSM
-
score

=



max




a
ij
,w
j
, for 1=1,2,3,….
n


i



j = 1


where,



A*
WSM
-
score

= the WSM of the best alternative




n

= the total number of criteria

a
ij

= the score of the i
-
th alternative in terms of the j
-
th criterion

w
j

= the weight of importance of t
he j
-
th criterion



National Aeronautics and Space Administration

Explicit Tradeoff Approaches


Decision analysis methods that are based on value functions


Methods in this category include:


Multi
-
Attribute Utility Theory (MAUT)


Simplified Multi
-
Attribute Rating Approach (SMART)


14

Jones, et al (2009)

National Aeronautics and Space Administration

Excluded Decision Analysis Methods


Avoided decision analysis methods that are not popular in U.S.


Real world applications are complex with large amounts of uncertainty


Specific decision analysis methods that were excluded from study:


Outranking methods (e.g., ELECTRE, TOPSIS)


Optimization methods


Analytic network process (ANP)

Jones, et al (2009)

15

National Aeronautics and Space Administration

Data Collection and Analysis Process

Refine List of
Candidate
Survey
Participants

Develop Web
-
Based Survey
Instrument

Develop List
of Pilot
Participants

Conduct Pilot
Survey


Review &
Analyze Pilot
Results

Modify Survey
Instrument


Analyze

Results


Conduct
Survey


Legend



Survey Development



Data Collection



Data Analysis

Jones, et al (2009)

16

National Aeronautics and Space Administration

Web
-
Based Instrument Development


Several web
-
based services examined


Questions developed based on several sources:


Short surveys at professional meetings


Validated research in decision analysis literature


Identities of survey participants remained anonymous

Jones, et al. (2009)

17

National Aeronautics and Space Administration

Pilot Survey


Conducted with subset of population (10 persons)


Think aloud cognitive interviewing techniques used


Solicitation of all thoughts and comments


Manual recording of information during completion of online survey


De
-
identification of subjects in final documentation

18

Jones, et al. (2009)

National Aeronautics and Space Administration

Survey Instrument Modification


Questions were modified, added or eliminated from the survey based on input from:


Pilot survey comments


Data analysis of pilot survey data


Additional comments from other reviews (e.g., ODU IRB)


Number of survey questions reduced from 70 to 65

19

Jones, et al. (2009)

National Aeronautics and Space Administration

Data Collection Overview


Approval to conduct survey was obtained from NASA Langley and ODU Institutional Review
Boards (IRB’s)


E
-
mail invitation was sent to 260 persons


154 total visits to survey website


16 partial responses


99 completes surveys


Out of the 99 completed surveys


76% male, 24% female


Highest degree level was 60% Masters, 21% Bachelors, 18% Doctorate, 1% Associates


72% employed as government civil servants

Jones, et al. (2009)

20

National Aeronautics and Space Administration

Job Functions

Jones, et al. (2009)

21

National Aeronautics and Space Administration

Aerospace Experience

Jones, et al. (2009)

22

National Aeronautics and Space Administration

Decision Trees

23

Jones, et al. (2009)

National Aeronautics and Space Administration

Influence Diagrams

24

Jones, et al. (2009)

National Aeronautics and Space Administration

Criteria Aggregation Methods

25

Jones, et al. (2009)

National Aeronautics and Space Administration

Explicit Tradeoff Approaches

26

Jones, et al. (2009)

National Aeronautics and Space Administration

Jones, et al. (2009)

27

Usage of Decision Analysis Methods
Outside of a Classroom Environment

National Aeronautics and Space Administration

Jones, et al (2009)

28

Categories of Non
-
ATA Usage of DA
Outside of a Classroom Environment

National Aeronautics and Space Administration

Additional Questions

29

Jones, et al. (2009)

National Aeronautics and Space Administration

Additional Questions (cont’d)

30

Jones, et al. (2009)

National Aeronautics and Space Administration

31

Jones, et al. (2009)

Additional Questions (cont’d)

National Aeronautics and Space Administration

Additional Questions (cont’d)

32

Jones, et al. (2009)

National Aeronautics and Space Administration

Additional Questions (cont’d)

33

Jones, et al. (2009)

National Aeronautics and Space Administration

Likelihood of Future Usage of DA Methods

34

Jones, et al. (2009)

National Aeronautics and Space Administration

Concluding Remarks


This is a subset of the total data


There are many additional questions in the study


More formal analysis of the data was conducted using structural equation modeling
techniques to test a set of hypotheses


Survey participants believed that the successful use of decision analysis methods depends on:


Selection criteria in the decision model


Experience of the person that implements the method


Reliability of the input data


Training/education does not guarantee future use of a decision analysis method


35

Jones, et al. (2009)

National Aeronautics and Space Administration

Questions?

36

National Aeronautics and Space Administration

Your Title Here

37

SignOffPage