Life Cycle Alignment Concerns in Acquisition of Software-Intensive Systems

goodyearmiaowMechanics

Nov 18, 2013 (3 years and 11 months ago)

125 views

© The Aerospace Corporation 2010

Life Cycle Alignment Concerns in Acquisition of
Software
-
Intensive Systems

Dr. Peter Hantos and Nancy Kern

The Aerospace Corporation

1

2010 COCOMO International Forum, Los Angeles, California

November 1
-
4, 2010

2

Acknowledgements


This work would not have been possible without the following:


Reviewers


Suellen Eslinger


Zane Faught


Rick Johnson


Dr. Leslie Holloway


Sponsors


Rosalind Lewis


Dr. Malina Hills


Funding Source


The Aerospace Corporation’s Independent Research & Development
Program

3

Outline




Objectives


Introduction


Basic Lifecycle Alignment Concerns


Alignment Concerns for Incremental Development and Evolutionary
Acquisition


… and Where Everything is Really Falling Apart


Conclusions


Acronyms


References


Referenced Standards and Government Resources

4

Objectives


The presentation’s objective is to lay the foundations for the
effective tailoring and use of Life Cycle Process Standards in
Software
-
Intensive System Acquisition


The focus of inquiry is on two interrelated problems:


The prevailing technical standards, i.e., systems engineering life cycle
processes and software life cycle processes, that should be guiding the
contractors’ development efforts are not properly aligned


These technical standards and the DOD 5000.02 Defense Acquisition
Framework are not properly aligned either


Disclaimer


Discussion of hardware concerns and detailed tailoring guidance are out
of scope for this presentation


5

Introduction

6

Basic Process Relationships

C
O
N
T
R
A
C
T
O
R
S
Acquisition
Domain
RFP
Proposal
System
Acquisition/
Engineering
SW HW
Acq
Acq
G
O
V
E
R
N
M
E
N
T
Engineering
Domain
System
Acquisition/
Engineering
SW HW
Acq Acq
Processes
Pre Contract
Post Contract
Processes
Systems
Engineering
Systems
Engineering
SW HW
Eng Eng
SW HW
Eng Eng
Processes
Processes
Contract
Milestone B
Legend: HW



Hardware
SW



Software
RFP



Request for Proposal

Clear delineation of acquisition and engineering processes

7

Basic Expectation Regarding Life Cycle Processes and
Life Cycle Models

Life cycle process standards are harmonized, life cycle models are aligned

Defense
Acquisition
System
(DOD 5000.02 and the Defense
Acquisition
Guidebook)
Systems Engineering
Life Cycle
Process Standard
(Life Cycle Processes
, Activities, and
Tasks)
Hardware
Engineering
Life Cycle
Process Standards
(Life
Cycle Processes, Activities, and
Tasks)
Software Engineering
Life Cycle
Process Standard
(Life
Cycle Processes, Activities, and
Tasks)
8

Why the Emphasis on Life Cycle Process Standards?


Our primary focus is Mission Success [MAG 2007]

-
Mission Success

is defined as the achievement by an acquired system
(or system of systems) to singularly or in combination meet not only
specified performance

requirements but also the expectations of the
users and operators in terms of safety, operability, suitability and
supportability

-
Mission Assurance

is the disciplined application of general systems
engineering, quality, and management principles towards the goal of
achieving Mission Success, and, towards this goal, this disciplined
application provides confidence in its achievement


How can it be ensured that high mission assurance processes are used to
develop the objective system?


Use a robust development standard [Eslinger 2006]


Note that even the use of so
-
called mature processes that are based on such
frameworks as the CMMI
®

is inadequate, and
the government must require
contractual compliance with a robust development standard


The foundation for this conclusion has been derived from the analysis of
Acquisition Reform
-
induced failures on numerous space programs



®

CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon


University

9

Basic Alignment Concerns

10

The Defense Acquisition Life Cycle Model


DOD 5000.02 represents a System Development Life Cycle tailored for the
Defense Acquisition context; life cycle phases are separated by acquisition
milestones (acquisition decision points)


However, it also mandates systems engineering phase
-
gates (reviews)


Red, curved arrows indicate how technical reviews support acquisition milestones


The model prescribes processes for both the Acquisition Program Offices
(APOs) and the contractors


Unfortunately, its structure and language are hardware
-
oriented


Software interpretation is frequently ambiguous



Pre
-
Systems Acquisition
Systems Acquisition
Sustainment
Engineering &
Manufacturing
Development
Operations &
Support
Technology
Development
Materiel
Solution
Analysis
Production &
Deployment
Program Initiation
IOC
FOC
Post
PDR A
Post
CDR A
FRP
Decision
Review
SRR
SFR
PDR
CDR
TRR
SVR
Materiel
Development
Decision
A
B
C
Legend:
Decision Point if PDR is not conducted before Milestone B
Decision Point
Contractor’s Technical Review
Milestone Review
A
DODI 5000.02 (8 December 2008)
Acronyms:
CDR

Critical Design Review
DODI

Department of Defense Instruction
FOC

Final Operational Capability
FRP

Full
-
Rate Production
IOC

Initial Operational Capability
OTRR

Operational Test Readiness Review
PDR

Preliminary Design Review
SFR

System Functional Review
SRR

System Requirements Review
SVR

System Verification Review
TRR

Test Readiness Review
SVR

System Verification Review
Disposal
Low
-
Rate Initial
Production
Approval
Technology
Development
Approval
OTRR
11

Aligning ISO/IEC 15288
-
2008 with DOD 5000.02


Systems Engineering starting point is clear: Milestone A


However, beginning of Development is ambiguous (
More on this later
)


End of Development is clear: OTRR


ISO/IEC 15288
-
2008 does not account for additional transitional effort if
the system will be operated by another entity



* Source: [
Valerdi

2008]

Pre
-
Systems Acquisition
Systems Acquisition
Sustainment
Engineering &
Manufacturing
Development
Operations &
Support
Technology
Development
Materiel
Solution
Analysis
IOC
FOC
SRR
Post
PDR A
Post
CDR A
Materiel
Development
Decision
A
DODI 5000.02 (8 December 2008
)
Disposal
C
Low
-
Rate Initial
Production
Approval
Technology
Development
Approval
SFR
TRR
CDR
FRP
Decision
Review
OT&E
Transition
to
Operation
Operate,
Maintain or
Enhance
Replaor
Disme
Production &
Deployment
Program Initiation
B
PDR
SVR
Competing
Contractors
Pre
-
MS B
OTRR
FRP
Decision
Review
Development
ISO/IEC 15288
-
2008 (With COSYSMO Tailoring)
?
Conceptualization
Where should
System
Development start?
*

12

Critical, Mandated Contractor Activity
Not

Accounted
for in ISO/IEC 15288
-
2008


The DOD is very concerned about the technologies associated with
the acquired systems and conducts technology assessments
beginning at Milestone A


Prior to SFR, all competing contractors must carry out aggressive
technology risk reduction activities and technology maturity
demonstrations via prototyping [Dahmann 2009]


These activities require substantial contractor effort; still, likelihood
of success is highly uncertain

Pre
-
Systems Acquisition
Systems Acquisition
Sustainment
Engineering &
Manufacturing
Development
Operations &
Support
Technology
Development
Materiel
Solution
Analysis
IOC
FOC
SRR
Post
PDR A
Post
CDR A
Materiel
Development
Decision
A
DODI 5000.02 (8 December 2008
)
Disposal
C
Low
-
Rate Initial
Production
Approval
Technology
Development
Approval
SFR
TRR
CDR
FRP
Decision
Review
OT&E
Transition
to
Operation
Operate,
Maintain or
Enhance
Replaor
Disme
Production &
Deployment
Program Initiation
B
PDR
SVR
Competing
Contractors
Pre
-
MS B
OTRR
FRP
Decision
Review
Conceptualization
Development
ISO/IEC 15288
-
2008 (With COSYSMO Tailoring)
Technology Risk
Reduction and
Prototyping
Competing
Contractors
Pre
-
MS B
13

Aligning ISO/IEC 12207
-
2008 with ISO/IEC 15288
-
2008
for Software
-
Intensive Systems


SW support activities required for system conceptualization are missing


System requirements supposed to be allocated and a high
-
level system
architecture should be available
before

SW requirements allocation


SW development should end prior to System Integration & Test

Pre
-
Systems Acquisition
Systems Acquisition
Sustainment
Engineering &
Manufacturing
Development
Operations &
Support
Technology
Development
Materiel
Solution
Analysis
IOC
FOC
SRR
Post
PDR A
Post
CDR A
Materiel
Development
Decision
A
DODI 5000.02 (8 December 2008
)
Disposal
C
Low
-
Rate Initial
Production
Approval
Technology
Development
Approval
SFR
TRR
CDR
FRP
Decision
Review
OT&E
Transition
to
Operation
Operate,
Maintain or
Enhance
Replaor
Disme
Production &
Deployment
Program Initiation
B
PDR
SVR
Competing
Contractors
Pre
-
MS B
OTRR
FRP
Decision
Review
System
Integration
& Test
Software Support
to
Conceptualization
?
In what state should
Software Development
be at PDR?
Conceptualization
Where should
Software
Development start?
Software Operation and Maintenance
In what state should
Software Development
be at CDR?
Software Development
Software
Development must
be completed
ISO/IEC 15288
-
2008 (With COSYSMO Tailoring)
Development
ISO/IEC 12207.02
-
2008
14

Using ISO/IEC 12207
-
2008 for Software
-
Only Systems


The beginning of SW Development is clearly aligned with MS A


Unless the process is Waterfall, there is no designated System
Integration and Test period at the end of Software Development

-
Software is integrated continuously, on a build by build basis

-
However, software must be completely integrated and tested by OTRR


Unfortunately, ISO/IEC 12207 and ISO/IEC 15288 are still inadequately
harmonized


Examples on the next slide

Pre
-
Systems Acquisition
Systems Acquisition
Sustainment
Engineering &
Manufacturing
Development
Operations &
Support
Technology
Development
Materiel
Solution
Analysis
IOC
FOC
SRR
Post
PDR A
Post
CDR A
Materiel
Development
Decision
A
DODI 5000.02 (8 December 2008
)
Disposal
C
Low
-
Rate Initial
Production
Approval
Technology
Development
Approval
SFR
TRR
CDR
FRP
Decision
Review
Operate,
Maintain or
Enhance
Replaor
Disme
Production &
Deployment
Program Initiation
B
PDR
SVR
OTRR
FRP
Decision
Review
Software Operation
and Maintenance
ISO/IEC 12207:2008(E)
Software Development
Competing
Contractors
Pre
-
MS B
OT&E
Transition
to
Operation
Conceptualization
Development
ISO/IEC 15288
-
2008 (With COSYSMO Tailoring)
15

Selected Alignment Concerns for ISO/IEC
12207:2008(E) and ISO/IEC 15288
-
2008*


Implementation


Clause 6.4.4 Implementation Process


6.4.4.1 The purpose of the Implementation Process is to realize a specified
system element


Clause 7.1.1 Software Implementation Process


“NOTE The Software Implementation Process is a conforming instance of
the Implementation Process of ISO/IEC 15288…”


Concern: Implementation in ISO/IEC 15288 should refer only to systems
engineering activities, not activities of all contributing (i.e., software) disciplines


The back
-
end of the Software Development Process


Clause 6.4.8 Software Acceptance Support Process


“NOTE The Software Acceptance Support Process … contributes to the
outcomes of the Transition Process of ISO/IEC 15288. Users may consider
claiming conformance to 15288 rather than the process in this standard.”


Concern: This is a recurring theme, i.e., claiming conformance selectively and
alternatively to the different standards



* Referenced quotes from ISO/IEC 12207:2008(E)

The relationship between the two standards is still ambiguous

16

Alignment Concerns for Incremental Development
and Evolutionary Acquisition

17

Incremental System Development


Actual planning of incremental system development can start after SFR


Planning of increments is a substantial, additional activity


All players must plan but only the winning contractor will actually continue


Every increment goes through all of the mandated technical reviews


By the time the program is to go to the Operation & Support acquisition
phase, the last increment must reach IOC

OTRR
1
CDR
1
OT&E
1
Transition
to
Operation
1
Replaor
Disme
Conceptualization
IOC
1
IOC
2
OT&E
2
Transition
to
Operation
2
Development
2
OTRR
2
CDR
2
PDR
2
Competing
Contractors
Pre
-
MS B
OT&E
n
Transition
to
Operation
n
Development
n
OTRR
n
CDR
n
PDR
n
Operate,
Maintain or
Enhance
n

Operate,
Maintain or
Enhance
1
Operate,
Maintain or
Enhance
2
IOC
n
Pre
-
Systems Acquisition
Systems Acquisition
Sustainment
Engineering &
Manufacturing
Development
Operations &
Support
Technology
Development
Materiel
Solution
Analysis
FOC
SRR
Post
PDR A
Materiel
Development
Decision
A
DODI 5000.02 (8 December 2008
)
Disposal
C
Low
-
Rate Initial
Production
Approval
Technology
Development
Approval
SFR
FRP
Decision
Review
Production &
Deployment
Program Initiation
B
PDR
ISO/IEC 15288
-
2008 (With COSYSMO Tailoring)
Development
Increment
Planning

18

Incremental Software Development


Note that the chart shows Waterfall methodology for increment
development; however, it can be any approach (e.g., iterative)


Increments are successively integrated with each other


Regression testing of software increments is necessary


The final increment is migrated to the System Integration & Test
environment

Pre
-
Systems Acquisition
Systems Acquisition
Sustainment
Engineering &
Manufacturing
Development
Operations &
Support
Technology
Development
Materiel
Solution
Analysis
IOC
FOC
SRR
Post
PDR A
Post
CDR A
Materiel
Development
Decision
A
DODI 5000.02 (8 December 2008
)
Disposal
C
Low
-
Rate Initial
Production
Approval
Technology
Development
Approval
SFR
TRR
CDR
FRP
Decision
Review
OT&E
Transition
to
Operation
Operate,
Maintain or
Enhance
Replaor
Disme
Production &
Deployment
Program Initiation
B
PDR
SVR
Competing
Contractors
Pre
-
MS B
OTRR
FRP
Decision
Review
System
Integration
& Test
Software Support
to
Conceptualization
?
In what state should
Software Development
be at PDR?
Conceptualization
Where should
Software
Development start?
Software Operation and Maintenance
In what state should
Software Development
be at CDR?
Software Development
Software
Development must
be completed
ISO/IEC 15288
-
2008 (With COSYSMO Tailoring)
Development
ISO/IEC 12207.02
-
2008
Software
Requirements
Reassessment
Software
Architecture
Update
Design/Code/Unit Test/
Software Qualification Test
Software Increment
1
Software
Requirements
Reassessment
Software
Architecture
Update
Design/Code/Unit Test
/
Software Integration & Test/
Software Regression Test/
Software
Qualification Test
Software Increment
n
System Architecture,
Allocated Software
Requirements
Software
Increment
Planning
Software
Requirements
Analysis
Software
Architecture
Development
To System
Integration &
Test

Software
Requirements
Reassessment
Software
Architecture
Update
Design/Code/Unit Test
/
Software Integration & Test/
Software Regression Test/
Software
Qualification Test
Software Increment
2
No standard
methodology for
increment
-
level
reviews

19

Evolutionary Acquisition (High Technology Readiness)


The contractor’s technical effort to support the government’s upgrade
decision is not reflected in ISO/IEC 15288
-
2008


Note that this effort is not the same as what “Enhance” refers to

OT&E
Transition
to
Operation
Operate,
Maintain or
Enhance
Replace or
Dismantle
Development
Systems Acquisition
Sustainment
Engineering & Manufacturing
Development
IOC
FOC
Post
PDR A
Post
CDR A
C
DODI 5000.02 (8 December 2008
)
Disposal
Low
-
Rate Initial
Production
Approval
TRR
OTRR
CDR
FRP
Decision
Review
Production & Deployment
PDR
ISO/IEC 15288
-
2008 (With COSYSMO Tailoring)
Pre
-
Systems Acquisition
Systems Acquisition
Sustainment
Engineering & Manufacturing
Development
Operations &
Support
Technology
Development
Materiel
Solution
Analysis
Program Initiation
IOC
FOC
Materiel
Development
Decision
A
B
C
DODI 5000.02 (8 December 2008
)
Disposal
Low
-
Rate Initial
Production
Approval
Technology
Development
Approval
FRP
Decision
Review
Production &
Deployment
Upgrade
Decision
Operations &
Support
Second
Acquisition
Increment
is Initiated
SVR
Potential
Upgrade
Decision
Contractor
support to
upgrade decision

Third
Acquisition
Increment
is Initiated

20

Evolutionary Acquisition (Low Technology Readiness)


Again, the contractor’s technical effort to support the government’s
upgrade decision is not reflected in ISO/IEC 15288
-
2008


As was shown, all Critical Technology Elements (CTEs) must reach
the mandated Technology Readiness Levels (TRLs) by Milestone B


This maturation effort is not reflected in ISO/IEC 15288
-
2008

OT&E
Transition
to
Operation
Operate,
Maintain or
Enhance
Conceptualization
Development
Pre
-
Systems Acquisition
Systems Acquisition
Sustainment
Engineering &
Manufacturing
Development
Technology
Development
Program Initiation
IOC
FOC
SRR
Post
PDR A
Post
CDR A
B
C
DODI 5000.02 (8 December 2008
)
Disposal
Low
-
Rate Initial
Production
Approval
SFR
TRR
CDR
FRP
Decision
Review
Production &
Deployment
PDR
SVR
Technology
Maturation
Pre
-
Systems Acquisition
Systems Acquisition
Sustainment
Engineering & Manufacturing
Development
Operations &
Support
Technology
Development
Materiel
Solution
Analysis
Program Initiation
IOC
FOC
Materiel
Development
Decision
A
B
C
DODI 5000.02 (8 December 2008
)
Disposal
Low
-
Rate Initial
Production
Approval
Technology
Development
Approval
FRP
Decision
Review
Production &
Deployment
Upgrade
Decision
Operations &
Support
Second
Acquisition
Increment
is Initiated
OTRR
ISO/IEC 15288
-
2008 (With COSYSMO Tailoring)
Contractor
support to
upgrade decision

21

… and Where Everything is Really Falling Apart

22

Systems Engineering Process


This is supposedly the
complete

Systems Engineering Process (SEP)*


However, some sources only view it as the Requirements Development
Process [EIA/IS 632]


Naming games with scope and content
-
related consequences


Systems Engineering Process [INCOSE 2003, MIL
-
STD
-
499B] vs. Processes
for Engineering a System (EIA/IS 632)


Due to the iteration loops it is very difficult to establish stage gates


The “Synthesis” part is ill
-
defined


The applicable Life Cycle Process Standards, ISO/IEC 15288,


ISO/IEC TR 19760, or ISO/IEC 12207, do not even mention it



* Source: [INCOSE 2003], [MIL
-
STD
-
499B].

Synthesis
(?)
Process
Output
Design Loop
Requirements Analysis
Process
Input
System
Analysis &
Control
Functional
Analysis/Allocation
Requirements Loop
Verification
Recursive

&

Iterative…

23

Contractor Technical Reviews


In numerous, previous slides it was shown that the positioning of the
system
-
level reviews is ambiguous


Originally they were meant to be phase
-
gates of a sequential process.
However, while the acquisition life cycle is indeed sequential, the underlying
technical processes are not. Consequently, the positioning of all technical
reviews is arbitrary


The content of the system
-
level reviews is ambiguous


The expected content of the reviews is different in different organizations


All references to reviews, e.g., the descriptions in the Defense Acquisition
Guidebook, are modeled after MIL
-
STD
-
1521B; However, that standard is
now obsolete


MIL
-
STD
-
1521B assumed a sequential development life cycle; Since the
development processes are not sequential anymore, review content is ill
-
defined and ambiguous


There is no standard methodology for increment
-
level (build) reviews


Unfortunately, the situation has not really change since 2004


For a detailed analysis of reviews, see [Hantos 2004]



24

Conclusions


Life cycle models and life cycle processes have a fundamental role
in program management


Mission assurance considerations also warrant the use of robust life
cycle process standards on acquisition contracts


However, the use of inadequate or out
-
of
-
synch life cycle processes
represents a major program management risk


To mitigate this risk, a thorough understanding and careful tailoring
of the involved technical standards are needed



25

Acronyms




ANSI

American National Standards Institute

APO

Acquisition Program Office

CDR

Critical Design Review

COCOMO

Constructive Cost Model

COSYSMO

Constructive Systems Engineering

Cost

Model

CTE

Critical Technology Element

DAG

Defense Acquisition Guidebook

DO
D

Department of Defense

FOC

Final Operational Capability

HW

Hardware

IEC

International Electro
-
technical Commission

INCOSE

International Council on Systems Engineering

ISO

International Standards Organization

IOC

Initial Operational Capability

MIL
-
STD

Military Standard

MS

Milestone

OT&E

Operational
Test & Evaluation

PDR

Preliminary Design Review

RFP

Request for Proposal

SEP

Systems Engineering Process

SFR

System Functional Review

SRR

System Requirements Review

SVR

System Verification Review

SW

Software

TR

Technical Report

TRL

Technology
Readiness Level

TRR

Test Readiness Review

26

References

Dahmann 2009


Dahmann, J.S., and Kelley, M.,
Systems Engineering During




the Materiel Solution and Technology Development Phases
,




Washington, DC: Office of the DDR&E/Systems Engineering,




2009, www.acq.osd.mil/sse

Eslinger 2006


Eslinger, S., Mission Assurance
-
Driven Processes for






Software
-
Intensive Ground Systems, The Aerospace






Corporation Technical Report ATR
-
2006(8056)
-
1, September




30, 2006

Hantos 2004


Hantos, P.,
System and Software Reviews Since Acquisition




Reform
, Southern California Software Process Improvement




Network, April 2, 2004

INCOSE 2003


INCOSE Systems Engineering Handbook, INCOSE
-
TP
-
2003
-




016
-
02, Version 2a, June 1, 2004

MAG 2007



Mission Assurance Guide, The Aerospace Corporation





Technical Operating Report, TOR
-
2007(8546)
-
6018 Rev. A ,




July 1, 2007

Valerdi 2008


Valerdi, R., The Constructive Systems Engineering Cost





Model (COSYSMO), VDM Verlag Dr. Mueller, 2008




27

Referenced Standards and Government Resources

Defense Acquisition Guidebook
, Fort Belvoir, VA, Defense Acquisition
University,
<
https://acc.dau.mil/dag>

DOD 5000.02
, Operation of the Defense Acquisition System, December 8, 2008

EIA/IS 632
-
1988
, Processes for Engineering a System

ISO/IEC 12207:2008(E)
, Systems and Software Engineering
-

Software life
cycle processes

ISO/IEC 15288
-
2008(E)
, Systems Engineering


System life cycle processes

ISO/IEC TR 19760:2003
(E), Systems Engineering


A guide for the application
of ISO/IEC 15288 (System life cycle processes)

MIL
-
STD
-
499B
, Draft Military Standard Systems Engineering, September 7,
1993

MIL
-
STD
-
1521B
, Military Standard Technical Reviews and Audits for Systems,
Equipments, and Computer Software, June 4, 1985



28

Use of Trademarks, Service Marks and Trade Names

Use of any trademarks in this material is not intended in
any way to infringe on the rights of the trademark holder.
All trademarks, service marks, and trade names are the
property of their respective owners.


Author of clip art on Slide 22 is Elee Kirk. Used with
permission.