Image Generation - APL Modeling and Simulation Asset Reuse ...

architectgroundhogInternet και Εφαρμογές Web

4 Δεκ 2013 (πριν από 3 χρόνια και 4 μήνες)

59 εμφανίσεις



I/ITSEC Tutorial #1019

The Basics of the M&S VV&A
Process





Simone Youngblood

The Johns Hopkins University Applied Physics Laboratory

Simone.Youngblood@jhuapl.edu

Agenda





Definitions


The VV&A Processes


VV&A as Applied to the Simulation
Categories


The Key Players


Conceptual Model


Documentation


Conclusion


Definitions

What is Verification?


The process of determining that a model or
simulation implementation and its
associated data accurately represents the
developer's conceptual description and
specifications.


Addresses the structural integrity and the
stability of the model


Goal: Did I build the model or simulation
RIGHT
?


What is Validation?


The process of determining the degree to
which a model or simulation and its
associated data are an accurate
representation of the real world from the
perspective of the intended uses of the
model.


Addresses the representational aspects of
the model or simulation


Goal: Did I build the
RIGHT

model or
simulation?



What is Accreditation?


The
official certification that a model
or simulation and its associated data
are acceptable for use for a specific
purpose.


The accreditation decision is based
on an accreditation assessment of the
V&V evidence.




Goal: Are the results of the M&S
FIT

FOR PURPOSE

and
USEFUL

for
making decisions?



The Balancing Act

Risk

Resources

VV&A

Balancing the
cost of knowing

against the
risk of assuming
.


The VV&A Processes

VV&A and the Problem Solving
Process

From the DOD RPG

Tailoring Guidance for VV&A :

Stay Focused on the Problem


Maintain a problem orientation


Understand the relationships between


The problem being solved


The role of M&S in the problem


The role of VV&A in establishing M&S credibility


If the problem is too complex, decompose it into
workable units


Ensure that you are still trying to solve the
same problem


Focus on accreditation


Ensure that the V&V evidence is clearly applicable
to the intended purpose
-

solving ‘this’ problem

Tailoring Guidance for VV&A :

Understand the Requirements


Define the explicit problem requirements


What specific questions are to be answered?


How will M&S be used?


What does the M&S need to do?


How good does the M&S need to be?


Define what question(s) need to be answered...


Assess the completeness, correctness, and consistency
of the M&S requirements


Define the acceptability criteria needed to assess M&S
attributes


Identify the M&S options, the VV&A credibility needs, and
the level of resources necessary to meet defined
requirements

An explicit statement of the requirements is the most important
aspect of cost
-
effective VV&A and the hardest to come by

Tailoring Guidance for VV&A :

Know the Level of Credibility Required


Credibility is Defined by the Customer
-

but it has a cost


Ensure that the User determines the level of credibility
needed


Ensure the level of credibility is explicitly defined by the
Acceptability Criteria


Credibility and Risk Assessment
-

know the limits &
tolerances


How will incorrect simulation results affect the ‘end user’
decision


What are the costs in time and resources


Tailor Risk Analysis to Level of Credibility Requirements


Determine how much evidence will be needed to accredit
M&S


Determine the impact of NOT doing (or failing) V&V


Provide mechanisms to facilitate development of
workaround strategies and assessment of these solutions


Tailoring Guidance for VV&A:

Determine How to Optimize Resources



Not all V&V supports Accreditation to the same extent


Understand what each type of V&V evidence ‘buys’ you
relative to the cost of producing it


Determine which techniques to apply to derive required
level of credibility


Try to avoid (re)doing it all yourself



Capitalize on existing VV&A information whenever
possible


Know how to extract useful data from software V&V
results


Cooperative application of resources : use common
VV&A data elements and report structure


The VV&A Templates provides a common structure for
documenting VV&A information


Each of the Services has adopted a common database
structure for their M&S Resource Repositories


VV&A as Applied to the
Simulation Categories

Simulation Categories

New M&S Development


Any stand
-
alone model or simulation under
development being built to address the
intended purpose or purposes of the User.

Legacy



Any model or simulation that was developed
either in the past or for a different purpose.

Federation


A system of interacting models and/or
simulations, with supporting infrastructure,
based on a common understanding of the
objects portrayed in the system.

VV&A and New M&S Development


`

ACCREDITATION PROCESS

PROBLEM SOLVING PROCESS

Establish

Objectives

Define
Problem

Accept &
Record

Solution

Analyze
Results

Non
-
M&S Methods

M&S USE
PROCESS

Select

Approaches

Apply

Results

Repository

Execute

and
Prepare
Results

yes

no

Make

Accreditation

Decision

Develop New M&S

Define

M&S
Rqmts

Plan

Approach

M&S
Method

M&S DEVELOPMENT/PREPARATION PROCESS

Refine
M&S
Rqmts

Plan
M&S
Develop
-
ment

Develop
Con
-
ceptual
Model

Imple
-
ment &
Test

Develop
Design

Collect and Evaluate Accreditation Information

Perform

Accreditation

Assessment

Develop
Accreditation
Plan

V&V PROCESS

Verify
Design

Validate
Concep
-
tual
Model

Verify
Imple
-
ment
-
ation

Validate
Results

Verify
Rqmts

Develop
V&V Plan

Prepare
M&S for
Use

Challenge Issues that Impact
VV&A Implementation


The lack of a well defined referent


Determining how to use and when to
use Subject Matter Experts


The lack of insight into the M&S


The lack of M&S stability


Starting the V&V/A process “late”


Keeping the second “V” in V&V


VV&A and Legacy M&S

Is There Sufficient

Info to Assess the

M&S?

NO

Refine Requirements

And Verify

Obtain Needed

Information and Verify

NO

YES

NO

Plan/Conduct

Accreditation

Assessment

Plan M&S

Modification

Modify and

Verify

Use As

-

Is

Use

As

-

Is or

Modify?

Reject

M&S

Are the

Requirements

Adequate?

Validate

Prepare M&S

for Use

Issue

Accreditation

Decision

AA

User

AA

Define

Requirements

User

AA

Select Legacy M&S

AA

AA

AA

VV

User

User

User

Prop

User

AA, VV

VV

User

Prop. User

Dev, VV

Prop

Dev

Dev, VV

Prop

Is the Info

on M&S Capability

Adequate?

Can the M&S be

Used?

Modify

V&V as

Needed

VV

NO

YES

Prop, Dev

AA, VV

[ AA ]

Accreditation Agent

[ User ]

User

[ Dev ]

Developer

[ VV ]

V&V Agent

[ Prop ]

M&S Proponent

[ AA ]

Accreditation Agent

[ User ]

User

[ Dev ]

Developer

[ VV ]

V&V Agent

[ Prop ]

M&S Proponent

VV

YES

YES

Advantages of Using Legacy M&S


Starting with a legacy simulation may be more expedient,
requiring less time and fewer resources than a new development


The experience of previous developers and users may be
leveraged to establish a knowledge base that can provide
guidance and support training



Testing and VV&A histories can provide evidence of simulation
capabilities and limitations allowing the current effort to focus
on high risk areas



A legacy simulation with a good reputation for providing
credible results in similar applications can increase the
simulation’s credibility (i.e., the User’s confidence in the
simulation’s ability to address the current situation) due to


the demonstrated ability to address similar requirements


the demonstrated capabilities of the existing simulation


the direct experiences of previous users who successfully
applied the simulation to related problems


Disadvantages of Using

Legacy M&S


The simulation was developed to address different
requirements



Unexpected and undesirable side effects may occur
during execution of the code


The simulation comes with specific hardware and
software, which may be obsolete



The simulation comes with specific data needs,
however the data to fulfill those needs may no longer
be authoritative or available



Information about the simulation and its previous
usage may be incomplete or inconsistent



Participants (e.g., Accreditation Agent, V&V Agent)
may need time up front to acquire knowledge about
the simulation



What Can Help Reduce

Legacy V&V Cost


Documentation that can directly support the
accreditation assessment can reduce the
burden of an extensive discovery effort



Documentation that completely and accurately
describes simulation capabilities and use
history can reduce the discovery and training
costs



A simulation under configuration control with
an established user community that can
provide necessary information can reduce the
support and discovery costs



Participants familiar with the simulation can
reduce the need for extensive training

Scope of the Distributed M&S
Problem

Distributed M&S:

Technical Interoperability

From “HLA and Beyond: Interoperability Challenges” by Dahmann, Salisbury, Barry, Turrell, and
Blemberg (99F
-
SIW
-
073)


Characterized by the ability of
federates to
physically connect and
exchange data


Involves the use of common standards,
compatible interfaces and coordinated
data structures


Elements of technical interoperability


Hardware compatibility


Standards compatibility


Time management coordination


Coordinated use of Runtime
Infrastructure (RTI) services


Security issues


Distributed M&S: Substantive
Interoperability Issues

Simulation Interoperability focuses on the
capability of federates, when connected, to
provide adequate, accurate & consistent
simulated representations that adhere to the
principles of “fair fight” & address the mission
objectives.



Elements of Substantive Interoperability


Logical interaction between entities
modeled in different federates


Temporal resolution


Spatial resolution



Coherent relationships between the
components of the physical environment


Top Level View of the VV&A Overlay
to the FEDEP (IEEE 1516.4)

FEDEP

VV&A

Overlay

to the

FEDEP

FEDEP INFORMATION & PRODUCTS

Plan,

Integrate

& Test

Federation

Step 5

Validate &

Accept

Federation






Phase 5


Validate the
integrated
federation
results against
the Federation
Acceptability
Criteria and
against the
Federation
Referent.


Formulate
federation
acceptance/
accreditation
recommen
-
dations.

Execute

Federation

& Prepare

Outputs

Step 6

V&V

Federation

Output






Phase 6


Verify the
federation
outputs
internally.


Validate
federation
output against
the Federation
Acceptability
Criteria and
against the
Federation
Referent.

Define

Federation

Objectives

Verify
Federation
Objectives







Phase 1


Verify the
Federation
Objectives
with the User
/Sponsor.


Define the
Federation
Acceptability
Criteria.


Assemble the
Federation
Referent.


Plan the
VV&A
activities.

Step 1

Perform

Conceptual

Analysis

V&V

Federation

Conceptual

Model





Phase 2


Verify the
Federation
Conceptual
Model
internally.


Validate the
Federation
Conceptual
Model
against the
Federation
Referent and
against the
Federation
Acceptability
Criteria.

Step 2

Design

Federation

Verify

Federation

Design







Phase 3


Verify the
Federation
Design
internally.


Verify the
Federation
Design
against the
Federation
Conceptual
Model.

Step 3

Develop

Federation

Verify

Federation

Development

Products





Phase 4


Verify the
federation
development
products
internally.


Verify the
federation
development
products
against the
Federation
Conceptual
Model.

Step 4

Analyze

Data &

Evaluate

Results

Consolidate

Federation

VV&A

Products





Phase 7


Collect all
VV&A
products
and lessons
learned.


Assemble
them into a
VV&A
package to
support
federation
reuse.

Step 7

Tasks Associated with each
VV&A Overlay Phase

Verify
Federation
Design



3.0

Support
Selecting
Federates

Support
Preparing
Federation
Design

Contribute to
Verifying
Federation
Design

Support
Preparing
Federation
Development &
Execution Plan

Verify
Federation
Objectives



1.0

Support
Identifying
User/Sponsor
Needs

Plan
Accreditation
Activities

Contribute to
Verifying
Federation
Objectives

Support
Developing
Federation
Objectives

Assemble
Federation
Referent

Define
Federation
Acceptability
Criteria

Plan V&V
Activities

V&V
Federation
Conceptual
Model


2.0

Support
Developing
Federation
Scenarios

Contribute to
Verifying
Federation
Scenarios

Contribute to
Verifying
Federation
Requirements

Support
Developing
Federation
Requirements

Support
Developing
Federation
Conceptual
Model

Contribute to
Verifying
Federation
Conceptual
Model

Validate
Federation
Conceptual
Model

Validate &
Accept
Federation



5.0

Support
Planning
Federation
Execution

Support
Integrating
Federation

Contribute to
Verifying
Integrated
Federation

Support
Testing
Federation

Perform
Acceptance
Assessment

Validate
Integrated
Federation
Results

V&V
Federation
Output



6.0

Support
Executing
Federation

Contribute to
Verifying Raw
Execution and
Derived
Output

Validate
Federation
Outputs

Consolidate
Federation
VV&A
Products


7.0

Support
Analyzing
Data

Prepare
Federation
VV&A
Products for
Reuse

V&V
Federation
Data Sets

Verify
Federation
Development
Products


4.0

Support
Developing
FOM

Contribute to
Verifying FOM

Support
Establishing
Federation
Agreements

Contribute to
Verifying
Federation
Agreements

Contribute to
Verifying
Federation
Infrastructure

Support
Implementing
Federate
Designs

Support
Implementing
Federation
Infrastructure


The Key Players

User


The User wants to use the simulation to provide
an answer or solve a problem; he is concerned
that the simulation can provide credible answers.


The User defines the objectives and the
requirements, scenarios and use cases, and
generally serves as the subject matter expert
(SME) for issues in the problem and user
domains.



Ultimately the User is the one who decides
whether the simulation is credible enough to use.





Accreditation Agent


The Accreditation Agent
prepares for and
conducts a cost
-
effective accreditation
assessment that results in a logical, sufficient,
and fully justified accreditation
recommendation to the User.


The Accreditation Agent
assists the User with
activities that help establish the scope of the
problem to be addressed.


The

Accreditation

Agent

serves

as

the

User’s

advocate

throughout

the

M&S

development

process
.

Accreditation Agent
Challenges


Ensuring the definition of the
intended use


Using existing V&V documentation


Coping with configuration
management difficulties


Locating appropriate Subject Matter
Experts


Overcoming delay in appointment of
the accreditation agent


Obtaining needed resources


Communicating the results of the
accreditation assessment

V&V Agent


The V&V Agent
plans and performs the
verification and validation (V&V) activities


The V&V Agent ensures that
sufficient
evidence is gathered to demonstrate that
the simulation meets the stated
requirements in terms of their associated
measures and acceptability criteria.





V&V Agent Challenges


Obtaining well
-
defined accreditation
information needs


Dealing with missing documentation


Establishing V&V support system and
infrastructure


Finding adequate resources


Selecting the right people


Managing the V&V effort


Leveraging configuration management
resources


Tracking and reporting V&V effort progress



Conceptual Model

Definition

“The conceptual model is a simulation
developer’s way of translating
modeling requirements into a detailed
design framework, from which the
software, hardware, networks, and
systems/equipment that will make up
the simulation can be built, modified,
or assembled.”



from SIW Paper 00F
-
SIW
-
019, “Simulation Conceptual Model




Development Issues and Implications for Reuse of Simulation



Components” by Dr. Dale Pace

Why Conceptual Models?


“The biggest indication that
Trespasser had game design
problems was the fact that it never
had a proper design spec….
Trespasser started and finished
weak in the game design, and this
affected every other part of the
project.”




Richard Wyckoff,

“Postmortem: DreamWorks Interactive’s
Trespasser”

Gamasutra, May 14, 1999.

From presentation 03F
-
SIW
-
080.ppt by Gustavson, Turrell,
Zimmerman at Fall 2003 SIW

Why Conceptual Models?


“We never had a proper design
document, which meant that we
generated a lot of code and art that we
later had to scrap. What’s worse,
because we didn’t have a detailed
outline of what we were trying to build,
we had no way to measure our progress
(or lack thereof) accurately. We only
realized that we were in trouble when it
became glaringly obvious. If we’d been
about the design rigorous up front, we
would have known that we were slipping
much sooner.”



Brian Upton,

“Postmortem: Red Storm's Rainbo
w Six”

Gamasutra, January 21, 2000.

From presentation 03F
-
SIW
-
080.ppt by Gustavson, Turrell,
Zimmerman at Fall 2003 SIW

Conceptual Model Components

(1) Simulation Context

Authoritative Information re:
relevant entities/processes, data,
algorithms, assumptions,
behaviors, etc.

Sets constraints/bounds on the
Simulation Concept

Simulation Concept

(2)

Mission Space
--

Representation Aspect


Simulation Elements

Entities/processes represented (tasks,
actions, behaviors, etc.) by
assumptions, algorithms, data, and
relationships (architecture)


(3)

Simulation Space

Operational/functional capabilities

Requirements

Constraints

Conceptual Model

Specifications

Benefits of a

Conceptual Model


A good conceptual model can improve requirements


Good requirements are important because

²
Faulty requirements cause half of software errors


Late error detection/correction is much more
expensive than early error detection/correction


A good conceptual model allows early

²
Estimation of simulation fidelity


Development of validation sampling methodology


Assessment of simulation validity for the intended
use


Determination of compatibility with other M&S/tools

Conceptual Model Documentation


Conceptual Model Portion Identification


Principal Simulation Developer POCs


Requirements and Purpose


Overview


General Assumptions


Identification of Possible States, Tasks,
Actions, and Behaviors, Relationships and
Interactions, Events, and Parameters and
Factors for Entities and Processes being
described


Identification of Algorithms


Simulation Development Plans


Summary and Synopsis


Documentation

Documentation Philosophy


Organize information according to a standard outline


Rationale


Eases the comparison of data with accreditation
requirements


Facilitates V&V data accumulation for future use


Reduces costs of multiple accreditations over time


Specific outlines vary with type of data element



Structure V&V results to facilitate comparison with
M&S requirements


Include description of how the results impact m&s use


State newly discovered discrepancies as assumptions,
limitations on model use, or as errors


Identify the range of applicability for all results (particularly
regarding any limitations or errors)


Ensure results are linked to the applicable M&S version



Configuration Management

MIL
-
STD 3022: VV&A
Documentation Templates

Accreditation Plan

V&V Plan

V&V Report

Accreditation Report

Executive Summary

Executive Summary

Executive Summary

Executive Summary

1 Problem Statement

1 Problem Statement

1 Problem Statement

1 Problem Statement

2 M&S Requirements and
Acceptability Criteria

2 M&S Requirements and
Acceptability Criteria

2 M&S Requirements and
Acceptability Criteria

2 M&S Requirements and
Acceptability Criteria

3 M&S Assumptions,
Capabilities, Limitations &
Risks/Impacts

3 M&S Assumptions,
Capabilities, Limitations &
Risks/Impacts

3 M&S Assumptions,
Capabilities, Limitations &
Risks/Impacts

3 M&S Assumptions,
Capabilities, Limitations &
Risks/Impacts

4 Accreditation Methodology

4 V&V Methodology

4 V&V Task Analysis

4 Accreditation Assessment

5 Accreditation Issues

5 V&V Issues

5 V&V Recommendations

5 Accreditation
Recommendations

6 Key Participants

6 Key Participants

6 Key Participants

6 Key Participants

7 Planned Accreditation
Resources

7 Planned V&V Resources

7 Actual V&V Resources

7 Actual Accreditation
Resources

8 V&V Lessons Learned

8 Accreditation Lessons Learned

Suggested Appendices:

A M&S Description

B M&S Requirements
Traceability Matrix

C Basis of Comparison

D References

E Acronyms

F Glossary

G Accreditation Programmatics

H Distribution List

Suggested Appendices:

A M&S Description

B M&S Requirements
Traceability Matrix

C Basis of Comparison

D References

E Acronyms

F Glossary

G V&V Programmatics

H Distribution List

I Accreditation Plan

Suggested Appendices:

A M&S Description

B M&S Requirements
Traceability Matrix

C Basis of Comparison

D References

E Acronyms

F Glossary

G V&V
Programmatics

H Distribution List

I V&V Plan

J Test Information

Suggested Appendices:

A M&S Description

B M&S Requirements
Traceability Matrix

C Basis of Comparison

D References

E Acronyms

F Glossary

G Accreditation
Programmatics

H Distribution List

I Accreditation Plan

J V&V Report

VV&A Data Archiving


VV&A Documents


Should be archived in central location:


Defense Department level or


Program level


Indexed in a electronic database


Possibly available on
-
line


Electronic Database


Should contain summarized,
unclassified data from VV&A reports


Should be sufficient for model
comparisons and selection



VV&A data are only useful if they are documented, and
the documentation is readily available to model users.


Conclusion

Summary of VV&A


Cost
-
effective accreditation requires


Complete problem definition


Clear application description


Explicit statements of M&S requirements and
acceptability criteria


A well
-
planned, tailored and focused accreditation
assessment effort


Efficient V&V to support accreditation requires



A clearly defined application context to permit
focused V&V efforts


The use of risk to focus and tailor the V&V effort


A clear understanding of how each V&V activity
contributes to M&S credibility


A means of extracting key accreditation support
information from previous V&V (and other M&S)
efforts


An efficient means to produce, document, and archive
all V&V information (from software and M&S V&V)


Simone Youngblood

Simone.Youngblood@jhuapl.edu

(240) 228
-
7958