Simulation Models for Engineering

downtownbeeΜηχανική

18 Νοε 2013 (πριν από 3 χρόνια και 8 μήνες)

58 εμφανίσεις

David J. Murray
-
Smith,

School of Engineering, Rankine Building,

University of Glasgow,

Scotland, U.K.



Keynote Tutorial:

Methods for Testing and Validation of
Simulation Models for Engineering
Applications

E
-
mail: david.murray
-
smith@glasgow.ac.uk

Keynote Tutorial: Model quality, testing and validation 0


The presentation

1.
Introduction

2.
Model testing and validation issues




Uncertainties, modelling errors, testing, verification and validation.




Model quality measures and model improvements.

3. Model management




Libraries of sub
-
models and development of generic models.




Model version control and model documentation.

4. Educational implications

5. Examples




External validation and upgrading of linear and nonlinear


helicopter models for handling qualities and flight control studies.

6.

Discussion and Conclusions


Keynote Tutorial: Model quality, testing and validation 1



To provide a basis for
design
.


To assist in human
decision making
.


To
explain

complex system behaviour.


For use within
fault detection
systems etc..


For
simulator

development (e.g. for operator training
or for engineering development applications).

The purpose of models in engineering

Keynote Tutorial: Model quality, testing and validation 2


1.
Conceptual

models

allow

investigation

of

performance

limitations

at

an

early

stage

of

the

design

process

for

normal

and

abnormal

operating

conditions
.


2.
Fully

developed

and

proven

models

can

provide

information

about

key

parameter

sensitivities

and

inter
-
dependencies



useful

for

design

decisions

and

optimisation
.

3.
Full

models

allow

virtual

prototypes

to

be

created

bef or e

any

har dwar e

pr ot ot ype

is

available

so

identifying

necessary

design

alterations

at

an

early

stage,

avoiding

expensive

changes

later

on
.

Design benefits with fit
-
for
-
purpose
models

Keynote Tutorial: Model quality, testing and validation 3


Models in
design: some pointers



“Improved modelling of physical and manufacturing
processes will improve our ability to predict the behaviour,
costs and risks of future systems, and dramatically reduce
the development timescale”
.
UK Office of Science and
Technology, Technology Foresight Panel Report (1995).


“Verification
, validation and accreditation”

(VV&A).


“Smart procurement”
methods.


Concept of
“the model as a specification”

(as promoted by
T.S.
Ericsen
, US Office of Naval Research
).



Keynote Tutorial: Model quality, testing and validation 4


Level

of

model

quality

necessary

is

of

critical

importance

for

any

given

application

........

an

inappropriate

model

is

less

than

useless

as

it

may

delay

the

project

and

lead

to

cost

escalation



Balance

needs

to

be

found

between

model

accuracy

and

the

cost

of

developing

the

model
.



Rigorous

consideration

of

model

quality

is

most

common

in

applications

involving

safety

critical

issues

(e
.
g
.

aeronautical

engineering,

automotive

engineering

etc
.
)

Levels of model quality

Keynote Tutorial: Model quality, testing and validation 5



Models are often used with very
little systematic testing.



Model documentation is often minimal
and is not recognised
as a vital part of the model development process.



In many organisations
models are passed from project to
project
and end up being used in ways that were never
intended by the original developer of the model.

Current problems with models

Keynote Tutorial: Model quality, testing and validation 6


Example: Hydro
-
turbine system

modelling


.


Figure from Bryce, G.W., Foord, T.R., Murray
-
Smith , D.J.
and Agnew, P., ‘Hybrid simulation of water turbine
governors’,
Simulation Councils Proceedings,
6(1), 35
-
44,
1976

Keynote Tutorial: Model quality, testing and validation 7


Photographs:
©
D. J. Murray
-
Smith

Keynote Tutorial: Model quality, testing and validation 8


Pipeline geometry


Figure from Bryce, G.W., Foord, T.R., Murray
-
Smith , D.J. and
Agnew, P., ‘Hybrid simulation of water turbine governors’,
Simulation Councils Proceedings,
6(1), 35
-
44, 1976

Keynote Tutorial: Model quality, testing and validation 9


First simplification




Figure from Bryce, G.W., Foord, T.R., Murray
-
Smith , D.J. and
Agnew, P., ‘Hybrid simulation of water turbine governors’,
Simulation Councils Proceedings,
6(1), 35
-
44, 1976

Keynote Tutorial: Model quality, testing and validation 10


Model comparisons

((
(Real
-
time approximation: continuous line


Non
-
real
-
time model: dashed line)


Figure from Bryce, Fo Murray
-
Smith and
Agnew,


Simulation Councils Proceedings,
Vol.6, No. 1,
35
-
44.


Figure from Bryce, G.W., Foord, T.R., Murray
-
Smith , D.J. and
Agnew, P., ‘Hybrid simulation of water turbine governors’,
Simulation Councils Proceedings,
6(1), 35
-
44, 1976

Keynote Tutorial: Model quality, testing and validation 11


Second approximation

F



Figure from Bryce, G.W., Foord, T.R., Murray
-
Smith , D.J. and
Agnew, P., ‘Hybrid simulation of water turbine governors’,
Simulation Councils Proceedings,
6(1), 35
-
44, 1976

Keynote Tutorial: Model quality, testing and validation 12


Model comparisons for second
approximation



( Real
-
time approximation: continuous line


Non
-
real
-
time model: dashed line)

Figure from Bryce, G.W., Foord, T.R., Murray
-
Smith , D.J. and
Agnew, P., ‘Hybrid simulation of water turbine governors’,
Simulation Councils Proceedings,
6(1), 35
-
44, 1976

Keynote Tutorial: Model quality, testing and validation 13


Site test and simulation comparisons


Figure from Bryce, G.W., Foord, T.R., Murray
-
Smith , D.J. and
Agnew, P., ‘Hybrid simulation of water turbine governors’,
Simulation Councils Proceedings,
6(1), 35
-
44, 1976

Keynote Tutorial: Model quality, testing and validation 14


Model predictions

Keynote Tutorial: Model quality, testing and validation 15




a) Model quality, uncertainties and modelling errors.


b) Testing, verification and validation of models.



c) External validation methods.


d) Model quality measures in external validation.

Part 2: Issues of model
testing
and validation

Keynote Tutorial: Model quality, testing and validation 16



Tests

only

deal

with

a

small

number

of

cases
.



General

statements

about

validity

are

impossible
.


All

testing

must

be

carried

out

in

the

context

of

the

application

and

especially

the

precise

range

of

operating

conditions

for

that

application
.


Should

start

from

a

well
-
understood

case
,

even

if

much

simplified
;

then

move

incrementally

to

testing

for

less

certain

situations

for

that

application
.



The

more

complex

the

model

the

harder

the

problem

of

quality

assessment

becomes
:

measures

of

model

performance

become

harder

to

define

and

visualisation

becomes

more

difficult
.

Testing of models:

fitness
-
for
-
purpose

Keynote Tutorial: Model quality, testing and validation 17



Establishing the
useful range
of the dynamic model for a
specific application.



Estimating the
limits of accuracy
of the model (usually both
for steady
-
state and transient conditions) in terms the
magnitude of expected errors in model predictions.

Aspects of model quality

Keynote Tutorial: Model quality, testing and validation 18




Sources

of errors and uncertainties


Incorrect modelling
assumptions



Errors in
a priori

information
(
e,g
, model
parameter values)



Errors in
numerical solutions
of model equations



Errors in
experimental procedures
and in the
measurements used for model testing
.

Keynote Tutorial: Model quality, testing and validation 19




Internal verification and


external

validation


Internal Verification


establishing that a
computer

based model is consistent with the
underlying mathematical model.


i.e.
“Is the simulation model right?”


External Validation


demonstrating that a final
(nonlinear)) model is adequate for the intended
application.



i.e.
“Is it the right simulation model
?”


Note that checks of identified linear models are
sometimes referred to as “external verification”.

Keynote Tutorial: Model quality, testing and validation 20




External validation

Need to distinguish between:



Functional

validation: where model is assessed in
terms of how well it mimics input
-
output behaviour
of the real system.



Physical/Theoretical

v
alidation where the model
is based on theory and intermediate variables in
model and system are compared. Approximations
and assumptions are investigated within this
process.

Keynote Tutorial: Model quality, testing and validation 21




Approaches to external validation



Holistic
approaches (e.g. subjective opinion of an
expert on the real system such as an operator).



Model component
approaches (e.g. each sub
-
system tested independently and compared with
corresponding components of the real system).

Keynote Tutorial: Model quality, testing and validation 22




Methods for external validation



Methods

involving

direct

comparisons

of

response

data

from

model

and

real

system
.


Methods

based

on

system

identification

and

parameter

estimation
.


Methods
involving
parameter sensitivity analysis
.


Methods

based

on

inverse

models

and

inverse

simulations
.


Whatever

method

is

used,

data

for

external

validation

must

be

appropriate

for

the

intended

application
.

Careful

experimental

design

is

essential
.



Keynote Tutorial: Model quality, testing and validation 23




Methods for system and model


comparisons


Graphical comparisons


Quantitative measures


System identification methods


Expert opinion



Different approaches can provide different kinds of
insight.

Keynote Tutorial: Model quality, testing and validation 24



Plots
of simulated and measured responses
against an independent
variable
(often time).



Plots

of
simulated

values against the corresponding
measured

values
(should be 45 degree line).



Different graphical methods
may emphasise different

aspects of the
simulation model performance so there are possible benefits from
combining different approaches.


Methods of system/model
comparison: some examples

Keynote Tutorial: Model quality, testing and validation 25


Typical time history comparisons



Keynote Tutorial: Model quality, testing and validation 26


Another

time history comparison: a
multi
-
input multi
-
output case


The original version of this
figure was published by the
Advisory Group for Aerospace
Research and Development,
North Atlantic Treaty
Organisation (AGARD/NATO)
in AGARD Advisory Report
280 ‘Rotorcraft System
Identification’, September
1991


BO
-
105
helicopter flight test data,

DLR SIMH simulation model

Keynote Tutorial: Model quality, testing and validation 27


Examples of simple cost functions


Mean absolute error



Mean absolute percent error

Weighted error

Keynote Tutorial: Model quality, testing and validation 28


Theil’s Inequality Coefficient (TIC)



Keynote Tutorial: Model quality, testing and validation 29


Use of quantitative measures:

an example



Keynote Tutorial: Model quality, testing and validation 30




Polar diagram form of display

From Smith, M.I., Murray
-
Smith, D.J. and Hickman,
D., ‘Verification and validation issues in a generic
model of electro
-
optic sensor systems ‘
J. Defense
Modeling and Simulation
, 4(1), 17
-
27, 2007

Keynote Tutorial: Model quality, testing and validation 31




Issues of identifiability in external

validation

Test

input

design

is

important

since

inputs

must

excite

the

system

and

model

over

an

appropriate

range

of

frequencies

and

amplitudes
.


The

concept

of

identifiability

is

central

to

issues

of

test

input

design

external

validation

and

is

thus

very

important

for

external

validation
.



Structural

identifiability

relates

to

situations

where

a

model

may

have

an

excess

of

parameters

so

that

some

specific

parameters

cannot

be

estimated

uniquely

for

any

possible

experimental

design

(e
.
g
.

Bellman,

R
.

and

Åström,

K
.
J
.
,

Mathematical

Biosciences
,

7
,

329
-
339
,

1970
)
.

Structural

identifiability

is

also

important

for

external

validation
.


Keynote Tutorial: Model quality, testing and validation 32


Numerical
unidentifiability

arises when a structurally identifiable
model is being used with data that is inappropriate for the application.


Numerical
identifiability


investigated
from
parameter information
matrix
M
, the
related
dispersion matrix
D

and the parameter
correlation
matrix
P
.
All
depend on the sensitivity matrix
X
where:








Inputs may maximise the overall accuracy of all parameter estimates or
may be chosen to maximise accuracy of specific parameter estimates.






Issues of numerical identifiability and
test input design in model validation

1
1
1




jj
ii
ij
ij
m
m
m
p



Keynote Tutorial: Model quality, testing and validation 33


Upgrading of simulation models


Following

comparison

of

model

and

system

behaviour

usually

need

to

analyse

discrepancies

and

propose

upgrades

for

model
.


Changes

must

be

evaluated

systematically

on

a

physical

basis



with

further

iterations

in

the

development

cycle
.


Parametric

changes

usually

considered

first
,

before

structure

.



Sometimes

possible

to

associate

model

deficiencies

with

specific

state

variables

model

(e
.
g
.

correlation

of

output

error

with

a

state

variable

may

help

identify

problem

source)
.


C
orrelation

of

model

errors

with

derivatives

of

state

variables

may

suggest

that

a

higher
-
order

description

would

be

more

useful
.


Optimisation

tools

(including

evolutionary

computing

methods

such

as

GA

and

GP)

may

be

useful

but

should

be

used

along

with

physical

knowledge

and

understanding

of

the

model
.


Keynote Tutorial: Model quality, testing and validation 34




Part 3: Model management


a)

Libraries of sub
-
models and generic models.


b) Model version control and model documentation.

Keynote Tutorial: Model quality, testing and validation 35




Generic and re
-
usable sub
-
models


Generally accepted that
system
design should be based on use of
generic descriptions and re
-
usable sub
-
models
.



Examples of the generic approach may be found in automotive
engineering, gas turbines
etc.
Issues inevitably arise in the external
validation of generic models


one approach is discussed in Smith,
M.I., Murray
-
Smith, D.J. and Hickman, D., ‘Verification and
validation issues in
a generic
model of electro
-
optic sensor systems’
J.
Defense

Modeling

and Simulation,

4(1), 17
-
27, 2007

Keynote Tutorial: Model quality, testing and validation 36




Model documentation and version control



Extra
costs

of creating good documentation should be
more than balanced by the resulting
re
-
usability of
models
.
Version control processes
should ensure that
changes are fully documented.



Documentation and version control well developed in
software engineering
field. Same principles should be
applied to the model development process.


Keynote Tutorial: Model quality, testing and validation 37




Items for documentation


Purpose

of model and the intended application.


Full

description of model
and corresponding computer
code.


List of all
assumptions and approximations
used
.



Details of all
tests

carried out on the real system to provide
information for model development.


Details of the
internal verification
process.


Details of the
external validation
process, with statements
about why model was accepted or rejected and information
about usable range for model.

Keynote Tutorial: Model quality, testing and validation 38




Part
4: Educational implications




Keynote Tutorial: Model quality, testing and validation 39



Engineering

students

encounter

mathematical

and

computer
-
based

modelling

repeatedly

in

their

university

education
.



Emphasis

is

most

often

on

development

of

models

from

physical

principles

and

on

using

models/simulations

in

place

of

experiments

on

real

systems
.



Modelling
and simulation in

engineering education

Keynote Tutorial: Model quality, testing and validation 40



Issues

of

model

quality

and

fitness
-
for
-
purpose

are

seldom

emphasised



in

the

teaching

of

modelling

and

simulation
.




Model

validation

is

neglected

in

education
.

The

teaching

of



system

modelling

and

simulation

should

include

much

more

on



model

validation

methods
.



Model

testing

should

become

second

nature

for

students
.



Documentation
,

model

re
-
use

and

libraries

of

models

must

be



given

much

more

emphasis

(especially

in

more

advanced

teaching)
.



Model quality and testing issues in
engineering education

Keynote Tutorial: Model quality, testing and validation 41




Part
5:
Examples



Drawn from external
validation and upgrading
of


helicopter
models for flight control system


design.

Keynote Tutorial: Model quality, testing and validation 42



Plant

model

limitations

often

impose

serious

performance

limitations

within

control

system,

especially

in

systems

with

high
-
performance

requirements
.




Particularly

important

to

have

highly

accurate

plant

models

for

the

part

of

the

frequency

range

close

to

the

“cross
-
over”

region

in

the

frequency

domain
.


Model limitations in control

Keynote Tutorial: Model quality, testing and validation 43




Examples from aircraft and helicopter
flight control system design


There

are

many

well

documented

aircraft

flight

control

examples

illustrating

problems

of

model

quality

and

model

limitations

in

integrated

system

design
.


Problems

are

often

identified

during

initial

testing

of

hardware
.

These

lead

to

development

of

improved

models

and

corresponding

control

design

changes
.


The

later

in

the

design

cycle

these

changes

have

to

be

made

the

more

costly

they

are

and

the

greater

the

delays

to

the

project
.


Keynote Tutorial: Model quality, testing and validation 44




Helicopter model requirements


Must perform well over a defined
range of frequencies
.



Must perform well over a defined
range of manoeuvre amplitudes.


For

flight

control

design

model

must

perform

especially

well

close

to

open
-
loop

gain

and

phase

cross
-
over

frequencies

(as

with

control

applications

involving

other

types

of

system)
.


Photograph:
©
D. Murray
-
Smith

Keynote Tutorial: Model quality, testing and validation 45



Nominal model: simulation and flight

data compared for same test inputs.

(Westland Lynx helicopter in 300 ft. quick
-
hop manoeuvre)

From Bradley, R., Padfield, G.D., Murray
-
Smith, D.J. and Thomson, D.G., ‘Validation of helicopter mathematical
models’,
Transactions of the Institute of Measurement and Control’
, 12(4), 186
-
196, 1990.



SS

Common test inputs used for system

identification and “external verification”

of the identified model
(BO
-
105 flight data)

The original versions of these figures were published by

the Advisory Group for Aerospace Research and Development,

North Atlantic Treaty Organisation (AGARD/NATO) in AGARD

Advisory Report 280 ‘Rotorcraft System Identification’, September 1991


...........and flight testing

Keynote Tutorial: Model quality, testing and validation 49


Simulation, identification and

“external verification” results

(BO
-
105 flight test data, DLR SIMH model)

The
original version of these figures
were published by the Advisory Group
for Aerospace Research and
Development, North Atlantic Treaty
Organisation (AGARD/NATO) in
AGARD Advisory Report 280
‘Rotorcraft System Identification’,
September 1991


Y
v

L
v

N
v

L
p

N
p

L
r

N
r

L
δ lat

N
δ lat


L
δ ped

N
δ ped


2ζω
0

Normalised HELISTAB
values

Values estimated from
flight data using system
identification methods.

Estimated and theoretical parameter values of the
identified model for lateral/directional
characteristics

(SA
-
330 Puma helicopter flight test
data : 80 knots straight and level flight)

Assessment of a theoretical

nonlinear model for a Puma helicopter


Parameter values for two
different flight conditions,
showing parametric trends from
a physically
-
based nonlinear
simulation model (HELISTAB)
and the trends in estimates from
flight tests involving system
identification of separate
linearised models for each
flight condition.

From Bradley, R., Padfield, G.D., Murray
-
Smith, D.J. and Thomson, D.G., ‘Validation of helicopter mathematical models’,
Transactions of
the Institute of Measurement and Control’
, 12(4), 186
-
196, 1990.

Keynote Tutorial: Model quality, testing and validation 53



Good vehicle models
are essential for design of
high
-
bandwidth

full
-
authority active flight control systems.



Published examples
show that the achievable performance of flight

control systems have, in some cases, been overestimated in initial

design studies, usually because of
limitations of the flight mechanics
model
of the vehicle (see e.g.
Tischler
, M. B.
Advances in Aircraft Flight
Control Systems,
Taylor & Francis, London 1996)
.



Although control systems can be made
robust
to compensate for poor

accuracy this is usually at the
expense of performance.
Improved modelling
procedures and improved models can offer significant benefits. Otherwise,
problems may not be apparent until the flight testing stage, leading to

costly redesign,
extended flight test programmes and
delays
in certification.





Summary of model quality issues for helicopter


flight control system design

Keynote Tutorial: Model quality, testing and validation 55




Part 5: Discussion and conclusions

Keynote Tutorial: Model quality, testing and validation 56




Discussion: Model quality in design


The

more

demanding

the

design

specification

the

more

important

is

the

fitness
-
for
-
purpose

of

models

used

in

design
.




More

attention

needs

to

be

given

to

the

external

validation

of

models

for

the

specific

application

in

question
.


a)

Establishing

the

useful

range

of

the

model
.



b)

Estimating

the

accuracy

of

the

model

within

that

range
.




Model

validation

is

part

of

the

model

building

process

and

external



validation

techniques

need

to

be

applied

repeatedly
.




Models

should

be

retained,

maintained

and

updated

throughout

the


whole

life
-
cycle

of

the

system

that

they

represent
.



Keynote Tutorial: Model quality, testing and validation 57



More attention
should be given to the
fitness
-
for
-
purpose
of models
used in design, especially for demanding applications.


Current methods for
external validation
are time
-
consuming and
difficult to apply in many situations. More effort should be devoted to
improving validation methods.


Techniques of
version control
and rigorous
documentation

should
borrowed from software engineering and applied to the model
development process.
Re
-
use
of proven models should be made easier
and more comprehensive model documentation should be available
within model libraries.


Issues of
model quality
should be given far more attention within
engineering education.


Recommendations

Keynote Tutorial: Model quality, testing and validation 58


Conclusions


With

suitable

structure

and

parameter

values

and

rigorous

external

validation

models

that

are

fit
-
for
-
the
-
purpose

of

a

given

application

can

be

developed

(iteratively

of

course)
.



Good

model

management

can

reduce

the

cost

of

design

and

development
.



There

are

no

quick

answers
:

a

systematic

approach

is

essential,

moving

incrementally

from

well
-
understood

cases

to

less

well

known

situations
.


Educational

and

cultural

changes

are

needed

as

well

as

improved

management

of

the

modelling,

simulation

and

design

processes

within

most

organisations
.




Keynote Tutorial: Model quality, testing and validation 59