Operators Versus Automation


Nov 5, 2013 (3 years and 7 months ago)


Operators Versus Automation

Andrew Rae

A couple of warm
up questions

Would you voluntarily accept a device added to your
car that detected the local speed limit and prevented
you exceeding that limit in all circumstances?

Do you believe that trains should be fitted with
vigilance devices which apply the brakes if the
driver isn’t aware/awake/conscious?

In different circumstances, people are willing to place
different levels of trust in automation.

Chernobyl 4 versus USS Thresher

Chernobyl: Automatic Shut
Down System was
disabled to allow special testing


Explosion and fire


should not

be allowed to turn off

vital safety systems

USS Thresher: Automatic Shut
Down System
removed power at a time when propulsion power
was the only thing that could save the submarine


Loss of submarine with all crew



be allowed to override vital

safety systems

So who are we going to trust?

For road vehicles: should speed limits be
automatically enforced, or should drivers have
ultimate control?

For military engines: should thermal limits be
permitted to be overridden using “battle shorts”?

For aircraft: should “alpha” (flight envelope)
protection be strict, or soft?

For railways: should a signal interlocking be flexible
in order to move trains out of a dangerous

The question of ultimate trust

In a given situation, should automation be designed
to prevent system states which the designers judge
to be dangerous, or should the interface provide
facility for the operator to execute any control at any

What conventional hazard analysis has to say

Hazard: Operator error in performing manual task

Mitigation: Automation

Hazard: Automation Error

Mitigation: Operator in the loop

Hazard: Inappropriate operator intervention

Mitigation: Interlock

Hazard: Unforeseen circumstances

Mitigation: Interlock override

Hazard: Inappropriate use of override

Mitigation: ???

Conventional Hazard Analysis (continued)

Conventional hazard analysis does not cope well
with situations where:

Mitigations / Safety Requirements are in conflict

Hazards cannot be quantified and thus directly compared

The risk of human error can be quantified (but not
very accurately without application

The risk of automation error is very difficult to

The risk of unforeseen circumstances (unknown
unknowns) cannot be quantified

The Fitts’ List Approach

Humans are good at:

Ability to perceive patterns

Ability to improvise and use flexible procedure

Ability to reason inductively

Ability to exercise judgment

Fitts’ List Continued

Machines are good at:

Ability to respond quickly to control signals

Ability to perform repetitive tasks

Ability to do many things at once

Ability to perform complex calculations quickly and

The “Fitts’ List” approach isn’t perfect

In reality, humans and machines co
operate to
perform tasks

The performance of humans and machines is
shaped by environmental factors, and their
interaction with each other

A “performance shaping” approach

In a given situation, the performance of humans and
machines will be moderated by:

Whether the agent is innately suited to the task

The internal state of the agent

The ability of the agent to process inputs

The ability of the agent to decide upon outputs

The ability of the agent to execute outputs

We call each of these considerations “performance
shaping factors”

Performance shaping (continued)

Performance shaping is a common concept in
centered design

The idea that performance shaping factors can be
applied to machines is less frequently addressed

Performance Shaping Factors for Humans

Task demands and characteristics

Instructions and procedures

Displays and controls


Individual skill and capacity

Individual condition

technical factors

Performance shaping factors for machines








Performance shaping factors for machines

Incorrect Internal Model

Incorrect assumptions

Incorrect or insufficient inputs

Incorrectly Applied Rule Set

Incorrectly Specified Rule Set

Error in Implementing Rule Set

Error in Transmitting Outputs


Automatic Train Operation

Train driving is a well
understood task

Most often performed by humans

Sometimes (e.g. Docklands Light Rail, Copenhagen
Metro) performed by computer

Sometimes performed by computer with a driver in
the cab “just in case”

A good candidate for testing our methodology


Automatic Train Operation

Performance Shaping Factors
for Train Drivers

Task demands are moderate

Instructions and procedures
are generally clear and well

Displays and controls can be
made simple and intuitive


job has some
pressures, and also some

Driver condition can be a

remember Waterfall

Performance Shaping Factors
for Train Computers

Task is well understood

Inputs are generally sufficient

note important exception
that drivers can see out of the
windscreen and computers

conscious industry /
development environment

Mechanical reliability is high

Automatic Train Operation


So should humans or computers be used to drive

Some supplementary points of interest

Adequate levels of safety and performance must be delivered in
a cost effective way

sometimes performance and cost will tip
the balance towards a particular solution

As automation increases, the drivers’ performance may

Lower situation awareness

Greater boredom

Greater stress when things do go wrong

There comes a point when things are so automated that the
driver lacks capacity to be an effective safeguard


It is important to make a conscious decision who (or
what) is responsible for making safety decisions

Ultimately, humans will be responsible

Either as operators, or as designers/maintainers of the system

The nature of the task, and the task environment,
dictate whether the operator or the automation is
best suited to making safety decisions

A sound argument can be formulated for assigning
the safety decision, based on performance shaping