Increased Virtual Human Intelligence

stagetofuΤεχνίτη Νοημοσύνη και Ρομποτική

29 Οκτ 2013 (πριν από 4 χρόνια και 8 μήνες)

133 εμφανίσεις

Virtual Human IQ: Near Term Support for a Human Participant’s Perception of
Increased Virtual Human Intelligence

Randy Boys

Network Centric Systems


This paper proposes a number of short term
for increasing the robustness and interactive intelligence
of a Virtual Human populated Virtual world, particularly
for team
based human immersion in military training
environments, specifically Infantry small unit
engagements across a wide rang
e of operations, including
the solicitation of leader/Soldier decision
making in
complex non
kinetic engagements.

Virtual Human
, Simulation Immersion,
Infantry Training
, Small Unit



Technology that introduces individu
al humans
vs. human
operated platforms

into Virtual space is increasingly
capable and commonplace. Applications of this
technology range from the creation of an animation
supported transaction using mobile devices, to role
representatives in massiv
ely multiplayer gaming
environments, to team
based, full
body motion capture in
real t ime. Although this latter capability has not yet
d the Holy Grail as set forth in

the holodeck of Star

fame [1],

it is clear that humans will increasingly
‘meet’ in Virtual space, and that the meetings will be with
both avatars and with Virtual Humans (VH, also as ‘AI’
for Artificial Intelligences)

In these applications, human
(i.e., avatar
interaction is limited by the
instrumentation of the
creating human(s)

… capture of speech/text,
capture of
gestures, and a
priori capture of the human’s
appearance … but is also limited by the depiction of those
avatars in the Virtual space. Even the crudest of these
avatars c
an, however, readily exhibit the ‘intelligence’ of
their controlling human … such as during the exchange
between two avatars at an informat ion kiosk supported by
Second Life

, using only a keyboard or digitized speech
as input.

In avatar interactions wi
th VH, the limited ‘intelligence’
of computer
generated entities rapidly compromises the
responsiveness of the VH and the overall quality of the
‘interpersonal’ engagement. Although this is less
noticeable in the dominantly kinetic

(i.e., ID
teractions typical of the popular ‘first person shooter’
gaming environment,
the limited VH capability
compromises the ability to use VH in more complex
scenarios. Just as the holodeck is the grail of immersion
into a Virtual world, the Turing Tes
t [2] would

be the
Holy Grail of human <
> VH interaction. The Turing
Test is a proposed setting in which the human participant,
screened from direct observation of
the ‘tested’ entities
cannot readily ascertain whether the participant in their
ion is another human, or an ‘intelligent’

There are many technical pursuits being applied (by a
growing commercial industry) to increasing VH
intelligence. These include speech recognition, natural
language processing,
adaptive (i.e., ‘

generally applicable only where a sufficient
training base exists), goal
directed behaviors

and other
reasoning mechanisms
terrain/threat trafficability and
elements of ‘mission

, group and
behaviors, etc.

In the hu
immersive Virtual space,
almost all approaches
that demonstrate VH ‘IQ’
down to a matter of VH responsiveness to the actions of
the human participant(s), and the inferred intelligence
demonstrated by that responsiveness.

There are means other tha
n programmed VH intelligence
by which the perception of intelligence can be influenced.
This paper proposes a number of short term ‘remedies’
for increasing the robustness of a VH
populated Virtual
world, particularly for team
based human immersion in
litary training environments, specifically Infantry s mall
unit engagements across a wide range of operations,
including the solicitation of leader/Soldier decision
making in complex non
kinetic engagements.


In the conventions of

the military simulation community
(training, wargaming, ops analysis, and materiel
development), VH is a Constructive simulat ion. VH
actions are computer
generated. An avatar ... an entity
representing a human controller …is a Virtual simulation.
the VH exists in Live or Virtual space has caused
them to be termed Virtual Humans


And, although the
label AI is common for VH in the gaming/entertainment
industry, the term AI in gaming also applies to platforms
or other lifeforms, not just to human
s (i.e., ‘VH’ and this
definit ion of
the ‘human’

AI are equivalent). Similarly,
Artificial Intelligence, the sub
discipline of computer
science (interdisciplinary with regard to at least
psychology and mathemat ics), is commonly referred to as
‘AI’ but is
not the same as ‘an’ artificial intelligence (AI)
as the VH or platform entity.

The term VH is therefore used here to describe all non
avatar human entities presented in Virtual space, despite
the possible confusion resulting from the Constructive
e of VH generation.

Also note that both military and academic simulat ion
disciplines have coined other conventions for topics
related to VH, and that the boundaries between Live,
Virtual, and Constructive are challenged when the
simulated (or enhanced) en
tity is a human. All instances
of ‘AI’ in the gaming definit ion are equivalent to
Computer Generated Forces (CGF) as defined in the
context of military simulation, all being Constructive
simulations. The term Semi
Autonomous Forces (SAF)
refers to a sub
set of CGF that accepts run
time input from
a human controller, but that input is not conducted by
assuming first person control of the CGF (e.g., detailed
motion, sensor/weapon control, etc.). In the gaming
industry, and increasingly other practical appl
icat ions
(such as training), the disciplines involved in creating
‘Artificial Life’ are numerous, and expanding rapidly at
both academic and application levels.


Only recently has the training o
f individuals in Virtual
space become pract ical, ‘individuals’ being walking,
talking humans, as opposed to representations of a man
machine platform. This has been brought about by
advances in two dom
ains: (1) no
mot ion or limited
motion immersion, enabl
ed by affordable/capable
, part icularly widespread personal computing
most recently
including mobile devices and gaming
consoles) and
(2) whole body
immersive individual/team
simulators, ‘immersion’ referring to the capture of body
dynamics to co
ntrol the Virtual representation of the

Although both domains create an avatar of the participant,
avatar in the first case may be as little as a state
representation to indicate
‘speaking/typing/touch[screen]ing’ vs. ‘wait ing’, impose
on an animated picture of the participant or the selection
of other graphical representations from an available
library of characters. In gaming systems, simple
movement and object control states are also added to this
animation, these being controlled
by keyboard, mouse,
speech, touchcreen/active digitizer, or specialized devices
like gaming props and Kinect™
like video recognition.
In the second type of avatar generation, whole body
control of the avatar in immersive simulat ion
is by more n
atural (and robust) means, and

to meet the military adage
‘Train as you Fight’. The
end state of this
development would be similar to
the capability set forth as the holodeck of Star Trek

noting that there was no avatar
creation on the holodeck

(Live part icipants entered the Virtual space)
In this e
state, the trainee(s) would enter the
training space
with their objective tactical gear and ‘just do it’.

A similar non
avatar type of immersive Virtual training

does exist in our non
Star Trek world, this being the case
where Live trainee
s interact with Virtual entities

VH technologies with regard to creat ing the perception of
‘intelligence’ are identical to those in the Virtual world.
It could also be he
ld that this Live environment is
arguably the proper end state of this type of immersive
although it will be constrained by the inability to
replicate necessary exercise
specific influences in that
Live world (at least, that is, until the matter
holodeck technologies exist!).
These hybrid and
augmented realities are
not described here.

The first domain, physically non
immersive simulation,
benefits from the rapidly growing commercial
entertainment industry, although it is increasingly
ttressed by related telepresence industries (e.g., virtual
meet ings, training (including VH t rainers), telerobotics,
t command/control
). Furthermore, the mot ion
capture and speech recognition capabilities of gaming
consoles, along with the addit
ion of props for interaction
with the Virtual world (generally as weapons) have
significantly increased the immersive nature of these

The latter domain,
trainee immersion in
Virtual space via specialized instrumentation, continues to

exist as a high end military or specialized industrial
application. Features of these systems include full 3D
motion capture, including gross mot ion and fine gesture.

Although the experience shares many technical attributes
with the
increasingly interac
tive gaming applicat ions, the
intended training capabilities continue to differentiate
them. [4]

In the intended military team training application, various
immersive elements are crit ical to the training experience,
not all of which is oriented to the tr
ansfer of procedural
knowledge. The immersion attempts to be as realistic as
possible in emot ional, perceptual, kinetic, and cognitive
dimensions. This is not to say that increased simulat ion
fidelity is equivalent to increased training capability, but
here are elements of the simulation’s replication of the
range of military operations that must be attained if leader
development and team coordination are to be supported.
Note that these skills are not all physical or cognitive in
nature (e.g., leadersh
ip development, situational
awareness). Similarly, stress inoculation … ensuring that
a Soldier’s psycho physiological readiness for their first
firefight is supported by (and no worse than) their last
simulation … is a goal of such ‘high end’’ immersive

[5] These

degrees of warfighter immersion
have been largely attained in plat form simulator training,
but are only recently available (and still emerging) for the
individual Soldier, s mall unit, and small unit leadership.
Unlike the platform
simulations, the capability of the VH
that populate this type of immersive space is critical to the

For the purposes of this paper, the term ‘immersive
simulation’ refers to the extent to which contemporary
Virtual simulation can replicate
this desired Infantry s mall
unit ‘walk in’ end state, noting, however, that immersion
can occur in any type and fidelity of simulation, and that
the gaming industry would welcome the widespread
availability of a gaming console variant of the holodeck!


In the immersive space, the training goal is to both allow
the team to jointly hone their individual and team skills
but, just as importantly, to introduce thinking, reactive
adversaries and other human elements to the otherwise
ntly maneuver and weapons
based Live exercises.
The changing nature of small unit operations …
peacekeeping, humanitarian, disaster response, embedded
insurgents, coalition forces, etc. … requires this wide
variety of human
based interactions, and
even ro
in Live training have extreme difficulty approximat ing
these conditions, and at high cost.


The goals behind using Virtual roleplayers in small unit
(i.e., Fire Team and Squad) Live training exercises
include cost avoidance (e.g., real role

players take t ime
and money to train, coordinate, and utilize; using Soldiers
in this role is also suboptimal), session
standardization/replicability, and the ability to present
kinetic or socio
cultural settings that are otherwise
difficult, dangerous, o
r expensive to replicate. In a
training exercise emphasizing leader development, the
ability to use VH as the subordinates would also be

Unfortunately, the state of the art of Virtual roleplayers
limits their applicability in immersive simula
Development to make them compelling is expensive and
specific, and even then is presented in a medium
that is only nominally interactive, let alone immersive.
They are also only in their infancy with regard to being
able to react to any s
session variations of the
trainees or the session events/geometries. The applicat ion
of ‘reasoning’ VH in narrow contexts has been achieved,
but not with regard to the general application within the
available small unit immersive simulation.

That is,
the avatars have entered the space, but the space remains
unpopulated. Similarly, the vector in gaming applicat ions
has emphasized the kinetic interplay of combatants (and
their appearance), a dimension along which avatars and
VH do not vary gr

Ult imately, each element of human socio
behavior that is not supported by the trainees’ interactions
with the Virtual roleplayers in the immersive Virtual
space (to include the physical presentation medium) can
contribute to lowering the t
rainee part icipants’ overall
interaction acceptance threshold (i.e., their ‘suspension of
disbelief’), against which skills development may (or may
not) occur.

Our recent experience with full
body, full team, real
t ime
immersion in Virtual space

icates that the actions
(and reactions) of virtual roleplayers are more important
to the trainees’ degree of immersion than is the
appearance of these roleplayers, or the overall functional
bandwidth (‘intelligence’) of those roleplayers. For
example, a V
irtual character with precise facial
expressions or other high fidelity rendering, or one that
recognizes a large vocabulary, is … if a trade must be
made … less important to the training experience than are
roleplayers that the trainees perceives as react
ing to them
in an immersive, real
time, fashion.

It is as easy to list the characteristics of compelling
human interaction as it is to note the limitat ions
of VH. Indeed, the lists are the same:

Verbal communicat ion (understanding, not


Verbal communication in lieu of shared language

Gesture recognition (overt) and ‘body language’
(gross, as in posture or use of space, or fine, as in
hand motion or facial expression)

Perceived intelligence (a ‘thinking’
exhibited independently, or in
reaction to other VH state, or

in react ion to
trainee state)

These features of ‘intelligence’ are perhaps obvious, but
are often neglected as attempts are undertaken to add
‘higher level’ skills (e.g., ‘reasoning’, display of

affect) to
the VHs. Indicators of ‘purposeful’ behavior (i.e.,
reactive and seemingly intelligent) include, for example:

I walk in to a room and you look at me (or
another appropriate cultural response); I point at
someone, you look at them; When we tal
k, you
may move towards, or away from, me

In group
group settings, the dominant actors
in each group position uniquely (to each other
and to their subordinates), and their actions
overtly and covertly control the act ions of other
members of their group

I point my weapon at you, you take cover or
react otherwise … immediately

I talk or call to you and you orient to me, or react

The use of space is deliberate: in kinetic
situations, geometries are optimized and risk is
avoided; in non
situations, evidence of
purpose may be as simple as how and when I
cross a clearing or move from behind a table,
when I put down a piece of paper, etc.

If I am busy with a task, you (usually) would not
prompt me to begin another task

Behavior between party

members is coordinated,
and in manners the origin of which is not
obvious to me; this infers that there is a shared
perception or understanding between these other


First, the bounding box. The
‘quick fix’
recommendations that follow are not addressing those
contexts in which deeper VH intelligence is currently both
applicable and capable (e.g., specialist procedural training,
potentially including perceivable evidence of VH
reasoning such as nat
ural language processing, highly
detailed display of affective reactions, the ability to guide
trainee actions, etc.).

In the more dynamic, less bounded, and team
Infantry full spectrum operations immersive training
space, a stop
gap for the lim
ited utility of contemporary
VHs can include the introduction of a human controller of
the VH.

The use of humans to control simulation
entities in simulation training is not new. In the
Constructive arena, these roleplay
ers are often called


The change needed for VH control, however,
is related to the types of cueing the controller requires,
often calling for near
Virtual perception while immersed
in the environment, and also the nature of the VH
behavior, often non
kinetic, that is

desired to be
introduced. Current individual player/roleplayer tools
such as VBS2 allow a user to generate SAF, or to control
an entity as a user
controlled avatar. Entertainment
gaming systems do the same, including rudimentary
control by body position
, gesture, and speech. Recent
studies, such as those at the USMC Warfighting Lab at
Quantico, have assessed the ability of operators to control
multiple computer
generated entities, and have achieved
acceptable performance in ratios of 1:5
1:10 ‘controlle
virtual humans [8], with the operator assuming avatar
control of ‘needed’ assets when deviation from scripted or
otherwise limited VH behavior is required. However,
most of these successes are in the population of traditional
kinetic engagements typica
l of the documented battle
drills, as opposed to excursions into the human dimension
of increasingly diverse and autonomous small team

In this reminder of the desired immersive Virtual training
time, the paradigm for building SAF platforms and

capable gaming adversaries/teammates needs to
change [in the author’s opinion] in emphasis from adding
tactical functionality or visual fidelity to, rather, the
addition of a number of available ‘human CGF’ specific
attributes … of which the subse
t of Virtual t raining that is
defined by small unit team training in a Full Spectrum
Operations context is almost 100%, given the current state
of VH intelligence/ responsiveness in these contexts.
Only in shoot/no
shoot situations do most VH play their
ole well, at least in free play small unit team training. Of
course, as VH ‘AI’ capability or visual fidelity increases,
it would be added, too.

Many of these ‘quick fix’ solutions are just ‘checks’ to
make sure that a VH’s dumb behavior is not made
us. These include:



to me


to others


to events

Immediacy of response


At least as critical as the accuracy of



Scripts must ensure variability




subtype of respo
nsiveness and
unpredictability that
can be
‘deep’, but
there are simple approaches/checks that
allow multiple VH to demonstrate
coordinated perception or purpose

Motion is natural (and not obviously driven by a
library of oft
repeated scripts … no response is
er than the same one)

Motion is purposeful with regard to terrain and


This does not have to equate to on
mission planning/replanning.

This and other types of goal
behavior are the subject of ‘big AI’,
the computational field (e.g.
, NP
complete plan optimization, goal
directed production systems, pattern
recognition, etc.), but are not the only
approaches to executing purposeful
and useful behavior based
on a
momentary set of variables

Preprocessing of the environment for

consumption by the VH is
readily available, and includes not just
simple trafficability analysis, but can
include preprocessing of spatial
overlays for available
cover/concealment, kill zones,
ingress/egress routes (under varying
and dynamic
modification of these preprocessed
zones/paths/objects given the VH and
avatar position/situation

So, without adding ‘real’ AI to the VH so as to increase
the human participant’s perception of that VH intelligence,
nor adding the hard
‘real’ facial, gesture, and
emotive expressiveness … all of which have a long way
to go before they pass the Turing Test … how can we
maintain the degree of immersion that is otherwise only
possible with interaction between only human

The simplest solution is to have human controllers of the
VH. If, however, this is on a 1
1 basis, they should just
have been introduced as avatars in the first place, as no
cost savings would otherwise have been attained. The
possibility exists that
a ‘service center’ can apply a
limited number of human controllers to the somewhat
predictable number of needed roleplayer avatar instances
in a scenario, or set of concurrent scenarios, including
from remote centers and by operators that are both SMEs
the training and in the
. This would be,
however, both complex to execute and of limited cost

Tethering VH to avatars in entertainment systems is
common practice, but then suffers from most of the
difficult ies that make the VH so eas
y to ‘pick out of the
crowd’. The more complex the situation (i.e., in other
than kinetic force
force engagements), the more
limited is the ability of the avatar to impart meaningful
independent behavior to the tethered VH.

If, however, the degree o
f immersion of the VH
controlling avatar in the Virtual world is high, the VH in
the environment can be cued by that avatar’s human
perceptual abilit ies and reasoning (responsive actions).
This can be done without the VH being ‘tethered’ to the
avatar in
the conventional Computer Generated Forces
(CGF) sense. By ‘degree of immersion’, the reference is
to the immediacy of the VH ‘controlling’ avatar’s (or
avatars’) ability to perceive the actions of the trainee
avatars and to respond to those perceptions.

Ideally, the
immersion would be full body, free motion, with binaural
sound and excellent visual immersion (e.g., HMD or 360
degree projection). If the controller is a typical
based avatar controller, the cueing provided
by the ‘watch me’ VH
controller would be more limited.

Control over the VH by Virtual roleplayers, who can see
and understand the trainees, is distributed to by both the
avatar’s movement and commands. Although notionally
only a single avatar is required to add the VH ‘perce
and control, there is no limit on the number of controller
avatars, subject to the aforementioned concern regarding
cost. The avatars can be portrayed as needed by the
exercise (uniform/dress, race, gender, etc, even voice

Under the
assumption that the VH controller is immersed
in a manner similar to those now available devices that
place trainees ‘in’ (not in front of) the Virtual
environment … such as the VIRTSIM, CombatRedi,
ExpeditionDI, etc. systems [9], examples of the cues that

the controller can provide to the VH include (as

Responsiveness, and immediacy of response:
Add cued motion or orientation (e.g., Trainee to
whom my attention is focused, places that I am
looking, direction in which any required group
should be oriented, and even overt
gestures to specific VH for actions to be taken.

Unpredictability: The ability to task, from within
the simulat ion, specific VH promotes
unpredictability. Similarly, the queuing of VH
behavior by the controlling avatar l
ends to
inherent human unpredictability, as well as the
opportunity to overtly inject different VH

Coordination: Tethering to the immersed
controller generates one level of coordinated
behavior, and the specific tasking of individual
VH yet ano
ther. To the extent that it is possible
to ‘hide’ from the trainee avatars the nature of
who is the VH
controlling avatar and who the
VH are (including the use of subtle commands
the trainees may not notice), the more dynamic
the interchange with the tota
l ‘group’ becomes.

Purposeful mot ion: Without the aforementioned
preprocessing of space with regard to
alternatively weighted movement ‘profiles’, it is
likely that VH movement, even if cued or
commanded by an immersed controlling avatar,
will continue to
‘give them away’. This can be
avoided by careful scripting of the scenario and
by the controlling avatar’s familiarity with the

Given a high degree of cued VH performance, and when
the roleplaying immersed avatar understands the scenario
and t
he VH capabilities, it is possible to conceal the
avatar vs. VH nature of the entities in the environment.
That is, each of the non
trainee players in the scenario
could be a human, or a VH. The benefit is that the trainee
avatars must attend fully … at
least initially (until the
limitat ions of a VH are ‘uncovered’) … to all entit ies, not
just the one(s) that may be thinking, reasoning humans.

In the absence of an immersed human controller (or
controllers) of the VH, there are maxims to observe in
ting the Virtual space if it is to become an
immersive dynamic decision space, demanding (and
allowing) meaningful interaction with the VH population.

Move when someone talks. Although mediated
by cultural norms, this typically includes looking
at the

Nobody stands still if a weapon

is pointed at
them. The response is immediate. Hands will
go up, talking will start, movement to cover will
be initiated, something. Anything.

Not limited to the cases above, VH must
immediately respond to avatar
s. It is possibly
more important that the response be immediate
than it is that it is appropriate. And, note that a
late response that is reasonable, but late, is not
appropriate if the intent is to continue the users’
immersion. The response does not h
ave to be
‘intelligent’ to at least demonstrate that real
reasoning has been triggered. That is, knowing
that it is appropriate to do

is half the
game. That said, in situations such as structured
interviews where timing is less critical, t
introduction of a delay … such as those
associated with the processing of affective
presentation or natural language processing …
can still allow t raining to occur, but these delays
are probably not appropriate for infantry small
team immersive training

Behaviors should not be repeated. Triggered
scripts, including position/posture changes, must
be spaced appropriately. This requires that the
script be triggered with some complexity (at
least with an awareness of the prior incidence of
the script) and

even possibly recursive.

As mentioned above, purposeful use of terrain
and objects implies ‘intelligence’. Much of this
can be preprocessed and applied at runtime. The
nuances of character positioning and other
dynamic events in those ‘response spaces’
(notionally as precomputed weighted space
grids/voxels) also allows for variability in
otherwise scripted behavior.

VH should respond to each other. Much of the
cueing mentioned above for the ‘VH
avatar’ is also available for VH
tion. If one entity stops what they’re
doing to orient to an event, a second VH that
may not have been triggered to that event can
still be triggered by the VH response. Of course,
the nature of shared responses can vary from the
simple to the complex, s
uch as that addressed in
the literature on agent
based behavior
(cooperative and self
based) and computer
supported cooperative work. The argument for a
like perceptual and spatial ‘front end’ for
initiat ing intelligent VH response is addressed as
ooperative perceptual models. [7] The
argument is made that endowing VH with a
reasonable understanding of space, perceptual
acuity, attention, and ‘internal’ filtering (object
knowledge and context
driven attenuation) is
useful in producing human
like V
H reactions to


Based in part on our experience interacting with other
humans in the immersive Virtual space, which was
surprisingly different than interacting with other humans
in gaming styles of Virtual space, recommendations f
the nature of VH in this more first person and complex
environment emerged.

Not all behavior to impart perceived ‘human
ness’ to VH
is related to intelligence. This includes not only motion
and demonstrations of affect, but also unintelligent
or. We are, admittedly, human! It is not only
acceptable, but possibly preferred, that VH perform … at
times … in apparently incorrect manners. (This, by the
way, is one of the cited faults of the Turing Test, in that it
is a measure of perceived intell
igence, not humanness.)

The intent is not to argue whether a VH can imitate a
human vs. cause an observer to believe it was human, nor
to measure VH IQ, but, rather, to point out that the
dynamics of a VH are complicated in a different, ‘human’
n, when compared to the Virtual and
Constructive modeling of warfighting platforms with
which we have grown comfortable in our development of
CGF, and even with regard to less immersive styles of
team Virtual exercises.

Similarly, in an attempt to ‘repl
icate reality’, there has
been little research on those factors of the immersive
Virtual space (as defined here) that are most salient to the
training intended to be performed in this space. This is in
part due to the relatively new ability to immerse tea
ms of
participants into this type of Virtual space, but also
because of the many successes achieved in *other* types
of immersive environments and with other types of VH,
which may or may not apply, or which may need to be
applied in a tailored manner. Si
milarly, the nature of
interpersonal interactions in is likely to place
developmental demands on distributed simulation
protocols such as LVC

The observations and recommendations presented here
have assumed that all players

are in the Virtual space.
The same mechanisms for increased avatar/VH
interaction can be introduced to Live training
environments in which a Virtual player/players are
introduced, whether those players are avatars or VH. In
the case where the immersed V
controlling avatar is in
both their Virtual world (beside and behind them) and in
the Live world (in front of them, via a video feed, raw or
augmented), there is very little difference from the all
Virtual example, other than the types of influence that
‘permeate’ the Live/Virtual boundary. Even that
boundary is quite permeable, with the proper
instrumentation of the presentation area.

A final observation: the freeplay nature of immersive
simulations must be tempered by the need to provide
zed training. The flexibility of Virtual
simulation can allow for a learner
centric pull of training,
in which exercises may be repeated, modified, interrupted,
and replayed as needed, and with automated feedback.
However, the scenarios (including backst
ory, VH
behavior, etc.) must be a part of a skills and knowledges
progression, much of which occurs not only in other
simulations (Live, Virtual, and Constructive) but in
classrooms and computer
based individual training. A
significant challenge lies in t
he assessment of Soldier,
unit, and
leader performance

(and training progression) in
the types of simulation
based training that has been
discussed here, across both the gaming and highly
immersive variants. Providing (and managing) this
training architec
ture is the intent behind much of the
recent issue of an Army pamphlet on training change
vectors over the next 5 years. [11]


[1] The holodeck was introduced in the pilot episode of Star
Trek: The Next Generation
, although it has been
ed in an episode of the animated version of Star
Trek that followed the original television series.
Similar technical applications had been envisioned in
earlier scientific, science fiction, and other popular
media, but none as pervasive nor as technicall
complete as the holodeck. The following reference
provides not only the history of the holodeck, but
describes many aspects of immersion in Virtual space
complementary to the discussion of VH addressed in
the body of this paper.

, Holodeck, 13
May 2011.

[2] Turing, Alan

(October 1950),
"Computing Machinery and


(236): 433


Note that the fact that a teletype was proposed as the
interface mechanism by Turing can readily be
extended to the presentation of both the avatar and the
VH into Virtual space, in that the ability to present
both can be made equivalent. The degree of mo
vs. transduced motion is otherwise the only difference,
and can be made equivalent for the purpose of such a

[3] Balcisoy, S. Kallmann, M. Torre, R. Fua, P. Thalmann, D.,
Interaction Techniques with Virtual Humans in Mixed


International Symposium on Mixed
Reality, Tokyo, Japan, 2001.

[4] Although there are many bodies of research regarding the
use of Virtual simulation for training, and particularly
dismounted infantry training, a particularly
comprehensive bibliography
was compiled by the
Army Research Institute as their contribution to the
NATO Research and Technology Organisation
compendium final report regarding “Human Factors
Issues in the Use of Virtual and Augmented reality for
Military Purposes,” as:

Goldberg, S
teve, Knerr, Bruce,
et al,
I (pp 7
19/28), Dec 2005 (also as ISBN 92

[5] The
to use immersive training for individual and small
unit training has appeared increasingly in the writings
of senior Army and Marine Corps
leadership. Rather
than cite one or more of these calls for action, the
following reference provides what may initially appear
to be a tangential analysis of the domain that is
immersive simulation, including application not just as
skills training, but a
s a method to inoculate against, or
treat, stress reactions. The proceedings address
required attributes of the immersive space, including
the intelligent adversaries and other VH participants.

line proceedings of the 2010 Navy Marine Combat
onal Stress Control Conference (COSC), San
Diego, CA, 18
20 May, 2010.

[6] A comprehensive analysis of Live and Virtual simulation for
Infantry small units was conducted in the recent Joint
Capability Technology Demonstration (JCTD)
“Future Immersive Training Environment” or FITE.
The FITE fi
nal report assessed not only the Live and
Virtual experiments conducted during the JCTD, but
also included a survey of available immersion
products and technologies.

Joint Forces Command, Spiral 1 FOTE JCTD,
“Limited Joint Operational Utility Assessment

June 2010.

[7] Although reference to whole body Virtual immersion via
means of networked camera participant digitization is
made in [6], the surveyed technologies did not include
the new product by Motion Reality, Incorporated
Prior camera
based, arbitrary
volume traversal technology was either not real
or not multiple participant. VIRTSIM is both. The
technology used by the VIRTSIM device is an
extension of the technology licensed by MRI to Giant
Entertainment, and with
which the Academy Award™
for Technical Achievement was won by MRI in 2005
for their work mapping human motions to the Gollum
creature, and with which the Avatar movie generated
the human
based character animations. The
observations made by the author as
reported in this
paper are from exercises conducted in the VIRTSIM

[8] The Navy and Marines have conducted research on the
number of controlli
ng humans that are required for the
introduction of intelligent behavior in both VH and
USV robotics. The cited ratios are from conversation
between the author and representatives of the Marine
Corps Warfighting Laboratory (MCWL, Quantico)
regarding unpub
lished work.

[9] The VIRTSIM full
body, real
time team immersion was
addressed in [7]. Additional approaches to body
motion capture and team immersion training exist.
The two cited here use head tracking and body
position/posture sensors, along with weapon
comotion input devices, to approximate full
motion input to the avatar. These and other
technologies are addressed in [4], [5], and [6]. In
addition to the instrumentation used to capture motion
in those systems not using natural motion (i.e., all b
the VIRTSIM device), the algorithms to map the
sensor and input device C2 to the motion of the avatar
can vary, as described by Templeman,
et al

Templeman, J.N., Sibert, L.E., Page, R.C., Denbrook,
P.S. (2006) Immersive Simulation to Train Urban
try Combat, in
Virtual Media for Military
(pp 23

16). Meeting Proceedings
136, Paper 23, Neuilly
France: RTO.

[10] Herrero, Pilar, de Antonio, Angelica, A human based
perception model for cooperative intelligen
t virtual
Facultad de Informática. Universidad
Politécnica de Madrid. Campus de Montegancedo S/N.
28.660 Boadilla del Monte.
Madrid. Spain

[11] TRADOC Pam 525
2, The U.S. Army Learning concept
for 2015, 20 January 2011.