V i r t u a l R e a l i t y A r t a n d E n t e r t a i n m e n t

juicebottleΤεχνίτη Νοημοσύνη και Ρομποτική

14 Νοε 2013 (πριν από 4 χρόνια και 6 μήνες)

101 εμφανίσεις

Virtual Realit ydanrt En tertainmen t
Joseph Bates
Sc ho ol of Computer Science
and College of Fine Arts
Carnegie Mellon Univ ersit y
Most existing researc h n o irtual v reality concerns issues close to the
interface primarily ho w t o present an underlying sim ulated w orld in a
con vincing fashion Ho w ev er or f virtual realit ytoac hiev e ts i promise as a
ric h and p opular a rtistic f orm as ha v e the no v el inema c and television
w e b eliev e it will b e necessary to xplore e w ell b ey ond the interface to those
issues of con ten t and st yle hat t ha v e made raditional t media so p o w erful
W e presen t a case for the imp ortance o f t his researc h then outline sev
eral t opics w e b eliev e are c entral to the inquiry dev eloping computational
theories for cognitivemotional agen ts presentation t s yle and drama
The F an tasy Mac hine
A writer of p opular science articles visits a ab l oratory to learn ab o ut the new
irtual realit y researc h The writer is ushered in to the d emo ro om and w arned
to mo v e slo un til adjusted o t the system The goggles lip s on then the data
glo v e and a switc h is t a w orld app ears the writer h ead turns and
slo wly the ligh t o f understanding d a wns
Three mon ths later the article app ears op ened m yey es and had b een
transp orted in time and space I ould c lo ok around and lmost a f eel things with
m y glo v ed hand as if I w ere l iving i n a diren tw orld Hollyw oo d w atc h out
so on VR will let us en ter into f tasies W el b e able to go a nywhere and do
an ything imaginable
The public is b eginning to understand that v irtual realit y p ortends a new
medium new en tertainmen t a n ew and v ery p o w erful t yp e of art It has a

Accepted t o Pr esenc e The Journal of T op er ators and Virtual Envir onments MIT Press
This is a revised v ersion of eep Structure for Virtual Realit y arnegie C Mellon c S ho ol
of Computer S cience echnical t rep ort MUS C

an our
As hrown
p oten tial that b c e k ons to the a v erage p erson as w ell as to the scien tists dev eloping
the tec hnology The vision of this writer to et l us o a n ywhere and d o n a ything
is an importan t ne o Indeed the wish to b uild suc ha fan tasy mac hine is the
fundamen tal driving force for man y VR researc hers
Ac hieving this wish requires us to examine i t closely s A engineers w eneedto
kno w hat w sorts of considerations re a in v olv ed in constructing suc ha mac
Do the goggles and glo v e bring us most of the w a y to success Some of the w a y
Let us consider sev eral highly d ev elop ed successful edia m for presen ting fan
tasy w orlds no v els television and lm S ome p eople are i n terested in w orlds
solely in v olving ph ysical space ob jects a nd natural f orces Films ab ut o these
topics are usually o d cumen taries with narration with an o casional c art lm pro
duced Ho w ev er almost alw a ys television mo vies and o n v els include at least
three other k ey elemen ts
First there are living creatures usually h uman and usually m e b o dying some
in telligence and emotion These let the v iew er see the w orld as a lace p of life
purp ose and feeling Second there is long term structure to the ev en ts p ortra y ed
whic his tosa y that some ind k of story is told The story giv es in tensit y and
meaning to the w orld Finally he w orld is presen ted in an ectiv e emotionally
po w erful st yle Cinematic and narrativ etce hnique are highly dev elop ed examples
of the art of presen ting w orlds
Eac h of these asp ects of traditional media la p y a great role in allo wing the
author to act the view er emotionally and i n tellectually ac king any one the
author will ha v e sp ecial diult y con v eying a ric h signian tw orld Th us when
our writer sa w ewlilsoongo na ywhere and d o n a ything these k ey elemen ts
of traditional media are c ertainly implicit in the vision
Let us no w c onsider the primary researc h directions n i o ur v ery y oung ld of
virtual realit y Most fo the w ell k no wn w ork concerns the in terface hardw are and
soft w are Dev eloping he t goggles and g lo v e and getting them to w ork together to
presen tan y kind of w orld at all has b een a attle b Muc h existing w ork suc has
that at NASA and U NC isher et l a Bro ks o has b een substantially
concerned with solving this problem
Animating the structure and b eha vior of ph ysical w orlds is also an activ e
researc h a rea d irectly relev nattoVWR ork on ph ysically ased b mo deling for
animation suc h as that rep orted n i Badler et al Witkin nd a W h
McKenna et al and elsewhere ma ymak eitm uc h easier to p ortra y and
sim ulate f amiliar ph ysical w orlds including he t ph ysical b o dies and lo w lev el
b eha viors of agen ts
Y et lo oking at what mak es traditional m edia so p o w erful for view ers and th us
what ma y mak e v irtual realit y a gen uinely p o w erful a nd p pular o artistic form
w e a re struc kb y the absence of researc hin to the VR analogs f o our three k ey
areas computational models of cognitiv emotional agents long term dramatic
structure and presen tation st yle While an insigh tful f ew ha v e suggested the im
p ortance f o hese t reas a sp ecially see Brenda Laurel w ork Laurel Lau


rel t o o ur kno wledge the AAI A W orkshop on In teractiv e iction F and
Syn thetic Realities ates b is the nly o p lace that w ers in the relev
lds ha v e come together to seriously discuss broad researc hto w ard i nhabited
dramatic virtual w orlds
In the e arly da ys of our ld erts it is natural that lmost a ll a w ork has
fo cused on the hardware n i terface and system soft w are Indeed some virtual
w orlds u ndoubtedly will exist solely to provide an apparently real p h ysical space
for h umans to inhabit The notion of yb erspace presen ted so orcefully f in the
w orks of William Gibson is of this c haracter So to o is t he use f o virtual realit y
as a v isualization tec hnology i n m edicine science arc hitecture and lsewhere e
If one iews v a VR system as pro ucing d surface lev el phenomena v ia the n i ter
face and asso ciated soft w are then the o rganization and con ten tof soft w are w
b ehind he t in terface constitute a deep structure f or the irtual v w orld This
is the arrangemen t f o c o de and data that pro duces the essen tial meaning of the
in teractiv e xp e erience F or the reasons suggested ab o v e w e b eliev e that for VR
to join the no v el cinema and television s a a b roadly successful artistic medium
the tec hnology m ust pro vide a suitably ric h eep d structure particularly con tain
ing computational theories for agen ts presen tation and rama d T his structure
is largely missing from existing VR w ork and is an area whose dev elopmen tw e
wish to encourage along with w terface and h p ysical mo deling to elp h
VR ac hiev e its p ten o tial of letting s u go an ywhere and d o anything
An Approac h to the Problem
Ha ving suggested that there a re applications o f v irtual realit y t r a
certain kind of deep structure let us examine ho w t i migh tbepor vided W e
no w b e f o cused solely on those w orlds that d emand this particular approac h
A v irtual w orld m ust esp r ond a ppropriately to u sers without assistance from
its original creators This means that agen t b eha vior dramatic on c ten t and
presen tation st yle m v ary a ccording to explicit a rtistic odels m b uilt i n to the
w orld b y its creators h T us t he primary task in building a w orld is to transfer
artistic kno wledge o f b oth general nd a sp eci kinds in to the m ac hine
The study of kno wledge represen tation ro p cessing and transfer i s ne o of the
cen tral topics of artiial n i telligence researc h W e b eliev ethat m uc h existing AI
tec hnology or f example n i cognitiv e modeling agen tarc hitectures story u nder
standing natural language pro cessing a nd adv ersary searc h can help pro a
framew ork for the deep structure w e s eek Note that this is primarily a problem
of applying kno wn AI results for t he b ene of VR W e do n ot prop ose to gain
lev erage on the undamen f tal roblems p of AI b y examining them in the con text of
Muc h f o this existing artiial n i telligence researc h aimed at building in telli
gen t systems for the real w orld has b een criticized for w orking only in small nar
equire hat
in on ork
ro wly deed micro orlds But virtual w orlds a re precisely suc h icro m orlds
so this is not an o b ection j as long s a the virtual w orlds are ric h nough e to hold
artistic in terest The question then is whether the b est of the created w orlds are
of suien t qualit ytow arrant the term art a nd whether existing tec hnology
is ric h enough to supp rt o con ued rtistic a nno i v ation within the m edium This
question eems s to us answerable only b y exp erimen t
The Oz pro ject at Carnegie Mellon i s attempting to p e rform this exp eri
men t W e are a group of facult y sta studen ts and visitors from the Sc ho ol of
Computer cience S the College of Fine Arts and elsewhere O ur goal is to bring
together appropriate existing tec hnologies esp ecially AI tec hnologies and on this
framew ork to help artists build dramatic sim ulated w orlds that include sim ulated
p eople Oz th us is an attempt to pro vide a eep d structure f or virtual realit y and
to explore the p o ten tial of v irtual w orlds built o n suc h foundations
Three en C tral Concerns
T o elp h con v ey our v iew of what i s needed in this researc h rea a w ewlli sk etc h
brie the approach tak en in Oz to the construction of computational theories
of agen ts presen tation and rama d T he in ten t here is to illuminate the areas of
in v estigation r ather than to presen t sp eci researc h results
Cognitiv emotional Agen ts
One of the k eys t o an ectiv e irtual v w orld i s for the user to b e able to susp end
disb elief T hat s i the user m ust b e able to imagine that the w orld p ortra y
real without b eing jarred out of this b elief b ythe w orld b eha vior
This requiremen t can b e turned to our a dv antage f i w etak e a nonraditional
view of the p roblem of building n i telligen t agen ts Instead of demanding that our
agen ts b e esp ecially activ e and smart w e require only that they not b e clearly
stupid or unreal n A agent that k eeps q uiet ma y app ear wise while a n gen a ttath
o v ersteps its abilities will probably destro y t he susp ension of disb elief Th us w e
prop ose to tak eavd an tage of the Eliza e ct eizen baum in whic h
p eople see ubtlet s y understanding a nd emotion in an agent as long as the agent
do es not activ ely destro y the illusion
In order t o foster this illusion of realit y e b eliev e ur o agents m ust hav e
broad though p erhaps shallo w capabilities T o this end w e are attempting t o
pro duce an agen t arc hitecture that includes goals nd a goal directed b eha vior
emotional state and its ects on b eha vior some natural language bilities a s
p ecially ragmatics p based language generation and some m emory and i nference
abilities Eac h of hese t capacities can b e as shallo w s a i s ecessary n to allo wusto
build a broadly capable in tegrated agen t
This approach is unlik e most AI e rts to build agen ts whic h enerally g result

is ed
in deep ut b narro waeng ts Rather than attempting to driv ean y particular asp ect
of agen tarc hitecture to ew n highs w e a re trying t o in tegrate existing tec hnology
in to a single un usually broad arc hitecture This is a l ittle s tudied area and w e
are hop eful that moderate ert in in tegration m a y y ield go o d results suc has
b eliev able hin ts of thought and emotion in our limited m icro orld domains
Some w ork n i this irection d w as discussed at the AAAI Spring ymposium S
on In tegrated In telligen t Arc hitectures V ere a nd Bic kmore B ates et al
a More detail o n our particular arc hitecture called T ok is pro vided in
sev eral tec hnical rep orts an tro witz Loy all nd a Bates Bates et al
Presen tation
It is clear that presen tation st p ys a cen tral role in traditional media In m
the feeling of the underlying w orld is greatly nenced i b y the ligh ting camera
angles fo cus editing and so on Music la p a k ey role in foreshadowing and
emphasizing the moo d of the visuals
The st ms w ere little more than v isual records of traditional theater and
naturally visible scenes As time passed and lm mak ers in v en ted new w a ys of
sho wing the cinematic language o f presen tation b ecame ric her Early audiences
accustomed to a certain kind of realit igh y tw ell not understand asp ects of
modern m t ec hnique
One can imagine a case b eing ade m i n the early da ys of cinema that suc ha
photographically accurate medium as lm could ot n p ssibly o em b o dy an ysbu
jectivit y This a rgumen t that m tec hnique made no conceptual sense of course
w ould ha v e urned t out to b e wrong
One can imagine the same kind o f ev olution in the art of virtual realit y While
VR app ears to b e a rst erson ealistic form w ebeeliv e that a language
of presen tation w ill dev elop o v er time omeone S accustomed to ha ving normal
con trol o v er their b o dies and p erceptions igh m t nd the artial p lac kof onc trol
implied b y suc h a dev elopmen t confusing and p erhaps unpleasan t w er w e
susp ect that a language of presen tation for R V w ill allo ww orld builders to enric h
the xp e erience for participan ts W orlds a v oiding suc hst yle a m y come to app ear
dull and emotionally t
The Oz group has made some ert to understand presen tation in the c on
of narrativ e W e are exploring language eneration g under t s ylistic constrain ts
suc has hte w ork b yHo vy o vy and generation arc hitectures that in te
grate text planning and urface s structure generation Our initial w ork is describ ed
in mith and Bates Kan tro witz but this w ork remains v ery pre
Language based presen tation researc h h as relev ance to virtual r ealit y but
there is clearly a n eed to study and automate visual st yle W e see the w ork of
Witkin and K ass on spaceime constrain ts as an excellen t example itkin and
ev Ho

Kass By adjusting simple con trol arameters p o f a ph ysical mo del they
w ere able mec hanically to repro duce certain famous asp ects of Disney st yle anima
tion W e op h e nimation a researc hers will con ue to d n i terest in automating
artistic st yles used in the visual presen tation of sim ulated w orlds
Film of course go es b ey ond the visual and i t eems s clear that m usic sp eec h
and other sound will b e similarly i mportan t o t the broad success of virtual realit y
A particularly relev an t and to our kno wledge unstudied researc h t opic w ould b e
generating sound as a b yro uct d o f ph ysically based animation
A al v ariation of the theme of p resen tation is the p ossibilit yofin terac
tiv e radio drama This is a r elativ ely l o w b andwidth p oten tially high impact
w a yofprnees ting a v irtual w orld W ekno w of n o existing w rkio ndve eloping
computational theories of presen tation for radio drama
Some w orlds are b est modeled as free running s im ulations The ph ysical s etting
and the c haracters re a enough to express the creator v ision Ho w er at least
some of the ime t the creator will w t to mpose i l onger term structure on the
ev olution of t he w orld nd a this in the resence p of the freeilled a nd activ e user
It is the job f o a computational theory of in teractiv e drama hic hw e refer to
here as the irector to m ak e this p o ssible
If the user had no inence on the w orld the creator could u se the riter w
traditional method of carefully planning the exact sequence of story v e en ts Ho w
ev er the ser u is not a p e ripheral elemen t Indeed usually the whole purp se o of
the w orld is to giv e the activ e user a ertain c kind of exp erience Th us the creator
m ust con v ey to the director enough g eneral i nformation ab out stories and enough
sp eci information ab out the in tended kind of exp erience to allo w the director
to ac hiev e the creator g oals This m ust b e d one n i suc haw a yasto ela v e the
user with an undiminished feeling of freeill
Ho w can w e v iew the elationship r b et w een the director and the user One
approach is to see them in a k ind of t w olay s has c hess The irector d
and user are aking t turns the u ser acting as a free agen t n i the w orld he t director
lo oking do wn from ab o v e and v ery gen tly pushing the elemen ts of the w orld in
v arious w a ys The director is constan tly trying to m aximi ze the c hances of a
pleasing o v erall exp erience no matter what the ser u do es along the w a f yw e
elab orate on the c hess analogy the pushes of the d irector and he t actions o f the
user are the mo v es of the game The irector d wins if the complete istory h of
the w orld is consisten t w ith the creator aesthetic goals thereb y presumably
pleasing the user
Computer c hess programs based on adv ersary searc hha v e radually g come
close to the capabilities of the w orld b est h uman pla y ers The t w olay er game
model for in teractiv e d y b e menable a o t the same inds k of searc htec h
nology and migh t yield similarly mpressiv i e r esults
ma rama

uc game er
F ormalizing the drama game for adv ersary searc h r equires us to mak e p recise
the mo v es and the ev aluation function The rst h as b en e studied n i t he form
of plot units b y raditional t narrativ e theorists nd a b y some computer scien tists
ehnert Leb o witz Dy er The second requires a theory of
the n ature of go o d stories whic h as h b een discussed also to some xten e t Laurel
Leb o witz Dy er
One o f the problems that arises in this o m del is the large n b r e of e
grained actions that ma y o ccur during an i n teractiv e exp erience p ossibly on the
order of thousands I n addition there is a wide v ariet y o f p ossible suc h actions at
eac h momen t T ogether these pro uce d a deep searc h with high branc hing factor
a computationally diult t ask
The diult y f o searc h i n the space of rimitiv p e a ctions suggests that w e t
searc h instead in a space of abstractions suc h as the plot unit space Ho w ev er
abstract adv ersary searc h is a tec hnique that is not w ell understo o d P erhaps in
compensation a p ossible adv an tage here o v t ypical t w ola y er games is that
the ser u is not particularly trying to m ak e the director w in or lose but is instead
mo ving through the w orld in ten t n o p ersonal goals Th us the director is really
not facing an adv ersary and a s hallo w searc hma y sue
Besides the computational complexit y of the problem a n atural oncern c ab out
an y heory t of drama is the exten t to whic h a director can inence the w orld with
out attracting the atten tion of the user T o study his t ssue i w eha v e p erformed
sev eral sim ulations of this odel m for v irtual w orlds placing a user in toarael
ph ysical en vironmen twithliv e actors and ith w a liv e director hidden a w ay from
view elso and Bates
Our reliminary p results suggest that he t director inence easily can b e
in trusiv e but that substantial inence can b e subtly expressed b y modifying the
actions of agents W e ound f that p eople engaged i n n a exp erience asily e accepted
o dd b eha vior in the actors ev en when that b eha w as judged as irrational or
manipulativ eb y third parties giv en access to the whole picture In retrosp ect
this particular method for wielding p o w er in subtle w ays ma y b e lready a w
kno wn in so ciet y
The Oz group is activ ely studying sev eral computational models of drama
including the adv ersary searc h o m d el ates a W e are excited b y the
p ossibilit y of ectiv e c omputer p erformance in this un usual omain d
Virtual realit y dev elop ed out of the tec hnical comm y from a vision of what
w as tec hnically p ssible o and from the requiremen ts of certain tec hnically demand
ing applications Of course some of the creators had visions o f applications far
bey ond the needs of their funding sources b ut generally the comm unit y h as ex
plored virtual realit yas a h umanomputer in terface tec hnology
W e s ee this fo cus on in terface s a something lik e studying celluloid nstead i of
cinema pap er instead of no v els catho e d ra y t ub es instead of television hardware
instead of soft w are While the in terface is importan t w em ust also lo ok b ey ond
it to the underlying structure of the w orlds w ew an t to odel m T raditional arts
and media pro vide excellen tealxmpes w orth y o f o ur insp ection S ince VR is new
it is natural that w eha v en explored the whole p roblem y et but w em ust indeed
go forth and carry ut o a road b exploration if w ew t virtual realit ytoac hiev e
its promise of letting us o an ywhere and o d n a ything
Ac kno wledgmen ts
W e ratefully g ac kno wledge t he supp ort of F ujitsu Lab ratories o Ltd for our
researc h
Badler N W ebb er B Esak o v J nd a Kalita J Making Them
Move Me chanics Contr ol and A nimation of A rticulate d Figur es Morgan
Bates J omputational C drama in Oz In Working N otes of the AAAI
Workshop on Inter active Fiction and ynthetic S R e alities Boston A M
Bates J editor Working otes N of the A AAI Workshop on Inter ac
tive Fiction and Synthetic R e alities Boston MA American Asso ciation of
Artiial In telligence
Bates J Lo y all A B and Reilly W S B road a gen n I Pr o c e e dings
of AAAI Spring Symp osium on Inte gr ate d Intel ligent A r chite ctur tanford S
CA Av ailable as a sp ecial issues of SIGAR T News Summer
Bates J Lo y all A B R eilly W S and anK tro witz M b T ok A
broad rc a hitecture for in telligen t emotional gen a ts T ec hnical rep rt o S c ho ol
of Computer cience S Carnegie Mellon niv U ersit y Pittsburgh P A
Bro oks Jr F Grasping realit y through i llusion i n teractiv e g raphics
serving science In CHI Pr o c e e dings pages
Dy er M Inepth Understanding T he MIT ress P Cam bridge MA
Fisher S McGreevy M Humphries J nd a Robinett W
en vironmen t displa y system A CM Workshop on Inter active
Gr aphics p ages Chap el Hill NC

Ho vy Gener ating Natur al L anguage under Pr agmatic Constr aints
La wrence Erlbaum Asso ciates Hillsdale NJ
Kan witz M Glinda Natural anguage l text generation in the O z n i ter
activ e ction pro ject T ec hnical Rep rt o CMUS Sc ho ol of Computer
Science Carnegie Mellon Univ ersit y P ittsburgh P A
Kelso M and Bates J Liv esim ulation of a model for dramatic virtual
w orlds W orking P ap er Oz Pro ject Sc ho ol of Computer Science arnegie C
Mellon Univ ersit y P ittsburgh P A
Laurel B T owar d the Design of a Computerase d Inter active F antasy
System PhD thesis Drama Departmen t Ohio State Univ ersit y
Laurel B Computers as The ater Addison esley Publishing Compan y
Reading MA
Leb o witz M Storyelling a s lanning p a nd learning Po etics
Lehnert W Plot units and narrativ e ummarization s Co gnitive Science

Loy all A B and Bates J Hap A eactiv r e adaptiv e arc hitecture for
agen T ec hnical Rep ort CMUS c S ho ol of Computer Science
Carnegie Mellon Univ ersit y P ittsburgh P A
McKenna J Piep er S and Zelter D C on trol of a virtual ctor a The
roac h Pr o c e e dings of he t Symp osium on Inter active Gr aphics
Sno wbird U tah
Smith S and B ates J T o w ard a theory of narrativ e for in teractiv e ction
T ec hnical Rep ort CMUS Sc ho ol of Computer Science arnegie C
Mellon Univ ersit y P ittsburgh P A
V ere S and Bic kmore T A b asic agen t Computational Intel ligence

W eizen baum J Eliza Communic ations of the A CM
Witkin A and K ass M Spacetime constrain ts Computer Gr aphics

Witkin A and W h W F ast nimation a and con trol of nonrigid struc
tures In Pr o c e e dings of SIGRAPH