Digital Interactive Entertainment

brasscoffeeAI and Robotics

Nov 17, 2013 (3 years and 10 months ago)

66 views

Digital Interactive Entertainment


Dr. Yangsheng Wang


Professor of Institute of Automation

Chinese Academy of Sciences

Yangsheng.wang@mail.ia.ac.cn

http://hci.ia.ac.cn/JointLab/onlinegame
-
En.htm

Overview of

Digital Entertainment Industry

Digital Entertainment Industry


Year 2008 the market size will be about
$100B and major products are TV games

PC games and mobile games.



Microsoft, Sony and
Nintendo provide
most
digital entertainment
products.



China has a very big potential market for
digital entertainment products.

Internet and Game in China


The total amount of Chinese Internet users is up to 87
millions in 2004, about 25% are online
game(OG)
users.


OGs

are

one of

major part
s

of digital entertainment
contents
in China
.

Almost half of t
he OGs on the domestic
market mainly come from Korea
.
Some Chinese game
companiers are beginning to develop their own OG
s.


PC Stand
-
alone games are losing their market.


Mobile games are developed rapidly, but the income is on
the low level at the present stage.


Overseas game companies enter Chinese market via
Joint
-
venture.

source

䍇P䄦䥄䌠㈰〴

Internet Users in China

Online Game Market in China

source

䍇P䄦䥄䌠㈰〴

Stand
-
alone Game Market in China

source

䍇P䄦䥄䌠㈰〴

New Research & Applications of


Natural Interactive Technologies


Traditional games mainly depend on mouse,
keyboard, joystick and special input device.



Game players desire more natural and intelligent
interaction methods, including voice, gesture,
expression, gaze, and intelligent sensor, etc.



User
-
oriented multi
-
model HCI (Human Computer
Interaction) technologies has become the hotspot of
research and application in many domains.

Natural Interactive Technologies (NIT)




Computer Vision for Computer Games and Graphics


Camera is used as a powerful interface device for games by
sensing user’s body position, head orientation, direction of
gaze, pointing commands, and gestures.


Developer: Mitsubishi Electric Research Laboratories.


Applications: Sega games

NIT Research and Application
-
I

Vision device


Vision
-
based Human
-
Computer Interface for Games


By a camera, the player's body movements are projected
right into the game as an unencumbered input and control
device.


Developer: Sony Computer Entertainment America


Applications: Sony PlayStation2 EyeToy games

NIT Research and Application
-
II

Camera


Voice Control for Games and Simulations


Many of today's games have a lot of keyboard commands
that can be hard to remember. Speech recognition
technologies are used in this area for game control.


Product: Game Commander (Mindmaker)


Product: Game Voice (Microsoft SideWinder)

NIT Research and Application
-
III


Interactive Entertainment

Research Group in CASIA


Auto
-
Construction of a Game Object;


Player
-
Orient
ed Game


Multi
-
model

Human & Computer
Interaction
;


Natural Interactive Entertainment
Platform.

The Outline of Our Research

Photo
-
stereo Based

Model Reconstruction


Images capture


Normal calculation


Surface reconstruction

flashlight

flashlight

flashlight

camera

bracket

control card

Images capture

Images of the surface under different
lighting conditions is captured in a very
short period, that the register is avoid.

Normal calculation


A photo
-
stereo method is used to
retrieve the surface normal based on
the reflect property of the surface.

Surface reconstruction


The relevant height
between the points can
be integrated from the
derivative field.




A pyramid integration
method is used to
optimize the results





L
qdy
pdx
p
z
p
z
)
(
)
(
0



2
|
)
,
(
ˆ
)
,
(
|
)
ˆ
,
ˆ
(
y
x
p
y
x
p
q
p
F
dxdy
y
x
q
y
x
q
2
|
)
,
(
ˆ
)
,
(
|


Orthogonal images Based

Model Reconstruction


Generic Model and its feature points

selection


Two images feature points editing


Generic model deformation


Texture merging and mapping

Generic model and its feature points
selection

79 feature points according MPEG4

Images feature points editing and

texture merging

frontal face

side face

texture merging result

Generic model deformation and

texture mapping

texture

generic model

model reconstruction result

deforming

Individual game characters

Individual faces give players more fun and immersion.

player A

player B


Face detection


Deformation & Texture mapping


Expression recognition


Expression retargeting & Animation

Video
-
Based

Model Reconstruction and Animation

Capture user’s frontal face with a
camera, then deform an existent
face model and map the face image
to create an individual face model.


Five kind of user’s expressions are
recognized real
-
time to drive the
relevant animations.

Auto
-
Extraction of

key points of a face

An RealBoost
-
based statistical learning
is adopted to build a
model for each facial point. During extraction, the models are
used to locate facial points for each face image.







Deformation & Texture mapping







Origin image

filling image

generic model

merging texture

reconstructed individual models

deforming


Expression recognition and retargeting












Expression
recognition

Source
expression

models

Individual

expression


results

surprise

anger

sad

happy

fear

A novel approach to real
time facial expression
recognition
using an
embedded HMM

which
uses observation vectors
that are composed of
DCT results.

Getting into a Game
-

Change face


Player’s frontal
face is captured by
a camera.


Player’s face
feature points are
detected to deform
character’s face
model.


Player’s face
image is used
change character’s
face texture.



Head detection and pose estimation



Fist detection and motion analysis



Other motion detection and analysis

Video
-
Based

Motion Detection and Analysis

Head detection and pose estimation













front

left

right

up

down

close

far


locating the position of a head


recognizing the pose of the head

The novel scheme based on
Markov model

fusing appearance,
color, and motion information.

Fist detection and motion analysis

prototype DEMO

training samples

Driving a car using your hands and voice


Player’s hands
control the car’s
direction.


Player’s voice
controls the car’s
speed and status.

Motion Detection and Analysis


Project video into game


Motion detection


Parameters calculation


Game objects control

Playing a game with your body


Player’s body
movement is projected
right into a game


The motion detection
and analysis results are
used to activate objects
in a game.

Online
-
Game

Strategic Research Group

Online
-
Game

Strategic Research Group

A

meeting of the group

The group includes
major digital
entertainment
companies,
universities,
research institutes,
and digital
entertainment
association in China.


Write strategic report of digital
entertainment for Chinese government.



Establish “China National Center for Digital
Entertainment”.

The goal of the Strategic
Research Group

Thank you!