BASED HEAD TRACKING
FOR CONTROL OF A
Supervised by Steve Maddock
This project aims
to use the
to track the head of a user to try and create a more immersive pseudo
There are already
of this technique such
Johnny Chung Lee’s
head tracking with
a Wii remote
head tracking with the Kinect
. Both of the
se projects show simple demos of
using head tracking to change the perspective of the image on the screen
head tracking example
Lee demonstrates how by tracking the head of a user, you can change the perspective of the camera in the 3D
er space in such a way that it turns the TV from a 2D image of a 3D scene, into a window into a 3D
of the user
from the screen and
position relative to the screen change
image to create the illusion of looking throug
h a window into a 3D space.
This is very different to current day
stereoscopic 3D technologies in
which the user is presented with a different image for
each eye that makes up
when the user moves their head laterally they are presented
with the same image.
differs to head tracking in that if you duck to avoid something jumping out from the screen, it will have an
effect on what is displayed on the TV.
This experience would be much more immersive than the current
technique in which t
he illusion is broken and the experience jarred the moment the user moves their head and
it becomes obvious that they are not looking at a 3D scene at all.
This project will
implement these techniques in an SDK format
demonstrate the potential of head
by creating an immersive
computer game in which the player dodges attacks in the physical
space which in turn changes the perspective on the computer screen
as well as having an effect on gameplay
Other similar current projects are
simply demos whereas this project will develop tools which will enable the
technique to be applied easily to a wide variety of other programs.
it could be used
to create a
virtual fish tank, or virtual window in which
user walking past wo
accordingly to show
This could be placed in a wall and
be visually appealing.
could also be used
in more practical ways
CAD designers view
by intuitively looking around
software could change the perspective on that object, providing a much more natural way of viewing the
An extension of this could be to add hand gestures in which the user could spin the object round, zoom
in or out or even make changes in a very ‘Minority Report’
or ‘Iron Man’
kind of way.
In modern day compu
ting there are
a growing number of
ways to interface with computers. From
the original keyboard and mouse, or other pointing device, to touch screens, voice commands, and input
devices especially for gaming, such as gamepads, joysticks and st
A trend amongst the
development of input devices is for a more natural way to interface with computers. For example, speech
recognition is a very large field of study as it would be very beneficial to be able to communicate with a
n the same way that we would communicate with another human.
With the advent of the Kinect and
depth camera and other sensors there is a very large scope for using
natural body movements and voice commands to interact in a very natural way with comput
Using a Kinect
as opposed to the Wii’s IR camera will allo
w any users to interact, rather
than those only equipped with the
right sensors or emitters.
The Kinect, born as a device to control a game console is currently used to ‘turn the use
r into the controller’
As such it provides many excellent tools to aid in this project, especially since the release of the Microsoft
Currently the Kinect is used primarily as a games controller for the X
Box 360, using your hands arms and
rest of your body to control the game. This project will be taking the core ideas of the Kinect but applying them
in a different way.
Today’s game engines mostly operate in the same way for each genre. For example in first person shooter
style games, the character is controlled by a mouse and keyboard or a gamepad. There is a control to look
around the world rotating the camera in a fixe
d place, or to move the character/camera in the y z plane. This
has persisted for
over two decades and the control of your character has changed very little
during that time. The potential is there however for other methods of input to be abl
e to control different
aspects of the game, as will be demonstrated in this project.
This project consists of three constituent parts: The input, the game and the mapping between the two.
The input will be
the data collect
from the K
inect, including informatio
n from the depth camera and
cam. The scope of the project is large enough that once methods of tracking the head’s position with the
have been developed, the
tilt, or pitch and yaw of the head
could also be invest
information from the webcam could
ly be used
and eye movements.
is the information
that will be made
available in the SDK.
The initial input will be using
heuristic methods to detect a player making a ducking movement
using the depth
camera, potentially using the Kinect SDK
The potential for this is demonstrated below with an example of
from the Kinect using the Microsoft supplied sample
skeletal viewer, so
this method will need
re is a lot of
extracting the head tracking data from this.
After the simple
heuristic approach to detect whether or not a player is ducking, more sophisticated ways of track
precise location of the head will be determined.
In the finished
from the position of the head
be used to move the camera
position and angle in the 3D space.
Potentially simulating the
the near v
iew plane, then the player
moving towards or further away from the TV would increase or decrease the amount that is viewable, and
could rotate the camera around the near view plane.
For this to be effective
the best point of
rotation for t
he best user experience
will need to be determined
; whether it is better to rotate the camera
around the near view plane, far clip plane or origin.
Since the Kinect SDK provides skeletal tracking, obtaining the position of the head shouldn
’t prove too diff
to try and track the pitch and yaw of the head
could prove challenging as
the depth data may well
not be accurate e
nough on its own. At this point
the accuracy of
the depth data
could be an issue, so
could be gathered
from the webcam.
could be used to
determine what suita
from the webcam.
With regards to a game engine the Unreal Developers Kit (UDK)
by Epic Games or the Source Engine
main object is to create a SDK for head tracking that can be used in many
this project will include a demonstration
game that is visually appealing in order to fully
demonstrate how immersive adding head tracking to ex
isting technology can really be.
This constitutes the bulk of the project. The importance of this project lies with the ability to map the inputs
from the Kinect to the output on the screen in such a way that it can be used meaningfully and eas
ily to create
a head tracking experience in a variety of different applications.
This will therefore require the most time spent investigating which data can be obtained from the Kinect with
regards to the position of the head and also the accuracy and ti
ming of such information.
In this section it will also need to be determined which functions or methods the SDK will provide.
previously mentioned methods that will return Boolean values for such actions as ducking, jumping or leaning
could be implemen
ted but the main aim is to have one to one real time tracking of the head so perhaps a
more useful output would be to provide co
ordinates, angle and pit
and yaw for a camera in the 3D space.
It will also be considered, for this stage of the project, us
ing heuristic methods versus machine learning
methods and the robustness of each. For this
will also need to consider whether there is any learning, setup,
or calibration time. Ideally the head would be tracked with no conscious input from the user as t
o not jar the
experience of the user.
With regards to evaluating the success of the projects it has to be considered whether this can be done
objectively or subjectively. Since the main object is to create an SDK which can be easily and widely
subjective evaluation of the ease of using this SDK must be consider
. But it must also be considered the
results from using the SDK as demonstrated in through the game. This must be done subjectively as it is the
end user experience that the pr
oject aims to improve by creating a very immersive experience.
As such the project will be demonstrated to users and then a survey completed, this will help to
immersive this is versus other technologies and also how intuitive it is.
Install Kinect and SDK
Investigate available data streams
Write Project Description
Review related literature relevant to the project
Extract data from the Kinect
Save or view data that can be obtained from the Kinect
Install Games Engine
Install a games engine and script a simple game
Find data relevant to tracking the head from the Kinect
Experiment with camera options available in games engine
Investigate machine learning techniques versus heuristic approa
Decide on the functions to be made available in SDK and feasibility of each
Use input from Kinect to control character or camera in games engine
Investigate the best way to change the camera to create correct perspective
Look at other
input device to use as a pointing device during game play
Summarise into survey and analysis document