Processing Based Feedback

peachpuceAI and Robotics

Nov 6, 2013 (3 years and 8 months ago)

73 views

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Development of Image
Processing Based Feedback
Systems for Interactive Gaming
Using Non
-
Traditional
Controllers


Adam

Hedji

Mantas Pulinas

Philip San

III. Viktorious

SSIP 2009, Debrecen, Hungary

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Project Natal

X
-
Box 360

Microsoft Corporation


Full Body Motion Detection

New project that
promises to use two
cameras for full
-
body
motion detection


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

CHECKERS!


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Overview of Code

Board Plane
Geometry Detection

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Overview of Code

Relaying of Data
with Checkers A.I.

Feedback

Identification of
Square Status by
Colour Classification

Segmentation of
Individual Game
Squares

Board Plane
Geometry Detection

This is blue!

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Geometry Detection


Find the corners


2 approaches
were used:


Edge
detection
and region
growing


Hough
transforms

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Geometry Detection


Edge Detection
and Region Growing


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Geometry Detection


Edge Detection
and Region Growing


Edge detection with Sobel
operator


Dilate the image to fill the
gaps in the border


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Geometry Detection


Edge Detection
and Region Growing


Edge detection with Sobel
operator


Dilate the image to fill the
gaps in the border


Thin the image to get the
true border

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Geometry Detection


Edge Detection
and Region Growing


Edge detection with Sobel
operator


Dilate the image to fill the
gaps in the border


Thin the image to get the
true border


Dilate the image several
times to remove useless
edges


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Geometry Detection


Edge Detection
and Region Growing


Edge detection with Sobel
operator


Dilate the image to fill the
gaps in the border


Thin the image to get the
true border


Dilate the image several
times to remove useless
edges


Region growing


Determine the corner tiles


Track corners in real
-
time


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Geometry Detection


Projective
Correction


Edge detection with Sobel
operator


Dilate the image to fill the
gaps in the border


Thin the image to get the
true border


Dilate the image several
times to remove useless
edges


Region growing


Determine the corner tiles


Track corners in real
-
time


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Geometry Detection


Projective
Correction

Calculate homography using corner coordinates

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Geometry Detection


Hough Transform


Hough transform


We had time so we developed a better
solution


Based on Hough transformations


Better real
-
time line detection


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Colour Classification

How can the computer classify the colours?


(171,154,158)

(32,61,105)

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Colour Classification

Use image processing algorithms to make the
RGB values only 0 or 255


(255,255,255)

(0,0,255)

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Colour Classification

(255,255,255)

(0,0,0)

(255,255,0)

(0,0,255)


Select individual tile



Analyse the predominant colour
inside to classify the square
state (white, black, blue, yellow)



Sample of pixels used as
opposed to whole square

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Colour Classification


How to determine if each R, G and B
values are 0 or 255?


-
Need to choose threshold value




e.g.


A given pixel of value (15, 19, 250)


A threshold of 126


Output is (0,0,255)

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Colour Classification
-

But how to
choose the threshold?


Webcams can be of relatively poor quality and
provide poor contrast.


For example, blue pieces used were relatively
hard to distinguish from the black tiles.

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

But how to choose the threshold?


Can instead normalise the
image and use a threshold of
127, given by (256/2)
-
1.


Select a white tile and take
an average of the colour
values.


Do the same for a black tile.


Use these averages to
normalise the image

p
1
=(241,209,210)


p
2
=(232,204,214)


p
3
=(240,211,205)


Black
White
Black
Input
B
G
R
Output



)
,
,
(
Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Colour Classification
-

Normalisation


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Colour Classification
-

Normalisation


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Colour Classification
-

Normalisation


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Colour Classification
-

Normalisation


Then reduce the image to absolute values of 0
and 255


Use a threshold of 126 (half of full intensity
value)

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Colour Classification
-

Normalisation


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Colour Classification
-

Normalisation


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Summary


Main results


Successfully used 2 approaches to chess
board detection


Edge detection and region growing


Hough transform


Removing perspective distortion


Identification of individual tiles and pieces,
including classification


Connection to engine interface with feedback
system

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Improvements


Aim towards real
-
time 60fps processing


Use a more efficient programming language
such as C++


Use of GPU using CUDA or Open CL
programming language


More complex algorithms


Motion detection of hand


Use of overlay of 3D structures onto camera
image.


Virtual humans...


Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

Unused Virtual Human

Development of Image Processing Based Feedback Systems for
Interactive Gaming Using Non
-
Traditional Controllers

The Team

Adam Hedji

University of Zagreb, Croatia

adam.hedi@fer.hr

Mantas Paulinas

Vilnius Gediminas Technical
University, Lithuania

mantas.paulinas@el.vgut.lt

Philip San

University College London, England

p.san@ucl.ac.uk

Viktor Blaskovics

University of Szeged, Hungary

blaskovics.viktor@stud.u
-
szeged.hu

Viktor Blaskovics

University of Szeged, Hungary

blaskovics.viktor@stud.u
-
szeged.hu

Philip San

University College London, England

p.san@ucl.ac.uk