FYPReportx - Snooker Score System Using Image Processing ...

farmacridInternet and Web Development

Feb 2, 2013 (4 years and 8 months ago)

165 views











Using Image
Processing to Create a
Snooker Score System


Nathan King

Supervisor


Dr. Martin Glavin

Co Supervisor


Prof. Ger Hurley

B.E. Electronic & Computer
Engineering Project Report

EE426

30
st

March 2012



i


1

Abstract


This project aims to automate the score keeping process in the game of snooker.
Nowadays, the majority of sports are aided

by various electronic devices
. We see goal
-
line technology making its way slowly into the world of professional football. Snooker
should be no different.

With the speed that snooker balls can potentially travel at, the identification of when a
ball
is pot
ted is a complex task. M
onitoring the pockets alone is not enough to identify if a
ball has been potted and
also to identify

what colour

ball

has been potted.

This project aims to use circle detection algorithms and a colour i
dentification algorithm
to ove
r
come the problem of

the fast travelling ball.

The aim of the project is to implement the algorithm in the C language using the OpenCV
libraries. Working in this way helps provide a high performance and low overhead system
that will be possible to implemen
t and run on a low processing powered machine.














ii


2

Acknowledgements


I would like to
thank my supervisor Dr. Martin Glavin for

the help and
guidance

he
provid
ed throughout this

project
.

I would

also

like to acknowledge my co
-
supervisor, Prof. Ger Hurley
, whose professional
opinion proved an invaluable asset.

Finally,
I would also like to thank my fellow 4
th
EE/ECE classmates who
provided constant
support

when it came to debugging, provided encouragem
ent and
always showed

a keen
interest in th
i
s project.


















iii


3

Declaration of Originality


I declare that this thesis is my original work except where stated.


Date: ___________________________________


Signature: ___________________________________
















iv


4

Table of Contents

1

Abstract

................................
................................
................................
..................

i

2

Acknowledgements

................................
................................
................................

ii

3

Declaration of Originality

................................
................................
.......................

iii

5

Table of Figures

................................
................................
................................
.....

vi

6

Table of Tables

................................
................................
................................
.....

vii

1.

Glossary

................................
................................
................................
.............

viii

2.

System Overview

................................
................................
...............................

-

1
-

3.

Background Technologies

................................
................................
..................

-

4
-

3.1

Computer Vision

................................
................................
..........................

-

4
-

3.2

OpenCV[1]

................................
................................
................................
...

-

4
-

3.2.1

OpenCV Functions

................................
................................
..........................
-

5
-

4

Project Structure

................................
................................
................................

-

8
-

4.1

Project Path

................................
................................
................................
.

-

8
-

4.2

Overall System FlowChart

................................
................................
..........

-

10
-

4.3

Process Specifics

................................
................................
........................

-

12
-

4.3.1

Capture Video Frame

................................
................................
..................

-

12
-

4.3.2

Operations Performed on Frames

................................
..............................

-

12
-

4.3.3

Snooker Rules Applied

................................
................................
................

-

16
-

4.3.4

Graphical Overlay

................................
................................
........................

-

18
-

4.3.5

Processing in Real Time

................................
................................
..............

-

19
-

5

Testing

................................
................................
................................
.............

-

20
-

5.1

Lighting

................................
................................
................................
......

-

20
-

5.2

Range of Colours

................................
................................
........................

-

21
-

5.3

Motion Blur

................................
................................
...............................

-

23
-

5.4

Finding the Threshold Values

................................
................................
.....

-

24
-

5.5

Intel Performance Primitives

................................
................................
......

-

27
-

5.6

Memory Management

................................
................................
...............

-

27
-

6

Results

................................
................................
................................
.............

-

30
-

6.1

Algorithm Performance

................................
................................
..............

-

30
-

6.2

Image Quality

................................
................................
............................

-

33
-

7

Further Work

................................
................................
................................
....

-

34
-

7.1

Camera Quality

................................
................................
..........................

-

34
-

7.2

Processing Time

................................
................................
.........................

-

34
-

7.3

Lighting

................................
................................
................................
......

-

35
-

7.4

Multi
-
Camera System

................................
................................
.................

-

35
-



v


7.5

Voice Recognition

................................
................................
......................

-

36
-

8

Conclusion

................................
................................
................................
........

-

37
-

9

References

................................
................................
................................
.......

-

38
-

10

Appendix


On the CD

................................
................................
....................

-

39
-





















vi


5

Table of Figures


Figure 1
-

a snooker table with the webcam
mounted above

................................
.....

-

3
-

Figure 2
-

shows the top
-
view that the mounted camera outputs

..............................

-

3
-

Figure 3
-

shows the flowchart for part 1 of the system

................................
...........

-

10
-

Figure 4
-

shows the flowchart for part 2 of the system

................................
...........

-

11
-

Figure 5
-

shows the gray scale image that the Hough] Transform uses to detect circles

.

-

13
-

Figure
6
-

shows the original unthresholded image

................................
.................

-

15
-

Figure 7
-

shows the thresholded image with the detected pixels in white

..............

-

15
-

Figure 8
-

shows a zoomed
-
in view of a detected snooker ball.

................................

-

18
-

Figure 9
-

shows a graphical overlay that di
splays the scores of each player

...........

-

19
-

Figure 10
-

shows the snooker balls casting shadows on the snooker table

..............

-

21
-

Figure 11
-

illustrat
es the result of an image that is too bright. The yellow, pink and
white are very similar as a result.

................................
................................
...........

-

22
-

Figure 12
-

shows the effects of motion blur on the white

ball at the bottom left of the
image

................................
................................
................................
.....................

-

24
-

Figure 13
-

illustates the correct result of thresholding

................................
............

-

26
-

Figure 14
-

illustrates the result of a threshold that is too rigid and doesn't detect pixels
and therefore converts the pixels to black instead of white

................................
....

-

26
-

Figure 15
-

shows the result of threshold that is much too flexible and converts most
pixels in the image to white

................................
................................
....................

-

27

-

Figure 16
-

illustrates the improvement in memory usage when de
-
allocation routines
are applied

................................
................................
................................
.............

-

29
-

Figure 17
-

Samples of the successful implementation of the system.

......................

-

31
-

Figure 18
-

shows the processing power taken up by each of the algorithms in
percentage for
m

................................
................................
................................
.....

-

32
-

Figure 19
-

describes the relationship between the compression ratio and the numbe of
balls detected.

................................
................................
................................
........

-

33
-









vii


6

Table of Tables


Table 1
-

Min/
M
ax RGB values of each colour

................................
...............................

-
23
-

Table 2
-

Allocation and De
-
allocation functions
................................
............................

-
28
-



















viii


1.

Glossary


[1] OpenCV


Open Computer Vision

[2] RGB


Red, Blue, Green

[3] HSV


Hue, Saturation, Value

[4]
C


low level, compiled programming language

[5]BSD


Berkeley Sotfware Distibution

[6]

Intel IPP


Intel Integrated Performance Primitives

[7]FPS


frames per second












-

1
-


2.

System Overview


In
today’s dynamic world
,
the time
of manual processing

is almost over
. These once
manual processes have been replaced by electronic devices that automate these
processes.

Snooker is one of the
few sports that are still in this

“old fashioned”
manual
way. There is
a referee who identifies when a ball has been potted. From the colour of the ball, the
scorekeeper increments the scoreboard accordingly depending on which player is at the
table.

T
his project aims to automate this

cu
rrent manual process of scorekeeping

within the
game of snooker
. Using
computer vision techniques and a web cam
, this can be achieved.

Computer vision is the process which

allows
computers to

see


the world

and respond
to the
it
[2]
. In the case of snooker
,
computer vision

would allow computers to
observe

what the referee sees during a game.

Using computer vision techniques can be
a
powerful

tool

when it comes to real time applications such as this.

Applications can be programmed so that they detect certain

shapes based on the
functions that the application is implementing. Computer vision can be used to provide
these functions so that they accurately detect the desired shape.

It is becoming common that sports make use of the power that computer vision
along
with a camera
can provide
.
As can be seen with football, Hawk Eye technology is to be
implemented to improve
the
game of football.
In addition to football,
Tennis and cricket
currently make use of this Hawk Eye technology which incorporates the use o
f

cameras to
track the ball. Computer vision is in this way helpful and becoming increasingly common
in applications like this.

Snooker is there
fore, a great contender to

be next on the agenda for computer vision.
With an effective ball detection and colou
r identification algorithm, steps can be taken to
detect
each ball

on the table and hence, detect when a ball is potted.



-

2
-


Research into the best way to detect the snooker balls,

led me to propose a system that
consists of
a single

downward facing camera mou
nted above the snooker table capturing
video images at 30 frames per second.
An alternative system was one

where 6 cameras
would be used to closely watch eac
h of the 6 pockets. As a ball was

potted, the camera
observing

the
receiving pocket

would detect this ball.

My proposed
system would be a lot more
efficient with only 1 c
amera as

there would be
severe processing and configuration overheads, which would render it unsuitable for use
with a low processing powered machine.

The single camera
system gives a birds
-
eye view
of the snooker table and can be used to identify what balls are moving and what
collisions occur, whereas the 6 camera system focuses on the pockets of the table rather
than the table as a whole.
For obvious reasons, a single
camera system would be much
more cost efficient than the alternative system.

Computer vision algorithms and functions are computationally intensive so implementing
these algorithms and functions in real time on a low processing powered machine

is not
reall
y viable, yet it is still possible to get accurate results at lower fps than that of the
camera.

Using the low level language of C helps to maintain as much processing power as
possible.

In concl
usion, the single camera system

is
a

more ideal system to dea
l with the
scorekeeping of snooker. The system has the ability to detect all balls and th
eir colours
while providing
information on the balls that the alternative solution
could

not.





-

3
-



Figure
1

-

a snooker table with the webcam mounted above


Figure
2

-

shows the top
-
view that the mounted camera outputs



-

4
-


3.

Background Technologies


This chapter gives an insight into the tec
h
nologies used in the project. It provides an
overv
iew of computer vision and the functions used to implement the system.

3.1

Computer Vision

Computer vision is

bec
oming more and more common nowa
days. The techniques used in
computer vision are computation
ally intensive and ordinary machines would not have
bee
n able to handle this expensive overhead. Computer vision is a field that includes
methods for obtaining, processing

and analysing images and videos from the real world in
order to produce numerical or symbolic results. A recen
tly well known example of
co
mpu
ter vision can be seen in many smartphones and cameras
today
. Face recognition
is
present

in most cameras

nowadays
. This allows the camera to detect faces in an image
and overlay the image with information. In the case of facial recognition, a rectangul
ar
box is usually drawn around the faces in the image.

Processing of this calib
re would not
have been possible
a few years ago.

Computer vision is becoming an integral part of gaming nowadays.

Microsoft released the
Kinect at the end of 2010, which utilis
es

computer vision to recognise gestures made by
the user to create a controller
-
free gaming experience.


3.2

OpenCV
[1
]

OpenCV (Open Source Computer Vision) is a library of programming functions for real
time computer vision. It was originally developed by
Intel in 1999, and it is now an Open
Source project released under a BSD licence.


The purpose of OpenCV is to provide open

a
nd optimized code for basic vision
infrastructure. OpenCV functions are written in C. These functions help provide robust,
quick routines for standard image processing techniques.
There are many techniques
used in image processing that are common and by u
sing OpenCV the programmer does
not have

to

continually write these techniques but merely call these functions.



-

5
-


Intel provides a multi
-
threaded software library of functions called Intel IPP
. The library
contains highly optimized functions for image proces
sing that OpenCV can use instead of
its own functions to speed up computation and increase performa
n
ce.

In this project, OpenCV greatly sped up the development process by providing functions
to detect circles, identify colour, convert images from one colo
ur spac
e

to another and to
threshold images for a particular colour.

Wrappers for languages such as C#, Python, Ruby and Java have been developed, but the
code for my system is written in C. The use of this low level language helps

to

increase
performance
and aids in the processing of the video frames in real time or as close to real
time as is possible .


3.2.1

OpenCV Functions

For this project, many OpenCV functions were utilised to accelerate the processing of the
video frames.

3.2.1.1

cvHoughCircles

This function is
used to find the circles in a grey
-
scaled image. It uses some modification
of Hough transform.

3.2.1.2

cvGet2D

The cvGet2D func
t
ion allows for

an examination

of a particular pixel and the extraction of
certain information about that particular pixel.


3.2.1.3

cvCvtColor

cvCvtColor function enables

the conversion of an image from one colour space to another.

3.2.1.4

cvThreshold

Performing a t
hreshold of an image is a fundamental technique when it comes to image
processing. It consists of examining the values of each pixel in the
image and then
performing an operation depending on the value of the pixel.



-

6
-


OpenCV provides multiple techniques when it comes to thresholding. Using the
cvThreshold function, there are several different thresholding options available.



CV_THRESH_BINARY



CV_
THRESH_BINARY _INV



CV_THRESH_TOZERO



CV_THRESH_TOZERO_INV



CV_THRESH_TRUNK


CV_THRESH_BINARY was the thresholding fu
n
ction that was
applied

in this project.
Thresholding worked best in a HSV colour space. When using this function, the image was
converted us
ing the cvCvtColor function described above. The min
imum

and max
imum

threshold values were manually

selected in HSV format. Depend
i
n
g on whether the pixel
value l
ay

with
in

the min and max value or out
side these values, that pixel waas

assigned a
value of 0
(black) or 255(white).




3.2.1.5

Drawing Functions

OpenCV provides many drawing functions which
enable

the information the m
achine has
gathered to be displayed

on the image for the user to see.

cvCircle was the function that I made use of. This function allowed
the programme to
draw the outline on the detected circles in the image and also annotated the image to
show w
here the circles centre point was

located. This was very useful as it let the user
see what balls were being detected.

3.2.1.6

Text Overlay


OpenCV allows
the programme to display certain information by outputting text onto the
image.

The function
cvPutText allows the user to display text on the image.



-

7
-


cvRectangle is another function provided by OpenCV that, as the name suggest, draws a
rectangle on the ima
ge. This along with cvPutText work
ed well as
the rectangle and text
were of contrasting colours.

In this way,

the text would be a lot clearer for the user to see
on the image.






















-

8
-


4

Project Structure


This chapter

describes the structure of the project and each step that was
undertook
through the development of the system. It gives a brief description of the flow of the
system before giving an in depth description of each step that was implemented to get to
the fin
ished system.

The project was divided into 6 steps that flowed into each other to give the final system.

4.1

Project Path


Step 1



Configure OpenCV to load a synthetic image of a snooker table and its balls.



Synthetic image contains a snooker table in ideal co
nditions:

-
Perfect lighting.

-
No shadows on the table.

-
Perfect circles.

-
Clear and distinct colour.



Detect all circles(snooker balls) on the table in the synthetic image.



Extract the colour of these detected circles by examining their pixel values


Step 2



Instead of a synthetic image, load in a real image of a snooker table and the
snooker balls.



Load images under different lighting conditions:

-
Bright image/dark image.

-
Light source directed from above the centre of the table

-
Light sources directed from

different angles from the snooker table.



Test the detection of the snooker balls in these conditions.



-

9
-




Test the extractio
n of the ball colours by examin
i
n
g their pixel values in the
various conditions.



The results show which conditions are best to progress

with the project.


Step 3



Identify which player is at the table so that scores can be awarded accordingly.
This is achieved by thresholding the image for two particular colours that are on
ly
present on each players’ cue
.


Step 4



Detect when a shot has tak
en place and when it has ended. This is done by
monitoring the movement of the white

(cue) ball. When the white ball begins to
move, a shot has taken place. After a certain amount of time, it is safe to say that
all the balls will have stopped moving and
the shot has ended.



Identify the colour of the ball that the white ball has come

in conta
ct with first.
This colour ball can be identified by detecting the ball that first moves after the
white ball has moved.

Step 5



Implement two scoreboards into the syst
em, one for each player. Scores awarded
to each player can then be applied to their respective scoreboard.

Step 6



Display the score
of each player to the screen by ovelaying graphics on the original
image to clearly illustrate when a play
er has been award
ed
points
and how man
y
points he has been awarded.





-

10
-


4.2

Overall System FlowChart



START
Detect num
_
balls
Wait for white to
move
,
track colour
it initially hits
,
wait
for balls to stop
Initial ball hit
?
Do a Detection for
num
_
balls
Opponent
awarded points
Potted Ball
Colour
?
Opponent
awarded points
Player awarded
1
point for each red
.
Num
_
balls
=
num
_
balls
-
1
(
for each
red potted
)
Do a detection for
num
_
balls
Wait for white to
move
,
track colour
it initially hits
,
wait
for balls to stop
Initial ball hit
?
Opponent
awarded points
Do a detection
for num
_
balls
Potted Ball
Colour
?
Opponent
awarded points
Player awarded
points
Num
_
Reds
=
0
?
Red
Colour
/
None
Num
_
balls
=
22
still
.
Change Player
Deteced balls
<
num
_
balls
Colour
/
White
Change Player
Change Player
Red
(
s
)
only
Red
/

none
/
other colours
Nominated Colour
Num
_
balls is the same as
previously detected
.
Change Player
Detected balls
<
num
_
balls
Not nominated colour
Change Player
Change Player
No
Yes
Nominated
Colour only
Num
_
balls is the number
of balls on the table
.
Starts
at
22
.

Figure
3

-

shows the flowchart for part 1 of the system



-

11
-





Detect num
_
balls
Wait for white to
move
,
track colour
it initially hits
,
wait
for balls to stop
Initial ball hit
?
Opponent
awarded points
Do a detection
Potted Ball
Colour
Opponent
awarded points
Player awarded
points
.
Num
_
balls
=
num
_
balls
-
1
Num
_
balls
=
1
?
(
is only white left
)
Show Player
1
and
2
scores and
highest breaks
END
Num
_
reds
=
0
Wrong colour
/
None
Change Player
Change Player
Correct Colour Only
Wrong Colour
Detected balls
<
num
_
balls
No Ball Potted
.
Change Player
Correct Colour
No
Yes
No reds left
.
Only Colours
left

Figure
4

-

shows the flowchart for part 2 of the system



-

12
-


4.3

Process Specifics


4.3.1

Capture Video Frame

Capturing of video frames is achieved with the use of O
penCV’s cvCapture function. This
function allows the creation of a cvCapture object which we can
request frames fr
om
either a vi
d
e
o file or from the camera connected to the machine.

If no frames are
available, an error is displayed to inform the user.


4.3.2

Operations

Performed on Frames

When
a

frame is received by the programme, there are several operations performed.

4.3.2.1

Ho
ugh Transform


In order for the cvHoughCircles funtion to be effective, it must have a gray
-
scaled image
to work with. In this way the function will be at its most accurate and can detect
all the
positive circles

in the frame
.

In OpenCV, the Hough Transfor
m works by taking a gray

scaled image and identifying large changes in colour intensity which represent an edge.
When it has iterated th
ro
ugh the image, it applies the fundamental parameters of a circle,
its x and y coordinates which repres
e
nt the centre
of the circle and the radius of the circle.
By applying these parameters to the edges it has already detected, it can
be
deci
d
ed
whether a shape is a circle or not. If it is, the centre points and radius are recorded. If not,
it just ignores those edges.




-

13
-



Figure
5

-

shows the gray scale image that the Hough
]

Transform uses to detect circles


As the circles

(snooker balls) are detected, their x and y coordinates are saved in two
arrays, one for the x coordinates and one for the y.


4.3.2.2

Threshold Image


The identification of which player is at the snooker table was a problem. This was an
imperative part of the programme as without the knowledge of which player is at the
table, points cannot be correctly allocated.

A solution to this
problem is to place two different coloured pieces of tape on each of the
players’ sno
okers cues. The incoming frames

can then be thresholded to detect a
particular colour.
In this project, the images were

thresholded to try and detect the two
colours on th
e pla
yers’ cues. Depending on the colour detected,
the programme

would
successfully identify

which player
was

at the table and whi
c
h player should be allocated
points.



-

14
-


In order for the thr
e
sholding of the image to be accurate, the colours on the players’ c
ues
must be unique to that of the rest of the image. To obtain these colours, various colours
were tested and the two colours that returned the most positive results were used.

The thresholding process functions in the

foll
owing way:



The image is split in
to its constituent channels

(H,S,V). From testing, it was found
that the HSV coloured worked better with thresholding function than RGB.



Use the built in OpenCV threshold function, CV_THRESH_BINARY. This function
converts pix
els in the real image, whose v
a
lues lie outside the min and max
threshold values to a value of 0(black). Pixels whose values lie within the
threshold range are converted to a value of 255(white).



This leaves us with a black and white
impression

of the original image. The white
pixels r
epresent the colour that we were looking for. In this case, that would be
the colours of the tape on the players’ cues.



To take into account any noise
, if the thresholded image returned an image where
the number of white pixels was small, it would be rejec
ted and the programme
would reject the notion that one of the players was at the table and would wait
until a larger number of white pixels
were

present.

An example of the thresholding algorithm at work can be seen in the following figures.






-

15
-



Figure
6

-

shows the original unthresholded image



Figure
7

-

shows the thresholded image with the detected pixels in white





-

16
-


4.3.3

Snooker Rules Applied


When the thresholding is complete, and hence, the player at the table identified, the
Hough Transform function is performed constantly identifying which snooker balls are on
the table.
From this moment,
the snooker rules are applied so that points can be
a
warded to the
appropri
ate player.

Snooker is played using a cue ball and snooker balls: one white cue ball, 15 red balls
worth one point each, and six balls of different colours


yellow(2points), green(3),
brown(4), blue(5), pink(6) and black(7).

A player

wins a frame

(individual game) of snooker by scoring more points than the
opponent, using the cue ball to pot the red and coloured balls.

The objective of the game of snooker is to strike the white cue ball with a cue in the
direction of other object ball
s and to pot these object balls in one of the six pockets
[2]
.
This must be done according to the rules of the game. By potting object balls points can
be scored.

The ball or balls that can be hit first by the white are called the ball(s) "on" for that
part
icular
shot
. The ball(s) "on" differ from shot to shot: a red ball, if potted, must be
followed by a colour. If a red is not potted, any red ball remains the ball "on"

for the next
shot
. Only a ball or balls "on" may be potted legally by a player. If a bal
l not "on" is potted,
this is a foul
.

At the beginning of the game, the red ball is the ball that is “on”.

If a foul shot occurs, the
opponent is awarded points.

If the player legally pots the ball that is “on”, they are
awar
d
ed points and stay at the tabl
e.

The game of snooker generally consists of two phases. The first phase is the situation in
which there are still red balls on the table. In the first phase, at the beginning of a player's
turn, the balls "on" are all remaining red balls. The player must

therefore attempt to first
hit and pot one or more red balls. For every red ball potted, the player will receive 1 point.
When a red
ball
has been potted, it will stay off the table and the player can continue his


-

17
-


break. If no red has been potted or a fou
l has been made, the other player will come into
play.

In case one or more red balls have been potted, the player can continue his break. This
time one of the six colours (yellow, green, brown, blue, pink and black) is the ball "on".
Only one of these can
be the ball "on" and the rules of the game state that a player must
nominate his desired colour to the referee, although it is often clear which ball the striker
is playing and it is not necessary to nominate.

When the nominated colour is potted, the playe
r will be awarded the correct number of
points (yellow, 2; green, 3; brown, 4; blue, 5; pink, 6; black, 7). The colour is then taken
out of the pocket by the referee and placed on its original spot.

If a player fails to pot a ball "on", it being a red or n
ominated colour, the other player will
come into play and the balls "on" are always the reds, as long as there are still reds on the
table.

The alternation between red balls and colours ends when all reds have been potted and a
colour is potted after the l
ast red, or a failed attempt to do so is made. All six colours
have then to be potted in ascending order of their points value (yellow, green, brown,
blue, pink, black). Each becomes the ball "on" in that order. During this phase, when
potted, the colours
stay down and are not replaced on the table, unless a foul is made
when potting the colour, in which case the colour is placed back on it’s original spot on
the table.

When the colours have been potted, the frame is over and the player who ha
s scored
most
points has won.

The programme has to take all these rules into account in order for it to be successful. An
initial detection is carried out on the first frame to detect all the balls on the table. Wh
en
a shot is taken and has been complete
, another detect
ion is

p
erformed. If the number of
balls
on the table now is less than the initial detection, then a ball is potted. By
subtracting the number of coloured balls from the number of coloured balls that are
present on the table now, the potted ball can be
identified and points can be allocated as
a result.



-

18
-


In order to identi
fy when a shot has been taken,
the coordinates of the centre of the
white ball, which have been recorded in an array, are monitored. If theses coordinates
differ in one frame from the ne
xt, then we can confidently say that the white ball has
begun to move.

By monitoring the centre point coordinates of all the coloured balls, we can identify
which ball moves first after the white ball and, therefore, identify which ball
the white
ball has
come in conta
ct with first.


Figure
8

-

shows a zoomed
-
in view of a detected snooker ball.

4.3.4

Graphical Overlay


Now that the programme understands how to calculate the players’ scores by
applying
the rules of snooker,
we can overlay

the original image to display these scores to the user.

In order to make the displaying of the scores clear to the user, a black rectangle box is
drawn on the original image at the bottom of the snooker table. Then we draw the scores
on top of this black
rectangle in white. In this way, contrast in maximised and this makes
it easier for the user to view the scores.

This graphical information provided to the user is positioned at the bottom of the image
just off the snooker table so that the movement of the

ball
s

on the table is still clear for
the user to see.

A blue circle is also drawn around the detected circles on the snooker table. Their centre
points are also highlighted with a green dot.



-

19
-







Figure
9

-

shows a graphical overlay that displays the scores of each player




4.3.5

Processing in Real Time


The processing and testing of the programme was primarily carried out on recorded
video. The aim was to try and get this programme functioning in real time or

as close to
real time as possible. Due to the physical constraints
of the machine that was used and
low maximum frame rate of the camera, processing in real time was not possible.

Every frame that was gathered had to be processed as a lot can happen in s
nooker in 1
frame. The movement of a ball by a couple of centimetres could be crucial in
determining what ball the white ball has touched.

The results of the ball detection and implementation of the rules of the game were not
affected by this. The playbac
k
of the processed video frames was

where the problem
occurred.








-

20
-


5

Testing


In order for this system to be fully functional, a number of tests had to be carried out

5.1

Lighting

Lighting was an important factor in this projec
t. It was discovered from tests

that
depending on the light source

the detection of both the amount of circles found on the
snooker table and the colour identified on each ball by our algorithm varied in different
conditions.

Initially three light sources were used (one pointing at cent
re of table, one pointing to the
left and one to the right).

When the circle detection algorithm was performed on images
with these light sources, it was found that there were a lot of shadows being cast on the
table and these shadows were being falsely de
tected as circles on the table.

A system that contained a single light source pointing directly at the centre of the t
able
proved to be a successful
alternative.

In this way, light was not sourcing from an angle on
the balls, so shadows were eliminated fro
m the snooker table.

Images of different brightness levels were then tested with the circle detection algorithm.
It was evident from this test that dark images resulted in the algorithm not detecting all
the balls.









-

21
-



Figure
10

-

shows the snooker balls casting shadows on the snooker table



5.2

Range of Colours


Detecting the correct colour of each snooker ball was a pri
mary objective in this project.
Without the correct identification of the colour of each snooker ball, the
system could
not allocate the correct points to a player.

Tests were carried out similar to those above. Images were taken in different condition
s,
varying
from 1 light source to 3 light sources and also in bright images and dark images.

It soon became
clear that again the 3 light source system would not work. Not only did
the this cause shadows, it also added a huge amount of glare on the snooker balls which
made the colour identification algorithm believe that most balls were of the

colour white
or clo
se to white.





-

22
-



Figure
11

-

illustrates the result of an image that is too bright. The yellow, pink and white are very similar as a result.



The single light source system again prevailed.

With this system, the brightness o
f the

image was satisfying and the results were positive. This system was tested with images at
sl
ightly different light intensities. In th
is way, when the system examined

the RGB values
of each snooker balls’ pixel value, it
would

return a range of RGB values
for each colour of
the snooker balls.

This gives us a minimum and maximum
RGB
value

for each of the colours. Most of the
balls’ range of RGB values were flexible. There were some colours that were similar to
each other and a strict range of values were req
uired in orde
r for the system to
confidently
identify the correct colour.

Before testing, it was thought that there may be a
similarity between the pink and red ball but that was not the case. It was the white,
yellow and pink ball that were similar in cer
tain conditions as well as the brown and red
ball.

These colours required a rigid range of RGB values. The following table show the min and
max RGB values of each colour.



-

23
-




Ball Colour

Red(R) min/max

Green(G) min/max

Blue(B) min/max

White

251/255

252/255

220/255

Yellow

246/252

243/250

120/140

Green

0/7

87/93

39/65

Brown

158/169

61/70

27/31

Blue

8/12

37/46

130/143

Pink

247/255

184/245

160/187

Black

0/24


0/36

0/16


Red

249/255

87/110

60/85


Table 1


min/max RGB values of each colour




5.3

Motion Blur


While testing the system, it became evident that at certain times, the circle detection
algorithm failed to pick up moving balls. After close inspection, it was clear that the
failure to detect the balls was due to motion blur.

Motion blur is the apparen
t streaking of rapidly moving objects in

an
image or sequence
of images
[3]
. This motion blur is inconvenient

because it obscures t
he exact position of
an object. It occurs as a result of frames being gathered at a low rate. At a frame rate of
30fps,

motion

blur would occur in the white cue ball when it travels at the typical speed
of 40km/hr.

To take this into consideration, when testing the system, it was ensured that the white
ball was hit at a speed that would minimise the effects of motion blur and the
refore,
return positive results.




-

24
-






Figure
12

-

shows the effects of motion blur on the white ball at the bottom left of the image





5.4

Finding the Threshold Values


In order to threshold the image to identify which player is at the table, the colours
of the
pieces of tape on the players’ cues

had to be known.

To correctly detect pixels on the player’s cue and not anywhere else in the image, two
unique
colours that we
re not present anywhere else on the image were used. Initially,
a
light blue (cyan) and purple were to be used.

To obtain the pixel value of these colours,
the colours were printed out and cut into a circle. They were placed on the snooker table
and the ci
rcle detection algorithm then detected and examined their pixel values.
These
images were tested in different lighting conditions in order to gain a wide range of figures
for the cyan and the purple

colour
.



-

25
-


From this,
the thresholding
routine
could be test
ed. The cyan piece of tape on
the
player’s cue returned satisf
ying results.
Only the pixels of the tape on the player’s
snooker cue were detected by the threshold and this the result we sought.
The purple
however, didn’
t work out. While
t
hresholding for pu
rple
, some of the pixels in the blue
snooker ball were being detected as purple pixels. The two colours were too similar to
work effectively.

Orange was chosen as an alternative. The

same tests were applied to the

orange colour
and when thresholding was pe
rformed on the image, the orange pixels were correctly
detected on the player’s cue and only on the player’s cue.

Following this, cyan and o
range

were
selected. To take into
account any noise in the
image,

a minimum number of pixels had to be detected befo
re the system could
confidently recognise which player was at the table if any.

10 pixels were

selected initially. This proved

to be too small and the system falsely
identified a pla
yer at the table.
Following this,
100 pixels were

tested but this seemed too
large
a value as the system could not recognise any players at the table. A value of 50
proved to be
a
successful

alternative
.

It allowed for noise in the image but wasn’t so large
as to not identify a player at the table.





-

26
-



F
igure
13

-

illustates the correct result of thresholding






Figure
14

-

illustrates the result of a threshold that is too rigid and doesn't detect pixels

and therefore converts the
pixels to black instea
d of white




-

27
-




Figure
15

-

show
s the result of threshold that is much too flexible and converts most pixels in the image to white

5.5

Intel P
erformance Primitives


Processing video frames in real time or as close to real time as possible was a major
concern. OpenCV provided optimized routines for
image processing, but there are more
optimized implementations of core functions available in a commercial package called
“ Intel Performance Primitives”
[4]
.
When OpenCV is deployed on an Intel processor, it is
able to take advantage of these performance routines to greatly accelerate the execution
of the code and therefore,
increase
the performance

of the system
. This projec
t was
developed and run on an Intel powered machine,
and
the use of these performance
primitives

could greatly accelerate the
performance of the system.

The package however, is a commercial product and was ruled out as an option for this
project
due to bud
get constraints
.


5.6

Memory Management




-

28
-


This project was completed in the C language
,

which does not include automatic memory
management like other languages. In order to prevent memor
y leaks, it was essential to
de
allocate any memory that was allocated when
necessary.

Memory was not in huge
supply in the machine that was used and therefore memory had to be closely monitored.

Memory use became a big concern during the testing stage of this project. Memory use
generally peaked around the 1.6GB mark when process
ing a 20 second video.

OpenCV provides several functions to allocate memo
ry. It also provides de
allocat
i
o
n
functions that must be used in order to avoid these memory leaks.

The table below
illustrates the most common memory allocation and de
-
allocation fun
ctions.






cvCreateImage

cvReleaseImage

cvCreateImageHeader

cvReleaseImageHeader

cvAlloc

cvFree

cvOpenFileStorage

cvReleaseFileStorage

cvCreateGraphScanner

cvReleaseGraphScanner

cvCreateSparseMe

cvReleaseSparseMe

cvCreateDara

cvReleaseData

cvMemStorage

cvReleaseMemStorage

cvCreateMat

cvReleaseMat

cvCreateMatND

cvReleaseMatND


Table 2


Allocation and De
-
allocation functions




W
hen these rules were applied, the

improvement in memory usage was immediate. The
following chart shows the impr
ovement in memory usage when de
allocation functions
were applied.



-

29
-





Figure
16

-

illustrates the improvement in memory usage when de
-
allocation routines are applied





















0
200
400
600
800
1000
1200
1400
1600
1800
Memory Usage (Mb)

1 De-allocation Function
2 De-allocation Functions
3 De-allocation Functions
4 De-allocation Functions
5 De-allocation Functions


-

30
-


6

Results


This section describes the results
obtained by the system as implemented in the system.


6.1

Algorithm Perfo
r
mance

The best way to evaluate the effectiveness of th
is
system was to test it in different scenarios.


It was
discovered

that the system functioned as expected in most conditions. The screenshots
below illustrate some of the scenarios where the system functioned as expected.








-

31
-







Figure
17

-

Samples of the successful implementation of the system.







-

32
-


Image p
rocessing algorithms are, in gen
e
ral, computationally intensive. The numerous
computer vision functions provided by OpenCV, all add up to put a huge load

on the
processor of

the machine.

In particular, the conversion of each frame from RGB to Grayscale, the circle detection
algorithm performed on this grayscale image and the
thresholding of the image to
identify the player at the table
were the most computationally intensive routines.

Ana
lysis was
completed

on the time taken to carry out each
of the steps above
. Initially,
measurements were
recorded

for

the total time taken to perform all 3 of the major
operation
s that are repeatedly performed.

The chart below shows the percentage of total

execution time taken by each
of
the 3
steps.




Figure
18

-

shows the processing power taken up by each of the algorithms in percentage form




As can be seen from the figure above, the circle detection algorithm is by far the
most
time consuming part of the algorithm.

0
10
20
30
40
50
60
Proccessing %

Circle Detection
Thresholding
Frame Conversion


-

33
-


Thresholding was the second most expensive operation.
The thresholding function
requires the examining of each pixel in the image and converting each pixel to black or
white based on a rule. This is performed seve
ral times in an attempt to identify the player
at the snooker table.

The conversion of each
frame from RGB to gray scale was the least extensive of the 3.



6.2

Image

Quality

In an attempt to increase the performance of the system, tests were carried out on a
n
image at different compression rates.

An image was used

where all

the snooker

balls were detected
on the snooker table on a
raw image.

The image was then compressed and the circle detection algorithm was run
again.

The below graph shows the ratio betwe
en compression ratio and the number of snooker
balls detected.


Figure
19

-

describes the relationship between the compression ratio and the numbe of balls detected.


This emphasised the need for a high quality image.


0
5
10
15
20
25
0
20
40
60
80
100
Compression Ratio Vs Number of Balls
Detected



-

34
-


7

Further
Work


Although the system is working efficiently and detects when a ball and what ball is
potted, there are some ways in which the system can be improved in the future.


7.1

Camera
Quality


In order to eliminate the effects of motion blur on an image, the
camera quality would
have to be increased. At the moment, a 30fps camera

is used. The max speed a snooker
ball can

travel at is approximately

50km/hr, which is also 14m/s.
With the use of a 30 fps
camera, the

ball

c
ould move 0.47m in 1
frame.

In snooker, a few centimetres can make a
big difference when it comes
to detecting if the white ball has moved and what ball it has
hit.

If a 5cm movement in the snooker ball is required at
the same max speed of
50km/hr, a
camera with a frame rate of 280
fps would be required. If a mor
e accurate reading of 1cm
movement is required, a frame rate of 1,400fps would be required.

These cameras were not used in this project
due to

financial
constraints
.


7.2

Processing Time


T
he

processing time of th
is system, can b
e improve
d in a number of ways:



Implemen
t white blob tracking to increase performance.

-
Instead of applying the circle detection algorithm every frame to detect
when a white bal
l

has moved, use blob detection to track the movement
of the white ball.

-

Whe
n the ball begins to move, the circle detection algorithm can then be
applied to detect which ball has been hit first.




-

35
-





Reduce
the
number of channels in image to be transformed, to one.

-
This will have the effect of reducing the computation required to
transform the (3channel) image being transfo
rmed to 1, and should
provide an
increase in

performance.




Change thresholding algorithm to use less memory.

-
In the system, as part of the thresholding operation, several extra data
structures are

allocated and
de
allocated.

-
This slows down computation and increases the amount of

memory used.
A more efficient algorithm using fewer resources would improve overall

processing time.

7.3

Lighting


A lot of testing was carried out to improve the lighting and eliminate
shadows on the
snooker table.

This was
achieved

by using a single light source directed at the centre of
the table.

This worked well but there were certain areas of the table c
lose to the edge of
the cushion
,

where the brightness of the image wasn’t the
same as that of the centre.


In the future a different type of lig
ht source, that would increase the brightness of the
whole table without the resulting shadows,
would be more beneficial to the system
.


7.4

Multi
-
Camera System


Although the single camera syste
m works well, a multi
-

camera system could be
implemented.
As well as the top
-
view camera that is in place at the moment, extra
cameras could be mounted at different angles around the table.

This could help detect


-

36
-


ball collisions when a snooker player is s
tretching over the table and blocking some of the
snooker balls.

In this way, a foul shot could be detected in nearly all situations.

7.5

Voice Recognition


Voice recognition could be implemented into the system. There are a few exceptions in
the rules of snoo
ker that only the referee can identify.

The system could process the
speech signals from the referee

so it can understand when these exceptions are to be
applied.

Voic
e recognition is a common tool
used today and could be
implemented to make this
system ev
en more complete.























-

37
-





8

Conclusion


It is evident from the successful implementation of the system that image processing can be used
to
automatically keep the score of a snooker game.

The system is functional an
d with a few more
refinements, t
he system should be ready t
o be tested in a competitive snooker game.


The project was greatly assisted by the OpenCV framework.

It has proven to be a powerful and
lightweight
computer
vision framework.


T
his single system camera
offers
an ef
fective
automatic
snooker score system. We see other
sports taking advantage of image processing to improve different aspects of their respective
sports.

Now,

snooker can take advantage
of these techniques too.

And with the implementation
of a multi
-
camera

system with voice recognition, the system could become very efficient and
could be deployed in snooker
clubs for early competitive testing.



















-

38
-




9

References


[1]


OpenCV
(
http://opencv.willowgarage.com/wiki/)

[2
]

-

The Rules of Snooker(
http://www.billiardworld.com/snooker.html
)

[3
]

-


Motion Blur(
http://lcdtvb
uyingguide.com/hdtv/motion
-
lag.html
)

[4
]



Intel Performance Primitives(
http://software.intel.com/en
-
us/articles/intel
-
ipp/
)

[5
]
-

Learning OpenCV


Computer Vision with the OpenCV

libraries
. Gary Bradski, Adrian
Kaehler.
2008. O.Reilly Media.






















-

39
-




10

Appendix



On the CD

Included on the submitted CD is the code developed throughout the project.

Code was developed for testing
both an image and a video and
have been
provided in
their respective folders.

Example videos of the final implementation of the system can be found in the results
folder.