Tagging of digital historical images

brasscoffeeAI and Robotics

Nov 17, 2013 (3 years and 8 months ago)

91 views

Tagging of digital historical
images

Authors
:

A. N. Talbonen

(antal@sampo.ru)

A. A. Rogov

(rogov@psu.karelia.ru)

Petrozavodsk state university

General tagging model

Object

selection

Tag

attribution

Indexing

Image

collection

Object

DB

File

Tags

I1

……

I2

……

Full
-
text

index

Tag DB

General research features


Research is based on analysis of
image collection of White Sea
-
Baltic
Sea Canal provided by National
museum of Karelia


Collection consists of about 8k images
with resolution 75 dpi.

1.
Face tagging

General features


P
redominance of small
-
sized objects (
width is less than

40
pixels
)


No database


Available expert

Distribution of object’s size

1.
Face tagging


General algorithm


Object (face) detection
.


Computing of pairwise distances
between objects
.


Tagging (for each object)
:


The system displays a list of the most
similar objects
.


The expert determines a relationship
between objects


Object tags are specified

1.
Face tagging

Face detection features


There is OpenCV library

(
OpenCvSharp in
C#
)
and it’s

method
cv
::CascadeClassifier
::
detectMultiScale

(haarDetectObject in C#) (Viola
-
Jones
implementation) being used for face
detection


Viola
-
Jones method parameters are used to
affect on precision and recall on face
detection results


There is face recognition method based on
Local Binary Patterns being used to
improve the quality of Viola
-
Jones results

Training set

Object

Recognition

Face

objects

Fake

objects

Object is

a face

Insert in result

collection

Yes

Face detection

Source image

1.
Face tagging

Face detection diagram

1.
Face tagging


Local binary patterns (LBP)

Original LBP filter

Advanced LBP filters

1.
Face tagging


Local binary patterns

Uniform codes

(patterns)

Rotation invariant

codes

1.
Face tagging


Local binary patterns

Weight matrix

Computing of face object histogram

1.
Face tagging


Face detection experiment


The purpose is to find the LBP modification


with the
best detection rates


Experiment features
:


Sample of

1070
images


Assessing features


Fake object when
:


Object is not a face


Faces are recognized weakly


Faces turned at an angle greater than 90 degrees


Face object when
:


Object is a face


Object is an image of people:
portraits, paintings,
sculptures


12 different LBP modifications were used

1.
Face tagging


Face detection experiment results

1.
Face tagging


Face recognition experiment


Purpose is to find the LBP
modification with the best face
recognition rates


Experiment features


Training set contains 19 objects including
3 relevant pairs of face objects and 1
relevant pair of fake objects


10

LBP modifications were used

1.
Face tagging


Face recognition experiment

1)

2)

3)

4)

5)

6)

7)

8)

9)

10)

11)

12)

13)

14)

15)

16)

17)

18)

19)

Pairs: {1, 15}, {3, 14}, {4, 13}, {7, 9}

1.
Face tagging


Face recognition experiment results

8,1
LBP
16,1
LBP
8,2
LBP
16,2
LBP
8,3
LBP
16,3
LBP
ri
16,3
LBP
riu
16,3
LBP
u
16,3
LBP
16,3
LBP
Взвешенный


Взвешенный


Взвешенный

Взвешенный


Modification

Precision

0
,
38

0
,
25

0
,
50

0
,
50

0
,
50

0
,
75

0
,
50

0
,
38

0
,
63

1
,
00

1.
Face tagging

Face comparing

Training set object’s histograms:

Objects at position (row, col): (1,1) and (3, 4) correspond to

fake objects and have similar histograms very different from the rest


2.
Texture tagging

General features


The classifier with tags based on
moments is built


Texture searching is based on the
built classifier


Search involves finding the segments
corresponding to different textures


Minimal segment size to be include in
result is 100 pixels

2.
Texture tagging

Moment
-
based segmentation

Moment calculation function
:

Source image
I

Moment image

M00

Moment image

M10

Moment image

M01

2.
Texture tagging

Moment
-
based segmentation

F00

F10

F01

Binary segmentation example

Precision
: 96,7 %

Moment feature calculation function
:

2.
Texture segmentation

Implementation features


Each moment is an image


Moment computing is based on
library OpenCV and it’s method

cv::filter2D


Parameter seek is based on
developed experiment

2.
Texture tagging


Parameter seek example

Moment

window size

Moment feature

Window size

Sigma

Precision

9

49

0,01

95,285

9

39

0,005

95,1782

9

39

0,02

95,1752

9

44

0,005

95,1355

9

49

0,015

95,1324

14

14

0,02

93,8416

14

14

0,005

93,7103

14

19

0,005

92,7826

14

19

0,015

92,7826

14

34

0,015

92,5293

14

29

0,015

92,395

14

34

0,02

92,3248

24

24

0,02

87,9639

39

19

0,01

87,9639

2.
Texture tagging

Classifier features


Set of textures of several classes is
given



Each class is assigned a set of tags


Each image is subjected to a
separate texture search


Each texture found adds appropriate
set of tags to the source image

2.
Texture tagging

Example

Source image

2.
Texture tagging

Example

Classifier example

Classifier textures example

2.
Texture tagging

Experiment


Purpose is to evaluate the search quality


Experiment features


Sample of

100
images


Classifier contains 2 textures

House roof

House wall

2.
Texture tagging

Search quality evaluate method







i
ij
i
ij
j
i
ij
i
ij
j
E
R
F
R
Re
;
Pr






j
i
ij
j
i
ij
j
i
ij
j
i
ij
E
R
F
R
,
,
,
,
Re
;
Pr
ij
E
ij
F
ij
ij
ij
R
E
R


-

Flag of belonging to

assessed collection

-

Flag of belonging to

search result

Flag of relevance

Single texture estimations:

General estimations:

2.
Texture tagging

Experiment results

Thanks for your attention
!