Storing and Retrieving Images

cowphysicistInternet και Εφαρμογές Web

4 Δεκ 2013 (πριν από 3 χρόνια και 10 μήνες)

188 εμφανίσεις

G52IIP, School of Computer Science, University of Nottingham

1

Storing and Retrieving Images


Content
-
based Image/Video Indexing and Retrieval



Selected Advanced Topics

G52IIP, School of Computer Science, University of Nottingham

2

Problem

Find all
images
contain
horses …..

G52IIP, School of Computer Science, University of Nottingham

3

Text
-
based technology


Annotation
: Each image is indexed with a
set of relevant text phrases, e.g.,







Retrieval
: based on text search technology

Appropriate phrases to
describe the content of this
image include:


Mother, Child, Vegetable,
Yellow, Green, Purple ….

G52IIP, School of Computer Science, University of Nottingham

4

Text
-
based technology
-

Drawbacks


Annotation
-

subjective


different people may use different phrases
to describe the same or very similar
image/content

G52IIP, School of Computer Science, University of Nottingham

5

Text
-
based technology
-

Drawbacks


Annotation
-

Laborious



It will take a lot of man
-
hours to label large
image/video databases with 1m+ items


G52IIP, School of Computer Science, University of Nottingham

6

Content
-
based Technology

Using Visual Examples

G52IIP, School of Computer Science, University of Nottingham

7

Content
-
based Technology

r%

g%

b%

Using Visual Features

G52IIP, School of Computer Science, University of Nottingham

8

Content
-
based Technology


Content
-
based image indexing and retrieval (CBIR), is
an image database management technique, which
indexes the data items (images, or video clips) using
visual features

(e.g., color, shape, and texture) of the
images or video clips.



A CBIR system lets users find pictorial information in
large image and video databases based on
visual cues
,
such as colour, shape, texture, and sketches.


G52IIP, School of Computer Science, University of Nottingham

9

Content
-
based Technology


The
visual features
, computed using image
processing and computer vision techniques are used
to represent the
image contents

numerically.



Image Content
-

a
high level concept
, e.g., this
image is a sunset scene, a landscape scene, etc.



Numerical Content Representations
-

Low level

numbers, often the same set of numbers can come
from very different images, making the task
very
hard!

G52IIP, School of Computer Science, University of Nottingham

10

Content
-
based Technology


Techniques for Computing Visual
Features/Representing Image Contents




some are very sophisticated, and many are still not matured


hence the computational processes


in some cases are automatic


but in other cases are semi
-
automatic


in the most difficult cases, it may have to be done manually


G52IIP, School of Computer Science, University of Nottingham

11

Content
-
based Technology


Comparing Image Content/Retrieving Images based
on Content



Simple approaches
-

compute the metric distance between
low level numerical representations



Advanced Approaches
-

using sophisticated pattern
recognition, artificial intelligence, neural networks, and
interactive (relevant feed
-
back) techniques to compare the
visual content (low level numerical features)


G52IIP, School of Computer Science, University of Nottingham

12

Content
-
based Technology

-

IBM QBIC
System


The IBM’s QBIC (Query by Image and Video Content)
system is one of the early examples of CBIR system
developed in 1990s.



The system lets users find pictorial information in
large image and video databases based on color,
shape, texture, and sketches.


G52IIP, School of Computer Science, University of Nottingham

13

Content
-
based Technology

-

IBM QBIC
System


The User Interfaces Module


Let user specify visual query by drawing, selecting from a color wheel,
selecting a sample image …


Display results as an ordered set of images



The Database Population and Database Query Modules


Database population
-

process images and video to extract features
describing their content
-

colors, textures, shapes and camera and object
motion, and store the features in a database


Database Query
-

let user compose a query graphically, extract features
from the graphical query, input to a matching engine that finds images or
video clips with similar features

G52IIP, School of Computer Science, University of Nottingham

14

Content
-
based Technology

-

IBM QBIC
System


The Data Model



Still image, or scene
-

full image


Objects contained in the full image
-

subsets of an image


Videos
-

broken into clips called shots
-

sets of contiguous
frames


Representative frames, the r
-
frames, are generated for each
shot


R
-
frames are treated as still image
-

from which features are
extracted and stored in the database.


Further processing of shots generates motion objects
-

e.g.,
a car moving across the screen.

G52IIP, School of Computer Science, University of Nottingham

15

Content
-
based Technology

-

IBM QBIC
System


Queries are allowed on



Objects
-

e.g., Find images with a red round object


Scenes
-

e.g., Find images that have approximately 30% red
and 15% blue colors


Shots
-

e.g., Find all shots panning from left to right


A combination of above
-

e.g., Find images that have 30%
red and contain a blue textured objects


G52IIP, School of Computer Science, University of Nottingham

16

Content
-
based Technology

-

IBM QBIC
System


Similarity Measures



Similarity queries are done against the database of pre
-
computed
features using distance functions between the features


Examples include, Euclidean distance, City
-
block distance, ….


These distance functions are intended to mimic human perception
to approximate a perceptual ordering of the database



But, it is often the case that a distance metric in a feature space
will bear little relevance to perceptual similarity.


G52IIP, School of Computer Science, University of Nottingham

17

Content
-
based Technology
-

Basic Architecture

Meta data

Imagery

color

texture

shape

positions

….

Record1

color

texture

shape

positions

….

Record2

color

texture

shape

positions

….

Record3

color

texture

shape

positions

….

Record4

color

texture

shape

positions

….

Record n

color

texture

shape

positions

….

Image Database

Query

Similarity Measures

G52IIP, School of Computer Science, University of Nottingham

18

Colour
-

An effective Visual Cue

Colors can be a more powerful visual cue than you initially thought!

What soft drink

Which fruit?

G52IIP, School of Computer Science, University of Nottingham

19

Colour
-

An effective Visual Cue

In many cases,
color can be very
effective.



Here is an example



Results of content
-
based image retrieval using 4096
-
bin color
histograms

G52IIP, School of Computer Science, University of Nottingham

20

Colour Spaces

Colour Models


RGB Model
: This colour model uses the three NTSC primary colours to describe
a colour within a colour image.

R

G

B

Yellow

Cyan

Magenta

White

gray

Sometimes in Computer Vision, it is

convenient to use rg chromaticity
space


r = R/(R+G+B)

g= G/(R+G+B)

G52IIP, School of Computer Science, University of Nottingham

21

Colour Spaces

YIQ Model
: The YIQ models is used in commercial colour TV broadcasting, which
is a re
-
coding of RGB for transmission efficiency and for maintaining compatibility
with monochrome TV standard.



































B
G
R
Q
I
Y
331
.
0
523
.
0
212
.
0
321
.
0
275
.
0
596
.
0
114
.
0
587
.
0
299
.
0
In YIQ, the luminance (Y) and colour information (I and Q) are de
-
coupled.

YCbCr Model




Y = 0.299R + 0.587G + 0.114B



Cb =
-
0.169R
-

0.331G + 0.500B



Cr = 0.500R
-

0.419G
-

0.081B

G52IIP, School of Computer Science, University of Nottingham

22

Perceived Color Differences


One problem with the RGB colour
system is that colorimetric
distances between the individual
colours don't correspond to
perceived colour differences.


For example, in the chromaticity
diagram, a difference between
green and greenish
-
yellow is
relatively large, whereas the
distance distinguishing blue and red
is quite small.


r = R/(R+G+B)

g= G/(R+G+B)

G52IIP, School of Computer Science, University of Nottingham

23

CIELAB


CIE
(
Commission Internationale de
l'Eclairage
)

solved this problem in 1976
with the development of the Lab colour
space. A three
-
dimensional color space
was the result. In this model, the color
differences which you perceive
correspond to distances when measured
colorimetrically. The a axis extends from
green (
-
a) to red (+a) and the b axis
from blue (
-
b) to yellow (+b). The
brightness (L) increases from the
bottom to the top of the three
-
dimensional model.


With CIELAB what you see is what you
get (in theory at least).

G52IIP, School of Computer Science, University of Nottingham

24

Colour Histogram



Given a discrete colour space defined by some
colour axes (e.g., red, green, blue), the colour
histogram is obtained by discretizing the image
colours and counting the number of times each
discrete colour occurs in the image.



The image colours that are transformed to a
common discrete colour are usefully thought of as
being in the same 3D histogram bin centered at
that colour.

G52IIP, School of Computer Science, University of Nottingham

25

Colour Histogram Construction



Step 1


Colour quantization (discretizing the image colours)



Step 2



Count the number of times each discrete colour
occurs in the image.

G52IIP, School of Computer Science, University of Nottingham

26

Colour Quantization



A true colour, 24
-
bit/pixel image (8 bit
-

R, 8 bit
-

G, 8 bit
-
B),
will have 2
24

= 16777216 bins !



That is, each image will have to be represented by over 16
million numbers



computationally impossible


in practice not necessary



Colour quantization
-

reduce the number of (colours) bins

G52IIP, School of Computer Science, University of Nottingham

27

Simple Colour Quantization


Simple Colour Quantization (Non
-
adaptive)



Divide each colour axis into equal length
sections (different axis can be divided
differently).



Map (quantize) each colour into its
corresponding bin

G52IIP, School of Computer Science, University of Nottingham

28

Simple Colour Quantization

0

31

63

95

127

159

191

223

255

0

31

63

95

127

159

191

223

255

0

31

63

95

127

159

191

223

255

R

G

B

Colour


Bin


Colour


Bin



(123,23,45)


(3, 0, 1 )


(122, 28, 46)

(3, 0, 2)


(132, 29,50)


(4, 0, 1)


(122, 172, 27)

(3, 5, 0)


(121,26,48)

(x, x, x)


(142, 28, 46)

(x, x, x)

Example: In RGB space, quantize each image colour into one of 8x8x8 =
512 colour bins

G52IIP, School of Computer Science, University of Nottingham

29

Advanced Colour Quantization


Adaptive Colour quantization (Not
required)



Vector Quantization


K
-
means clustering


K representative colours


The colour histogram consists of K bins, each
corresponding to one of the representative
colours.


A pixels is classified as belonging to the nth
bin if the nth representative colour is the one
(amongst all the representative colours) that
is closest to the pixel.

R

G

B

A pixel is a point
in the 3D colour
space

Representative colours

G52IIP, School of Computer Science, University of Nottingham

30

Colour Histogram Construction
-

An Example


A 3 x 3, 24
-
bit/pixel image has following RGB planes







Construct an 8
-
bin colour histogram (using simple colour
quantization, treating each axis as equally important).

Red


23

24

77

11

24

69

22

12

12

Green


213

24

77

11

232

239

22

12

12

Blue


23

24

77

12

24

69

22

123

123

Bin (0,0,0) =


Bin (0,0,1) =


Bin (0,1,0) =


Bin (0,1,1) =


Bin (1,0,0) =


Bin (1,0,1) =


Bin (1,1,0) =


Bin (1,1,1) =

G52IIP, School of Computer Science, University of Nottingham

31

Colour Histogram Construction
-

An Example


Quantized Colour Planes






Count the number of times each discrete colour
occurs in the image.


Red


0

0

0

0

0

0

0

0

0

Green


1

0

0

0

1

1

0

0

0

Blue


0

0

0

0

0

0

0

0

0

Bin (0,0,0) = 6


Bin (0,0,1) = 0


Bin (0,1,0) = 3


Bin (0,1,1) = 0


Bin (1,0,0) = 0


Bin (1,0,1) = 0


Bin (1,1,0) = 0


Bin (1,1,1) = 0

G52IIP, School of Computer Science, University of Nottingham

32

Colour Based Image Indexing

# of pixels

# of pixels

10

0

0

0

100

10

30

0

0

Color

Distribution

=

(10,0,0,0,100,10,30,0,0)

The histogram of colours in an image defines the image colour distribution

G52IIP, School of Computer Science, University of Nottingham

33

Colour based Image Retrieval


Images are similar if their histograms are similar!

Colour

Distribution

=

(10,0,0,0,100,10,30,0,0)

Colour

Distribution

=

(10,0,0,0,90,10,40,0,0)

Colour

Distribution

=

(0,40,0,0,0,0,0,0,110,0)

Dissimilar

Similar!

G52IIP, School of Computer Science, University of Nottingham

34

Formalizing Similarity

Colour

Distribution

=

(10,0,0,0,100,10,30,0,0)

1
H

Colour

Distribution

=

(0,40,0,0,0,0,0,110,0)

2
H

Similarity(Image
1
, Image
2
) = D (H
1
, H
2
)


where D( ) is a distance measure between vectors (histograms)
H
1
and H
2

1

2

G52IIP, School of Computer Science, University of Nottingham

35

Metric Distances

A distance measure D( ) is a good measure if it is a metric!

D(a,b) is a metric if

D(a,a) = 0 (the distance from a to itself is 0

D(a,b) = D(b,a) (the distance from a to b = distance from b to a)

D(a,c) <= D(a,b) + D(b,c)

( triangle inequality [ the straight line distance is always the least!] )

D(a,b) + D(b,c) should be no smaller

than D(a,c)

a

b

c

G52IIP, School of Computer Science, University of Nottingham

36

Common Metric Distance measures



n
i
i
i
H
H
1
2
1
)
,
min(
HI(H
1
,

H
2
) =

H
1
=
(10, 0, 0, 0, 100, 10, 30, 0, 0)

H
2
=

( 0, 40, 0, 0, 0, 6, 0, 110, 0)

Similarity = HI(H
1
, H
2
) = 0 + 0 + 0 + 0 + 0 + 6 + 0 + 0 = 6

Histogram Intersection, HI

G52IIP, School of Computer Science, University of Nottingham

37

Common Metric Distance measures

Euclidean or straight
-
line distance or L2
-
norm,
D
2




2
2
1
2
2
1
2
1
2
)
,
(
H
H
H
H
H
H
D
i
i
i





Root
-
mean square

error

H
1
=
(10, 0, 0)

H
2
=

( 0, 40, 0)

Similarity = D
2
(H
1
, H
2
) = sqrt(100 + 1600 +0) = 41.23

G52IIP, School of Computer Science, University of Nottingham

38

Common Metric Distance measures

Manhattan or city
-
block or L1
-
norm,
D
1

1
2
1
2
1
2
1
1
)
,
(
H
H
H
H
H
H
D
i
i
i





sum of absolute

differences

H
1
=
(10, 0, 0)

H
2
=

( 0, 40, 0)

Similarity = D
1
(H
1
, H
2
) = (10 + 40 +0) = 50

G52IIP, School of Computer Science, University of Nottingham

39

Histogram Intersection vs City
Block Distance

Theorem
:
if

H
1

and H
2

are colour histograms and the total count

in each is N (there are N
-
pixels in an image)
then:

1
2
1
2
1
5
.
0
H
H
H
H
N




(Histogram Intersection inversely proportional to a metric distance!)

Proof



i
i
i
H
H
H
H
)
,
min(
2
1
2
1

(by definition)

1
)
,
max(
)
,
min(
b
a
b
a
b
a



(1)

(2)

G52IIP, School of Computer Science, University of Nottingham

40

Histogram Intersection vs City Block Distance

)
,
min(
)
,
max(
b
a
b
a
b
a



(3)

Substituting (2) and (3) in (1)















i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
i
H
H
H
H
H
H
H
H
H
H
H
H
1
2
1
2
1
2
1
1
2
1
2
1
2
1
)
,
min(
)
,
max(
)
,
min(
(4)

1
2
1
2
1
)
,
min(
2
H
H
N
N
H
H
i
i
i





(5)

1
2
1
2
1
5
.
0
H
H
H
H
N





G52IIP, School of Computer Science, University of Nottingham

41

Build

Histogram

Histogram

Database

(1)

(2)

Colour Histogram Database

G52IIP, School of Computer Science, University of Nottingham

42

How well does Color histogram intersection work ?

66 test histograms in the database

31 query images

Recognition rate almost 100%

Indeed, because color indexing worked so well it is at

he heart of almost all image database systems

Swain Original Test:

G52IIP, School of Computer Science, University of Nottingham

43

Google Image Search

G52IIP, School of Computer Science, University of Nottingham

44

Google Image Search

After clicking
this colour
patch

G52IIP, School of Computer Science, University of Nottingham

45

Problems with color histogram matching

1.

Color Quantization problem:

Colour

Distribution

=

(0,40,0,0,0,0,0,110,0)

Colour

Distribution

=

(0,0,40,0,0,0,0,0,110)

Because, the

two images have

slightly different

color distributions

their histograms have

nothing

in common!

0 intersection!

Sources of quantization error: noise, illumination, camera

G52IIP, School of Computer Science, University of Nottingham

46

Problems with color histogram matching

2.

The resolution of a color histogram

Colour

Distribution

=

(0,40,0,0,0 … ,0,0,110,0)

For the best results, Swain quantized colour

space into 4096 distinct colours => Each colour

distribution is a 4096
-
dimensional vector.

=> Histogram intersection costs O(4096) operations

(some constant * 4096)

4096 comparisons per database histogram => histogram intersection

will be very slow for large databases

Many newer methods work well using 8
-

64 D features

G52IIP, School of Computer Science, University of Nottingham

47

Problems with color histogram matching

3.

The colour of the light

Under a yellowish light all image colours are

more yellow than they ought to be

G52IIP, School of Computer Science, University of Nottingham

48

Problems with color histogram matching

All four images have the same color distribution
-

need to take into
account spatial relationships!

4.

The structure of colour distribution

G52IIP, School of Computer Science, University of Nottingham

49

Problem solution => Use statistical moments

1st order statistics

=

Mean

B
G
R



,
,
=

Variance/

Covariance

GB
RB
RG
B
G
R






,
,
,
,
,
2
2
2
2nd order statistics

2
2
)
(
/
1
R
i
i
R
R
N





)
)(
(
/
1
G
i
R
i
i
RG
G
R
N







G52IIP, School of Computer Science, University of Nottingham

50

Statistical similarity

Colour

Distribution

=

(50,50,50)

Colour

Distribution

=

(20,70,40)









....
]
[
...
)]
(
)
(
[
)]
(
)
(
[
2
2
2
2
2
1
2
2
1
G
R
G
G
R
R
I
I
I
I






Statistical similarity =

Compare mean RGBs

(In general compare all statistical measures)

(Euclidean distance between corresponding statistical measures)

G52IIP, School of Computer Science, University of Nottingham

51

Histogram vs Statistical Similarity

histogram

Low order stats

Low and high

order stats

Completeness of

representation

#params/

Match speed

Sensitivity to
Quantization error

Many/slow

Few/fast

Many/moderate

complete

incomplete

complete

(or over complete)

sensitive

insensitive

sensitive

G52IIP, School of Computer Science, University of Nottingham

52

Advanced Topics


Fast Indexing


Interactive/Relevant Feedback


Reducing the Semantic Gap


Visualization, Navigation, Browsing


Internet scale image/video retrieval


Flickr


billions of photos


Youtube


billions of videos