Image Processing Basics

molassesitalianAI and Robotics

Nov 6, 2013 (4 years and 3 days ago)

70 views

The University of








Ontario

CS
4487/9587

Algorithms for Image Analysis

Image
Processing Basics

Lecture 3


Lena Gorelick, substituting for Yuri
Boykov


Acknowledgements: slides from Steven Seitz,
Aleosha

Efros
, David Forsyth, and Gonzalez & Woods

The University of








Ontario

CS
4487/9587
Algorithms for Image Analysis


Image Processing Basics


Image Processing


Domain Transformation


Resize, Rotate etc.



Range Transformation


Point
Processing


gamma correction


window
-
center correction


histogram equalization


)
,
(
)
,
(
))
,
(
),
,
(
(
)
,
(
y
N
x
f
y
x
g
y
x
t
y
x
t
f
y
x
g
y
x



2
/
)
,
(
)
,
(
))
,
(
(
)
,
(
,
y
x
f
y
x
g
y
x
f
t
y
x
g
y
x


Window = 800

Center = 1160

Pixel location

Pixel Intensity

The University of








Ontario

CS
4487/9587
Algorithms for Image Analysis


Image Processing Basics


Image Processing


Domain Transformation


Resize, Rotate etc.



Range Transformation


Point
Processing


gamma correction


window
-
center correction


histogram equalization


Neighborhood Processing (Filtering)


)
2
,
2
(
)
,
(
))
,
(
),
,
(
(
)
,
(
y
x
f
y
x
g
y
x
t
y
x
t
f
y
x
g
y
x


2
/
)
,
(
)
,
(
))
,
(
(
)
,
(
,
y
x
f
y
x
g
y
x
f
t
y
x
g
y
x


The University of








Ontario

Neighborhood Processing (or filtering)


What is wrong with this image?






How can we remove noise?


Look spatially around each pixel
-

neighborhood


Can we use Point Processing?

Readings: Forsyth & Ponce, chapters 8.1
-
8.2


Courtesy of Carlo
Tomasi

Courtesy of Neel Joshi

The University of








Ontario

Neighborhood Processing (or filtering)


Lets reshuffle
all pixels within the
image





Point Processing will have the same effect


Gets intensity of a single pixel as an input


Unaware of spatial information


Neighborhood Processing


Takes “spatial information” into account


Images contain “spatial information”


Readings: Forsyth & Ponce, chapters 8.1
-
8.2


Has Spatial Information

Looks noisy

The University of








Ontario

Neighborhood Processing (filtering)

Linear image transforms


Assume 1D function














]
[
i
f

























0
1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
M





j
j
f
j
i
M
i
g
f
M
g
]
[
]
,
[
]
[
?
The University of








Ontario

Neighborhood Processing (filtering)

Linear image transforms


Assume 1D function














]
[
i
f


























1
1
0
0
0
1
1
0
0
0
1
1
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
0
0
0
1
1
0
0
0
1
1
0
0
0
2
2
/
1
M





j
j
f
j
i
M
i
g
f
M
g
]
[
]
,
[
]
[
?
The University of








Ontario


A commonly used pattern



Represented by a kernel

(or a mask)



For 2k+1 size kernel



(
i

-

1)

Neighborhood Processing (filtering)

Linear shift
-
invariant filters

matrix M

]
[
c
b
a
h


























*
*
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
*
*
b
a
c
b
a
c
b
a
c
b
a
c
b
a
c
b
a
M








k
k
u
u
i
f
u
h
i
g
f
M
g
]
[
]
[
]
[
i

(
i

+ 1)

The University of








Ontario


A 2D function

Can be filtered by a 2D filter (kernel)

To get a new image




This is called a
cross
-
correlation

operation

Or a
sliding dot
-
product


Neighborhood Processing (filtering)

2D filtering







f
h
g












k
k
u
k
k
v
v
j
u
i
f
v
u
h
j
i
g
]
,
[
]
,
[
]
,
[
]
,
[
j
i
f
]
,
[
v
u
h
The University of








Ontario


Cross
-
correlation in which the filter is flipped
horizontally and vertically

is called
convolution




Neighborhood Processing (filtering)

2D filtering







f
h
g
*











k
k
u
k
k
v
v
j
u
i
f
v
u
h
]
,
[
]
,
[












k
k
u
k
k
v
v
j
u
i
f
v
u
h
j
i
g
]
,
[
]
,
[
]
,
[
The University of








Ontario


If the kernel is symmetric



convolution = cross
-
correlation

Neighborhood Processing (filtering)

Convolution vs. Cross
-
Correlation







)
,
(
)
,
(
v
u
h
v
u
h



The University of








Ontario


Types of noise:


Salt
and pepper
noise


Impulse noise


Gaussian noise


Due to


transmission errors


dead CCD pixels


specks on lens


can be specific to a sensor

Neighborhood Processing

2D filtering

Noise

The University of








Ontario


How can we

remove noise?



Replace each pixel with

the average of a

k
x
k

window around it


0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

100

130

110

120

110

0

0

0

0

0

110

90

100

90

100

0

0

0

0

0

130

100

90

130

110

0

0

0

0

0

120

100

130

110

120

0

0

0

0

0

90

110

80

120

100

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Neighborhood Processing

Practical Noise Reduction

104

The University of








Ontario

Neighborhood Processing (filtering)


Mean filtering

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

0

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

]
,
[
y
x
f
]
,
[
y
x
g
The University of








Ontario

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

0

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Neighborhood Processing (filtering)


Mean filtering

0

10

20

30

30

30

20

10

0

20

40

60

60

60

40

20

0

30

60

90

90

90

60

30

0

30

50

80

80

90

60

30

0

30

50

80

80

90

60

30

0

20

30

50

50

60

40

20

10

20

30

30

30

30

20

10

10

10

10

0

0

0

0

0

]
,
[
y
x
f
]
,
[
y
x
g
The University of








Ontario

Neighborhood Processing (filtering)

Effect
of
mean
filters

3x3

5x5

7x7

Salt & Pepper noise

Gaussian noise

Side effect
-

blur

The University of








Ontario

Neighborhood Processing (filtering)


Mean kernel


What’s the kernel for a 3x3
mean

filter?


0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

0

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

]
,
[
y
x
f
]
,
[
v
u
h
The University of








Ontario

Neighborhood Processing (filtering)


Mean kernel


What’s the kernel for a 3x3
mean

filter?


0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

0

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1/9

1/9

1/9

1/9

1/9

1/9

1/9

1/9

1/9

]
,
[
y
x
f
]
,
[
v
u
h
The University of








Ontario

Neighborhood Processing (filtering)


Mean kernel


What’s the kernel for a 3x3
mean

filter
?








Equal weight to all pixels





within the neighborhood


0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

0

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

1

1

1

1

1

1

1

1

]
,
[
y
x
f
]
,
[
v
u
h
1/9


The University of








Ontario

Neighborhood Processing (filtering)


Gaussian Filtering


A
Gaussian kernel
gives less weight to pixels





further from the center





of the window

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

0

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

is a discrete approximation


of a Gaussian function:

1

2

1

2

4

2

1

2

1

]
,
[
y
x
f
]
,
[
v
u
h

16
1
The University of








Ontario

Neighborhood Processing (filtering)


Mean vs. Gaussian filtering

Input image

Gaussian filtering

Mean Filtering

Gaussian filter, 20x20,

σ

= 5

Mean filter, 20x20

The University of








Ontario

Neighborhood Processing (filtering)


Mean vs. Gaussian filtering

The University of








Ontario

Neighborhood Processing (filtering)


Gaussian Filtering


Low
-
pass filter


Smooth color variation (low
frequency) is preserved


Sharp edges (high frequency) are
removed

The University of








Ontario

Neighborhood Processing (filtering)


Median filters


A
Median Filter

operates over a window by
selecting the median intensity in the window.



Image credit: Wikipedia


page
on Median Filter

The University of








Ontario

Neighborhood Processing (filtering)


Median filters


Is
a median filter a kind of
convolution?

No, median filter is an example of non
-
linear
filtering

))
(
(
))
(
(
))
(
)
(
(
2
1
2
1
x
f
Median
x
f
Median
x
f
x
f
Median



2
1
1
)
0
1
0
1
2
1
0
1
0
(
)
1
0
1
0
1
0
1
0
1
(
























Median
Median


































)
0
1
0
1
2
1
0
1
0
1
0
1
0
1
0
1
0
1
(
)
1
1
1
1
3
1
1
1
1
(
1
Median
Median
The University of








Ontario

Salt&Pepper

Noise

The University of








Ontario

Gaussian

noise

The University of








Ontario

Neighborhood Processing (filtering)


Median filters


What
advantage does a
median filter
have
over a
mean filter
?

Better at removing Salt & Pepper noise


Disadvantage:

Slow


The University of








Ontario

Neighborhood Processing (filtering)

Derivatives and Convolution


Image Derivatives and Gradients


Used for Edge/Corner Detection


Computed with Finite Differences Filters


Laplacian

of Gaussians (
LoG
) Filter


Used for Edge/Blob Detection and

Image Enhancement


Approximated using Difference of Gaussians



The University of








Ontario

Reading: Forsyth & Ponce, 8.1
-
8.2

First Derivative


Recall
Sharp changes in gray level of the input
image correspond to “peaks or valleys” of the
first
-
derivative of the input signal.



http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of








Ontario





How can we approximate it

for a discrete function?



Is this operation shift
-
invariant?



Is it linear?


Reading: Forsyth & Ponce, 8.1
-
8.2

First Derivative and
convolution













h
y
x
f
y
h
x
f
f
h
x
)
,
(
)
,
(
lim
0
?

?

?

?

?

?

?

?

?

]
,
[
v
u
x

The University of








Ontario

Reading: Forsyth & Ponce, 8.1
-
8.2

First Derivative
and convolution


Finite Difference





x

x
-
h

x

x+h

x

x+h
/2

x
-
h/2

)
(
'
)
(
)
(
0
]
[
x
f
h
x
f
h
x
f
h
h
h
f









)
(
)
(
b
x
f
a
x
f



Slide credit: Wikipedia

The University of








Ontario

Reading: Forsyth & Ponce, 8.1
-
8.2

First Derivative
and convolution


Finite Difference


The order of an error

can be derived using Taylor Theorem





Slide credit: Wikipedia

The University of








Ontario

Reading: Forsyth & Ponce, 8.1
-
8.2

First Derivative
and convolution


Pixel Size


Using Finite Central Difference




We need a kernel such that

x
y
x
x
f
y
x
x
f
f
x










2
)
,
(
)
,
(
f
f
x
x





0

0

0

1

0

-
1

0

0

0

]
,
[
v
u
x



x
2
1
h
x


x

The University of








Ontario

Neighborhood Processing (filtering)


Finite differences


responds to edges

Dark = negative

White = positive

Gray = 0

f
x


The University of








Ontario

Neighborhood Processing (filtering)


Finite differences
-

responding to noise

Increasing noise
-
>

(this is zero mean additive gaussian noise)

f
x


f
x


f
x


The University of








Ontario

Neighborhood Processing (filtering)


Finite
differences and noise


Finite difference filters respond strongly to noise


Noisy pixels look
very different from their
neighbours


The
larger the noise



the stronger is
the
response



How can we eliminate the response to noise?


Most pixels in images look similar to their
neighbours

(even at an edge)


Smooth the image (mean/
gaussian

filtering)

The University of








Ontario

Neighborhood Processing (filtering)


Smoothing
and Differentiation


Smoothing
before
differentiation = two convolutions




Convolution
is associative

)
(
H
x


f
H
f
H
x
x







)
(
)
(
)
(
H
y


)
(
f
H
x



The University of








Ontario

The scale of the smoothing filter affects derivative estimates, and also

the semantics of the edges recovered.

1 pixel

3 pixels

7 pixels

f
H
x



)
(
Neighborhood Processing (filtering)


Smoothing and Differentiation

The University of








Ontario


Yet another approximation frequently used

Sobel

kernels


1

0

-
1

2

0

-
2

1

0

-
1

]
,
[
v
u
x



x
8
1
f
x


1

2

1

0

0

0

-
1

-
2

-
1

]
,
[
v
u
y



y
8
1
f
y


Neighborhood Processing (filtering)



The University of








Ontario


Recall for a function of
two variables


The
gradient

at a point (
x,y
)





Gradient Magnitude





Gradient Orientation


direction of the “steepest ascend”


orthogonal to object boundaries in the image








x
f


y
f


Neighborhood Processing (filtering)


Image Gradients












f
f
y
x














y
f
x
f
f
)
,
(
y
x
f
x

y

small image

gradients in low

textured areas

2
2
2
2
)
(
)
(
)
(
)
(
||
||
f
f
f
y
x
y
f
x
f













The University of








Ontario

Neighborhood Processing (filtering)


Image Gradient for Edge Detection


Typical application where image gradients are used
is
image edge
detection


find points with large image gradients




Canny edge detector
suppresses

non
-
extrema

Gradient points

The University of








Ontario

Reading: Forsyth & Ponce, 8.1
-
8.2

Second derivatives
and convolution


Peaks or valleys of the first
-
derivative of the input
signal, correspond to “zero
-
crossings” of the
second
-
derivative of the input signal.


http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of








Ontario

Neighborhood Processing (filtering)


Second derivatives and convolution

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of








Ontario

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering)


Second derivatives and convolution


Better localized edges


But more sensitive to
noise

The University of








Ontario

Neighborhood Processing (filtering)


Second derivatives and convolution

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of








Ontario

Neighborhood Processing (filtering)


Second Image Derivatives


Laplace operator

f
f
f
2







f
f
y
f
x
f
f
y
x
y
x




























]
[
2
2
2
2
0

0

0

-
1

2

-
1

0

0

0

0

-
1

0

0

2

0

0

-
1

0


0

-
1

0

-
1

4

-
1

0

-
1

0


“divergence

of gradient”

rotationally invariant

second derivative for 2D functions

Finite Difference

Second Order

Derivative in x

Finite Difference

Second Order

Derivative in y

The University of








Ontario

Neighborhood Processing (filtering)


Second Image Derivatives


Laplacian

Zero Crossing



Used for edge detection

(alternative to computing Gradient
extrema
)


http://homepages.inf.ed.ac.uk/rbf/HIPR2/zeros.htm

magnitude of

image gradient

image

intensity


Laplacian


(2
nd

derivative)


of the image

2
2
2
2
y
f
x
f
f







f

||
||
f

f
The University of








Ontario

Neighborhood Processing (filtering)

Laplacian

Filtering

-
Zero on uniform regions

-
Positive on one side

of an edge

-
Negative on the other side


-
Zero at some point in between

on the edge itself



band
-
pass filter (
Suppresses both high and low frequencies
)

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of








Ontario


Neighborhood Processing (filtering)


Laplacian

of a Gaussian (
LoG
)


Smooth before differentiation

(remember associative property of convolution)

LoG
G



2
2
2
2
2
2
2
4
2
1
1
)
,
(



y
x
e
y
x
y
x
LoG













http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of








Ontario


Neighborhood Processing (filtering)


Laplacian

of a Gaussian (
LoG
)

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Suppresses both high and low frequencies

The University of








Ontario


Neighborhood Processing (filtering)


Laplacian

of a Gaussian (
LoG
)

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of








Ontario


Neighborhood Processing (filtering)


Laplacian

of a Gaussian (
LoG
)

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of








Ontario


Neighborhood Processing (filtering)


Laplacian

of a Gaussian (
LoG
)


Can be approximated by a difference of two

Gaussians (
DoG
)

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of








Ontario


Neighborhood Processing (filtering)


DoG

vs.
LoG


Separable (product decomposition)


more efficient


Can explain band
-
pass filter since Gaussian is a low
-
pass filter.

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of








Ontario

Neighborhood Processing (filtering)


LoG

for Blob Detection


Cross correlation with a filter can be viewed as
comparing a little “picture” of what you want to find
against all local regions in the image.



Scale of blob (size ; radius in pixels)

is determined by the sigma parameter

of the
LoG

filter.

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of








Ontario


Neighborhood Processing (filtering)


LoG

for Blob Detection

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of








Ontario

Reading: Forsyth & Ponce ch.7.5


Filters
and
T
emplates


Recall: filtering = cross
-
correlation = dot
-
product








Measures the angle between two vectors:

the filter and the image window



Measures visual similarity


f
h
*
(
x,y
)

Filter Mask

Input Image

f(
x,y
)

g(
x,y
)

Output Image

(
x,y
)

Correlation

)
cos(
)
,
(
||
||||
||
arccos
)
,
(






y
x
g
f
h
f
h
y
x
f
f
w
w
w
w
The University of








Ontario


Since we divide by the norm

Normalized cross
-
correlation



Bring the template and the image to a uniform scale:

subtract the mean and divide by the variance







Neighborhood Processing (filtering)


Template Matching

||
||||
||
arccos
)
,
(
w
w
w
w
f
h
f
h
y
x
f
f




http://scien.stanford.edu/pages/labsite/2008/psych221/projects/08/MariaJabon/index.htm

)
,
(
y
x
h
)
,
(
y
x
f
The University of








Ontario


Normalized cross
-
correlation




Extremely time consuming due to sliding window


Neighborhood Processing (filtering)


Template Matching

h
f
1
|
|
)
cos(


g
g

Slide

credit:
Matlab

manual

||
||||
||
arccos
)
,
(
w
w
w
w
f
h
f
h
y
x
f
f




The University of








Ontario

Neighborhood Processing (filtering)


Template Matching

h
f
)
cos(


g
Slide

credit:
OpenCV

manual