# Image Processing Basics

AI and Robotics

Nov 6, 2013 (4 years and 6 months ago)

105 views

The University of

Ontario

CS
4487/9587

Algorithms for Image Analysis

Image
Processing Basics

Lecture 3

Lena Gorelick, substituting for Yuri
Boykov

Acknowledgements: slides from Steven Seitz,
Aleosha

Efros
, David Forsyth, and Gonzalez & Woods

The University of

Ontario

CS
4487/9587
Algorithms for Image Analysis

Image Processing Basics

Image Processing

Domain Transformation

Resize, Rotate etc.

Range Transformation

Point
Processing

gamma correction

window
-
center correction

histogram equalization

)
,
(
)
,
(
))
,
(
),
,
(
(
)
,
(
y
N
x
f
y
x
g
y
x
t
y
x
t
f
y
x
g
y
x

2
/
)
,
(
)
,
(
))
,
(
(
)
,
(
,
y
x
f
y
x
g
y
x
f
t
y
x
g
y
x

Window = 800

Center = 1160

Pixel location

Pixel Intensity

The University of

Ontario

CS
4487/9587
Algorithms for Image Analysis

Image Processing Basics

Image Processing

Domain Transformation

Resize, Rotate etc.

Range Transformation

Point
Processing

gamma correction

window
-
center correction

histogram equalization

Neighborhood Processing (Filtering)

)
2
,
2
(
)
,
(
))
,
(
),
,
(
(
)
,
(
y
x
f
y
x
g
y
x
t
y
x
t
f
y
x
g
y
x

2
/
)
,
(
)
,
(
))
,
(
(
)
,
(
,
y
x
f
y
x
g
y
x
f
t
y
x
g
y
x

The University of

Ontario

Neighborhood Processing (or filtering)

What is wrong with this image?

How can we remove noise?

Look spatially around each pixel
-

neighborhood

Can we use Point Processing?

Readings: Forsyth & Ponce, chapters 8.1
-
8.2

Courtesy of Carlo
Tomasi

Courtesy of Neel Joshi

The University of

Ontario

Neighborhood Processing (or filtering)

Lets reshuffle
all pixels within the
image

Point Processing will have the same effect

Gets intensity of a single pixel as an input

Unaware of spatial information

Neighborhood Processing

Takes “spatial information” into account

Images contain “spatial information”

Readings: Forsyth & Ponce, chapters 8.1
-
8.2

Has Spatial Information

Looks noisy

The University of

Ontario

Neighborhood Processing (filtering)

Linear image transforms

Assume 1D function

]
[
i
f

0
1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
1
0
0
0
0
1
0
0
0
0
M

j
j
f
j
i
M
i
g
f
M
g
]
[
]
,
[
]
[
?
The University of

Ontario

Neighborhood Processing (filtering)

Linear image transforms

Assume 1D function

]
[
i
f

1
1
0
0
0
1
1
0
0
0
1
1
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
0
0
0
1
1
0
0
0
1
1
0
0
0
2
2
/
1
M

j
j
f
j
i
M
i
g
f
M
g
]
[
]
,
[
]
[
?
The University of

Ontario

A commonly used pattern

Represented by a kernel

For 2k+1 size kernel

(
i

-

1)

Neighborhood Processing (filtering)

Linear shift
-
invariant filters

matrix M

]
[
c
b
a
h

*
*
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
*
*
b
a
c
b
a
c
b
a
c
b
a
c
b
a
c
b
a
M

k
k
u
u
i
f
u
h
i
g
f
M
g
]
[
]
[
]
[
i

(
i

+ 1)

The University of

Ontario

A 2D function

Can be filtered by a 2D filter (kernel)

To get a new image

This is called a
cross
-
correlation

operation

Or a
sliding dot
-
product

Neighborhood Processing (filtering)

2D filtering

f
h
g

k
k
u
k
k
v
v
j
u
i
f
v
u
h
j
i
g
]
,
[
]
,
[
]
,
[
]
,
[
j
i
f
]
,
[
v
u
h
The University of

Ontario

Cross
-
correlation in which the filter is flipped
horizontally and vertically

is called
convolution

Neighborhood Processing (filtering)

2D filtering

f
h
g
*

k
k
u
k
k
v
v
j
u
i
f
v
u
h
]
,
[
]
,
[

k
k
u
k
k
v
v
j
u
i
f
v
u
h
j
i
g
]
,
[
]
,
[
]
,
[
The University of

Ontario

If the kernel is symmetric

convolution = cross
-
correlation

Neighborhood Processing (filtering)

Convolution vs. Cross
-
Correlation

)
,
(
)
,
(
v
u
h
v
u
h

The University of

Ontario

Types of noise:

Salt
and pepper
noise

Impulse noise

Gaussian noise

Due to

transmission errors

specks on lens

can be specific to a sensor

Neighborhood Processing

2D filtering

Noise

The University of

Ontario

How can we

remove noise?

Replace each pixel with

the average of a

k
x
k

window around it

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

100

130

110

120

110

0

0

0

0

0

110

90

100

90

100

0

0

0

0

0

130

100

90

130

110

0

0

0

0

0

120

100

130

110

120

0

0

0

0

0

90

110

80

120

100

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Neighborhood Processing

Practical Noise Reduction

104

The University of

Ontario

Neighborhood Processing (filtering)

Mean filtering

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

0

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

]
,
[
y
x
f
]
,
[
y
x
g
The University of

Ontario

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

0

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Neighborhood Processing (filtering)

Mean filtering

0

10

20

30

30

30

20

10

0

20

40

60

60

60

40

20

0

30

60

90

90

90

60

30

0

30

50

80

80

90

60

30

0

30

50

80

80

90

60

30

0

20

30

50

50

60

40

20

10

20

30

30

30

30

20

10

10

10

10

0

0

0

0

0

]
,
[
y
x
f
]
,
[
y
x
g
The University of

Ontario

Neighborhood Processing (filtering)

Effect
of
mean
filters

3x3

5x5

7x7

Salt & Pepper noise

Gaussian noise

Side effect
-

blur

The University of

Ontario

Neighborhood Processing (filtering)

Mean kernel

What’s the kernel for a 3x3
mean

filter?

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

0

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

]
,
[
y
x
f
]
,
[
v
u
h
The University of

Ontario

Neighborhood Processing (filtering)

Mean kernel

What’s the kernel for a 3x3
mean

filter?

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

0

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1/9

1/9

1/9

1/9

1/9

1/9

1/9

1/9

1/9

]
,
[
y
x
f
]
,
[
v
u
h
The University of

Ontario

Neighborhood Processing (filtering)

Mean kernel

What’s the kernel for a 3x3
mean

filter
?

Equal weight to all pixels

within the neighborhood

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

0

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

1

1

1

1

1

1

1

1

]
,
[
y
x
f
]
,
[
v
u
h
1/9

The University of

Ontario

Neighborhood Processing (filtering)

Gaussian Filtering

A
Gaussian kernel
gives less weight to pixels

further from the center

of the window

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

90

0

90

90

90

0

0

0

0

0

90

90

90

90

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

90

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

is a discrete approximation

of a Gaussian function:

1

2

1

2

4

2

1

2

1

]
,
[
y
x
f
]
,
[
v
u
h

16
1
The University of

Ontario

Neighborhood Processing (filtering)

Mean vs. Gaussian filtering

Input image

Gaussian filtering

Mean Filtering

Gaussian filter, 20x20,

σ

= 5

Mean filter, 20x20

The University of

Ontario

Neighborhood Processing (filtering)

Mean vs. Gaussian filtering

The University of

Ontario

Neighborhood Processing (filtering)

Gaussian Filtering

Low
-
pass filter

Smooth color variation (low
frequency) is preserved

Sharp edges (high frequency) are
removed

The University of

Ontario

Neighborhood Processing (filtering)

Median filters

A
Median Filter

operates over a window by
selecting the median intensity in the window.

Image credit: Wikipedia

page
on Median Filter

The University of

Ontario

Neighborhood Processing (filtering)

Median filters

Is
a median filter a kind of
convolution?

No, median filter is an example of non
-
linear
filtering

))
(
(
))
(
(
))
(
)
(
(
2
1
2
1
x
f
Median
x
f
Median
x
f
x
f
Median

2
1
1
)
0
1
0
1
2
1
0
1
0
(
)
1
0
1
0
1
0
1
0
1
(

Median
Median

)
0
1
0
1
2
1
0
1
0
1
0
1
0
1
0
1
0
1
(
)
1
1
1
1
3
1
1
1
1
(
1
Median
Median
The University of

Ontario

Salt&Pepper

Noise

The University of

Ontario

Gaussian

noise

The University of

Ontario

Neighborhood Processing (filtering)

Median filters

What
median filter
have
over a
mean filter
?

Better at removing Salt & Pepper noise

Slow

The University of

Ontario

Neighborhood Processing (filtering)

Derivatives and Convolution

Used for Edge/Corner Detection

Computed with Finite Differences Filters

Laplacian

of Gaussians (
LoG
) Filter

Used for Edge/Blob Detection and

Image Enhancement

Approximated using Difference of Gaussians

The University of

Ontario

-
8.2

First Derivative

Recall
Sharp changes in gray level of the input
image correspond to “peaks or valleys” of the
first
-
derivative of the input signal.

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of

Ontario

How can we approximate it

for a discrete function?

Is this operation shift
-
invariant?

Is it linear?

-
8.2

First Derivative and
convolution

h
y
x
f
y
h
x
f
f
h
x
)
,
(
)
,
(
lim
0
?

?

?

?

?

?

?

?

?

]
,
[
v
u
x

The University of

Ontario

-
8.2

First Derivative
and convolution

Finite Difference

x

x
-
h

x

x+h

x

x+h
/2

x
-
h/2

)
(
'
)
(
)
(
0
]
[
x
f
h
x
f
h
x
f
h
h
h
f

)
(
)
(
b
x
f
a
x
f

Slide credit: Wikipedia

The University of

Ontario

-
8.2

First Derivative
and convolution

Finite Difference

The order of an error

can be derived using Taylor Theorem

Slide credit: Wikipedia

The University of

Ontario

-
8.2

First Derivative
and convolution

Pixel Size

Using Finite Central Difference

We need a kernel such that

x
y
x
x
f
y
x
x
f
f
x

2
)
,
(
)
,
(
f
f
x
x

0

0

0

1

0

-
1

0

0

0

]
,
[
v
u
x

x
2
1
h
x

x

The University of

Ontario

Neighborhood Processing (filtering)

Finite differences

responds to edges

Dark = negative

White = positive

Gray = 0

f
x

The University of

Ontario

Neighborhood Processing (filtering)

Finite differences
-

responding to noise

Increasing noise
-
>

(this is zero mean additive gaussian noise)

f
x

f
x

f
x

The University of

Ontario

Neighborhood Processing (filtering)

Finite
differences and noise

Finite difference filters respond strongly to noise

Noisy pixels look
very different from their
neighbours

The
larger the noise

the stronger is
the
response

How can we eliminate the response to noise?

Most pixels in images look similar to their
neighbours

(even at an edge)

Smooth the image (mean/
gaussian

filtering)

The University of

Ontario

Neighborhood Processing (filtering)

Smoothing
and Differentiation

Smoothing
before
differentiation = two convolutions

Convolution
is associative

)
(
H
x

f
H
f
H
x
x

)
(
)
(
)
(
H
y

)
(
f
H
x

The University of

Ontario

The scale of the smoothing filter affects derivative estimates, and also

the semantics of the edges recovered.

1 pixel

3 pixels

7 pixels

f
H
x

)
(
Neighborhood Processing (filtering)

Smoothing and Differentiation

The University of

Ontario

Yet another approximation frequently used

Sobel

kernels

1

0

-
1

2

0

-
2

1

0

-
1

]
,
[
v
u
x

x
8
1
f
x

1

2

1

0

0

0

-
1

-
2

-
1

]
,
[
v
u
y

y
8
1
f
y

Neighborhood Processing (filtering)

The University of

Ontario

Recall for a function of
two variables

The

at a point (
x,y
)

direction of the “steepest ascend”

orthogonal to object boundaries in the image

x
f

y
f

Neighborhood Processing (filtering)

f
f
y
x

y
f
x
f
f
)
,
(
y
x
f
x

y

small image

textured areas

2
2
2
2
)
(
)
(
)
(
)
(
||
||
f
f
f
y
x
y
f
x
f

The University of

Ontario

Neighborhood Processing (filtering)

Typical application where image gradients are used
is
image edge
detection

find points with large image gradients

Canny edge detector
suppresses

non
-
extrema

The University of

Ontario

-
8.2

Second derivatives
and convolution

Peaks or valleys of the first
-
derivative of the input
signal, correspond to “zero
-
crossings” of the
second
-
derivative of the input signal.

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of

Ontario

Neighborhood Processing (filtering)

Second derivatives and convolution

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of

Ontario

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Neighborhood Processing (filtering)

Second derivatives and convolution

Better localized edges

But more sensitive to
noise

The University of

Ontario

Neighborhood Processing (filtering)

Second derivatives and convolution

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of

Ontario

Neighborhood Processing (filtering)

Second Image Derivatives

Laplace operator

f
f
f
2

f
f
y
f
x
f
f
y
x
y
x

]
[
2
2
2
2
0

0

0

-
1

2

-
1

0

0

0

0

-
1

0

0

2

0

0

-
1

0

0

-
1

0

-
1

4

-
1

0

-
1

0

“divergence

rotationally invariant

second derivative for 2D functions

Finite Difference

Second Order

Derivative in x

Finite Difference

Second Order

Derivative in y

The University of

Ontario

Neighborhood Processing (filtering)

Second Image Derivatives

Laplacian

Zero Crossing

Used for edge detection

extrema
)

http://homepages.inf.ed.ac.uk/rbf/HIPR2/zeros.htm

magnitude of

image

intensity

Laplacian

(2
nd

derivative)

of the image

2
2
2
2
y
f
x
f
f

f

||
||
f

f
The University of

Ontario

Neighborhood Processing (filtering)

Laplacian

Filtering

-
Zero on uniform regions

-
Positive on one side

of an edge

-
Negative on the other side

-
Zero at some point in between

on the edge itself

band
-
pass filter (
Suppresses both high and low frequencies
)

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of

Ontario

Neighborhood Processing (filtering)

Laplacian

of a Gaussian (
LoG
)

Smooth before differentiation

(remember associative property of convolution)

LoG
G

2
2
2
2
2
2
2
4
2
1
1
)
,
(


y
x
e
y
x
y
x
LoG

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of

Ontario

Neighborhood Processing (filtering)

Laplacian

of a Gaussian (
LoG
)

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

Suppresses both high and low frequencies

The University of

Ontario

Neighborhood Processing (filtering)

Laplacian

of a Gaussian (
LoG
)

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of

Ontario

Neighborhood Processing (filtering)

Laplacian

of a Gaussian (
LoG
)

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of

Ontario

Neighborhood Processing (filtering)

Laplacian

of a Gaussian (
LoG
)

Can be approximated by a difference of two

Gaussians (
DoG
)

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of

Ontario

Neighborhood Processing (filtering)

DoG

vs.
LoG

Separable (product decomposition)

more efficient

Can explain band
-
pass filter since Gaussian is a low
-
pass filter.

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of

Ontario

Neighborhood Processing (filtering)

LoG

for Blob Detection

Cross correlation with a filter can be viewed as
comparing a little “picture” of what you want to find
against all local regions in the image.

Scale of blob (size ; radius in pixels)

is determined by the sigma parameter

of the
LoG

filter.

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of

Ontario

Neighborhood Processing (filtering)

LoG

for Blob Detection

http://www.cse.psu.edu/~rcollins/CSE486/lecture11_6pp.pdf

The University of

Ontario

Filters
and
T
emplates

Recall: filtering = cross
-
correlation = dot
-
product

Measures the angle between two vectors:

the filter and the image window

Measures visual similarity

f
h
*
(
x,y
)

Input Image

f(
x,y
)

g(
x,y
)

Output Image

(
x,y
)

Correlation

)
cos(
)
,
(
||
||||
||
arccos
)
,
(

y
x
g
f
h
f
h
y
x
f
f
w
w
w
w
The University of

Ontario

Since we divide by the norm

Normalized cross
-
correlation

Bring the template and the image to a uniform scale:

subtract the mean and divide by the variance

Neighborhood Processing (filtering)

Template Matching

||
||||
||
arccos
)
,
(
w
w
w
w
f
h
f
h
y
x
f
f

http://scien.stanford.edu/pages/labsite/2008/psych221/projects/08/MariaJabon/index.htm

)
,
(
y
x
h
)
,
(
y
x
f
The University of

Ontario

Normalized cross
-
correlation

Extremely time consuming due to sliding window

Neighborhood Processing (filtering)

Template Matching

h
f
1
|
|
)
cos(


g
g

Slide

credit:
Matlab

manual

||
||||
||
arccos
)
,
(
w
w
w
w
f
h
f
h
y
x
f
f

The University of

Ontario

Neighborhood Processing (filtering)

Template Matching

h
f
)
cos(

g
Slide

credit:
OpenCV

manual