Mathematical Beer Goggles or The Mathematics of Image Processing

pancakesnightmuteAI and Robotics

Nov 5, 2013 (3 years and 11 months ago)

113 views

Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Mathematical Beer Goggles
or
The Mathematics of Image Processing
Melina Freitag
Department of Mathematical Sciences
University of Bath
Postgraduate Seminar Series
University of Bath
12th February 2008
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
1
Motivation
2
How images become numbers
3
Compressing images
4
The image deblurring problem
5
Blurring and Deblurring images
The blurring function
Deblurring
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Outline
1
Motivation
2
How images become numbers
3
Compressing images
4
The image deblurring problem
5
Blurring and Deblurring images
The blurring function
Deblurring
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Outline
1
Motivation
2
How images become numbers
3
Compressing images
4
The image deblurring problem
5
Blurring and Deblurring images
The blurring function
Deblurring
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Grayscale images
One dimensional matrix
X =








0 0 0 0 0 0 0 0 0
0 8 8 0 4 4 0 2 0
0 8 8 0 4 4 0 2 0
0 8 8 0 4 4 0 2 0
0 8 8 0 4 4 0 2 0
0 0 0 0 0 0 0 0 0








imagesc(X),colormap(gray)
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
MATLAB image
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
RGB images
Three dimensional matrix
X(:,:,1) =




0 0 0 0 0 0 0 0 0
0 1 1 0 1 1 0 0 0
0 1 1 0 1 1 0 0 0
0 0 0 0 0 0 0 0 0




X(:,:,2) =




0 0 0 0 0 0 0 0 0
0 0 0 0 1 1 0 0 0
0 0 0 0 1 1 0 0 0
0 0 0 0 0 0 0 0 0




X(:,:,3) =




0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 1 0
0 0 0 0 0 0 0 1 0
0 0 0 0 0 0 0 0 0




imagesc(X)
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
MATLAB image
X = imread(’pic.jpg’),imwrite(X,’pic.jpg’)
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Outline
1
Motivation
2
How images become numbers
3
Compressing images
4
The image deblurring problem
5
Blurring and Deblurring images
The blurring function
Deblurring
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
The singular value decomposition (SVD)
Existence and Uniqueness
Let X ∈ C
m,n
,m≥ n Then


X



v
1
|v
2
|...|v
n

=

u
1
|u
2
|...|u
m








σ
1
σ
2
.
.
.
σ
n
0







or
X = UΣV
T
,
where U
T
U = I,with columns of U called left singular vectors
and V
T
V = I with right singular vectors as columns of V and
Σ = diag(σ
1
,...,σ
n
) called
singular values
ordered such that
σ
1
≥ σ
2
≥...≥ σ
n
≥ 0.
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Low-rank approximations
Theorem (The rank of a matrix)
The rank of X is r,the number of nonzero singular values in
X = UΣV
T
= U










σ
1
σ
2
.
.
.
σ
r
0   
0
.
.
.










V
T
.
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Low-rank approximations
Theorem (The rank of a matrix)
The rank of X is r,the number of nonzero singular values in
X = UΣV
T
= U










σ
1
σ
2
.
.
.
σ
r
0   
0
.
.
.










V
T
.
Theorem (Another representation)
X is the sum of r rank-one matrices
X =
r
X
j=1
σ
j
u
j
v
T
j
.
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Low-rank approximations
Theorem
For any ν with 0 ≤ ν ≤ r,define
X
ν
=
ν
X
j=1
σ
j
u
j
v
T
j
,
Then
kX −X
ν
k
2
= inf
B∈C
m,n
,rank(B)≤ν
kX −Bk
2
= σ
ν+1
.
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Low-rank approximations
Proof (m= n)
kX −X
ν
k
2
= k
r
X
j=ν+1
σ
j
u
j
v
T
j
k
2
= kU





0
σ
ν+1
.
.
.
σ
n





V
T
k
2
= σ
ν+1
Remains to show that there is no closer rank ν matrix to X.
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Low-rank approximations
Proof (m= n)
kX −X
ν
k
2
= k
r
X
j=ν+1
σ
j
u
j
v
T
j
k
2
= kU





0
σ
ν+1
.
.
.
σ
n





V
T
k
2
= σ
ν+1
Remains to show that there is no closer rank ν matrix to X.
Let B have rank ν,
null space of (B)
has dimension n −ν
{v
1
,...,v
ν+1
}
has dimension ν +1
Let h be a unit vector in their intersection:
kX −Bk
2
≥ k(X −B)hk
2
= kXhk
2
= kUΣV
T
hk
2
= kΣ(V
T
h)k
2
≥ σ
2
ν+1
kV
T
hk
2
≥ σ
2
ν+1
.
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Example m= 604,n = 453,m∗ n = 273612
Compression ratio
c:=
(m+n)ν
mn
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Rank-1 approximation m= 604,n = 453,
m+n = 1057
Compression ratio
c:= 3.8631e −03
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Rank-2 approximation
Compression ratio
c:= 7.7263e −03
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Rank-3 approximation
Compression ratio
c:= 1.1589e −02
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Rank-4 approximation
Compression ratio
c:= 1.5453e −02
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Rank-5 approximation
Compression ratio
c:= 1.9316e −02
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Rank-10 approximation
Compression ratio
c:= 3.8631e −02
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Rank-20 approximation
Compression ratio
c:= 7.7263e −02
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Rank-30 approximation
Compression ratio
c:= 0.11589
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Rank-40 approximation
Compression ratio
c:= 0.15453
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Rank-60 approximation
Compression ratio
c:= 0.23179
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Rank-80 approximation
Compression ratio
c:= 0.30905
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Outline
1
Motivation
2
How images become numbers
3
Compressing images
4
The image deblurring problem
5
Blurring and Deblurring images
The blurring function
Deblurring
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Blurred and exact images
Let
X
be the exact image
Let
B
be the blurred image
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Blurred and exact images
Let
X
be the exact image
Let
B
be the blurred image
If the blurring of the columns is independent of the blurring
in the rows then
A
c
X
A
T
r
=
B
,A
c
∈ R
m,m
,A
r
∈ R
n,n
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Blurred and exact images
Let
X
be the exact image
Let
B
be the blurred image
If the blurring of the columns is independent of the blurring
in the rows then
A
c
X
A
T
r
=
B
,A
c
∈ R
m,m
,A
r
∈ R
n,n
First attempt at deblurring
X
Naive
= A
−1
c
BA
−T
r
.
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
First attempt at deblurring
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
X
Naive
= A
−1
c
BA
−T
r
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
What is the problem?
A noisy blurred image
B = B
exact
+E = A
c
XA
T
r
+E
and therefore
X
Naive
= X +
A
−1
c
EA
−T
r
.
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
What is the problem?
A noisy blurred image
B = B
exact
+E = A
c
XA
T
r
+E
and therefore
X
Naive
= X +
A
−1
c
EA
−T
r
.
Error
The naive solution satisfies
kX
Naive
−Xk
F
kXk
F
≤ cond(A
c
)cond(A
r
)
kEk
F
kBk
F
.
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Deblurring using a general model
The blurring process as a linear model
We assume the blurring process is
linear
,i.e.
x = vec(X) =



x
1
.
.
.
x
N



∈ R
N
,b = vec(B) =



b
1
.
.
.
b
N



∈ R
N
N = m∗ n are related by the linear model
Ax = b
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Deblurring using a general model
The blurring process as a linear model
We assume the blurring process is
linear
,i.e.
x = vec(X) =



x
1
.
.
.
x
N



∈ R
N
,b = vec(B) =



b
1
.
.
.
b
N



∈ R
N
N = m∗ n are related by the linear model
Ax = b
b = b
exact
+e
x
Naive
= A
−1
b = A
−1
b
exact
+A
−1
e = x +A
−1
e
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Separable two dimensional blurs
The Kronecker product
If horizontal and vertical flow can be separated then
A
x
=
b
⇔ A
vec(X)
=
vec(B)
=
vec(A
c
XA
T
r
)
(A
r
⊗A
c
)vec(X) = vec(A
c
XA
T
r
),
where
A = A
r
⊗A
c
=



a
r
11
A
c
...a
r
1n
A
c
.
.
.
.
.
.
.
.
.
a
r
n1
A
c
...a
r
nn
A
c



,
(U
r
Σ
r
V
T
r
) ⊗(U
c
Σ
c
V
T
c
) = (U
r
⊗U
c
)(Σ
r
⊗Σ
c
)(V
r
⊗V
c
)
T
.
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
x
Naive
= x +A
−1
e
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Outline
1
Motivation
2
How images become numbers
3
Compressing images
4
The image deblurring problem
5
Blurring and Deblurring images
The blurring function
Deblurring
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Taking bad pictures
Sources of bad pictures
defocus the camera lens (limitations in the optical system)
motion blur
air turbulence
atmospheric blurring
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Taking bad pictures
Sources of bad pictures
defocus the camera lens (limitations in the optical system)
motion blur
air turbulence
atmospheric blurring
Noise E
background photons from both natural or artificial sources
signal represented by finite number of bits (quantisation
error)
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Modelling the blurring matrix A
Single bright pixel
x = e
i
⇒Ae
i
= column i of A
Figure:
Point source
(single bright pixel)
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Modelling the blurring matrix A
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Modelling the blurring matrix A
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Boundary conditions and structured matrix
computations
Boundary conditions
Zero boundary conditions
Periodic boundary conditions
Reflexive boundary conditions
The matrix A which is obtained from P by convolution becomes
Block Toeplitz matrix
Block Circulant matrix
Sum of Block Toeplitz and Block Hankel and Block Toeplitz
plus Hankel matrices
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Spectral filtering
The SVD again
With
A = UΣV
T
=

u
1
|u
2
|...|u
N






σ
1
σ
2
.
.
.
σ
N








v
T
1
.
.
.
v
T
N



we have
x
Naive
= A
−1
b = V Σ
−1
U
T
b =
N
X
i=1
u
T
i
b
σ
i
v
i
X
Naive
=
N
X
i=1
u
T
i
b
σ
i
V
i
=
N
X
i=1
u
T
i
b
exact
σ
i
V
i
+
N
X
i=1
u
T
i
e
σ
i
V
i
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Spectral filtering
Behaviour of singular values
σ
i
→0 as i grows
the more “blurry” the function,the faster the decay rate
cond(A) = σ
1

N
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Spectral filtering
Behaviour of singular values
σ
i
→0 as i grows
the more “blurry” the function,the faster the decay rate
cond(A) = σ
1

N
The regularised solution
Introduce filter factors Φ
i
x
Naive
=
N
X
i=1
Φ
i
u
T
i
b
σ
i
v
i
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Two methods
TSVD
Φ
i
=

1 i = 1,...,k
0 i = k +1,...,N
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Two methods
TSVD
Φ
i
=

1 i = 1,...,k
0 i = k +1,...,N
Tikhonov regularisation
Φ
i
=
σ
2
i
σ
2
i

2
where α > 0 is a regularisation parameter,This choice of filter
factor yields solution to the minimisation problem
min
x
{kb −Axk
2
2

2
kxk
2
2
}.
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Regularisation and perturbation errors
Regularised solution
x
filt
= V ΦΣ
−1
U
T
b
= V ΦΣ
−1
U
T
Ax
exact
+V ΦΣ
−1
U
T
e
= V ΦV
T
x
exact
+V ΦΣ
−1
U
T
e
x
exact
−x
filt
= (I −V ΦV
T
)x
exact
|
{z
}
Regularisation error
− V ΦΣ
−1
U
T
e
|
{z
}
Perturbation error
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Regularisation and perturbation errors
Regularised solution
x
filt
= V ΦΣ
−1
U
T
b
= V ΦΣ
−1
U
T
Ax
exact
+V ΦΣ
−1
U
T
e
= V ΦV
T
x
exact
+V ΦΣ
−1
U
T
e
x
exact
−x
filt
= (I −V ΦV
T
)x
exact
|
{z
}
Regularisation error
− V ΦΣ
−1
U
T
e
|
{z
}
Perturbation error
Oversmoothing and undersmoothing
small regularisation error,large perturbation error leads to
undersmoothed solution
large regularisation error,small perturbation error leads to
oversmoothed solution
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Regularisation and perturbation errors
Parameter choice methods
Discrepancy Principle
Generalised Cross Validation
L-Curve Criterion
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
A second attempt at deblurring
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
X
Naive
= A
−1
c
BA
−T
r
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Filtered solution using TSVD
Figure:
k = 4801,
N = 83000
Figure:
k = 6630,
N = 238650s
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
Tikhonov regularisation
Figure:
α = 0.0276
Figure:
α = 0.0137
Deblurring
images
Melina
Freitag
Outline
Motivation
How images
become
numbers
Compressing
images
The image
deblurring
problem
Blurring and
Deblurring
images
The blurring
function
Deblurring
P.C.Hansen,J.G.Nagy,and D.P.O’Leary,
Deblurring Images - Matrices,Spectra and Filtering,SIAM,
Philadelphia,1st ed.,2006.