Conversion of Standard Broadcast Video Signals for HDTV Compatibility

stingymilitaryΗλεκτρονική - Συσκευές

27 Νοε 2013 (πριν από 3 χρόνια και 6 μήνες)

156 εμφανίσεις

Conversion of Standard Broadcast
Video Signals for HDTV Compatibility

Ph.D. Defense Presentation

Elham Shahinfard

Advisor: Prof. M. A Sid
-
Ahmed

16 July 2009

Contributions


Proposing a motion adaptive deinterlacing method.
It includes:

1.
An accurate hierarchical motion detection algorithm

2.
A recursive threshold optimization method

3.
A motion adaptive interpolation algorithm for estimating
the missing lines.

2

Outline

1.
Introduction

2.
Background Information & Review of Previous
Works

3.
Proposed Motion Detection (MD) Method

4.
Proposed Motion Adaptive Deinterlacing Method

5.
Evaluation of Algorithm Performance and
Comparison with Other Methods

6.
Summary and Future Works

3

Problem Definition


Digital high definition TV is replacing analog TV all over
the world


Canada and U.S. have adopted the Advanced Television System
Committee fully digital system as their new TV standard







Analog TV standards use Interlaced format


Deinterlacing for converting interlaced to progressive has
attracted attention


4

Vertical
lines

Horizontal
pixels per line

Aspect
Ratio

Progressive/
Interlaced

720

1280

16:9

Progressive

1080

1920

16:9

Progressive

1080

1920

16:9

Interlace

Video Scanning Format














Interlacing

5

Deinterlacing








De
-
Interlacing

Estimation

Deinterlacing Objective: To find the best estimation for
the missing lines with minimum distortion

6

Outline

1.
Introduction

2.
Background Information & Review of Previous
Works

3.
Proposed Motion Detection (MD) Method

4.
Proposed Motion Adaptive Deinterlacing Method

5.
Evaluation of Algorithm Performance and
Comparison with Other Methods

6.
Summary and Future Works

Deinterlacing Categories


Spatial deinterlacing (Intra
-
frame)


Temporal deinterlacing (Inter
-
frame)


Hybrid deinterlacing (Inter
-
frame)


Vertical
-
Temporal (VT) median deinterlacing


Motion compensated deinterlacing


Motion adaptive deinterlacing

8

Spatial Deinterlacing


Uses only spatial data from current field of video








: location of a pixel in the field





: spatial displacement vector


Examples


Line Repetition


Line Averaging


9

Spatial Deinterlacing Example

(Line Averaging)

10

Original Progressive Frame

Interlaced Field

Deinterlaced Frame






Advantages


Simplicity


Motion robustness

×
Disadvantages


Low quality; blurring
effect


Temporal Deinterlacing


Uses only temporal data from previous and/or
subsequent fields




Examples


Field Insertion


Bilinear Field Averaging

11

Temporal Deinterlacing Example

(Field Insertion)

12

Original Progressive Frame

Interlaced Field

Deinterlaced Frame






Advantages


Simplicity


Perfect for static regions

×
Disadvantages


Severe distortion in
dynamic regions


VT Median Deinterlacing


A median filter is used to interpolate the missing
lines


The median operation is done in both temporal and
vertical directions


Example:


Three tap VT median deinterlacing:

13

VT Median Deinterlacing Example

14

Original Progressive Frame

Interlaced Field

Deinterlaced Frame






Advantages


Easy implementation


Superior to temporal
and spatial methods

×
Disadvantages


Low quality


Motion Compensated Deinterlacing


Video sequence is virtually converted to a stationary
sequence


A motion estimation method estimates the motion


The estimated motion is removed consequently


A deinterlacing method, which performs well in static
regions, is applied to the stationary sequence


Motion data is added to the deinterlaced sequence
in a later stage

15

Motion Compensated Deinterlacing

16


Advantages


High quality results


Perfect for videos with translational motion, such as panning
camera

×
Disadvantages


Performance is highly dependant to motion estimation results


Vulnerable to common motion estimation obstacle, such as:


Object deformation


Appearance and disappearance of objects


Fast motions which goes beyond the search area


Sub
-
pixel motions



Motion Adaptive Deinterlacing


Benefits from both interframe and intraframe
deinterlacing methods by:


Using a motion detector to divide a video sequence into
static and dynamic regions


Using an interframe deinterlacing in static regions


Using an intraframe deinterlacing in dynamic regions


Combining the results to obtain best estimation


It has proven to be a proper choice for high quality
deinterlacing


A high quality motion adaptive deinterlacing has
been designed in this research


17

Outline

1.
Introduction

2.
Background Information & Review of Previous
Works

3.
Proposed Motion Detection (MD) Method

4.
Proposed Motion Adaptive Deinterlacing Method

5.
Evaluation of Algorithm Performance and
Comparison with Other Methods

6.
Summary and Future Works

Design Objectives & Contributions


Design Objectives


Accurately detecting the presence of motion in video
sequence


Measuring motion activity level of video sequence with
high precision


Contributions


Proposing an accurate motion detection algorithm which:


Uses five consecutive fields of interlaced video


Has a hierarchical structure


Utilizes two LPF to improve algorithm accuracy

19

Input Sequence

20


Five consecutive fields of a video
sequence are used for motion
detection


The optimum number of correlated
interlaced video fields is 5.


Relative position of missing lines in
five consecutive fields



Motion Detector


21



Increase the accuracy of motion detector.



Assumptions:

1.
Signal is large and noise is small

2.
Low frequency energy in signal is greater than
low frequency energy in noise and alias



2D square averaging filters are appropriate choices



Motion detection goal is detecting the
possibility of motion



Motion direction is not important



Improves the consistency of the output



Assumption: moving objects are larger than pixels



an
m
x
m

median filter is an appropriate choice

Hierarchical Block


Receives
dif
1
, dif
2
, dif
3


Partitions them into


data blocks


Calculates the average intensity value for each block


22



1
2
3
4
5
6
0.5
1
1.5
2
2.5
3
3.5
4
4.5
2
4
6
8
10
12
14
Hierarchical Block


Compares the average intensity value of each block with its
corresponding data blocks


Finds the maximum average intensity value of each three
corresponding data blocks


Compares the maximum value with a predefined threshold
value


If less than threshold value, the data block is considered static and its
motion value is set to zero


If greater than threshold value, the data block is dynamic


A dynamic data block will be recursively partitioned to smaller
data blocks


The recursive procedure may continue up to pixel level


The final output is motion value matrix




23

Hierarchical Block

24



1
2
3
4
5
6
0.5
1
1.5
2
2.5
3
3.5
4
4.5
2
4
6
8
10
12
14


1
2
3
4
5
6
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0
2
4
6
8
10
12
14


2
4
6
8
10
12
1
2
3
4
5
6
7
8
0
5
10
15
20
25


2
4
6
8
10
12
1
2
3
4
5
6
7
8
0
5
10
15
20
25


5
10
15
20
2
4
6
8
10
12
14
0
5
10
15
20
25


5
10
15
20
2
4
6
8
10
12
14
0
5
10
15
20
25


5
10
15
20
25
30
35
40
45
5
10
15
20
25
30
0
10
20
30
40
50


5
10
15
20
25
30
35
40
45
5
10
15
20
25
30
0
10
20
30
40
50


10
20
30
40
50
60
70
80
90
10
20
30
40
50
60
0
10
20
30
40
50
60


10
20
30
40
50
60
70
80
90
10
20
30
40
50
60
0
10
20
30
40
50
60


20
40
60
80
100
120
140
160
180
20
40
60
80
100
120
0
10
20
30
40
50
60
70
80
90
100


20
40
60
80
100
120
140
160
180
20
40
60
80
100
120
0
10
20
30
40
50
60
70
80
90
100
Average Intensity Value before
Thresholding

Average Intensity Value after
Thresholding

Threshold Value Determination


Threshold values have been found by experimental
tests


A video sequence with tractable moving objects is a proper
starting point


Average intensity values have been monitored for several
test sequences to find initial values


Initial values have been applied to motion detection
methods and recursively optimized for error minimization



25

Outline

1.
Introduction

2.
Background Information & Review of Previous
Works

3.
Proposed Motion Detection Method

4.
Proposed Deinterlacing Method

5.
Evaluation of Algorithm Performance and
Comparison with Other Methods

6.
Summary and Future Works

Proposed Motion Adaptive Deinterlacing


27

Reduces the possibility of missing
motions.

Improves motion detection
consistency by reducing
distortion

Non
-
Linear Transformation


Converts Motion
-
Value (MV) to motion possibility
value



and are predefined threshold values found by
recursive error minimization.


28

Threshold Calculation


Pre Conditions:



are in the same range as pixel intensity values (0
-
255 in a
general case)





Initial setup


Initial threshold values:


Initial step size: 10



Procedure:


Initial values have been applied to the proposed motion adaptive
deinterlacing method


Deinterlacing error has been calculated and recorded for each initial
value


Initial values have been changed based on calculated error.


Finer step size has been applied to the area with minimum error

29

Threshold Calculation


Same procedure has been applied to several test
sequences



has proven to be optimum values
for a general setup











30

5
10
15
20
5
10
15
20
25
11.1
11.15
11.2
11.25
11.3
MV1/10
MV2/10
MSEdB
Interpolation Algorithm





: motion possibility value


: intensity value of a pixel using
Spatial deinterlacing

method; Linear interpolation is chosen as spatial
deinterlacing method


: intensity value of a pixel using
Temporal deinterlacing

method; Median filtering is chosen as temporal
deinterlacing method

31

Deinterlacing Results


Stennis

Original Progressive Video


Stennis

Deinterlaced Video



Sflowg

Original Progressive Video


Sflowg

Deinterlaced Video


32

Implementation Results

(a)

Grandmom
;

Original

progressive

frame


(b)

Grandmom
;

Deinterlaced

frame

(c)

Mom
;

Original

progressive

frame


(d)

Mom
;

Deinterlaced

frame

33

34

(e)

MomDaughter
;

Original

progressive

frame


(f)

MomDaughter
;

Deinterlaced

frame

(g)

Stennis
;

Original

progressive

frame


(h)

Stennis
;

Deinterlaced

frame

35

(
i
)

Heart
;

Original

progressive

frame


(j)

Heart
;

Deinterlaced

frame

(k)

Sflowg
;

Original

progressive

frame


(l)

Sflowg
;

Deinterlaced

frame

36

(m)

Movi
;

Original

progressive

frame


(n)

Movi
;

Deinterlaced

frame

(o)

Disku
;

Original

progressive

frame


(p)

Disku
;

Deinterlaced

frame

Motion Detection Results

(a)
Grandmom

(b) Mom

(c)
MomDaughter

(d)
Stennis

37

38

(e)
Heart

(f) Sflowg

(g) Movi

(h)
Disku

Outline

1.
Introduction

2.
Background Information & Review of Previous
Works

3.
Proposed Motion Detection (MD) Method

4.
Proposed Motion Adaptive Deinterlacing Method

5.
Evaluation of Algorithm Performance and
Comparison with Other Methods

6.
Summary and Future Works

Performance Evaluation Method


40

Evaluation Criterion


Objective Evaluation Criterion: Peak Signal to Noise Ratio






Subjective Evaluation Criterion (According to ITU
-
R
BT.500
-
11):


Mean Score & its

associated confidence

Interval





41

Grade

Impairment level

5

4

3

2

1

Imperceptible

Perceptible but not annoying

Slightly annoying

Annoying

Very Annoying

Objective Evaluation Results

42

0
10
20
30
40
50
Line repetition (Li-Rep)
Bilinear line averaging (Li_Ave)
Field insertion (Fi_Ins)
Bilinear field interpolation (Fi_Ave)
VT median filtering
Weighted & edge dependant median filter
MC field insertion
MC bilinear field interpolation
MA with 3-field MD, Fi_Ave, Li_Ave
MA with 4-field MD, Fi_Ins, Li_Ave
MA method proposed in [46]
GA-HDTV MA method
Proposed MA method
PSNR(dB)

Average PSNR
-

Type 1
-
3 Videos

Type-3 (Sflowg, Movi, Disku)
Type-2 (Stennis, Hearth)
Type-1 (Mom, Grandmom, MomDaugther)
43

33%

24%

10%

4%

11%

25%

6%

3%

7%

11%

9%

6%

19%

10%

14%

15%

10%

14%

12%

7%

10%

5%

9%

1%

17%

7%

39%

50%

11%

13%

39%

42%

32%

24%

16%

11%

0%
10%
20%
30%
40%
50%
60%
Line repetition (Li-Rep)
Bilinear line averaging (Li_Ave)
Field insertion (Fi_Ins)
Bilinear field interpolation (Fi_Ave)
VT median filtering
Weighted & edge dependant median filter
MC field insertion
MC bilinear field interpolation
MA with 3-field MD, Fi_Ave, Li_Ave
MA with 4-field MD, Fi_Ins, Li_Ave
MA method proposed in [46]
GA-HDTV MA method
%PSNR improvement

Performance Improvement for Type 1
-
3

Type-3 (Sflowg, Movi, Disku)
Type-2 (Stennis, Hearth)
Type-1 (Mom, Grandmom, MomDaugther)
44

0
10
20
30
40
Line repetition (Li-Rep)
Bilinear line averaging (Li_Ave)
Field insertion (Fi_Ins)
Bilinear field interpolation (Fi_Ave)
VT median filtering
Weighted & edge dependant median filter
MC field insertion
MC bilinear field interpolation
MA with 3-field MD, Fi_Ave, Li_Ave
MA with 4-field MD, Fi_Ins, Li_Ave
MA method proposed in [46]
GA-HDTV MA method
Proposed MA method
PSNR(dB)

Average PSNR
-

All video sequences

45

24%

15%

19%

18%

11%

18%

16%

14%

15%

13%

11%

6%

0%
5%
10%
15%
20%
25%
30%
Line repetition (Li-Rep)
Bilinear line averaging (Li_Ave)
Field insertion (Fi_Ins)
Bilinear field interpolation (Fi_Ave)
VT median filtering
Weighted & edge dependant median filter
MC field insertion
MC bilinear field interpolation
MA with 3-field MD, Fi_Ave, Li_Ave
MA with 4-field MD, Fi_Ins, Li_Ave
MA method proposed in [46]
GA-HDTV MA method
%PSNR improvement

Performance Improvement over all test sequences

Single Frame of Mom Seq.

(a)

Original

progressive

frame

(b)

Deinterlaced

by

bilinear

field

averaging

(c)

Deinterlaced

by

GA
-
HDTV

method

(b)

Deinterlaced

by

proposed

MA

method

46

Zoomed on a Moving Region

(a)

Original

progressive

frame

(b)

Deinterlaced

by

bilinear

field

averaging

(c)

Deinterlaced

by

GA
-
HDTV

method

(b)

Deinterlaced

by

proposed

MA

method

47

Algorithm Robustness to Frame Rate

48

0.0%

7.5%

8.8%

4.4%

3.9%

6.0%

6.5%

7.8%

7.8%

10.4%

0.7%

0.4%

0.0%

0.0%

16.0%

16.7%

7.8%

6.2%

13.9%

11.4%

11.6%

15.1%

14.3%

4.6%

1.5%

Line repetition (Li-Rep)
Bilinear line averaging (Li_Ave)
Field insertion (Fi_Ins)
Bilinear field interpolation (Fi_Ave)
VT median filtering
Weighted & edge dependant median filter
MC field insertion
MC bilinear field interpolation
MA with 3-field MD, Fi_Ave, Li_Ave
MA with 4-field MD, Fi_Ins, Li_Ave
MA method proposed in [46]
GA-HDTV MA method
Proposed MA method
%PSNR decrease

Percentage of Performance Fluctuation
vs

Frame
Rate

60fps downsampled to 15fps
60fps downsampled to 30fps
Evaluation of Proposed Motion Detection

49

0
10
20
30
40
50
Grandmom
Mom
MomDaughter
Stennis
Hearth
Sflowg
Movi
Disku
PSNR(dB)

Average PSNR
-

Different MD method

2F MD
3F MD
4F MD
GA-HDTV
Proposed method
50

9%

11%

10%

10%

9%

10%

18%

11%

11%

12%

13%

13%

11%

12%

44%

14%

10%

14%

12%

14%

12%

11%

52%

13%

14%

18%

13%

55%

15%

60%

28%

15%

0%
20%
40%
60%
80%
Grandmom
Mom
MomDaughter
Stennis
Hearth
Sflowg
Movi
Disku
%PSNR

Performance Improvement over other MD

2-Field MD
3-Field MD
4-Field MD
GA-HDTV
Subjective Evaluation Results


25 observers have evaluated the algorithm


Non
-
professional random observers


Both male and female


Ages 15 to 65


Overall mean score is 4.74


Its 95% confidence interval

51

52

3.8
4
4.2
4.4
4.6
4.8
5
5.2
Mean Score

Test Sequence

Subjective Quality Test

Mean Score-Delta (95%
confidence interval)
Observers Mean Score per
Seq
Mean Score+Delta (95%
confidence interval)
Outline

1.
Introduction: Deinterlacing Problem Statement

2.
Review of Existing Methods

3.
Proposed Motion Detection (MD) Method

4.
Proposed Motion Adaptive Deinterlacing Method

5.
Evaluation of Algorithm Performance and
Comparison with Other Methods

6.
Summary and Future Works

Summary


A high accuracy motion detection algorithm was
proposed


Has a hierarchical structure


Uses 5 consecutive video fields for motion detection


Is capable of detecting a wide range of motions from slow to
fast motions


Provides superior PSNR compared to other
mthods

and
improves deinterlacing overall performance by 18% on average


A motion adaptive deinterlacing method was proposed


Uses motion possibility values to combine line averaging with
Vertical
-
Temporal median filtering and benefits from both


Has high performance and obtain high quality deinterlaced
video


54

Recommendations for Future Works


Improvement in deinterlacing


Study of human eye frequency response to


Changes in video contents


Changes in motion speed while tracking an object


Improvement in motion adaptive deinterlacing


Including video content information


Automatic recognition of the type of the video for performance
improvement


Combining video texture information with motion detection
results


Combining motion detection results with motion estimation


Utilizing a motion compensated method instead of temporal method



Hardware Implementation (Architecture, Software/Hardware
partitioning)


55