Progress Paper.docx - ipTV-reu2008

looneyvillestaticSoftware and s/w Development

Aug 15, 2012 (4 years and 10 months ago)

244 views

Outline


1.

Introduction

1.

Why is it Needed?

1.

Channel surfing

2.

Lower Higher Quality

3.

Drift

2.

What causes Drift?

1.

Encoding Introduction

2.

Intra vs. Inter Encoding

3.

Frame Types

1.

I
-
Frame

2.

B
-
Frame

3.

P
-
Frame

4.

Summary

2.

Goal of Paper

1.

Help the Next Group

1.

Difficult Project

2.

Researching Right Areas

3.

Complex VLC Code

4.

Getting the Correct Work Environment

2.

Future of the Project

1.

Next Group Might Build Off Our Work

2.

Focusing on Coding

3.

Links for the Future

3.

Implementation or Design

1.

Systems Level

1.

Server

1.

Multi Track Files

2.

File Switching

3.

RTSP

1.

SET PARAMETER

2.

Track Parameter

2.

Client

1.

RTSP

2.

SPS/PPS NAL Units

3.

Decoder

3.

Summary

2.

Decoder

1.

SPS/PPS Switching

1.

Live555 Transcript

2.

File Merging

3.

Existing Code

2.

Interpolation

1.

Last Frame of Data

2.

New Instance of Decoder

3.

Changing Parameters of Current
Decoder

3.

Testing and Modularization

1.

Goal of Decoder

2.

Simple Appending of NAL Unit
Stream

3.

Local Decoding

4.

Test Bed

4.

Advanced Switching

1.

P
-
Frame Switching

2.

Video Switching

3.

Summary ?

4.

Researched

1.

VLC

1.

Modules

1.

Allows Generic Commands

2.

Function Points Warning

2.

Resources

1.

Hacker's Guide

2.

Doxygen

3.

Netbeans

2.

RTSP

1.

RTSP Live555 Server

2.

RTSP Live555 Client

1.

SET PARAMETER

3.

NAL Unit Structure

1.

NAL Unit Marker

2.

NAL Unit Type

3.

NAL Unit Syntax

4.

Sequence Parameter

Set (SPS)

5.

Picture Parameter Set (PPS)

4.

Multi Track Files

1.

Boxes

2.

Atom Parsley

3.

Hex Level

1.

Enabling Tracks

5.

Interpolation Algorithms

1.

Types

1.

Nearest Neighbor

1.

Blocky

2.

Sample ?

2.

Catmull
-
Rom

1.

Tangential Prediction

3.

Other Types

2.

Where it Fits

1.

VLC avcodex

2.

ffmpeg

3.

Previous Frame


Data Buffer

5.

Coded So Far

1.

Local Bit Stream Switching

1.

INPUT_CONTROL_SET_ES

2.

demux_Control()

2.

RTSP Client

1.

INPUT_CONTROL_SET_ES

2.

demux_Control()

1.

DEMUX_SWITCH_VIDEO_TRACK

3.

Live 555 Demuxer

1.

setMediaSessionParameter()

2.

SET PARAMETER

3.

RTSP Server

1.

Switches Track Being Sent

2.

Lacks Sending of Extra SPS/PPS Data.

1.

Period Sends Already Implemented In
VLC

1.

VLC Error Ticket #1384

3.

Elementary Stream Sent Separately in VLC

1.

Command Line Options

4.

Summary

1.

Quality Changes Possible

2.

Resolution Size Changes Needs
SPS/PPS resend

4.

Interpolation Algorithm

1.

C
-
Image Interpolation Viewer

2.

C Code for a Catmull
-
Rom

3.

C Code for a Nearest Neighbor

6.

Key Results

1.

Multi
-
track Switch Locally

2.

RTSP Track Parameter

1.

Design

2.

Partial Implementation

3.

C
-
Image Interpolation Viewer

4.

Multi
-
track MP4s

5.

Basic GUI

7.

Group Members

1.

Grae Cullen

1.

Email (in case someone wants to contact
me.)

2.

Miguel Amador

3.

Phillip

4.

Kevin

5.

Steve

8.

Thanks and Remarks

1.

Dr. Wenjun (Kevin) Zeng

2.

REU & NSF

3.

God


Annex A



Code Changes With Comments



VLC


Local Track Switching


RTSP Client Track Switching


RTSP Server Track Switching



CIMage


Annex B


Sources



Web Links



Standards


MP4


H264



7.3.1



Annex B


Presentations


Annex C


RTSP Transcript


Paper


1.

Introduction

Bit
-
stream switching is analogous to changing the channel on a TV. However, it
also has other uses, also. For example, switching qualities, from high resolution video
to low resolution video, can be a form of bit
-
stream switching.
It is simply switching

from
one bit
-
stream, with one set of characteristics to another bit
-
stream with different
characteristics.

Bit
-
stream switching has a drawback, called drift. Drift is a video distortion caused
by bit
-
stream switching. It can also be caused by missing

frames of video data. It
corrects itself slowly, as the video continues. However, it can make the video look quite
bad.

Figure x:





First frame after
switch or lost.












Example of drift distortion

Drift is caused by missing data, normally the previous frame. So if you switch from
one switch to other you do not have the previous frame of the new stream, and you can
get drift. Switching to the middle of a stream will cause drift unless you
as a
prog
rammer take special precautions.


This drift effect is caused by the way video data is encoded. Rather than
c
ompress each frame, only the changes from frame to frame are encoded. This type of
encoding is called inter
-
frame compression, because it is in b
etween two frames. H264
also does intra
-
frame encoding, the
compressing of frames individually. (Elemental
Blog)


There are several types of frames; such as, I
-
frames, P
-
frames, and B
-
frames. I
-
frames are an intra
-
frame,
they do n
ot depend on any other images,
and they reset the
video quality to full quality with no drift
.


P
-
frame
s rely on previous frame(s), normally
just the previous frame; however

they can rely on several frames. As each P
-
frame is
added, the quality will slowl
y increase reducing drift if there is drift, and after a variable
amount of time the quality will be back to normal, full quality.


We attempted to in our project to produce a bit
-
stream switching system, and
could handle different types of switching and a
lso handle the drift problems they occur.
Drift is an import consideration in bit
-
stream switching
. In order, to gain the full
advantages of bit
-
stream switching
, the problem of drift must be addressed. We have
started the development of a system, for bi
t
-
stream switching, and made some progress
with different problem the system has to address.

2.

Goal of
this Section


Mainly, this section of the paper is to help the next group, am whoever else would
like to build off of our research. It is a difficult proje
ct, that requires a great deal of
research, and
hopefully

this will help the next avoid wasting time.
Certainly, the next
group will still have to research, but perhaps this paper will alleviate some of the burden.


The next group will mostly not want t
o do redundant research. Honestly, some
time was spent researching things that turned out to be unnecessary. Some things
probably are researched enough and do not need to be found, simple read from our
documentation. For example, although learning the M
P4 file format was formative, and
helped lead to the discovery of multi
-
track files, now that we have a method for making
multi
-
track MP4 with different qualities or resolutions, a future research would not have
read the MP4 specification, but rather might

be able to read our tutorial about making
multi
-
track mp4 files.


Working with VLC has a number of problems, they we can help the next group
avoid. For example, si
mply telling the next group to compile on Ubuntu Linux

will
probably save them hours of pro
blem
s
.

Further, VLC is a very complex system of
threads and pointer functions, and we would like them to have a some of the hints and
help we found useful.


This goes hand in hand with having the right work environment. We probably
could have saved week
or more just by know to use the same version of Ubuntu that
the developers of VLC use, currently ‘Hardy.’ Also, ‘Netbeans’ is invaluable integrated
design environment, (IDE) that will save the next group a lot of time.


Certainly we hope that what we
have coded can be used by the future groups.
Hopefully, it can at least, as starting point for coding, or a
n

example for their code.

We
have a number of patches, and this section will include instructions on how to apply
them to the code. Also instructi
ons of where to find the sources we used.


Also, there are a number of link
s

we collected in the process of researching.
Some of the links are listed as references on this page, and more are listed on our
project wiki.
Additionally we have a few
tutorials written, and on the wiki. Since, some
of the information is hard to find, or somewhat obscure, these links might help future
students.


Implementation or Design


It may be helpful to layout the basic design. This it not meant as a statement of
rule or specification of design. However, it might be helpful in understanding our overall
goal.


First,

this is a
systems level overview

of the procedure and tools used in our
design. The basic
premise

is to make a system to adjust quality and resolutio
n on the
fly. There are two main components of this system; the client and server.


The server should be able
use multi
-
track files. Multi
-
track files are
simply

MP4
with more than one
video
track. Each track is a raw stream of h264 data. (Tracks may
a
lso be s
ound data, or other information.
) One file can have several resolutions in the
file, and bit rate qualities
,
1

one for each track in the video.

Although we have a tentative
standard for the order of tracks, the order is not at this
time required f
or operability with
the current code.


The server also will have ability to switch between files on the fly. This will allow
channel changes between different movies, and picture
-
in
-
picture features.
This
functionality is not nearly as completed as the
track switching. However, some of the
underling mechanics of the switching are already planned, or partially completed.



The server must respond to
real
-
time streaming protocol (
RTSP
)

commands.
Videos are normally streamed using real
-
time transport prot
ocol (RTP). VLC uses the
RTP with RTSP control commands. However, it is necessary to add a non standard
parameter to the server code. The client sends a standard SET_PARAMETER
message. The parameter the client sets is the nonstandard “Track” parameter
which
we made up. The server will also have to at some point handle a special request for a
file change. VLC code can certainly currently play a file if you give it an rtsp address,
such as, rtsp://127.0.0.1/test_file, however, for the switching system
to be complete you
will probably way to send a special new command for starting the file and sending it in
the same stream. For more information about this, player see the transcript of dialog
with live555
, the maker of the RTSP library that VLC uses. He

is a very helpful guy, and
think was useful. However, most of our work was really with the same file at different
qualities or resolutions.


The client naturally sends the RTSP SET_PARAMETER command to the server.
Further, it must handle the change in q
uality or resolution. Testing showed that
changes in quality, even at a P
-
frame caused little drift. However, future systems, could
switch to an I
-
frame first to reduce drift even more.


The client does have major drift at resolutions changes.

Impleme
ntation or Design

1.

Systems Level

1.

Server

1.

Multi Track Files

2.

File Switching

3.

RTSP

1.

SET PARAMETER

2.

Track Parameter

2.

Client

1.

RTSP

2.

SPS/PPS NAL Units

3.

Decoder

3.

Summary

2.

Decoder

1.

SPS/PPS Switching

1.

Live555 Transcript

2.

File Merging

3.

Existing Code

2.

Interpolation

1.

Last Frame of Data

2.

New Instance of Decoder

3.

Changing Parameters of Current
Decoder

3.

Testing and Modularization

1.

Goal of Decoder

2.

Simple Appending of NAL Unit
Stream

3.

Local Decoding

4.

Test Bed

4.

Advanced Switching

1.

P
-
Frame Switching

2.

Video Switching

Summary ?



1.

A note about resolution, in general this paper will refer to changing height and
width sizes as resolution changes. Video quality is sometimes referred to as resolution
on the internet and other sources. However, we refer to lower bit
-
rates at the same
d
imensions as lower quality, not lower resolution. Lower resolution is actual smaller
sizes.