Multimedia Projects Our Experience

mittenturkeyElectronics - Devices

Nov 26, 2013 (3 years and 6 months ago)

60 views

Multimedia Projects

Our Experience

Research Projects

NSF Multimedia Laboratory at
Florida Atlantic University

(1995
-
2001)

Director Dr. Borko Furht

Projects


Content
-
Based Image Retrieval Using Relevance
Feedback (Oge Marques)


IP Simulcast
-

An Innovative Video and Audio
Broadcasting Technique Over the Internet (Ray
Westwater and Jeff Ice)


XYZ Video Encoding Technique (Ray Westwater
& Joshua Greenberg)


An Innovative Motion Estimation Algorithm for
MPEG Codec (Joshua Greenberg)


A Fast Content
-
Based Multimedia Retrieval
Technique (P. Saksobhavivat)


Interactive Progressive Encoding System
(Joe Celi)


Internet Broadcasting or


Webcasting


Broadcasting multimedia data
(audio and video) over the Internet
-

from a server (sender) to a large
number of clients (receivers)


Applications include:




radio and television broadcasting
real
-
time broadcasting of critical data
distance learning
videoconferencing
database replication
electronic software distribution


Broadcast Pyramid Applied


in IP Simulcast

SERVER
Client
1
Client
2
Client
3
Client
2
2
Client
4
Client
5
Client
6
Client
9
Client
10
Client
11
Client
12
Client
13
Client
8
Client
14
Client
7
IP Simulcast
-

An Innovative Technique


for Internet Broadcasting


IP Simulcast reduces the server (or
sender) overhead by distributing the
load to each client (or receiver)


Each receiver becomes a repeater,
which rebroadcasts its received
content to two child receivers


The needed network bandwidth for
the server is significantly reduced

Characteristics of IP Simulcast


It is a radically different model of digital
broadcast, referred to as repeater
-
server
model


The server manages and controls the
interconnection of repeaters


Each repeater not only plays back the
data stream, but also transmits the data
to two other repeaters


IP Simulcast provides guaranteed
delivery of packets, which is not the case
with IP Multicast

Product: AllCast

www.allcast.com

Broadcasting Tree

Once the AllCast
Broadcaster is
configured, many users
can connect to hear/view
content. The bandwidth
usage is distributed
across the participants,
as illustrated by the
dynamic, self
-
healing
dissemination tree shown
in the AllCast main
window.





Microsoft Media Player

with AllCast Plug
-
in


Users can connect to a broadcast using the Microsoft
Windows Media Player together with a small, seamlessly
integrated AllCast Plug
-
in.


The plug
-
in enables the Windows Media Player to participate
in peer
-
to
-
peer broadcasts.



XYZ
-

New Video
Compression Technique


The XYZ video compression
algorithm is based on 3D Discrete
Cosine Transform (DCT)


It provides very high compression
ratios and excellent video quality


It is very suitable for real
-
time video
compression


Forming Video Cube for


XYZ Compression

8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
Frames
0
1
2
3
4
5
6
7
8X8X8
video
cube

Block Diagram of


the XYZ Compression

Forward
3-D
DCT
Quantizer
Entropy
Coder
Quantizing
Tables
Huffman
Table
Video
Cube
Compressed
video
sequence

Key Encoding Equations


Both encoder and decoder are symmetrical, which makes
the algorithm suitable for VLSI implementation





















F
u
v
w
C
u
C
v
C
w
f
x
y
z
x
u
y
v
z
w
z
y
x
(
,
,
)
,
,
cos
cos
cos












0
7
0
7
0
7
2
1
16
2
1
16
2
1
16



(4.1)







F
u
v
w
F
u
v
w
Q
u
v
w
q
,
,
,
,
,
,







(4.3)

XYZ Versus MPEG

Video
Compression
Technique
Compression
Ratio
Normalized
RMS
Error
Execution
Time [min]
(8 frames, 320x240)
XYZ
(QT1)
34.5
0.079
6.45
XYZ
(QT2)
57.7
0.097
6.45
XYZ
(QT3)
70.8
0.105
6.45
XYZ
(QT4)
101.7
0.120
6.45
XYZ
(QT5)
128.1
0.130
6.45
MPEG
Logarithmic Search and
Error Correction
11.0
0.080
21.35
MPEG
Exhaustive Search and
Error Correction
15.6
0.080
163.0
MPEG
Logarithmic Search and
No Error Correction
27.0
0.140
21.35
MPEG
Exhaustive Search and
No Error Correction
32.9
0.125
163.0

Complexity of Video


Compression Techniques

Compression
Algorithm
Encoder
Complexity
Decoder
Complexity
Total
Complexity
H.261/H.263
970
200
1,170
MPEG
No B Frames
750
100
850
MPEG
70% B Frames
1,120
120
1,240
XYZ
240
240
480

XYZ Versus MPEG

Original

MPEG

Cr=11,NRMSE=0.08

MPEG, Motion Est. only

Cr=27, NRMSE=0.14

XYZ

Cr=45, NRMSE=0.079

Examples of XYZ Compression


Original

XYZ
-
compressed

Cr=51

Examples of XYZ Compression


Original

XYZ
-
compressed

Cr=110


Sensitivity of the XYZ Algorithm


to Various Video Effects

Movie Clip
Video Effect
Compression
Ratio
Normalized RMSE
Error
Dick Tracy
Typical Motion
34.5
0.079
Interview with the
Vampire
Camera Break
32.5
0.087
Interview with the
Vampire
Camera Panning
26.1
0.085
Total Recall
Camera Panning
17.5
0.049
Total Recall
Camera Zoom
23.9
0.042
Interceptor
Fast Motion
26.0
0.025

Characteristics of


XYZ Video Compression


XYZ gives significantly better
compression ratios than MPEG for
the same quality of video


For similar compression ratios, XYZ
gives much better quality than
MPEG


XYZ is faster than MPEG (lower
complexity)


XYZ is simple for implementation


Applications of the XYZ


Interactive TV and TV set
-
top boxes


TV phone


Video broadcasting on the Internet


Video
-
on
-
demand applications


Videoconferencing


Wireless video



TV Phone


Videophone is a box on the top of TV with a
small camera, modem, and video/audio
codec.

Videophone
Videophone

Conventional
telephone network
TV set
TV set
Design of the TV Phone

VIDEOPHONE
TV
Video/audio capture
and compression
Video-in
Video/audio out
Video/audio out
Video/audio in
Modem
Telephone
lines

User's
control panel
Video/audio
decompression
and conversion
to TV format
VLSI chips
implement XYZ
algorithm
Camera


A Fast Content
-
Based
Multimedia Retrieval Technique



Two main approaches in indexing
and retrieval of images and videos


Keyword
-
based indexing and
retrieval


Content
-
based indexing and
retrieval

Keyword
-
Based

Retrieval and Indexing


Uses keywords or descriptive text, which
is stored together with images and
videos in the database


Retrieval is performed by matching the
query, given in the form of keywords,
with the stored keywords


This approach is not satisfactory
-

the
text
-
based description is incomplete,
imprecise, and inconsistent in specifying
visual information

New Algorithm for Similarity
-
Based Retrieval of Images


Images in the database are stored as
JPEG
-
compressed images


The user submits a request for search
-
by
-
similarity by presenting the desired
image.


The algorithm calculates the DC
coefficients of this image and creates the
histogram of DC coefficients.


The algorithm compares the DC
histogram of the submitted image with
the DC histograms

of the stored images.

Histogram of DC Coefficients
for the Image “Elephant”

Comparison of Histograms

of DC Coefficients

Example of Similarity
-
Based
Retrieval Using the DC Histograms

Similarity
-
Based Retrieval of
Compressed Video


Partitioning video into clips
-

video
segmentation


Key frame extraction


Indexing and retrieval of key frames

DC Histogram Technique
Applied for Video Partitioning

0
5
10
15
20
25
30
35
40
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Frame number
NSD x 100 [%]
Example of Similarity
-
Based Retrieval
of Key Frames Using DC Histograms

Interactive Progressive

Encoding System



Users submit requests for imagery to the
image database via a graphical user
interface


Upon an initial request, a DCT image
(version of the image based on DC
coefficients only) is transmitted and
reconstructed at the user site.


The user can then isolate specific
regions of interests within the image and
request additional levels of details.

Band Transmission in Interactive JPEG
System Based on Spectral Selection

All blocks of the image
Amplitude of
Coefficients
Blocks of the selected region of the image
Band-1 Band-2 Band-3 Band-4
DC AC
1
AC
2
AC
3
…… AC
6
AC
7
………. AC
63
Transmission
Prototype System
-

IPES and
Experimental Results


Original image “Airport”

Interactive Progressive
Transmission in Four Scans

Selection of Two Regions

Cumulative Number of
Transmitted Bits

0
200
400
600
800
1000
1200
1400
1600
1800
2000
1
2
3
4
Scans
Cumulative transmitted bits [Kbits]
Whole Image
1 Region
2 Regions
Extracted Images From

a Group of Images

Applications


Retrieval and transmission of complex
images over low bandwidth
communication channels (image
transmission over the Internet, real
-
time
transmission of medical images)


Archiving and browsing visually lossless
image databases (medical imaginary,
space exploration

and military
applications)

Content
-
Based Retrieval


Large, complex, and ever growing,
distributed, mostly unstructured,
multimedia repositories


Three ways of retrieving multimedia
information:


Free browsing

(inefficient, time
-
consuming,
doesn’t scale well)


Text
-
based retrieval

(relies on metadata, time
-
consuming, subjective)


Content
-
based retrieval

(requires intelligent
interpretation of the contents)

Design of MUSE System

Image Analysis


Image Feature
Extraction

-
Color

-
Shape

-
Texture

Image Representation
&

Feature Organization

Image

Archive

User

GUI

-
Image

selection

-
Result viewing

Probability
recalculation &
candidate ranking

Feature Extraction

Similarity comparison

Interactive learning

& Display update

Off
-
line

Online

Query By Example

Example
image

Best result

Similarity
Score [0,1]



Relevance Feedback

Good


Bad


Neither


Relevance Feedback
-

Next

Technology Behind the MUSE System
Feature extraction


Extraction of relevant image features
impacts the overall performance of the
system.


MUSE uses:


color
-
related features

(color histograms, color
space partitioning and/or quantization, color
moments, color coherence vectors)


texture
-
related features

(Multiresolution
Simultaneous Autoregressive Model
-

MSAR)


frequency
-
related features

(DFT, DCT)

Technology Behind the MUSE System
Bayesian formulation


MUSE is based on a Bayesian framework for
relevance feedback.


During each iteration of a MUSE session, the
system displays a subset of images from its
database, and the user takes an action in
response, which the system observes.


Based on the user’s actions, the probability
distribution over possible targets is refined.
(Most systems refine the user’s query)


The best candidates are then displayed back.