A Quest for an Internet Video Quality-of-Experience Metric

chardfriendlyΤεχνίτη Νοημοσύνη και Ρομποτική

16 Οκτ 2013 (πριν από 3 χρόνια και 9 μήνες)

80 εμφανίσεις

1

A Quest for an Internet Video

Quality
-
of
-
Experience Metric

Athula

Balachandran
,
Vyas

Sekar
,

Aditya

Akella
,
Srinivasan

Seshan
,

Ion
Stoica
,
Hui

Zhang

Internet Video is taking off

2

Improve Users’ Quality of Experience

Video Quality Metrics: The State of the Art

3

Objective Score


(e.g., Peak Signal to Noise Ratio)

Subjective Scores

(e.g., Mean Opinion
Score)

Problem 1: New Effects, New Metrics

4

PLAYER

STATES

EVENTS

Joining

Playing

Buffering

Playing

Buffer

filled up

Buffer

empty

Buffer

f
illed up

Switch

bitrate

Problem 1: New Effects, New Metrics

5

PLAYER

STATES

EVENTS

Joining

Playing

Buffering

Playing

Buffer

filled up

Buffer

empty

Buffer

f
illed up

Switch

bitrate

Problem 1: New Effects, New Metrics

6

PLAYER

STATES

EVENTS

Joining

Playing

Buffering

Playing

Buffer

filled up

Buffer

empty

Buffer

f
illed up

Switch

bitrate

Problem 1: New Effects, New Metrics

7

PLAYER

STATES

EVENTS

Joining

Playing

Buffering

Playing

Buffer

filled up

Buffer

empty

Buffer

f
illed up

Switch

bitrate

Problem 1: New Effects, New Metrics

8

PLAYER

STATES

EVENTS

Joining

Playing

Buffering

Playing

Buffer

filled up

Buffer

empty

Buffer

f
illed up

Switch

bitrate

Problem 1: New Effects, New Metrics

9

PLAYER

STATES

EVENTS

Joining

Playing

Buffering

Playing

Buffer

filled up

Buffer

empty

Buffer

f
illed up

Switch

bitrate

Problem 1: New Effects, New Metrics

10

PLAYER

STATES

EVENTS

Joining

Playing

Buffering

Playing

Buffer

filled up

Buffer

empty

Buffer

f
illed up

Switch

bitrate

Join Time

Buffering Ratio

Rate of buffering

Rate of switching

Average bitrate

Problem 2: Opinion Scores


Engagement

Opinion Scores


-

Not representative of “in the wild”
experience


-

Combinatorial explosion of parameters


Engagement as
replacement
for opinion score.


(e.g., Play time, customer return rate)






11

Internet Video
QoE

12

Objective Scores

PSNR

Subjective Scores

MOS

Internet Video
QoE

13

Objective Scores

PSNR

Subjective Scores

MOS

Engagement

(e.g., Fraction of video viewed)

Internet Video
QoE

14

Objective Scores

PSNR

Join Time, Avg. bitrate, …?

Subjective Scores

MOS

Engagement

(e.g., Fraction of video viewed)

Internet Video
QoE

15

Objective Scores

PSNR

Join Time, Avg. bitrate, …?

f(Join Time, Avg. bitrate, …)

Subjective Scores

MOS

Engagement

(e.g., Fraction of video viewed)

Internet Video
QoE

16

Objective Scores

PSNR

Join Time, Avg. bitrate, …?

f(Join Time, Avg. bitrate, …)

Subjective Scores

MOS

Engagement

(e.g., Fraction of video viewed)

Outline


Need for a unified
QoE



What makes this hard?



Our proposed approach


17

18


Challenge: Complex Engagement
-
to
-
metric
Relationships

Engagement

Quality Metric

[
Dobrian

et al.
Sigcomm

2011]













19


Challenge: Complex Engagement
-
to
-
metric
Relationships

Engagement

Quality Metric

Non
-
monotonic


Engagement


Average bitrate

[
Dobrian

et al.
Sigcomm

2011]













20


Challenge: Complex Engagement
-
to
-
metric
Relationships

Engagement

Quality Metric

Non
-
monotonic


Engagement


Average bitrate


Engagement


Rate of switching

Threshold

Challenge: Complex Metric Interdependencies

21

Join Time



Bitrate


Rate of


buffering


Rate of


switching


Buffering



Ratio

Challenge: Complex Metric Interdependencies

22

Join Time



Bitrate


Rate of


buffering


Rate of


switching


Buffering



Ratio

Challenge: Complex Metric Interdependencies

23

Join Time


Rate of


buffering


Rate of


switching


Buffering



Ratio



Bitrate

Challenge: Complex Metric Interdependencies

24

Join Time

Avg. bitrate


Rate of


buffering


Rate of


switching


Buffering



Ratio

25

Need to learn these complex engagement
-
to
-
metric
relationships and metric
-
to
-
metric dependencies

Casting as a Learning Problem

26

MACHINE LEARNING

Engagement

Quality Metrics

QoE

Model

Need to learn these complex engagement
-
to
-
metric
relationships and metric
-
to
-
metric dependencies

Impact of the ML algorithm

27


Classify engagement into uniform classes


Accuracy = # of accurate predictions/ # of cases

ML algorithm must be expressive enough to handle the
complex relationships and interdependencies

Challenge: Confounding Factors








28

Live and VOD sessions experience similar quality

Challenge: Confounding Factors








29

However, user viewing behavior is very different

Challenge: Confounding Factors

30

Devices

User Interest

Connectivity

Need systematic approach to identify and handle confounding factors

Domain
-
specific Refinement

31

MACHINE LEARNING

Engagement

Quality Metrics

QoE

Model

Domain
-
specific Refinement

32

MACHINE LEARNING

Engagement

Quality Metrics

QoE

Model

Confounding


Factors

Improved prediction accuracy

33

Refined ML models can handle confounding factors

Concluding Remarks


Internet Video needs unified quantitative
QoE



What makes this hard?


Complex engagement
-
to
-
metric relationships


Complex metric
-
to
-
metric interdependencies


Confounding factors (e.g., genre, device)



Promising start


Machine learning + domain
-
specific refinements



Open Challenges


Coverage over confounding factors


System Design



34