20130516-EXPERIMEDIA-Technical-Overview-compressedx

licoricehealthAI and Robotics

Nov 14, 2013 (3 years and 9 months ago)

85 views

Technical Overview

Open Call Information
Teleconferences

16
-
17 May 2013


Thanos Voulodimos, ICCS/NTUA

(thanosv@mail.ntua.gr)


EXPERIMEDIA Components


The functional pieces of software in
EXPERIMEDIA have been grouped into logical
“components”


Components


Provide extension points


Interface to each other


Will interface to your technology


Can be adapted to fit your technology or have existing
parts replaced


Experiments drive the evolution of the facility


Pushing the components to provide new capabilities

2

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

Five EXPERIMEDIA Components


Experiment Content Component (ECC)


Social Content Component (SCC)


Audio Visual Content Component (AVCC)


Pervasive Content Component (PCC)


3D Content Component (3DCC)

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

3

Experiment Content Component (ECC)


Experiment management system


Support for automatic deployment


Local machine or Cloud


Automatic configuration available


Web based dashboard:


Experiment control via UI


Live
QoS
/
QoE
/
QoC

metric visualisation


Instrumentation API
(Java, C#, Ruby, [C++ in development])


Security modelling




4

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

Instrumentation & the ECC

5

EXPERIMEDIA baseline components

Your technologies

AVCC

PCC

SCC

3DCC

Metrics

Metrics

Metrics

Metrics

Internet services

Desktop apps

Mobile apps

Metrics

Metrics

Metrics

FMI Content

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

The ECC dashboard


Web based management:


C
onnections to instrumented systems


Experimental process


Support for NAGIOS views


Live visualisation of in
-
coming data


Metric model & database persistence


Data export

6

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

Social Content Component (SCC)


SCC

publishes,

gathers,

manages

and

monitors

data

generated

on

social

networks

during

an

experiment


Provides

access

to

different

social

networks

in

terms

of



Authentication


Reading

content


Publishing

content


Communicates

monitoring

metrics

to

the

ECC

for

analyzing

social

network

activities

during

the

experiment


Two

sub
-
components
:


SocialIntegrator


Social

Analytics

Dashboard

(SAD)


7

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

SocialIntegrator

API


Java

library

for

accessing

different

social

networks

transparently

in

terms

of
:


Content

retrieval

(posts,

photos,

comments,

likes,

etc
)


Publishing

and

modifying

capabilities


Extends

SocialAuth

Core

and

SocialAuth

Android

Java

API


Currently

supports

Facebook

and

Twitter


To

be

extended

for

more

social

networks


Supports monitoring of multiple
QoE

metrics (fully integrated with ECC)


Two applications developed for FHW:


Android Social Integrator App (Visitor App)


Web
-
based Social Integrator App (Expert App)

8

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

Social Analytics Dashboard


Configurable, extensible web service for gathering
and analyzing data from social networks.


Provides a plugin framework with supporting
database and scheduling services along with a
connection to the ECC for reporting metrics.

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

9

Audio Visual Content Component (AVCC)
Overview

10

AVCC

CONSUMER

PRODUCER

Atos provides AVCC as
a
service

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

AVCC Services



11


Publish a live stream


Live recording


Play a video on demand


Play a live stream (SD/HD)


Publish
live metadata streaming service


Adaptive
live stream


Live
Timeshift


Ingest


Professional editing/directing


UGC editing/directing


Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

AVCC Service Combination Example



12

CONSUMER

(Player)

PRODUCER
(Encoder)

Publish an adaptive live stream
+

timeshift
+

playing

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

Industrial robust services:

SMART
player

@
Paralympic

Games

13

87.395
XML,

565
Megabytes of
Metadata


1.9TB
traffic


300.000 hours

130 Terabytes
live content


170.000 unique users

2.400.000 sessions

179 countries
.


7

users from
Kyrgyzstan
!


13.500
users watching the
ceremony with
English

Audio and
870

users in
Spanish
.

smart.paralympic.org

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

Pervasive Content Component (PCC)
Overview

Location
tracking (Tracker)

Live games (Creator)

POI Data

Management

Augmented Reality Platform

QoE measurement

(Babylon)

14

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

PCC: Augmented Reality Platform


Overlay camera with information


POI data for example


Show lots of information even on old HW


Customizable icons for POIs


Render icon size according to distance


Interaction


E.g. pop up info box links to detail view


Smooth out user/device movement (jitter)


Show POI according to orientation (compass)


Use data from location manager for positioning


Android


Maven deployment



Copyright JOANNEUM RESEARCH Forschungsgesellschaft mbH and other members of the EXPERIMEDIA consortium

15

Augmented Reality Viewer screenshot

Copyright JOANNEUM RESEARCH Forschungsgesellschaft mbH and other members of the EXPERIMEDIA consortium

16

PCC: The Creator


Rapid prototyping support for game mechanics and game
structure


Collaborative, accessible from any web browser


Creator makes it easy to
author interactive
experiences in the
physical domain using a wide variety of sensors, services and
devices


Creator acts as a content management system for required
assets and game objects. Users can edit the game in real time
and changes are propagated immediately.


A rule engine allows the Creator to support custom game
logic. The rule engine is extendable using plugins and users
can change or update rules in real
-
time.


Details
:
http://
www.slideshare.net/kallep/pervasive
-
game
-
development
-
with
-
the
-
creator



Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

17

PCC: Babylon


Continuous in
-
game / in
-
experiment measurements of
subjective, qualitative QoE


Complemented by objective, quantitative measures (data
logging), and...


pre
-

and post
-
experiment subjective, qualitative measures
(questionnaires, interviews, focus groups).


A server component gathers information and feedback
from participants in real
-
time. Data is visualized using
maps, graphs and timelines.


Babylon supports integration into
iOS

and Android
mobile clients. The API is accessible over HTTP for
custom integrations.


Babylon is
integrated
with
the ECC.



18

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

Babylon server and client screenshots

19

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

PCC: Tracker


Tracker is an open location
-
tracking server
which provides an open API for external
reports to post locations.


Currently
, both mobile clients and custom
hardware integrations
exist.


Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

20

3D Content Component


3DCC supports experimenters in acquiring and
manipulating 3D information from depth
sensing devices.



Various types of information can be acquired:
raw depth images, human skeleton, RGB
images registered to the depth images.



3DCC
comprises three
different levels (low,
mid and high).


Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

21

3DCC: Low level functionality


Depth Acquisition:


Provides
the experimenter with the raw depth
information of a scene.


Is the middle layer between the Kinect device and the
experimenter.


Has simple and easy to use functions that provide the
experimenter with per pixel information about the depth
of the scene.



Skeleton Acquisition:


3DCC can provide robust skeleton extraction for up to 15
joints.



RGB Acquisition:


Images coming from Kinect, along with their registration
to depth pixel transformation, can be provided to the
experimenter


Texturing of a post produced 3D model can be made
possible through this information.


Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

22

3DCC: Mid level functionality


Depth Enhancement:


Several filtering algorithm are provided to smooth and de
-
noise the raw information
so that more accurate depth measurement can be made possible.


Skeleton Enhancement:


Jerky
(noisy) skeleton joints are detected and tracked and therefore corrected through
a sophisticated tailored filtering framework to provide a more realistic skeleton
.


Biomechanical Measurements:


Biomechanical
measurements that are inferred from both depth and skeleton
information as angles between bones, human joints and calibrated objects’ velocities,
human body parts surface areas and calibrated objects’ surface area.



Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

23

3DCC: High level functionality


Avatar Creation:


The 3DCC provides an avatar authoring tool so
that experimenters can create their own avatars
that can be easily integrated into a virtual world
.


Avatar Motion:


The 3DCC can also

interactively
move the

avatar using a Kinect alone.



Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

24

Thank you!

Key

Component name

ECC Interface

Configuration

ECC Interface

Monitoring

ECC Interface

Security

Content Interface

Content interface

Sub
-
component

Sub
-
component

Extension point

Extension point

26

Copyright University of Southampton IT Innovation Centre and other members of the EXPERIMEDIA consortium, 2012

Live
streaming

27

Video
Codec

Player/Device

H.264

Adobe Flash player

iOS

based device

QuickTime player

Microsoft Silverlight

VideoLAN

VLC player

Set
-
top boxes

3GPP devices

VP6

Adobe Flash player

SorensonSpark

Adobe Flash player

Screen Share

Adobe Flash player

Audio Codec

Player/Device

AAC

Adobe Flash player

iOS based device

QuickTime player

Microsoft Silverlight

VideoLAN VLC player

Set
-
top boxes

3GPP devices

AAC LC, HE
-
AAC

Adobe Flash player

iOS

based device

QuickTime player

Microsoft Silverlight

MP3

Adobe Flash player

iOS

based device

QuickTime player

Microsoft Silverlight

Speex

Adobe Flash player

Nelly Moser ASAO

Adobe Flash player

Player/Device

Protocol (OUTPUT)

Adobe Flash player

RTMP/RTMPE/RTMPT/RTMPTE/RTMPS

Flash HDS

iOS based device

Apple HLS

QuickTime player

Apple HLS

Microsoft Silverlight

Microsoft Smooth Streaming

VideoLAN VLC player

RTSP/RTP

MPEG2
-
TS

Set
-
top boxes

RTSP/RTP

MPEG2
-
TS

3GPP devices

RTSP/RTP

MPEG2
-
TS

AVCC modules



Input Manager
:
manages the reception
of all live content


Live
Metadata

Manager
:
manages the
reception of metadata and time stamps


MoreVideo


Video Analysis


Device

Adaptation

Manager
:
adapts the
media content to different platforms


Multi

Quality

Manager
:
coordinates the
mutiquality

content





28



Timeshift

Manager
:
continuously

records live streams for immediate playback on
deferred


VoD

Manager
: records
live

content

and
delivers
pre
-
recorded video and audio streams


Ingest

Manager
:
stores

new
content

from

a media file


Media
Distribution
:
produces
all multiplexes media output of the
distribution


UGC

VoD

ingest process

29

AV content
upload

Content
description and
adaptation
instructions

Content
adaptation

Content
publication

&
distribution

Content
description
and access
URLs
forward to
Exp. CMS

Codec

accepted

video/
avi

video/mp4

video/
webm

video/
ogg

video/
quicktime

Output
profiles

supported
*

video/mp4

video/
ogg

video/
webm

audio/
mpeg

(mp3)

video/x
-
flv

(
adaptive
)

Img
/JPEG

(
thumbnail
)

Distribution

Flash / Flex / HTML5 / Mobile
native

applications

PC /
Android

/
iOS
,

etc…

Progresive

download
, RTMP,
HDS, HLS, Smooth
Streaming, RTSP/RTP, MPEG2
-
TS

* The actual profiles can be optimised for each
experiment

AVCC
Monitoring


Focus on
Monitoring


Easy to
deploy java
software

30

RabbitMQ

Server

ECC

Monitoring control

signals

Experiment

data

AVCC EM
Interface

AVCC

Clients

AVCC Server

Playback client

Publish client

Client Count

I/O Bytes Total

I/O Bytes Rate

Publisher Count

Stream Count

Connection Count per Stream
Type

I/O Bytes Total per Stream
Type

I/O Bytes Rate per Stream Type

I/O Bytes Total per Stream

I/O Bytes Rate per Stream

Audio, Video and Data
Bitrate


([
audio,video,data
]
Audio, Video and Data Buffer
Length (bytes and seconds)

A/V Data Input Bitrate

Playback Rate


Dropped Frames

Network capacity

AVCC Server RTT

(SRTT)

Last Metadata Received

FPS (from camera)

Publish/Playing time

A/V Activity Level (0
-
100)

Video Resolution

Key Frame Interval



AR Integration


For Android Apps


Android 2.3+


Devices with camera


Necessary permissions:


Camera access


Data retrieval from Internet


If used: GPS, cell,
wifi

information for localization


Maven deployment





Copyright JOANNEUM RESEARCH Forschungsgesellschaft mbH and other members of the EXPERIMEDIA consortium

31

Creator screenshot

Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013

32