IDIAP in Brief

judgedrunkshipServers

Nov 17, 2013 (3 years and 7 months ago)

77 views


Pierre Wellner

IDIAP and IM2


SCSC ‘04

Browsing Recorded Meetings


IDIAP & IM2


Smart Meeting Room


Visual & Audio Tracking


Browsing Meetings


Browser Evaluation






Outline

IDIAP in brief


A private non
-
profit research institute.


Located in Martigny, Valais since 1991.


80+ persons, 65+ researchers.


Annual budget: ~8 MCHF.


Three missions:


Research


Education


Technology Transfer

IDIAP in Brief

IDIAP Research Themes


Machine Learning


Algorithms for classification, regression and density estimation, etc.


Speech Processing


Spoken language understanding, noise
-
robust speech recognition,
large vocabulary speech recognition, low bit rate transmission.


Computer Vision


Object recognition, motion analysis, text recognition.


Media Indexing


Structuring audio and video, noisy text information retrieval.


Biometric Authentication


Speaker verification, face verification, multimodal fusion.


Multimodal Interaction


Meeting browser, HCI design, brain
-
computer interfaces


IDIAP Research Themes



Quickly
find

what happened in meetings.


Recording is easy, but finding is hard.


Technology for three steps:

Browsing Recorded Meetings

1.

Recording

2.

Analysis

3.

Browsing


Synchronized recording:


24
audio

channels


microphone arrays


binaural
mannequin



3
video

channels


Computer projector


White board


Notes on paper


Recording:
Smart Meeting Room

Recording:
Smart Meeting Room

Talker:

Pierre Wellner, Spiderphone

Callers: 8

Talker:

Mike Flynn, IDIAP

Callers: 8

Talker:

----

Callers: 8

Talker:

Pierre Wellner, Spiderphone

Callers: 8

Talker:

Mike Flynn, IDIAP

Callers: 8

Talker:

----

Callers: 8

Talker:

Pierre Wellner, Spiderphone

Callers: 8


Recording: Conference Calls

Analysis
:
Visual Tracking

Analysis
: Audio Tracking

Analysis:
Audio+Video Tracking



Display multimodal Analysis results:


Speaker tracking (who is talking)


Recognized speech


Meeting Actions
(e.g. presentation, discussion)


Interest levels, etc…


Control audio and video playback




Meeting Browsers:

bringing it all together

XMLtoSV
G Servlet

XMLtoSV
G Servlet

XMLtoSV
G Servlet

Processin
g

Processin
g

Processin
g

Various

Text
Transcripts

Ferret Architecture

Internet Explorer


Media


Real
Server

Real
Player

Servlets,
CGI & JSP

Processin
g

Apache
Tomcat

SVG
Viewer

XML
Data

Server Client

Browser Architecture

Demo

The Browser Evaluation Problem


No evaluation, or...


Tested by unique scheme


Often very subjective

[from Cutler et al, “Distributed Meetings: A Meeting Capture and Broadcasting System”, ACM Multimedia, 2002]


“I was able to get the information I needed […]”


“I would use this system again if I had to miss a meeting.”


“I would recommend the use of this system to my peers.”


No standard Browsing task




Objective comparisons not possible



The browser evaluation problem

Aims for a good BET


Performance, not judgment.


Independent of experimenter perception.


Directly comparable numeric scores.


Replicable.


Aims for a good BET

(Browser Evaluation Test)

The Media Browsing Task

Find a
maximum
number of

observations of interest

in a
minimum
amount of

time
.

But what are “
observations of interest
”?


The Media Browsing Task

test

sampling

BET

overview

observations

answers

observers

playback

system

subjects

media
browser

scoring

scores

meeting

participants

corpus

recording
system


IDIAP and the IM2 project


Recording in the Smart Meeting Room


Visual & Audio Processing


Browsing Meetings


Browser Evaluation






Summary

Related Research Projects


EC
-
FP5
-
IST

MultiModal Meeting Manager (M4)

http://www.m4project.org



EC
-
FP6
-
IST Integrated Project

Augmented Multi
-
party Interaction (AMI)

http://www.amiproject.org



National (CH) Research Competence Center on:


Interactive Multimodal Information
Management” (IM2)
:
http://www.im2.ch



DARPA EARS (US):

http://www.darpa.mil/iao/EARS.htm