Smart Meeting Systems

brasscoffeeAI and Robotics

Nov 17, 2013 (3 years and 6 months ago)

63 views

Smart Meeting Systems

Josh Reilly

Why are Smart Meeting
Systems worth studying?

Objectives of a Smart Meeting System


Improves the productivity of a team by automating
the:


Capture of the meeting


Processing of the meeting for valuable information


Displaying of that information accurately and
effectively to the end user through a client
application

Organization of Smart Meeting System
Processes


A smart meeting system can be decomposed into three
sets of processes


Meeting Capture


Meeting Recognition


Semantic Processing

Organization of Smart Meeting System
Processes

Meeting Capture


Gathering raw inputs from the meeting


Video Capture


Audio Capture


Other Context

Video Capture


Video feeds from:


Cameras for the attendees


Could use a single static camera


Could use a single camera with pan, tilt, zoom (PTZ)
capabilities


Recommend camera view of every contributor's face


Visual Aids


Separate camera


Digital feed from device

Microsoft Distributed Meetings Project
Camera Placement

Microsoft Distributed Meetings Project

Video Capture

RingCam


Array of 90
º

Cameras


360
º

Panoramic view

Audio Capture


Use an array of microphones


Placed on the table


Placed on the ceiling


Worn on the person


Levels need to be controlled so that they are similar
levels for each contributor

Microsoft Distributed Meetings Project
Audio Capture


RingCam


Has an array of
microphones on its
base.

Other Context Capture


RFID to track attendees


Attendees swipe their RFID cards when they enter
the meeting to add their ID to the list of people
attending this meeting


Motion Detectors


to track the locations of attendees within the room

Organization of Smart Meeting System
Processes

Meeting Recognition


The processing of the raw capture before it is
organized into something useful


Steps:


Person Identification


Attention Detection


Activity Recognition


Hot Spot Recognition


Summarization

Person Identification


Person Identification

is associating sections of video, audio, and the visual
aids that were captured from the meeting with the
attendee(s) that they belong to



Face Recognition



Face Tracking



Speech Recognition



SSL



Beamforming

Person Identification

Face Recognition



Facial Recognition


Identify the person speaking from a list of attendees


Eigenface Approach


Challenges


Poor Quality Images


Poor Room Lighting


Continuously changing facial expressions


Occlusion

Face Recognition

The Eigenface Approach



All faces are assumed to
be made up of different
percentages of different
eigenfaces



A set of eigenfaces is a
set of very generalized
pictures of faces that
were generated so that
each has a basic
ingredient that can be
used to make a face

Eigenfaces from AT&T
Laboratories Cambridge

Person Identification

Speech Recognition


Speech Recognition


Match the voice of the person speaking to someone on
the list of attendees


Using Voice recognition in conjunction with face
recognition allows for an accurate identification of the
speaker


Sound Source Localization (SSL)


Used to determine which camera is pointed at the speaker


Could be used to point PTZ camera


Beamforming

Person Identification

Writer Recognition



Writer Recognition


When someone writes on the whiteboard, they may
not be in clear view of the cameras


Writing recognition algorithms can be used to
identify who wrote what during a meeting

Attention Detection



Attention Detection


Attempt to determine who is looking at whom during
a meeting.


Provides information used for activity recognition
and hot spot recognition


Done using:


Hidden Markov Models (HMM)


Sound Source Localization (SSL)


Known layout of room

Activity Recognition



Determine what is happening during the meeting


Step 1:


Determine what each individual is doing at each
point during the meeting


Person Identification, Attention Detection, SSL,
Gesture Recognition


Step 2:


Take that information to determine what activity the
entire group is engaging in at each point during the
meeting

Hot Spot Recognition



Find the important parts of the meeting


Using sound queues


Ex: Changes in pitch


Using activity recognition


When people are nodding


When their focus changes

Summarization



Takes all of the information that the smart meeting
system has learned about the meeting and creates a
quick overview of the events that took place during
that meeting.



This information will be used in the semantic
processing stage

Organization of Smart Meeting System
Processes

Semantic Processing



Takes the information from the meeting recognition
step and makes it usable by the end user.


Meeting Annotation


Meeting Indexing


Meeting Browsing

Meeting Annotation



Describe the raw data from the meeting from each
viewpoint



Attempt to label all meeting segments


Implicitly


Automatically


Explicitly


By Hand

Meeting Annotation

Implicit



Automated Annotation


Assumes that the meeting recognition processes
performed with relatively high efficiency


Tags every person in the video


Narrates what was happening during the meeting


Has not been achieved

Meeting Annotation

Explicit



Annotation By Hand


When the recognition processes fail to gather
sufficient correct information about the raw data


Users will have to go through the meeting and tag
the people attending as well as indicate what events
are happening all through the meeting

Meeting Indexing



Indexing is done at all levels of data from a raw
audio feed to the annotations



The best form of indexing to use is the event
-
based
indexing


An index is created every time an event occurs


This is the best way for users to find a specific spot
in the meeting when performing a query

Meeting Browsing



The interface that the end user uses to retrieve
information from the meetings



Functions:


Can browse/search a list of all meetings for a
specific meeting


Can browse/search the contents of the chosen
meeting


Aided by tools like bookmarks, a meeting outline, and
queries (content, people, camera angles, visual aids,
etc...)

Meeting Browsing

Microsoft Distributed Meetings

Remote Attendee



Use the smart meeting system as the attendee's eyes
and ears



Microsoft's PING project


Uses a monitor and speaker to display the remote
attendee's voice and audio during the meeting


However, the remote attendee is often ignored

Carnegie Mellon University’s

Meeting System Architecture

Lacks


Activity Recognition


Hot Spot Recognition


Annotations


University of California, San Diego

AVIARY System Architecture


2 PCs


4 Static Cameras


4 PTZ Cameras


No SSL

Ricoh

Portable Meeting Recorder

Ricoh

Portable Meeting Recorder

Doughnut Camera

Ricoh

Portable Meeting Recorder

Meeting Browser

Technology Limitations



Speech recognition and facial recognition algorithms
are not yet as efficient as they should be in order for
a smart meeting system to perform accurately

Workspace Limitations


Cameras and microphones can block view, distract,
or intimidate attendees during the meeting


Security and Privacy needs to be addressed

References

[1]

Zhiwen Yu and Yuichi Nakamura. 2010. Smart meeting systems: A survey of state
-
of
-
the
-
art and open
issues. ACM Comput. Surv. 42, 2, Article 8 (March 2010), 20 pages. DOI=10.1145/1667062.1667065
http://doi.acm.org/10.1145/1667062.1667065

[2]

Ross Cutler , Yong Rui , Anoop Gupta , Jj Cadiz , Ivan Tashev , Li
-
wei He , Alex Colburn , Zhengyou
Zhang , Zicheng Liu , Steve Silverberg. (2002). Distributed Meetings. A Meeting Capture and
Broadcasting System. 10 pages.
http://research.microsoft.com/en
-
us/um/people/yongrui/ps/mm02.pdf

[3]

Harold Fox. 2004. The eFacilitator: A Meeting Capture Application and Infrastructure. 89 pages.
http://hdl.handle.net/1721.1/17672

[4]

Yong Rui, Eric Rudolph, Li
-
wei He, Rico Malvar, Michael Cohen, Ivan Tashev. 2006. Ping: A Group
-
To
-
Individual Distributed meeting System. 4 pages.
http://research.microsoft.com/apps/pubs/default.aspx?id=76779

[5]

Dar
-
Shyang Lee, Berna Erol, Jamey Graham, Jonathan Hull, Norihiko Murata. 2011. Portable Meeting
Recorder. 10 pages. http://rii.ricoh.com/sites/default/files/Portable_Meeting_Recorder.pdf