Mobile Media Metadata:

superbshelfMobile - Wireless

Nov 12, 2013 (3 years and 5 months ago)

71 views


2004.08.23
-

SLIDE
1


MOBILE MEDIA WORKSHOP PRESENTATION

Prof. Marc Davis

University of California at Berkeley

School of Information Management and Systems

Garage Cinema Research

http://garage.sims.berkeley.edu


Mobile Media Metadata:

The Future of Mobile Imaging


2004.08.23
-

SLIDE
3


MOBILE MEDIA WORKSHOP PRESENTATION

Garage Cinema Research


Research and develop technology and
applications that will enable daily media
consumers to become daily media producers



Theory, design, and development of digital
media systems that


Create descriptions of media content and structure
(metadata)


Use metadata to automate media production and
reuse


2004.08.23
-

SLIDE
4


MOBILE MEDIA WORKSHOP PRESENTATION

What is the Problem?


Today people cannot easily find, edit, share, and
reuse media



Computers don’t understand media content


Media is opaque and data rich


We lack structured representations



Without metadata (descriptions of media content
and structure), manipulating digital media will
remain like word
-
processing with bitmaps


2004.08.23
-

SLIDE
5


MOBILE MEDIA WORKSHOP PRESENTATION

Signal
-
to
-
Symbol Problems


Semantic Gap


Gap between low
-
level signal analysis
and high
-
level
semantic descriptions


“Vertical off
-
white
rectangular blob on
blue background”
does not equal
“Campanile at UC
Berkeley”


2004.08.23
-

SLIDE
6


MOBILE MEDIA WORKSHOP PRESENTATION

Signal
-
to
-
Symbol Problems


Sensory Gap


Gap between how an object appears and what it is


Different images of same object can appear
dissimilar


Images of different objects can appear similar


2004.08.23
-

SLIDE
7


MOBILE MEDIA WORKSHOP PRESENTATION

Computer Vision and Context


You go out drinking with your friends


You get drunk


Really drunk


You get hit over the head and pass out


You are flown to a city in a country you’ve never been to
with a language you don’t understand and an alphabet
you can’t read


You wake up face down in a gutter with a terrible
hangover


You have no idea where you are or how you got there


This is what it’s like to be most computer vision
systems

they have no
context



Context

is what enables us to understand what we see


2004.08.23
-

SLIDE
8


MOBILE MEDIA WORKSHOP PRESENTATION

METADATA

Traditional Media Production Chain

M E T A D A T A

PRE
-
PRODUCTION

POST
-
PRODUCTION

PRODUCTION

DISTRIBUTION

Metadata
-
Centric Production Chain


2004.08.23
-

SLIDE
9


MOBILE MEDIA WORKSHOP PRESENTATION

Research Projects


Media Streams


A framework for creating metadata throughout the media
production cycle to enable media reuse


Active Capture


Automates direction and cinematography using real
-
time audio
-
video analysis in an interactive control loop to create reusable
media assets


Adaptive Media


Uses adaptive media templates and automatic editing functions
to mass customize and personalize media


Mobile Media Metadata


Leverages the spatio
-
temporal context and social community of
media capture to automate metadata creation for mobile media


Social Uses of Personal Media


Analysis of social uses of media to predict future uses and shape
the design of next
-
generation personal media devices and
applications



2004.08.23
-

SLIDE
10


MOBILE MEDIA WORKSHOP PRESENTATION

Asset Retrieval

and Reuse

Automated Media Production Process


Web Integration

and

Streaming Media

Services

Flash

Generator

MMS

XHTML

Print/Physical

Media

Active

Capture

1

Automatic

Editing

3

Personalized/

Customized

Delivery

4

Adaptive Media Engine

2

Annotation

and Retrieval

Reusable Online

Asset Database

Annotation of

Media Assets


2004.08.23
-

SLIDE
11


MOBILE MEDIA WORKSHOP PRESENTATION

The Coming Media Revolution


Media capture devices become programmable
and networked


Metadata creation and use become integrated
throughout media production and reuse


Media production changes from being a
mechanical process to a computational process


Media becomes programmable and networked



Daily media consumers become daily media
producers


2004.08.23
-

SLIDE
12


MOBILE MEDIA WORKSHOP PRESENTATION

Research Projects


Media Streams


A framework for creating metadata throughout the media
production cycle to enable media reuse


Active Capture


Automates direction and cinematography using real
-
time audio
-
video analysis in an interactive control loop to create reusable
media assets


Adaptive Media


Uses adaptive media templates and automatic editing functions
to mass customize and personalize media


Mobile Media Metadata


Leverages the spatio
-
temporal context and social community of
media capture to automate metadata creation for mobile media


Social Uses of Personal Media


Analysis of social uses of media to predict future uses and shape
the design of next
-
generation personal media devices and
applications



2004.08.23
-

SLIDE
13


MOBILE MEDIA WORKSHOP PRESENTATION

Moore’s Law for Cameras

2000

Kodak DC40

Nintendo GameBoy Camera

$400

$ 40

2002

Kodak DX4900

SiPix StyleCam Blink


2004.08.23
-

SLIDE
14


MOBILE MEDIA WORKSHOP PRESENTATION

Capture+Processing+Interaction+Network


2004.08.23
-

SLIDE
15


MOBILE MEDIA WORKSHOP PRESENTATION

Camera Phones as Platform


Media capture (images, video,
audio)


Programmable processing using
open standard operating
systems, programming
languages, and APIs


Wireless networking


Personal information
management functions


Rich user interaction modalities


Time, location, and user
contextual metadata


2004.08.23
-

SLIDE
16


MOBILE MEDIA WORKSHOP PRESENTATION

Camera Phones as Platform


In the first half of 2003, more
camera phones were sold
worldwide than digital cameras


By 2008, the average camera
phone is predicted to have 5
megapixel resolution


Last month Casio and Samsung
introduced 3.2 megapixel
camera phones with optical
zoom and photo flash


There are more cell phone users
in China than people in the
United States (300 million)


For 90% of the world their
“computer” is their cell phone



2004.08.23
-

SLIDE
17


MOBILE MEDIA WORKSHOP PRESENTATION

Campanile Inspiration


2004.08.23
-

SLIDE
18


MOBILE MEDIA WORKSHOP PRESENTATION

Mobile Media Metadata Idea


Leverage the spatio
-
temporal
context

and social
community

of media capture in mobile devices


Gather all automatically available information at the
point of capture (time, spatial location, phone user,
etc.)


Use metadata similarity and media analysis
algorithms to find similar media that has been
annotated before


Take advantage of this previously annotated media to
make educated guesses about the content of the
newly captured media


Interact in a simple and intuitive way with the phone
user to confirm and augment system
-
supplied
metadata for captured media


2004.08.23
-

SLIDE
19


MOBILE MEDIA WORKSHOP PRESENTATION

MMM Demo Video


2004.08.23
-

SLIDE
20


MOBILE MEDIA WORKSHOP PRESENTATION

From Context to Content


Context


When


Date and time


Where


CellID refined to
semantic place


Who


Cellphone user


What


Activity as product of
when, where, and who


Content


When was the photo
taken?


Where is the subject of
the photo?


Who is in the photo?


What are the people
doing?


What objects are in
the photo?


2004.08.23
-

SLIDE
21


MOBILE MEDIA WORKSHOP PRESENTATION

Space


Time


Social Space

SPATIAL

SOCIAL

TEMPORAL


2004.08.23
-

SLIDE
22


MOBILE MEDIA WORKSHOP PRESENTATION

What is “Location”?


2004.08.23
-

SLIDE
23


MOBILE MEDIA WORKSHOP PRESENTATION

Camera Location vs. Subject Location


Camera Location =
Golden Gate Bridge


Subject Location =
Golden Gate Bridge



Camera Location =
Albany Marina


Subject Location =
Golden Gate Bridge


2004.08.23
-

SLIDE
24


MOBILE MEDIA WORKSHOP PRESENTATION

Kodak Picture Spot


2004.08.23
-

SLIDE
25


MOBILE MEDIA WORKSHOP PRESENTATION

Themed Kodak Picture Spot


2004.08.23
-

SLIDE
26


MOBILE MEDIA WORKSHOP PRESENTATION

Location Guesser Concept


Calculate weighted sum of features


Most recently “visited” location


Most “visited” location by me in this CellID
around this time


Most “visited” location by me in this CellID


Most “visited” location by “others” in this
CellID around this time


Most “visited” location by “others” in this
CellID


2004.08.23
-

SLIDE
27


MOBILE MEDIA WORKSHOP PRESENTATION

Mobile Media Metadata Project (MMM)


Develop and test our technology for
mobile media metadata creation, sharing,
and reuse


Develop application scenarios that use this
technology


Study the usage patterns of applications
for mobile media metadata creation,
sharing, and reuse



2004.08.23
-

SLIDE
28


MOBILE MEDIA WORKSHOP PRESENTATION

MMM Fall 2003 Project


Numerous application ideas for mobile media
capture and sharing (at least seven documented
in detail with persona and scenario descriptions
and annotated storyboards)


A metadata framework for describing and
sharing mobile media


A database of annotated captured media


A prototype of one of the student application
scenarios


A write
-
up of the project including student
feedback, assessment of application ideas, and
recommendations for further research and
development



2004.08.23
-

SLIDE
29


MOBILE MEDIA WORKSHOP PRESENTATION

MMM Research Issues


Metadata and media creation, sharing, and
reuse


Ontology design


Vocabulary control


Leveraging spatial, temporal, and social
context to infer media content


Determining similarity


Application ideas and use scenarios


User interfaces


2004.08.23
-

SLIDE
30


MOBILE MEDIA WORKSHOP PRESENTATION

MMM Research Issues


Ontology of photography


Who owns/accesses/changes pictures of me?


Who owns/accesses/changes my pictures?


Who owns/accesses/changes my metadata?


Who owns/accesses/changes metadata about
me and/or my media?


Mobile media application design methods


HCI methodology for mobile media design


Social science methodology for design of
future applications


2004.08.23
-

SLIDE
31


MOBILE MEDIA WORKSHOP PRESENTATION

MMM Initial Application Ideas


Metadata creation sharing, and reuse of media and
metadata from various users connected in space, time,
and affinity


Content
-
based access to mobile media assets enabling


Networked photo albums


Personalized media messages


Personalized media CallerID


Matchmaking services


Ecommerce


Location
-
based and travel applications for using
networked media assets and metadata


Networked travel albums


Networked travel guidebooks


2004.08.23
-

SLIDE
32


MOBILE MEDIA WORKSHOP PRESENTATION

Over 150 MMM Application Ideas


Real time party ratings with pictures


Forgotten historical site cataloguing


Mapping wireless access on campus for
public



Picture/video comparison to map location


Tagging people



"Audio graffiti" audio tour of the world


Game: Spot the terrorist



Amateur photo contest.



Mobile American Idol



Affair detection



Do I sue or not? Online car estimates



Family tree builder/viewer


Human Clock: Pictures of people
showing the time


Human Weather: What are people
wearing in San Francisco?


Reality show that is following someone
around for a night going out and party
that people could watch online



Family tree builder/viewer


Human Clock: Pictures of people
showing the time


Human Weather: What are people
wearing in San Francisco?


Reality show that is following someone
around for a night going out and party
that people could watch online


Translation. Take a picture of something,
and get the word for it in the local
language. Or enter a word in your own
language to get a picture which you can
show to a native


Directions/locations. Illustrated travel
guides, an interactive visual Lonely
Planet



Indie reporting. News coverage via
amateur reporters. The blackout provided
some interesting examples


Some "leveling game" like taking photos
of people wearing gap jeans on the
street, if you get enough, you win a pair,
and go onto the next level


2004.08.23
-

SLIDE
33


MOBILE MEDIA WORKSHOP PRESENTATION

MonkeyBotster


Monkeybotster

aims to
allow you to access
interesting events that
have occurred in
someone's past as well
as contribute things from
the present through two
degrees of social
separation (your friend's
friends can see your
events but not your
friend's friend's friends,
etc.)


2004.08.23
-

SLIDE
34


MOBILE MEDIA WORKSHOP PRESENTATION

LunchMeister


LunchMeister

enables
users to share
information, opinions,
and photos of local
restaurants in real
time, allowing them to
maximize the use of
an often limited lunch
hour


2004.08.23
-

SLIDE
35


MOBILE MEDIA WORKSHOP PRESENTATION

DARE


DARE

will leverage the
camera phones as tools
that allow users to play
games with and against
each other


The main objective of
DARE is to bring
different social
networks together
informally, or as ice
breakers and team
builders for
corporations, school
groups and other such
groups


2004.08.23
-

SLIDE
36


MOBILE MEDIA WORKSHOP PRESENTATION

Recipe Box


Recipe Box

will facilitate
cooking for people who
wish to spend less money
and time making
presentable meals


Users will be able to
select from various
recipes in an annotated
media database on their
mobile phones


Users can add recipes
illustrated with annotated
media to the database
from their mobile phones



2004.08.23
-

SLIDE
37


MOBILE MEDIA WORKSHOP PRESENTATION

HouseBuddies


HouseBuddies

transforms the mobile
phone into a
collaborative apartment
-
hunting device that
enables real
-
time
information sharing


It facilitates a group of
friends in their search for
a place to live by
allowing them to
assemble a list of house
ads from various
sources and collaborate
on investigating their
housing options


2004.08.23
-

SLIDE
38


MOBILE MEDIA WORKSHOP PRESENTATION

Recreation Evaluation Interface


Recreation Evaluation
Interface

enables camera
phone users to see what's
happening where they are
not, and let others in their
community know what's
happening where they are


Using real
-
time data and a
reference database, this
community can connect
with other people with
similar interests to
maximize their local
knowledge, enabling better
decision
-
making about
how and where to spend
one's time


2004.08.23
-

SLIDE
39


MOBILE MEDIA WORKSHOP PRESENTATION

Wishter


Wishter

would manage


Lists of potential gifts for
friends and family


A “wishlist” of items for
oneself


It would allow the user to
annotate a potential gift
with a physical
description, the event for
which the gift is being
considered, and the
location where the item
was seen




























click


2004.08.23
-

SLIDE
40


MOBILE MEDIA WORKSHOP PRESENTATION

Wishter Rapid Development


2004.08.23
-

SLIDE
41


MOBILE MEDIA WORKSHOP PRESENTATION

MMM User Studies


Deployed MMM prototype with 40 graduate
students and 15 researchers for 4 months


Weekly surveys administered to MMM users


Current usage patterns gathered


5 subjects for usability testing with MMM users


User interaction and user motivation addressed


8 subjects for first focus group of MMM users


Current annotation habits discussed


User motivation addressed


7 subjects for second focus group


Current use of photos (capture, annotation, sharing)


2004.08.23
-

SLIDE
42


MOBILE MEDIA WORKSHOP PRESENTATION

User Interaction Findings


Network unpredictability


Slow and intermittent connectivity


Frequently dropped service with little
feedback


User work to that point was often lost



Proposed Solution:

Limit continual network
interaction by creating a full
-
client application
that can use network in background


2004.08.23
-

SLIDE
43


MOBILE MEDIA WORKSHOP PRESENTATION

User Interaction Findings


Mobile UI usability


Hardware buttons were not remapped to
support MMM UI functions


Desktop
-
based prototyping tools and methods
don’t predict usability issues with mobile
devices



Proposed Solution
: Use a prototyping
methodology that better simulates the
specifics of mobile interaction to design user
experiences and test usability


2004.08.23
-

SLIDE
44


MOBILE MEDIA WORKSHOP PRESENTATION

User Interaction Findings


Presentation of faceted hierarchical
structure


Lists of 12
-
15 items should not be exceeded
as selection devices



Proposed Solution
: Create smaller application
specific hierarchies to present to the user,
(which correlate into a broader general
structure on the backend)


2004.08.23
-

SLIDE
45


MOBILE MEDIA WORKSHOP PRESENTATION

Use Patterns and Motivation Findings


Power
-
of
-
Now


Camera phones are always available


Take pictures of ad
-
hoc subjects


Phototaking happens in daily lived context


Currently Not Search
-
Centric


For our users, sharing and browsing are more
important than search and retrieval


Role of the Desktop


A desktop component adds great value to the mobile
application by easing browsing


Challenges in Offloading Media


Non
-
MMM users find it difficult and frustrating to try to
get their photos off their cameraphone


2004.08.23
-

SLIDE
46


MOBILE MEDIA WORKSHOP PRESENTATION

Use Patterns and Motivation Findings


Funnel Effect:
Selective metadata
annotation


Two types of
annotations:


Content: 1
-
2 facets
describing the
photograph (e.g.,
Person, Location)


Comment: a personal
remark about the
photograph, why they
took it, a witty
comment, something
personal to share






PHOTOS TAKEN

PHOTOS SHARED

ANNOTATED

PHOTOS KEPT


2004.08.23
-

SLIDE
47


MOBILE MEDIA WORKSHOP PRESENTATION

Use Patterns and Motivation Findings


Integrate or motivate annotation


Tie annotation to some other activity the user
already does in a seamless way


Sharing


Storytelling


Motivate users to annotate


Demonstrate clear benefit of later reuse


Make annotation intrinsically fun (e.g., image
annotation games)



2004.08.23
-

SLIDE
48


MOBILE MEDIA WORKSHOP PRESENTATION

Application Design Findings


Input
-
output application design


Combination of simple small applications into more
powerful applications


Network effects


Power of many users sharing media and metadata


Media and coordination/collaboration


Using media to coordinate and collaborate among
groups


Mobile media and games


Using media to document games


Using games to make media and metadata


2004.08.23
-

SLIDE
49


MOBILE MEDIA WORKSHOP PRESENTATION

Nokia Platform Issues


Hardware


Keypad layout


Camera


Development environment


Access to files


Access to camera


Access to contextual metadata (time, CellID, username)


Access to contacts


Browser


Nokia web browser (caching)


Prototyping tools


Emulator


Wizard of Oz



2004.08.23
-

SLIDE
50


MOBILE MEDIA WORKSHOP PRESENTATION

Research Projects


Media Streams


A framework for creating metadata throughout the media
production cycle to enable media reuse


Active Capture


Automates direction and cinematography using real
-
time audio
-
video analysis in an interactive control loop to create reusable
media assets


Adaptive Media


Uses adaptive media templates and automatic editing functions
to mass customize and personalize media


Mobile Media Metadata


Leverages the spatio
-
temporal context and social community of
media capture to automate metadata creation for mobile media


Social Uses of Personal Media


Analysis of social uses of media to predict future uses and shape
the design of next
-
generation personal media devices and
applications



2004.08.23
-

SLIDE
51


MOBILE MEDIA WORKSHOP PRESENTATION

Social Uses of Personal Photos


Looking not just at
what

people do with digital
imaging technology, but
why

they do it


Goals


Identify social uses of photography to predict
resistances and affordances of next generation
mobile media devices and applications


Methods


Situated video interviews


Review of online photo sites


Sociotechnological prototyping (magic thing,
technology probes)



2004.08.23
-

SLIDE
52


MOBILE MEDIA WORKSHOP PRESENTATION

Preliminary Findings


Social uses of personal photos


Memory


Creating and maintaining relationships


Self
-
expression



Media and resistance


Materiality


Orality


Storytelling


2004.08.23
-

SLIDE
53


MOBILE MEDIA WORKSHOP PRESENTATION

From MMM
-
1 To MMM
-
2


MMM
-
1 asked


“What did I just take a picture of?”


MMM
-
2 adds


“Who do I want to share this picture with?”

Community

Context

Content

Community


2004.08.23
-

SLIDE
54


MOBILE MEDIA WORKSHOP PRESENTATION

MMM
-
2 Goals


Integrated photo
sharing

and mobile
media metadata application


Metadata as by product of sharing


Metadata enables new types of sharing


Advanced spatio
-
temporal
-
social
inferencing


Integrated media analysis functions


RDF backend for ontology and metadata
sharing and reuse


Connected handset and web applications


2004.08.23
-

SLIDE
55


MOBILE MEDIA WORKSHOP PRESENTATION

Sharing


Metadata


From sharing to metadata


A birdwatcher takes a photo in a bird
sanctuary and sends it to her birdwatching
group


What is the photo of?



From metadata to sharing


A parent takes a photo of his child on the
child’s birthday


Who does he share it with?


2004.08.23
-

SLIDE
56


MOBILE MEDIA WORKSHOP PRESENTATION

MMM
-
2 Use Scenarios


Share photo


Rank share lists


Add/delete members of share list


Create new share list


Annotate photo


Annotate photo on handset


Annotate photo on web


Annotate photo while sharing on handset


Annotate photo while sharing on web


2004.08.23
-

SLIDE
57


MOBILE MEDIA WORKSHOP PRESENTATION

Number

of

Sources

100
M

1

100
M

Photos

Per

Source

100
K

Scaling Up Photo Sharing

100
K


2004.08.23
-

SLIDE
58


MOBILE MEDIA WORKSHOP PRESENTATION

MMM
-
2 Prototype


Phase 1: MMM
-
2 Prototype Design


In Process: Spring 2004


Phase 2: MMM
-
2 Prototype Development


Summer 2004


Phase 3: MMM
-
2 Prototype Deployment


Fall 2004


Phase 4: MMM
-
2 Prototype Evaluation


Fall 2004


Spring 2005


2004.08.23
-

SLIDE
59


MOBILE MEDIA WORKSHOP PRESENTATION

MMM Publications


CHI 2004


Anita Wilhelm, Yuri Takhteyev, Risto Sarvas, Nancy Van House, Marc
Davis. "Photo Annotation on a Camera Phone." In: Proceedings of the
Conference on Human Factors in Computing Systems (CHI 2004) in
Vienna, Austria. ACM Press, 2004.


MobiSys 2004


Risto Sarvas, Erick Herrarte, Anita Wilhelm, and Marc Davis. “Metadata
Creation System for Mobile Images.” In: Proceedings of the Second
International Conference on Mobile Systems, Applications, and Services
(MobiSys2004) in Boston, Massachusetts. ACM Press, 2004.


Marc Davis, Nathan Good, and Risto Sarvas. "From Context to Content:
Leveraging Context for Mobile Media Metadata (Workshop Paper)."
Presented At:
MobiSys 2004 Workshop on Context Awareness at the
Second International Conference on Mobile Systems, Applications, and
Services in Boston, Massachusetts
, 2004.


Marc Davis. "Mobile Media Metadata (Video)." Presented At:
Second
International Conference on Mobile Systems, Applications, and Services
(MobiSys 2004) in Boston, Massachusetts
, 2004.


2004.08.23
-

SLIDE
60


MOBILE MEDIA WORKSHOP PRESENTATION

MMM Publications


ICME 2004


Marc Davis and Risto Sarvas. “Mobile Media Metadata for Mobile
Imaging.” In:
Proceedings of IEEE International Conference on
Multimedia and Expo (ICME 2004) Special Session on Mobile Imaging in
Taipei, Taiwan
, IEEE Computer Society Press, 2004.


ACM Multimedia 2004


Marc Davis, Simon King, Nathan Good, and Risto Sarvas. "From Context
to Content: Leveraging Context to Infer Media Metadata." In:
Proceedings
of 12th Annual ACM International Conference on Multimedia (MM 2004)
Brave New Topics Session on "From Context to Content: Leveraging
Contextual Metadata to Infer Multimedia Content" in New York, New
York
, ACM Press, Forthcoming 2004.


Marc Davis.
"Mobile Media Metadata: Metadata Creation System for
Mobile Images (Video)." In:
Video

Proceedings of 12th Annual ACM
International Conference on Multimedia in New York, New York
, ACM
Press, Forthcoming 2004.


Marc Davis.
"Mobile Media Metadata: Metadata Creation System for
Mobile Images (Video Description)." In:
Video

Proceedings of 12th
Annual ACM International Conference on Multimedia in New York, New
York
, ACM Press, Forthcoming 2004.


2004.08.23
-

SLIDE
61


MOBILE MEDIA WORKSHOP PRESENTATION

http://garage.sims.berkeley.edu