Visual Information Retrieval - NTHU MAKE LAB

builderanthologyAI and Robotics

Oct 19, 2013 (3 years and 8 months ago)

89 views

Visual Information
Retrieval

Chapter 1


Introduction

Alberto Del Bimbo

Dipartimento di Sistemi e Informatica

Universita di Firenze

Firenze,
Italy


Visual Information Retrieval


Information retrieval, image/video analysis
and processing, pattern recognition and
computer vision, visual data modeling and
representation, multimedia database
organization, multidimensional indexing,
psychological modeling of user behavior,
man
-
machine interaction and data
visualization

Visual Information Retrieval


Types of associated information


content
-
independent metadata (CIM)


format, author's name, date


content
-
dependent metadata (CDepM)


low
-
level features concerned with perceptual facts:
color, texture, shape, spatial relationship, motion


content
-
descriptive metadata (CDesM)


high
-
level content semantics: cloud, good weather,
白雲蒼狗

Visual Information Retrieval


First
-
generation visual information retrieval
systems


CIM by alphanumeric strings, CDepM and
CDesM by keywords or scripts

Visual Information Retrieval


find images of paintings by Chagall with a blue
background


Select IMAGE# from PAINTINGS where
PAINTER = "Chagall" and BACKGROUND =
"blue"


find images of paintings by Chagall with a girl
in red dress and a blue background


full text retrieval


Visual Information Retrieval


find images of paintings depicting similar
figures in similar positions as in
收割景緻


it is difficult for text to capture the perceptual
saliency of some visual features


text is not well suited for modeling perceptual
similarity


perception is mainly subjective, so is its text
descriptions

Visual Information Retrieval


New
-
generation visual information retrieval
systems


retrieval not only by concepts but also by
perception of visual contents


objective measurements of visual contents and
appropriate similarity models


automatically extract features from raw data by
image processing, pattern recognition, speech
analysis and computer vision techniques

Visual Information Retrieval

Visual Information Retrieval


Image retrieval


by perceptual features


for each image in the database, a set of features
(model parameters) are precomputed


to query the image database


express the query through visual examples

»
authored by the user

»
extracted from image samples


select features and ranges of features


choose a similarity measure


compute similarity degrees, ranking, relevance
feedback


Visual Information Retrieval


system architecture


extraction of perceptual features (CDepM)


extraction of high
-
level semantics (CDesM) from
low
-
level features


manual annotation of CIM and CDesM


index structure


graphical query tool


retrieval engine


visualization tool


relevance feedback mechanism

Visual Information Retrieval


Video retrieval


special characteristics


frames are linked together using editing effects


color, texture, shape and position (camera or object)
are changed in multiple frames


richer semantics


different types of video

Visual Information Retrieval


by structure


Figure 1.4


frame: basic unit of information


shot: elementary segment of video with perceptual
continuity


clip: set of frames with some semantic meaning


scene: consecutive shots with simultaneous space,
time and action


episode: specific sequence of shot types such as a
news episode

Visual Information Retrieval


by content


perceptual properties, motion and type of an object


situations between objects


motion of camera


semantics of shots by color
-

or motion
-
induced
sensations


semantics of scenes


stories


audio properties: dialogue, music or storytelling


textual information: caption or text recognized from
video

Visual Information Retrieval


system architecture


extraction of shots and the associated semantics,
key
-
frames or mosaics


extraction of scenes and stories


manual annotation tool


browsing/visualization tool


video summarization


graphical query tool


index structure


retrieval engine

Visual Information Retrieval


3
D image and video retrieval


WWW visual information searching


efficiency has to be emphasized due to limited
network bandwidth


operate in compressed domain


visual summarization


visualization at different levels of resolution

Visual Information Retrieval


Research directions


tools for automatic extraction of low
-
level
features


tools for automatic extraction of high
-
level
semantics


models for representing visual content


effective indexing


effective database models

Visual Information Retrieval


visual interfaces


allow querying and browsing


allow querying by text and visual information


similarity models


fit human similarity judgement


psychological similarity models


Web search


3
D image and video retrieval