Parker Dunlap

rodscarletSoftware and s/w Development

Dec 14, 2013 (4 years and 5 months ago)


Parker Dunlap


Semantic image analysis techniques can
automatically detect high level content of

Lack of intuitive visualization and analysis

Allow users to effectively browse/search in
large databases

Allow analysts to evaluate their annotation
process through interactive visual exploration

Target search

User knows exactly what they want, a precise image

Search by association

Find interesting things related to certain image

Category search

Retrieve images that are representative of a certain

Semantic contents of images are more useful
for image exploration than low level features
but in most large scale image collections
(internet) semantics are usually not described

This has given rise to techniques that enable
automatic annotation of images according to
their semantic concepts

Contains semantic image classification
process that automatically annotates large
image collections

Contains coordinated visualization techniques
that allow interactive exploration

Contains visualization techniques that allow
analysts to evaluate and monitor annotation

Annotation Engine

Image Browsing Interface

Visual Image Analysis

Abstract Image content by detecting
underlying salient objects (distinguishable

Associate salient objects with corresponding
semantic objects according to their
perceptual properties

Keywords for semantic objects are used to
annotate the image

Highlighted regions are salient objects detected and associated

semantic object “sand field”

Goal is to bridge the gap between low
visual features and high
level semantic

Annotation engine has set of predefined
salient objects and functions to detect them
from images

Uses techniques like image segmentation and SVM

Annotation engine assigns a semantic
concept to the data based on semantic

Sand, Field, Water →

Flowers, Trees → Garden

Image overview using MDS

Use the annotations to calculate distance matrix
and input into MDS algorithm

Distance between each pair of images in the content

Algorithm outputs a 2D position for each image
based on similarity with other images

Maps image miniatures onto the screen based
on their content similarities

Similar images placed closer to each other

Goal of MDS is to map some high dimensional
data into lower dimension (in our case 2D)

To learn more about MDS see
MDS Overview

Visually represents the contents of the entire
collection of images

Correlations of different contents and
detailed annotations are displayed

Interactively exploring large datasets with
real time response (high dimensionality)

Block of pixels to represent images contents

Each image is mapped to a pixel whose color
indicates if the image contains/doesn’t
contain the content for that block

Pixel representing the same image is the
same for all blocks

Allows us to observe content of image
collection by scanning labels of the blocks

Can see correlations among the contents

Can also select images to see them
highlighted in the view

Position of the blocks are determined by
similarity with neighboring contents

Pixels are generally created in a spiral
arrangement starting from the center and
moving out

Pixel order can greatly effect the looks of


To increase scalability, interface users
miniature versions of images

High res original pictures would increase load times

Load image miniatures as textures objects in

Allows all interactions to be done in real time

To reduce clutter in the MDS overview, the
system provides many interactions


Dynamic Scaling



Showing Original Image




Randomizing order of all images allows each frame
to have an equal probability of being visible

User can also explicitly bring certain image to the
front by selecting it

Dynamic Scaling

Interactively reduce image miniature size to reduce
overlap or increase image size to examine detail


Manually change position of individual image by
dragging and dropping


Enlarge size of certain image(s) while retaining size
of all others

Showing Original Image

Actual image (instead of scaled down image used
by OpenGL) opens at full resolution in new window

Only loaded when requested to save space/time


Zoom in/out and pan left/right

Can use multiple techniques at once to
achieve some goal

Use Dynamic Scaling with zooming in to examine
local details with less clutter


Interactively select a sample image to see similar
images in display

Can change similarity threshold via a slider to
increase/decrease number of results


Images can be sorted by concepts or similarity to
selected image

Inspired by rainfall animation

Correlations between image of interest and
other images are modeled through an

Focus image is on the bottom (ground) and
the other images fall to the ground (rain) at
accelerations related to their similarity

Search for images with/without certain

Reduce a selected subset by requiring images
must/not contain certain content

Increase selected subset by adding new

All these functions done by clicking on
images while holding certain function key

Offers many similar interactions as MDS as

Each image has its visual representations in
both MDS and


Selected images are highlighted in both views

Can use appropriate view as needed

MDS to select image based on relationship to
sample image


to select image based on content

Common strategy is to start from

switch to MDS after number of images has
been greatly reduced

We can use the MDS and

views to see
how well the annotations of images
correspond to their actual content

Select “red
flower” images from

view and
verify using MDS view to see if the images
are actually red flowers

If automatic annotation makes a mistake,
user can manually annotate image to fix it


display also shows the reliability of the
annotation by surrounding it with a colored

Green is safe to use, Yellow means lower reliability

Reliability measure can be determined from
annotation process or manually set up by

Comparison of SIB to the sequential
thumbnail view from Microsoft Explorer

Modes used in Microsoft Explorer

Random Explorer

images are randomly sorted

Sorted Explorer

images are sorted according to
semantic concepts generated by the classification

10 participants from varying fields

Each subject used both Sorted Explorer and

Random Explorer was only tested on 3 participants
since expected results were so low

Participants given 3 tasks to perform on 2
data sets

180 second timeout window

Presented with a particular image and asked
to search for it from the 1100 images in the
data set

Asked to find images containing particular
features (sand, water, sky, etc…)

Asked to approximate what proportion of
the images in the dataset contained
particular contents (% that contain

Random Explorer

2/9 trials failed

81 seconds was average time with 29 seconds
standard deviation

Sorted Explorer

2/30 trials failed

29 seconds was average time with 20 seconds
standard deviation


6/30 trials failed

45 seconds was average time with 26 seconds
standard deviation

Failure in SIB was due to inaccuracy in the
annotation process

SIB tended to be slower than Sorted Explorer
because content names could be confusing

This advantage will decrease as the data set grows
because Explorer provides no overview model

Task 2 had similar results to Task 1

Task 3 was where SIB became dominant

Positive feedback for SIB

Enjoyed Search by content feature the most

Enjoyed MDS overview over Windows explorer
to see entire collection of images at once

Suggested side
side views, example
image next to blocks in


Semantic Image Browser was introduced that
attempts to bride information visualization
with automatic image annotation

MDS image layout that groups images based
on semantic similarities


content display to represent large image

Semantic Image Browser: Bridging
Information Visualization
with Automated
Intelligent Image

Value and Relation Display for Interactive
Exploration of High
Dimensional Datasets

MDS Overview