Parker Dunlap

rodscarletΛογισμικό & κατασκευή λογ/κού

14 Δεκ 2013 (πριν από 3 χρόνια και 10 μήνες)

62 εμφανίσεις

Parker Dunlap

11/15/2013


Semantic image analysis techniques can
automatically detect high level content of
images


Lack of intuitive visualization and analysis
techniques



Allow users to effectively browse/search in
large databases


Allow analysts to evaluate their annotation
process through interactive visual exploration


Target search


User knows exactly what they want, a precise image


Search by association


Find interesting things related to certain image


Category search


Retrieve images that are representative of a certain
class


Semantic contents of images are more useful
for image exploration than low level features
but in most large scale image collections
(internet) semantics are usually not described


This has given rise to techniques that enable
automatic annotation of images according to
their semantic concepts


Contains semantic image classification
process that automatically annotates large
image collections


Contains coordinated visualization techniques
that allow interactive exploration


Contains visualization techniques that allow
analysts to evaluate and monitor annotation
process

1.
Annotation Engine

2.
Image Browsing Interface

3.
Visual Image Analysis


Abstract Image content by detecting
underlying salient objects (distinguishable
regions)


Associate salient objects with corresponding
semantic objects according to their
perceptual properties


Keywords for semantic objects are used to
annotate the image

Highlighted regions are salient objects detected and associated
with

semantic object “sand field”


Goal is to bridge the gap between low
-
level
visual features and high
-
level semantic
concepts


Annotation engine has set of predefined
salient objects and functions to detect them
from images


Uses techniques like image segmentation and SVM
classifiers



Annotation engine assigns a semantic
concept to the data based on semantic
content


Sand, Field, Water →
Seaworld


Flowers, Trees → Garden


Image overview using MDS


Use the annotations to calculate distance matrix
and input into MDS algorithm


Distance between each pair of images in the content
space


Algorithm outputs a 2D position for each image
based on similarity with other images


Maps image miniatures onto the screen based
on their content similarities


Similar images placed closer to each other


Goal of MDS is to map some high dimensional
data into lower dimension (in our case 2D)


To learn more about MDS see
MDS Overview


Visually represents the contents of the entire
collection of images


Correlations of different contents and
detailed annotations are displayed


Interactively exploring large datasets with
real time response (high dimensionality)



Block of pixels to represent images contents


Each image is mapped to a pixel whose color
indicates if the image contains/doesn’t
contain the content for that block


Pixel representing the same image is the
same for all blocks


Allows us to observe content of image
collection by scanning labels of the blocks


Can see correlations among the contents


Can also select images to see them
highlighted in the view


Position of the blocks are determined by
similarity with neighboring contents


Pixels are generally created in a spiral
arrangement starting from the center and
moving out


Pixel order can greatly effect the looks of
VaR

view


To increase scalability, interface users
miniature versions of images


High res original pictures would increase load times


Load image miniatures as textures objects in
OpenGL


Allows all interactions to be done in real time


To reduce clutter in the MDS overview, the
system provides many interactions


Reordering


Dynamic Scaling


Relocation


Distortion


Showing Original Image


Zoom


Pan


Reordering


Randomizing order of all images allows each frame
to have an equal probability of being visible


User can also explicitly bring certain image to the
front by selecting it


Dynamic Scaling


Interactively reduce image miniature size to reduce
overlap or increase image size to examine detail


Relocation


Manually change position of individual image by
dragging and dropping


Distortion


Enlarge size of certain image(s) while retaining size
of all others


Showing Original Image


Actual image (instead of scaled down image used
by OpenGL) opens at full resolution in new window


Only loaded when requested to save space/time


Zoom/Pan


Zoom in/out and pan left/right


Can use multiple techniques at once to
achieve some goal


Use Dynamic Scaling with zooming in to examine
local details with less clutter


Selection


Interactively select a sample image to see similar
images in display


Can change similarity threshold via a slider to
increase/decrease number of results


Sorting


Images can be sorted by concepts or similarity to
selected image


Inspired by rainfall animation


Correlations between image of interest and
other images are modeled through an
animation


Focus image is on the bottom (ground) and
the other images fall to the ground (rain) at
accelerations related to their similarity


Search for images with/without certain
content


Reduce a selected subset by requiring images
must/not contain certain content


Increase selected subset by adding new
images


All these functions done by clicking on
images while holding certain function key


Offers many similar interactions as MDS as
well


Each image has its visual representations in
both MDS and
VaR

views


Selected images are highlighted in both views


Can use appropriate view as needed


MDS to select image based on relationship to
sample image


VaR

to select image based on content


Common strategy is to start from
VaR

and
switch to MDS after number of images has
been greatly reduced


We can use the MDS and
VaR

views to see
how well the annotations of images
correspond to their actual content


Select “red
-
flower” images from
VaR

view and
verify using MDS view to see if the images
are actually red flowers


If automatic annotation makes a mistake,
user can manually annotate image to fix it


VaR

display also shows the reliability of the
annotation by surrounding it with a colored
frame


Green is safe to use, Yellow means lower reliability
measure


Reliability measure can be determined from
annotation process or manually set up by
analysts


Comparison of SIB to the sequential
thumbnail view from Microsoft Explorer


Modes used in Microsoft Explorer


Random Explorer


images are randomly sorted


Sorted Explorer


images are sorted according to
semantic concepts generated by the classification
process


10 participants from varying fields


Each subject used both Sorted Explorer and
SIB


Random Explorer was only tested on 3 participants
since expected results were so low


Participants given 3 tasks to perform on 2
data sets


180 second timeout window

1.
Presented with a particular image and asked
to search for it from the 1100 images in the
data set

2.
Asked to find images containing particular
features (sand, water, sky, etc…)

3.
Asked to approximate what proportion of
the images in the dataset contained
particular contents (% that contain
mountains)


Random Explorer


2/9 trials failed


81 seconds was average time with 29 seconds
standard deviation


Sorted Explorer


2/30 trials failed


29 seconds was average time with 20 seconds
standard deviation


SIB


6/30 trials failed


45 seconds was average time with 26 seconds
standard deviation


Failure in SIB was due to inaccuracy in the
annotation process


SIB tended to be slower than Sorted Explorer
because content names could be confusing


This advantage will decrease as the data set grows
because Explorer provides no overview model


Task 2 had similar results to Task 1


Task 3 was where SIB became dominant


Positive feedback for SIB


Enjoyed Search by content feature the most


Enjoyed MDS overview over Windows explorer
to see entire collection of images at once


Suggested side
-
by
-
side views, example
image next to blocks in
VaR

view


Semantic Image Browser was introduced that
attempts to bride information visualization
with automatic image annotation


MDS image layout that groups images based
on semantic similarities


VaR

content display to represent large image
collections


Semantic Image Browser: Bridging
Information Visualization
with Automated
Intelligent Image
Analysis


Value and Relation Display for Interactive
Exploration of High
Dimensional Datasets


MDS Overview