Final Progress Report 2011x

hundredcarriageΛογισμικό & κατασκευή λογ/κού

3 Νοε 2013 (πριν από 3 χρόνια και 7 μήνες)

70 εμφανίσεις

i


IT
-
SNAPS

Java

A Client
-
Server Implementation


Benjamin Bryant

Applied Vision Lab

Texas Tech University


ii


A
BSTRACT

This paper provides

the

technical and operational information of
a multi
-
user
, multi
-
GPU

implementation of IT
-
SNAPS in Java. The system uses
Java

for a small client application,

and Java, C,
and compiled Matlab code for the Server side. NVIDIA’s
Compute Unified

Device Architecture

(CUDA)

is also

used to reduce processing time

and increase the number of potential users per server.
This
report

discusses

the hardware, software, and concepts used in its creation.



1


1.

I
NTRODUCTION

IT
-
SNAPS
[
1
]

has been
expanded to an online, Java
-
based program

to make it accessible to
i
ndividual users
who do not
have
Matlab or
a GPU
1

card
.
In this way
, IT
-
SNAPS

can be

hosted on

a
remote server

that

has a GPU

and the

free Matlab Runti
me. The server

p
erform
s

the bulk of the
computations

and host
s

the client side application for deployment.
U
sers
are

no
t

required to install
anything on their local machine, assuming they already have a web browser and Java’s runtime. Clients
only need t
o open a webpage
,

which will automatical
ly launch IT
-
SNAPS ready to use. The system
requirements for both client and server can be
found

in
Appendix A: System Requirements
.

The server running IT
-
SNAPS
is

capable of managing multiple clients
, as well as multiple GPUs
.
After a user has selected and uploaded an image to be segm
ented, only
the cursor location

and the path to
be displayed
are

transmitted between the host and client
(s)
. The client application

s only requirement
is
to

display the
points and
the corresponding
path on the image

in its GUI
2
. This

will

allow feeble
computer syste
ms to still have

access to
IT
-
SNAPS
, since the client application has little to proces
s
.

2.

B
ASE
D
ESIG
N

IT
-
SNAPS generates a segmentation
path

in a similar
method to
I
ntelligent
S
cissors
[
2
]
.This path
connects

a
“seed point”

and a “free point”

using
Dijkstra’s
Algorithm
[
3
]
.


In Intelligent Scissors,
Dijkstra’s

algorithm

uses a cost of transition matrix based o
nly

on

one feature plane,
i.e.,
the gra
y level
image. IT
-
SNAPS works on a higher level by using a
dynamic
line
ar combination of several co
st
matrices
:

one for each feature

image
[
1
]
.
F
eature images can be anything from the gray plane

to the color
planes,
or any other feature planes the us
er has generated. IT
-
SNAPS then
adjusts

this lin
ear combination
of cost matric
es

to

best follow
the

path the

user appears to be
segmenting

for a more accurate and
rapid

segmentation experience.

2.1.

Cos
t Matrix

In this implementation, as in Intelligent Scissors, the cost matrix defines the “cost” of tran
sition
between one center pixel

and the 8
adjacent

connected

pixels
.
The cost matrix

is created

using the
gradient magnitude, gradient direction, and th
e Laplacian Zero
-
Crossings
,

as defined in
[
2
]
.

For an image
of size MxN
, the

full

cost matrix would require
MxNx8x4

bytes of memory as float values. To create the
super
3

cost matrix, there would need to
be a cost matrix of this size for each feature plane, consuming
an
inordinate amount of
memory.
For instance,
if a user wanted to act on a 9 Megapixel (3000x3000 pixel)
image, using 7 feature plans would
require

over 2 Gigabytes of free memory.

To counter this limitation, a dynamically created cost matrix is used. Only the cost matrix within
the vicinity of the cursor is created during segmentation,
which is then
used to update
Dijkstra’s

path

before being
released. This method trades memory c
onsumption for proces
sor loading during
segmentation
,

and requires a little less initial processing time. In order to create the cost matrix
dynamically, parameters such as the maxi
mum/minimum gradient magnitudes and other thresholds must
be pre
-
calculate
d. This is one of the many reasons CUDA is used

to

reduce the

preprocessing

time
.

2.2.

Dijkstra’s
Algorithm

Since the cost matrix in this implementation is dynamically created,
Dijkstra’s
Algorithm
[
3
]

must also act dynamically. Normally, the distance to every possible point is initialized to infinity

and
the
cost of
propagating

from the starting point to every other point is
calculated
iteratively
, resulting

in a



1

GPU: Graphics Processing Unit

2

GUI:
Graphical
U
ser
I
nterface

3

Cost matrix that is the linear combination of all other cost matrices.

2


distance map
,

as shown in

Figure
1
(b)
. Here
,
r
ed
represents

high
and blue represents low
distance

values
,

respectively
.

(
a)



(
b)


(
c)


Input Image

Original Distance

Dynamic Distance

Figure
1
:
(a) Input image
. (b)
R
esults from generating th
e distance to every pixel.

(c)
D
ynamically generated distance
matrix.

In this implementation, only the
cost matrix for the
region directly around the c
urrent cursor
location is known

and
,

therefore
,

Dijkstra’s
Algorithm is only updated with this information
; see

Figure
1
(c)
. The iterations are performed

on the CPU

in a spiraling pattern from the center of the cu
rrent cursor
location
. This method has the additional advantage of allowing contours around regions where the
distance matrix is still infinity; therefore
,

a p
ath cannot be created that cuts through the region
.

Since the
user should be dragging the curs
or
over, or
near
,

the boundary of interest, only the area around the
boundary
is

available for segmentation.

Figure
2

show
s

the track of a user’s cursor in red as th
e user
segments
an

object. The yellow boxes represent the areas where
Dijkstra’s
Algorithm has been
performed

and
,

therefore
,

the

area enclosed

by the boxes

contains the only potential paths.
Without this
feature, the calculated path could undesirably cu
t through the object.


(
a)

(
b)

Cursor Path

Region of Evaluated
Dijkstra’s
Alg潲it桭

Figure
2
:
Dynamic Dijkstra’s Algorithm
. If a user traces the path shown in red in (a), then only the region in yellow in (b)
will be used for segmentation.

2.3.

Feature Images


The
feature images
that
are provided with IT
-
SNAPS Java

are
all generated in CUDA.

RGB
features are the red, green

and

blue channels of the image, and LAB features are the
lightness
, A, and B
channels of the image in the LAB color space. Each of the LA
B and RGB planes are treated as a

feature
image
.

3


2.3.1.

Co
-
Occurrence

Features

Co
-
occurrence matrices are two dimensi
onal histogr
ams of co
-
occurring values

with a given
offset

[
4
]
. A co
-
occurrence matrix

generated from an image with n
-
bit resolution
is

2
n
by 2
n
,

tabulating the
occurrences of pixel values relative to other pixel values
.
To generate a feature image

from a co
-
occurrence matrix
, a window around each pixel
is
first
used to generate
the co
-
occurrence matrix. T
hen
,

properties
or features
of that matrix

are
computed

to
represent

the
center
pixel in the feature space
.
In this
implementation,

these windows
are

relatively
small

in size
,
giving

rise to
a
very sparse
co
-
occurrence

matri
x

for each pixel.
H
ence, to speed up the process by

avoiding the unneces
sary calculation of the full
matrix,
new formulas for direct image to co
-
occurrence matrix features were derived
.
This is illustrated
below for the contrast feature.















{










































(1)

Given the above definition of the co
-
occurrence matrix, c
ontrast

is
calculated using

the following
expression
[
4
]
:




|



|












(2)


In our work, t
his
is

calculated directly from the image by setting











,










and














. This is applicable since the original summation only includes areas where

























, and the
expression

is linear with respect to C
.

Hence, g
iven


(
p,q
)

a
s the current

center

pixel


(
w
)

a
s the
variable window size around the current
center
pixel


(

x,


y) a
s the co
-
occurrence pixel offset









































































(














)







the modified expression used t
o compute the contrast can be
written as






|



















|







(3)


Using this same
idea
, formulas for Correlation (
Eqn.
4) and Homogeneity (
Eqn.
5) can be derived from
the formulas in

[
4
]

as







(














)
(


























)


























(4)







|


















|





(5)



4


This feature generation has been implemented in CUDA and uses the average

feature

value for
the co
-
occurrence
pixel
offsets

{ (
-
1,0) , (1,0) , (0,1) , (0,
-
1) }
. Whenever this feature is chosen, all three
features

(
contrast, correlation, and homogeneity
)

are generated and used for segmentation.

2.3.2.

Gabor

Features

Gabor filters are band
-
pass filters for
spatial
frequen
cies
[
5
]
. These filters vary in frequency,
orientation, and bandwidth, and
can

be used to separate areas of different textures based on their
frequency responses. The general formulas for Gabor filter mask generation can be seen
below. Filtering
is achieved
by convolving the target image with two masks (real and imaginary

parts
:
Figure
1
)

and
taking the magnitude of the
response
. A
post
-
processing Gaussian

lo
w
-
pass

filter is
also

used to blend
regions distorted by the phase variations within the image. This feature generati
on has been implemented
in CUDA

and uses 3 central wavelengths (

) and four orientations (

) to generate 12 feature images for
segmentatio
n.

An example of these features can be seen in

Figure
4
.

These images show h
ow the Gabor
filter can be used to separate different regions with varying frequency components.


7








2
2
ln
2
2
ln
cos
sin
'
sin
cos
'


















bandwidth
y
x
y
y
x
x



'
2
2
'
'
2
2
2
)
,
(
x
j
y
x
e
e
y
x
g
















Figure
3
:
Example of
Gabor Filter Masks




0
20
40
0
5
10
15
20
25
30
35
-0.5
0
0.5
1
0
20
40
0
5
10
15
20
25
30
35
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
5



Input Image


wavelengths (

)‽″

潲ie湴ati潮猠(

)‽‴


w慶敬敮et桳


)‽‶

潲ie湴ati潮猠(

)‽‴


w慶敬敮et桳


)‽‱㈠

潲ie湴ati潮猠(

)‽‴


w慶敬敮et桳


)‽″

潲ie湴ati潮猠(

)‽‰



f
w慶敬敮et桳h(

)‽㘠

潲ie湴ati潮猠(

)‽‰


w慶敬敮et桳


)‽‱㈠

潲ie湴ati潮猠(

)‽‰


Figure
4
:
Example of
Gabor Features
.

T
op image is the test input with low frequencies in the center

that increase
to high frequencies at the
outer
edge
s
.
O
ther
images are feature images generated
using the labeled parameters.

6


3.

W
EB
I
NTERFACE

This application has been created for deployment on
Oracle
’s

GlassFish Server

and is distributed
in 2 parts.
The server side application uses the Java API for XML Web Services (JAX
-
WS) under the
Metro web service stack. The client applic
ation consumes th
is web service

and is

deployed from the
server

to be executed

on the user’s machine.

Multiple
clients can use the
server simultaneously and all
data is transmitted using the http protocol
over port 80
to maximize the potential of client access
.
Installat
ion instructions for the server are provided in
Appendix B: Installation
.

While

the client a
pplication runs entirely in Java, the server side
uses Java,
C, and compiled
Matlab libraries

through the Java Native Interface
. Java
is
, in general,

slower than

C and is not supported
by

NVIDIA’s

CUDA
[
6
]
.

This is
the reason
a
lmost all of the complex computations are preformed in C

on the GPU

with the exception of image compression/decompression

and
Dijkstra’s
Algorithm
.
Dijkstra’s
Algorithm

is preformed on the CPU, since it is too iterative of a pro
cess to benefit from

parallel
processing in CUDA
.


3.1.

Startup

To begin using IT
-
SNAPS Java, the user must simply open the
associated
webpage
. This page
uses JNLP
4

to download and launch the client applet with permissions to access the local file system.
If
the user does not have Java,

JNLP will
automatically
prompt

them to

get it

for free
.
This client applet
uses a Java Swing API
5
-
based

GUI to interact with

the user, which for ease of use is
launch
ed

in a
separate window from the web browser

(
Figure
5
)
.


Figure
5
: Startup Window


3.2.

Initialization


When the

IT
-
SNAPS Java

client first starts
, the applet requests a slot
on

the
server
, which will be
assigned or rejected

if no slots are free.

This slot contains place holders (pointers) f
or the user’s data in
both Java

and C
, and binds the user to the GPU with the most available memory
. A single thread must
also be assigned to the user f
or executing

the

native C

code
.
With

this thread
, there exists

a
n

inherited
CUDA context

with
its own address space.

Having this

preassigned
single thread, one per user, allows
this

application to

not only use

the GPU, but
store data on
it
. Otherwise

Java’s
natural

multithrea
d
ing

would cause segmentation faults as random threads attempt to access stored data

in the native code
.

This
also allows the application to compute feature images on the GPU using
various DLLs
6
.

Then
, i
nstead of
having to
copy

feature images back and forth between the CPU and GPU multiple times, having one



4

JNLP:
Java Network Launching Protocol

5

API:
Application
P
rogramming
I
nterface

6

DLL:
Dynamic
-
L
ink
L
ibrary

7


single thread per user allows feature images generated on the GPU to be passed as a GPU memory pointer
into IT
-
SNAPS for segmentation.


Now that the user has a place to wor
k on the server, they need to
l
oad an image

for segmentation
by clicking on “Load Image”
. After selection
, the

image

is

asynchronously

uploaded to the server while
the user selects the features they wish to use for segmentation. The image is
uploaded in
its original
format and

streamed directly into the server’s memory

as a compress
ed binary attachment using

MTOM
7
.


3.3.

Select Features

Once an image has been uploaded, a window will spawn allowing the user to select features for
segmentation

(
Figure
6
)
. While s
ome predefined features are available
,

the user
is allowed to
upload
custom feature images. If not done so already, the selected image will be up
loaded when this window is
displayed. Once fully uploaded to the server, the user will be able to submit their selections, preparing
the server for segmentation.


Figure
6
: Feature Selection Window

O
nce the features for segmen
tation h
ave been selected and uploaded, the server generates any
necessary
feature

images,

store
s them on the
GPU
, and initializes
. Initialization consists of calculating
Laplacian
zero
-
cross
ing

thresholds, and the maximum/minimum gradients

for each featu
re image
.
Without these parameters, the dynamic cost matrix generation would have unstable outputs since the local
cost matrix parameters do not represent the full cost matrix.
If there happens not to be enough available
GPU memory during the loading of a

feature or initialization of that f
eature, then it
is

removed from
the
segmentation

process
.

3.4.

Segmentation


After the user has

setup the server for

segmentation,
they

may

place the first seed point anywhere
on the image

by left or right clicking on the image
. This point is

then

transmi
tted to the server,
which

uses it
s location to initialize
Dijkstra’s
Algorithm for
a

new active path. As the user moves the cursor
around the image, they are manipulating the free point l
ocation. The client applet periodically sends the
location of the free point to the server, who replies with the active path back to the seed point

as a
compressed binary stream
.
The

periodicity

is varied based
on the time it takes for a point to be
tran
smitted and received. Without this

regulation,
the client application
could

flood the server w
ith
requests, and the server could

saturate the
connection with replies.




7

MOTM:
Message Transmission Optimization Mechanism

8


The

first active path is

generated by equally weighing selected feature images. IT
-
SN
APS
is then

used to dynamically adjust these weights using the multi
-
objective
function
optimization as defined in
[
1
]

by right
-
clicking.

The user

can

also

left
-
click to continue using equal weights.

The last active equal
weights segment before a right click will be used for training IT
-
SNAPS, and continuing to right click
will continue with those same dynamic weights. A left click will re
set all weights to equal values

for use
as another training s
egment

(see
Figure
7
)
.



Figure
7
: Segmentation

Every time the user clicks on the image,
Dijkstra’s
Algorithm is reinitialized

and the last active
path is “frozen” in place. The user may finish segmentation by double
clicking, or click back on the

original
seed point.

A window will then
spawn
,

prompti
ng the user to
name

the segmented area
.
C
ancelling this window will cause this segmented object to be
removed
. Another area may be segmented
by adding another seed point, and all segmented areas will be stored for later use. The user can view the
saved
segmentation
(s)

by clicking on the “Display all” checkbox.

Segmentation results are saved in the same format as “L
abel
M
e
” by MIT

[
7
]
, as shown in
Appendix
C
.

For every image segmented,
one

XML file is saved. Every object is defined by the pixel locations of
the

delineation
, and every object is defined individually.

3.5.

View Features

T
he user
may
view the actual features loaded onto the GPU for segmentation. When the user
clicks on the “View Features” button,
the features loaded on the GPU are copied to the website
.

The
n the
“V
i
ew Features” window

will be
spawned
,
which

will

access
the feature im
ages

as a web browser.

4.

C
ONCLUSION

AND

F
UTURE
W
ORK

The server is cap
able of managing multiple users

on multiple GPUs. The client application
is

easily acce
ssed on the server by end users

and does not require installation on the clients' machine.

Howev
er, communication

between the client and server are

currently unencrypted

and use

TCP
. Some
users may object to using a
n

unencrypted service, sinc
e it
is

possible for their images to be intercepted.
E
ncryption scheme
s

such as TLS or SSL
could be used with

this server

to secure communications
, but
9


would require
further

development. This extra layer of complexity would also require more processing
po
wer on the client and server.

S
witching
communication protocols to UDP, or a combination of UDP and TCP
,

could reduce the
data transmitted
between the client and server.
For instance, d
uring segmentation, the client and server
exchange
masses

of packets
,

but many of the
features
offered by TCP
are
not
required while the user is
manipulating the free point, since if a packet is lost, another will replace it

shortly
.

IT
-
SNAPS works well with feature images
that
hav
e

high gradients, such as the RGB and LA
B
features.
On the other hand, its performance suffers when working with low
-
gradient features, such as
those computed from the Gabor fil
t
er outputs.

A possible solution to this
problem
would be to subsample
the feature
images with known low gradients

to

produce a
feature image with high gradients. This would
be more difficult to implement, but IT
-
SNAPS could greatly benefit from it. It also has the additional
benefit of requiring less memory for feature storage.

Figure
8

shows the final overall workload distribution when using IT
-
SNAPS Java. Except for
managing the graphi
cal interface, including drawing the segmentation path on the image, the server
performs all the work. This allows
feeble

systems with an active network connection to use IT
-
SNAPS
for a faster
segmentation
experience.


Figure
8
: Client Server Workload


Figure
7

10


A
PPENDIX
A:

S
YSTEM
R
EQUIREMENTS



Server

o

Windows XP or Windows Server 2003: Later v
ersions of windows use Session 0 Isolation
which prevents access to graphics drivers. A cumbersome workaround can be created if
necessary.

o

MATLAB Compiler Runtime

o

Java Development Kit (JDK)

o


Microsoft Visual C++ 2010 Redistributable Package (x64)

http://www.microsoft.com/download/en/confirmation.aspx?id=14632

o

Java
GlassFish Server Open Source Edition

(
Full Platform
)

or
Oracle GlassFish Server

o

NVIDIA CUDA GPU: CUDA versio
n 1.3 or higher which has compatible drivers for
the
chosen

operating system.



Client

o

Web Browser

o

Java Runtime



11


A
PPENDIX
B:

I
NSTALLATION

1.

I
nstall GPU drivers, i
f windows

update

has not already.

2.

Install

the
MATLAB Compiler Runtime

3.

Install JDK 7

or higher version. Make note of the installation path for later use.

4.

Install GlassFish

a.

Select

"Custom Installation"



b.

Select "Install and Configure"


12


c.

Enter the full path to JDK



d.

The Update tool is not required, but recommended.



e.

Select "Create a
server domain"



f.

Select "Create Operating System Service" to allow the operating system to manage
GlassFish. This can be used to automatically start

GlassFish

and restart in case of failure.

Change Http port to 80.

13




5.

Deploy
IT
-
SNAPS

a.

Open a web browser
and
go to

localhost:4848

or an alternate admin port



14


b.

Click on "Applications"



c.

Click on "Deploy" then

"C
hoose

File."

Select
"itSnapsServer.war" then click "OK" in the
upper right hand corner.



d.

Repeat deployment with "itSnapsWebApplet.war"

IT
-
SNAPS

should now be available for use at

http://
localhost
:80/itSnapsWebApplet
. During the first
launch, there may be a delay as
IT
-
SNAPS

initializes itself.




15


A
PPENDIX
C:

XML

F
ORMAT

<annotation>

<filename>


</filename>

<folder>


</folder>

<source>

<
sourceImage> </sourceImage>

<sourceAnnotation>

IT
-
SNAPS

JAVA </sourceAnnotation>

</source>

<imagesize>

<nrows></nrows>

<ncols></ncols>

</imagesize>

<object>

<name></name>

<deleted></deleted>

<verified></verified>

<date>
MM

DD
,
YYYY
</date>


<id>0</id>

<
polygon>

<username>anonymous</username>

<pt> <x>

</x> <y>

</y> </pt>

<pt> <x>

</x> <y>

</y> </pt>

<pt> <x>

</x> <y>

</y> </pt>



</polygon>


</object>

</annotation>




16


R
EFERENCES


[1]

A. Gururajan, H. Sari
-
Sarraf, and a. E. Hequet, "Interactive Texture Segmentation Via IT
-
SNAPS,"
2010 IEEE Southwest Symposium
, pp. 129
-
132, May 2010.

[2]

E. N. Mortensen and W. A. Barrett, "Interactive Segmentation with Intelligent Scissors," Brigham
You
ng University.

[3]

E. W. Dijkstra, "“A Note on Two Problems in Connexion with Graphs,”,"
Numerische Mathematik
,
vol. 1, p. 269

270, 1959.

[4]

R. M. Haralick, K. Shanmugam, and I. Dinstein, "Textural features for image classification,"
IEEE
Transactions o
n Systems, Man, and Cybernetics
, vol. SMC
-
3, no. 6, pp. 610
-
621, Nov. 1973.

[5]

The Computer Vision Lab at GET . (1998, Nov.) Gabor Filters. [Online].
http://homepa
ges.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/TRAPP1/filter.html

[6]

NVIDIA. (2008, Jun.) NVIDIA CUDA Compute Unified Device Architecture Programming Guide.
PDF. [Online].
http://developer.download.nvidia.com/compute/cuda/2_0/docs/NVIDIA_CUDA_Programming_Guid
e_2.0.pdf

[7]

B. Russell, A. Torralba, and W. T. Freeman. (2005, Jul.) LabelMe. [Online].
http://labelme.csail.mit.edu/