SEE-GRID Techical Overview and Demos

farrightSoftware and s/w Development

Aug 15, 2012 (5 years and 2 months ago)

403 views

www.see
-
grid.eu

SEE
-
GRID
-
2

The SEE
-
GRID
-
2 initiative is co
-
funded by the European Commission under the FP6 Research Infrastructures contract no. 031775

Bringing Interactivity to 3D
Visualisation on the Grid

Branko Marović

Branislav
Čukanović, Dragan Okiljević, Ivica Barišić,
Jovana Vuleta, Danijel Bašić, Vladimir Čakarević et al.


University of Belgrade

Belgrade University Computing Center
-

RCUB

http://www.rcub.bg.ac.yu

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

2

Introduction

Objective

Complete interactive analysis of 3D datasets from medical
imaging devices (CT, MR, PET, SPECT…)

Easy remote access over the Internet

Generalized 3D tools for diagnosis, surgical planning, and
therapy evaluation

Virtual distant examination based on 3D datasets


Basic idea

Split the system into a client and server part

Dedicate the server part to access and processing of
volumetric data

Produce a lightweight client for user interaction and 3D
rendering

Exploit grid resources and paradigms

to overcome
implementation issues

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

3

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

4

Screnshots
-

From Atoms to
Galaxies

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

7

VIVE Gridification

General benefits


Distribution of processing and data over several nodes


Data transfer and replication


Security


high sensitivity of medical data


Application
-
specific monitoring

New grid
-
specific components


TCP binder (Client
-
server


Three
-
tier)


Mediating between clients and server jobs


Pool of server jobs


Persistence


Collecting monitoring and performance data


3D image management (LFC web front end, spatially segmented file format)


Measurement archive and web view of collected data

Adaptations


Server (lcg
-
ManageSoftware, file I/O to both nearby classic and remote
DPM/SRM SEs)


Glue web user interface


Client


longer delays and protocol changes

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

9

Performance Patterns

Start
-
up delay must be
avoided at any cost!

Network delay


Must be made acceptable
for small transactions


OK for long
-
running
requests

Texture Generation (ms)
28
29
44
48
50
213
282
30
35
40
44
57
22
63
25
14
19
34
49
0
50
100
150
200
250
300
AEGIS02-RCUB
AEGIS01-PHY-SCL
BG02-IM
SZTAKI
AEGIS03-ELEF-LEDA
BG01-IPP
RO-01-ICI
BA-01-ETFBL
AL-02-FIE
BA-02-ETFIS
BA-04-PMFSA
AL-01-INIMA
Communication
Processing
Mesh Generation (ms)
2826
6361
16104
3907
2036
57850
21953
26493
12553
3220
0
10000
20000
30000
40000
50000
60000
70000
AEGIS02-
RCUB
AL-02-FIE
BA-04-
PMFSA
BA-01-
ETFBL
BG01-IPP
Communication
Processing
Server Startup Delay (ms)
108
120
124
127
133
143
150
156
188
224
250
250
380
515
0
100
200
300
400
500
600
AL-02-FIE
BA-01-ETFBL
AEGIS02-RCUB
BG01-IPP
BA-04-PMFSA
BA-02-ETFIS
BG02-IM
AEGIS03-ELEF-LEDA
AEGIS01-PHY-SCL
RO-01-ICI
HR-01-RBI
TR-03-METU
AL-01-INIMA
SZTAKI
CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

10

VIVE Grid Architecture

VO Services
Grid Site
Computing Element
Worker Node
Field Server
Job
Resource
Broker
Browser
Java VM
Applet
VRML
Engine
3D
Behavior
Job Submission
VIVE Portal
TCP Binder
Web
Application
Grid
User
Interface
Web
Server
Job
Control
Tool
Updates
(TCP)
Scene
Setup
(HTTP)
DBMS
Storage
Element
LFC Replica
Manager
Information
System
Grid
Site
Selection
Image
Data
(RFIO)
Gatekeeper
Tool
Updates
(TCP)
CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

11

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

12

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

13

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

14

TCP Binder

Pool of server jobs


Pre
-
emptive submission
prevents delays


Adapt to site
responsiveness


Prevent flooding of
queues


Job lifetime management


Maintaining the
satisfactory number of
jobs in the pool


Per
-
site policy

SUBMITTED
READY
BUSY
AGED
MaxSubmissionTime
recalculated
response
arrival
MaxSubmissionTime
elapsed
MaxAgedTime
elapsed
user session
finished
job allocated
for user
response
arrival
job
discarding
CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

15

Performance measurement

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

16

Stripping:

Partitioned Volume Processing

Split a complex request into subvolumes

Allocate satellite jobs from TCP binder

Issue several parallel requests as client

Retrieve partial results

Integrate and return the final result






Site A

Site B

Site C

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

17

Developer Resources

Grid environment is constantly evolving, but


Useful features persist


New are constantly being added


Bugs are being fixed


Gained knowledge remains relevant, must be updated


Applications can be easily migrated to new/updated APIs


LCG
-
2 User Guide


https://edms.cern.ch/file/454439//LCG
-
2
-
UserGuide.html

SEEGRID Wiki


http://wiki.egee
-
see.org/index.php/SEE
-
GRID_Wiki

Waiting for gLite User Guide

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

18

Software/Job Management

Software installation management using ESM tools

Experiment Software Installation in LCG
-
2


http://goc.grid.sinica.edu.tw/gocwiki/Experiments_Software_Installation

SEEGRID Wiki “SEE
-
GRID ESM Software Installation Guide”


http://wiki.egee
-
see.org/index.php/SEE
-
GRID_ESM_Software_Installation_Guide


Site configuration


Using ESM tools


Example JDL and shell scripts for installation, validation, and removal


Job management

EDG WMS API (LCG
-
2)


C++ & Java


http://www.to.infn.it/grid/workload_management/apiDoc/edg
-
wms
-
api
-
index.html

Exec/spawn of shell scripts that create and submit JDL


simple, but no
direct feedback on jobs

gLite


http://egee
-
jra1
-
wm.mi.infn.it/egee
-
jra1
-
wm/glite
-
wms
-
api
-
index.shtml

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

19

Data Management

LCG
-
2 (LCG
-
2 User Guide, “man” pages)


LCG
-
UTILS API


C/C++


LFC API


C/C++, Python


GFAL API


C/C++, Python


http://grid
-
deployment.web.cern.ch/grid
-
deployment/gis/GFAL/GFALindex.html

SEEGRID Wiki “SG Using file replicas and RFIO: UI configuration, rfiod,
usage in apps, limitations and workarounds”


http://wiki.egee
-
see.org/index.php/SG_Using_file_replicas_and_RFIO:_UI_configuration%2C_rfiod%
2C_usage_in_apps%2C_limitations_and_workarounds


Configuring UI, SE, RB


Site testing of RFIO/GFAL


Typical problems and solutions


Java access to LFC and LCG
-
UTILS

Java LFC/GFAL wrapper


http://grid02.rcub.bg.ac.yu/LFCJavaAPI/index.html

Customizable LFC web front end (upload, list, replicate, delete)


http://grid02.rcub.bg.ac.yu/repmngr/


gLite


http://grid
-
deployment.web.cern.ch/grid
-
deployment/documentation/DataManagement/R3.0/

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

20

What VIVE Can Offer to Developers?

Ready

Experience in ESM software installation management

Java LFC/GFAL wrapper

Customizable LFC web front end

In development

Performance measurement facility


Service implementing measurement archive


Event recording API used by applications, sensors, and monitoring points or agents to
put events into measurement archive


Basic archive querying API suitable for creation of reports


HTML
-
based web application for querying of the archive and display of reports


Advanced features, like generation and display of aggregations, a Web service, and
MRTG graphs support could also be added.

3D Java environment framework


NetBeans/Swing based GUI: view and tool management


Integration of tool GUI and 3D appearance


Scene, lighting, navigation


Java 3D sensors, 3D and GUI events

VIVE


Visualisation of other volumetric data sources
: Earth sciences (atmospheric science, geology,
oceanography), aerodynamics, astrophysics, thermodynamics, and dynamics of fluids.

App
-
ware for interactive applications?

Based on TCP binder experience

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

21



Java Web Start


NetBeans


Java 3D


EJB3/Hibernate


MySQL


Servlets/JSP


Java JNI


Unix shell scripts


Grid APIs (GRAM, MDS, WMS, GFAL, LCG
-
UTILS, LFC,…)


DICOM


JPEG


Applets


Sockets


Java


CGI
-
Bin


VRML

C++


Implementation Technology
Evolution


t

CE EGEE and SEE
-
GRID
-
2 Summer School on Grid Application Support


Budapest 6th July 2006

22



Thank you.