Dreher Matthieu, INRIA Grenoble

gradebananaSoftware and s/w Development

Dec 2, 2013 (3 years and 11 months ago)

67 views

Dreher

Matthieu, INRIA Grenoble

Piuzzi

Marc, IBPC

Turki

Ahmed, Université d’Orléans


and
Chavent

Matthieu, IBPC

1

Index


Introduction


The
FvNano

framework


Real case
study


Conclusion

FvNano

with

a
small

peptide (191
atoms
)

2

Why

using

interactive
molecular

dynamic

systems
?

3

Molecular

dynamic

simulation

Packages
that

provide

numerical

simulation for
molecules

(batch mode).


Gromacs

[ Hess08]


NAMD [ Phil05 ]


Amber [ Case05 ]




MDs

are
computationnaly

intensive for large
models
:
recent

versions use
OpenMP
, MPI and/or GPU
on large cluster

Production simulations
currently

run

during

several

weeks

on large
clusters (1000
-
10000
cores
)
with

very

large
models

(5
-
10 million
atoms

in
biology
)

4

Problematic

:
Study

of a
particular

biological

event

Prepare

the
simulation

Run

the
simulation

Event
occured
?

No

Analyse

Yes

How
can

the
biologist

insert
his

expertise
during

the simulation to guide
the system?

5

Steeered

molecular

dynamic

Prepare

the
simulation

Run

the
simulation
with

user forces

Event
occured
?

No

Analyse

Yes

Prepare

script
describing

forces

SMD[Izra98] :
Add

external

forces to guide a simulation to a
defined

direction

6

Still

batch mode but not
convenient

for the user

Interactive
molecular

dynamic
(IMD)

7

IMD
[Stone01]
:
Connect

a
visualization

and
haptic

device

to the simulation


Very

convenient

for the user…

But not in
batch mode
anymore
!

Current

limitations of IMD

8

IMD usages


Limited to
demonstration

and
educationnal

purposes


Molecule

size < 60000

Main
difficulty


Performant
link

between

a
parallel

simulation and the
visualization

FvNano


Scalable

framework

to
handle

model up to 1 million
atoms


Focus on a performant pipeline
between

simulation and
visualization

9

FvNano

overvie
w

Simulation


Gromacs

Visualisation


Hyperballs

[Chav11]


VMD

VRPN


Phantom


SpaceNavigator

10

FlowVR


Modules :
indiviual

codes
encapsulated

inside

modules. A
module
can

receive

and
send

messages


Channel :
Communication
link

between

modules. By
default FIFO but
can

create

more
complex

policies

11

Extraction of the
atoms

from

Gromacs

Sim0

Sim1

Sim2

Sim3

Sim4

Sim5

Sim7

Sim6

Visu

How do
we

send

the
atom

positions to
the
visualization
?

12

Sim0

Sim1

Sim2

Sim3

Sim4

Sim5

Sim7

Sim6

The
atoms

positions are
extract

from

Gromacs

structures and
send

asynchronously

outside

the simulation
within

FlowVR

messages

13

Asynchronous

message
containing

atom

positions (TCP/IP)

Sim0

Sim1

Sim2

Sim3

Sim4

Sim5

Sim7

Sim6

Merge

Merge

Merge

Merge

A first
merge

step

concatenate

the
atom

positions
from

a single host

14

Sim0

Sim1

Sim2

Sim3

Sim4

Sim5

Sim7

Sim6

Merge

Merge

Merge

Merge

Filter

Filter

Filter

Filter

We

use a
parallel

filter

which

can

remove

atoms

from

the pipeline base on
its

name
,
residue
,
atomID
, ….

15

Sim0

Sim1

Sim2

Sim3

Sim4

Sim5

Sim7

Sim6

Merge

Merge

Merge

Merge

Filter

Filter

Filter

Filter

Merge

Merge

Merge

Visu

A final
step

merges

all the
remaining

atom

positions
using

a
tree

structure and
sends

the
result

to the
visualization

16

17

Edel

cluster


72
nodes


2
CPUs

Intel@2.27GHz
, 4
cores
/CPU


Infiniband

network (40G)

Grimage interactive cluster (
integrated

in Grid5000 infrastructure)


10
nodes


2
CPUs

Intel@2.40GHz
, 4
cores
/CPU


Infiniband

network (40G)


680 GTX
graphics

cards


Screen
, Virtual Reality
Devices

Connection


Infiniband

link

(20G)

Hardware setup
at

Grenoble

18

Sim0

Sim1

Sim2

Sim3

Sim4

Sim5

Sim7

Sim6

Merge

Merge

Merge

Merge

Filter

Filter

Filter

Filter

Merge

Merge

Merge

Visu

18

Edel

cluster

Grimage cluster

FvNano

network exemple (8
Gromacs

processes
)

19

Performance
evaluation


Simulation (GMX) speed
is

given

in
fps


Use 4
cores

per machine
for simulation


Interactivity

achieved

with

1.7 million
atoms
!


Makes

sense

to have the
simulation
faster

than

the
visualization

20

0
5
10
15
20
25
30
35
40
45
50
0
50
100
150
200
250
Speed (fps)

nbCore

Performances for various models

GMX_570K
GMX_1M7
GMX_860K
GMX_1.1M
Visu @13
fps

Visu @41
fps

Visu @25
fps

Visu @21
fps

Importance of
Infiniband

network

21


Source : biowolf.nih.gov/
apps
/
gromacs
/bench
-
4.5.3.html


Use 4
cores

per machine


In a
lesser

mesure,
high

bandwitch

is

also

required

between

the visualisation and
simulation cluster (3.5Gbit/s for the
larger

model)


Infiniband

16G


Infinipath

8G


Gige

1G

22

Real case
study

: the
Fepa

channel

23


The
complex

can

go
trough

the
channel

with

«

brute force

»

Hypothesis


Fepa

model (70K
atoms
)


Simulation on 128
cores

(160
fps
)


Total forces
limited

to 1000 kJ/nm

Setup


Confirm

2
paths

accessible by the
iron

complex

Result

24

25

Grid5000
experience

26


Integration

of visualisation and
computationnal

cluster
inside

Grid5000


Performance


Usability

(
same

architecture,
nfs
, …)


One
reference

engeneer


Specific

usage : interactive (
latency
,
cards
,
devices
)


Reservation

for interactive session
with

biologist


Deployment

(driver
phantom
, lib for
gromacs
,
flowvr
,
ssh
)



Make

my

experiements

easy

=> happy
biologist

=> happy
thesis


Scalable

extraction pipeline (200
cores
)


Possibility

to
interact

with

models

up to 1.7
million
atoms


Low

overhead

on simulation

Achieved


Link
with

dedicated

visualization

tools

(VMD,
Megamol
) for
better

representation


In Situ
analysis


ExaViz

project

(http://exaviz.simlab.ibpc.fr/)

Future
work

27

28

29

30

31

Importance of the
shading

in the
understanding

of
molecules

TODO :
add

4
screen

meme

molecule

avec 4
shading

différents VMD

32

33


Screen

1M7 sans filtrage


Screen

1M7 avec filtrage


Tableau
rentabilite

gromacs


Tableau latence


Tableau complet perfs



TODO:


Ajout
video

trajectoire +
presentation

IEEEVR


Article Nature


Contact
gency

(cluster
cury
)

Communication
between

modules

34


Use TCP/IP
between

daemons


Work

in
progress

: use
MPI
instead

(
better

support of
Infiniband
)