Presentation - SLAC

clappingknaveΛογισμικό & κατασκευή λογ/κού

14 Δεκ 2013 (πριν από 3 χρόνια και 10 μήνες)

66 εμφανίσεις


AliEn

uses

bbFTP

for

the

file

transfers
.

Every

FTD

runs

a

server,

and

all

the

others

FTD

can

connect

and

authenticate

to

it

using

certificates
.

bbFTP

implements

its

own

transfer

protocol

which

is

optimized

for

large

files

(larger

than

2
GB)
.

Its

main

features

are
:



SSH

and

Certificate

authentication

modules



Big

windows

as

defined

in

RFC
1323


Multi
-
stream

transfer



On
-
the
-
fly

data

compression



Automatic

retry



Customizable

time
-
outs



Transfer

simulation



RFIO

interface


AliEn Resource Brokers


AliEn

Services


AliEn

has

been

implemented

having

the

idea

of

‘web

services’

in

mind
.

Most

of

the

services

are

built

using

SOAP

( S i mp l e

Ob j e c t

Ac c e s s

P r o t o c o l ),

wh i c h

a l l o ws

easy

communication

among

processes

in

a

distributed

environment
.




AliEn

(ALICE

Environment)

is

a

lightweight

GRID

framework

developed

by

the

Alice

Collaboration
.

When

the

experiment

starts

running,

it

will

collect

data

at

at

a

rate

of

2

PB

per

year,

producing

more

than

10

9

files

per

year
.

All

these

files,

including

all

simulated

events

generated

during

the

preparation

phase

of

the

experiment,

must

be

accounted

and

reliably

tracked

in

the

GRID

environment
.

The

backbone

of

AliEn

is

a

distributed

file

catalogue,

which

associates

universal

logical

file

name

to

physical

file

names

for

each

dataset

and

provides

transparent

access

to

datasets

independently

of

physical

location
.

The

file

replication

and

transport

is

carried

out

under

the

control

of

the

File

Transport

Broker
.

In

addition,

the

file

catalogue

maintains

information

about

every

job

running

in

the

system
.

The

jobs

are

distributed

by

the

Job

Resource

Broker

that

is

implemented

using

a

simplified

pull

(as

opposed

to

traditional

push)

architecture
.

This

poster

describes

the

Job

and

File

Transport

Resource

Brokers

and

shows

that

a

similar

architecture

can

be

applied

to

solve

both

problems
.



Predrag

Buncic,

Andreas

Peters,

Pablo

Saiz

for

the

ALICE

Collaboration

ClassAds

Classified

Advertisements

(
classads
)

are

the

lingua

franca

of

Condor
.

They

are

used

for

describing

jobs,

workstations,

and

other

resources
.

Classads

are

exchanged

by

Condor

processes

to

schedule

jobs
.

AliEn

uses

classads

for

the

definition

of

jobs,

machines

and

file

transfers
.

Two

examples

of

classads

are

shown

below
.


FOR MORE
INFORMATION:
http://alien.cern.ch

IS

Proxy

Logger

Authen

CPUServer

Only

one

per

organization

Cluster

Monitor

CE

Process

Monitor

SE

FTD

At

least

one

per

site

Broker

TransServer

TBroker

Optimizer

Toptimizer

Job

status

A

job

in

AliEn

goes

through

the

stages

shown

in

the

diagram
.

First

WAITING
;

then

a

CE

picks

it

up

and

it

is

ASSIGNED
;

QUEUED

when

it

is

submitted

to

the

local

batch

system
;

RUNNING

when

the

job

starts

to

execute

on

the

worker

node,

and

finally

DONE

when

it

finishes

AliEn

also

provides

a

method

to

validate

jobs
.

If

a

job

has

to

be

validated,

a

new

process

for

the

validation

will

start

when

the

job

finishes
.

Then,

the

status

of

the

job

will

be

VALIDATED

if

it

passes

the

validation

procedure,

otherwise,

it

will

be

marked

as

FAILED








Broker

Yes
:

Select

Match

?

No
:

Next

Resource


(CE

or

FTD)

Tasks

(Jobs

or

Transfers)


Requirements

=

(

other
.
Type

==

"machine"

)&&(member(other
.
Packages,"AliRoot")

)
;


Packages

=

"AliRoot"
;


Arguments

=

"
--
round

2002
-
02

--
run

071

--
event

269

--
version

v
3
.
08
.
02

-

grun

G+F"
;


Executable

=

"/Alice/bin/AliRoot
.
sh"
;


InputFile

=



{LF
:
/alice/simulation/
2002
-
02
/v
3
.
08
.
02
/
00071
/Config
.
C",


"LF
:
/alice/simulation/
2002
-
02
/v
3
.
08
.
02
/
00071
/grun
.
C"}
;


Type

=

"Job"
;


Broker

FTD

FTD

FTD

SE



DB

SE

Optimizer

SE

Server

New transfer

requested

Castor, HPSS, HIS,

Tape,adsm,

File Disk…

AliEn

Computing

model

The

AliEn

Computing

model

is

made

of

a

central

server,

which

keeps

track

of

all

the

jobs

that

have

to

be

executed

in

the

system
.

Each

site

has

at

least

one

ClusterMonitor
.

This

service

will

ask

the

Broker

for

jobs

to

execute

(taking

into

account

the

specifications

of

the

site,

like

packages

installed

or

close

storage

elements),

and

will

send

them

to

the

Computing

Elements
.

They

will

send

them

to

batch

queues

(like

LSF

or

PBS),

and

register

the

output

in

the

system
.

There

is

as

well

an

optimizer,

that

checks

the

jobs

waiting

to

be

executed
.

The

optimizer

is

in

charge

of

modifying

the

waiting

jobs
.

For

example,

replicating

the

data

that

some

jobs

require

in

sites

with

more

computer

power
.

AliEn

Transfer

model

The

main

component

of

the

transfer

model

is,

like

in

the

case

of

the

computing

model,

a

database

with

all

the

transfers

that

have

to

be

done
.

A

server

inserts

new

entries

in

the

database,

and

an

optimizer

checks

the

entries

and

sets

the

priorities

for

the

transfers
.


Each

site

runs

a

FTD

(File

Transfer

Daemon)
.

Each

FTD

contacts

the

Broker

every

minute

and

requests

transfers
.


Brokers

The

AliEn

Brokers

have

a

list

with

all

the

classads

of

the

tasks

that

have

to

be

done,

and

a

priority

for

each

task
.

Whenever

a

new

resource

is

free,

it

will

send

its

classads

to

the

corresponding

broker,

The

Broker

will

do

the

matching

with

the

highest

priority

task

that

matches

all

the

requirements
.

If

no





Requirements

=

(

other
.
Type

==

"Job"

)
;

Packages

=


{

"ROOT
::
3
.
03
.
07
",

"ROOT",

"AliRoot
::
3
.
08
.
Rev
.
03
",


"AliRoot",

"AliRoot
::
3
.
08
.
Rev
.
04
"

}
;


CloseSE

=


{

"Alice
::
CERN
::
Castor",

"Alice
::
CERN
::
File",



"Alice
::
CERN
::
scratch",

"Alice
::
CERN
::
pcepaip
19
"

}
;

CE

=

"Alice
::
CERN
::
LXSHARE"
;

Host

=

"lxplus
051
.
cern
.
ch”
;

GridPartitions={

“Production”}
;

tasks

are

found,

the

resource

will

wait

for

a

while

and

request

a

new

task


Broker

Cluster

Monitor

Cluster

Monitor

Cluster

Monitor

CE

CE



DB

CE

Optimizer

CE

OTHER

VIRTUAL ORGANISATIONS


OR GRID

IMPLEMENTAIONS

Server

New Job

Submitted

LSF,PBS,CONDOR,…

The

transfers

have

to

be

made

in

several

steps
:

first,

the

source

FTD

has

to

bring

a

copy

from

the

local

mass

storage

system

into

a

scratch

directory

(this

step

is

not

necessary

if

bbFTP

can

access

the

storage

directly)
;

then

the

remote

FTD

can

fetch

the

file

and

put

it

into

its

own

storage
;

finally,

the

source

FTD

deletes

the

scratch

copy

of

the

file
.


*******************************************

* *

* W E L C O M E to R O O T *

* *

* Version 3.03/09 3 December 2002 *

* *

* You are welcome to visit our Web site *

* http://root.cern.ch *

* *

*******************************************

Compiled for linux with thread support.

CINT/ROOT C/C++ Interpreter version 5.15.61, Oct 6 2002

Type ? for help. Commands must be C++ statements.

Enclose multiple statements between { }.

root [0]
newanalysis
-
>Submit();


Analysis
Macro

MSS

MSS

MSS

MSS

MSS

CE

CE

CE

CE

CE

merged Trees
+ Histograms


?

Query for
Input Data

Job

splitting

AliEn

also

offers

the

possibility

of

splitting

jobs

that

require

data

spread

over

several

sites
.

Each

site

will

compute

their

local

available

data,

and

then

the

output

of

all

the

jobs

will

be

merged,

giving

back

the

output

as

if

it

had

run

in

only

one

place
.

The

user

should

not

care

at

all

where

the

job

is

executed
.


GP

CALMA