NDGF: Distributed Tier1

hordeprobableΒιοτεχνολογία

4 Οκτ 2013 (πριν από 4 χρόνια και 10 μέρες)

84 εμφανίσεις

Nordic Grid experience,
NDGF Tier1

Oxana Smirnova

NDGF/Lund University

ARC Meets
SwiNG
, June 25, 2008, Bern

Outlook


Distributed Tier1 for:


Politicians


Sysadmins


Users

2008
-
06
-
25

2

www.ndgf.org

How did Grid appear up North

2008
-
06
-
25

3


Back in
2001, High
Energy Physics Institutes from Scandinavia
wanted to share their computing resources and jointly contribute to
CERN/LHC computing


We needed Grid


The Grid hype just
begun…


… and we created a
NorduGrid

project (Lund, Uppsala, Copenhagen,

Oslo, Helsinki and many others)


No
production ready
grid software

(
middleware
) was
available or
seen

on
the
horizon in fall 2001


In
February 2002, NorduGrid
boldly

decided
to develop own
Grid

middleware


Was baptized ARC, for Advanced Resource Connector


Since May 2002 ARC is
extensively used
in ATLAS
production and
other
scientific
computing
projects


Now ARC is used to make a distributed computing center
for High Energy Physics: the NDGF “Tier1”

www.ndgf.org

Nordic
DataGrid

Facility

2008
-
06
-
25

4

Provides a unique distributed
“Tier1”
center

via NorduGrid/ARC

Involves 7 largest Nordic
academic HPC
centers

…plus a handful of University
centers

(Tier2 service)

Connected to CERN directly with
GEANT 10GBit
fiber

Inter
-
Nordic shared 10Gbit
network from
NORDUnet

Budget: staff only, 2 MEUR/year,
by Nordic research councils

www.ndgf.org

April 2008

Example of country

contribution:
SweGrid

Investment

Time

Cost,
KSEK

Six
clusters (6x100 cores)
including 12 TB FC disk

Dec 2003

10 173

Disk storage part 1, 60 TB
SATA

May 2004


2 930

Disk storage part 2, 86.4
TB SATA

May 2005


2 119

2008
-
06
-
25

5

Centre

Tape
volume,
TB

Cost,
KSEK


HPC2N

120

1000

PDC

120

1000

NSC

120

1000

SweGrid

in 2003
-
2007

Location

Profile

HPC2N (
Umeå
)

IT

UPPMAX (Uppsala)

IT, HEP

PDC (Stockholm)

IT

C3SE (Gothenburg)

IT

NSC (
Linköping
)

IT

Lunarc

(Lund)

IT, HEP


Co
-
funded by the
Swedish
Research Council and the Knut
and Alice Wallenberg foundation


One technician per center


Middleware: ARC,
gLite


1/3 allocated to LHC Computing

www.ndgf.org

SweGrid

and NDGF usage

2008
-
06
-
25

6

LHC
Chemistry
Other Physics
Amanda/Icecube
Geo Science
Fluid Mechanics
Bioinformatics
Medicine
Computer Science
Biotechnology
Mathematics
Statistics
Pharmacology
Material Chemistry
Electronics
www.ndgf.org

Grids in
LHC experiments

2008
-
06
-
25

7


Almost all Monte Carlo and data processing today is
done via Grid


There
are 20+ Grid flavors out
there


Almost
all are tailored for a specific application and/or specific
hardware


LHC experiments make
use of
3 Grid middleware
flavors:


gLite


ARC


OSG


All experiments develop own higher
-
level Grid
middleware layers

www.ndgf.org

ATLAS Experiment at CERN
-

Multi
-
Grid Infrastructure

2008
-
06
-
25

8

Graphics from a slide by
A.Vaniachine

www.ndgf.org

2008
-
06
-
25

9

Compute elements

Distributed over wide area


RedHat
, Ubuntu, Scientific Linux,
CentOS
, Rocks

Operating systems


LoadLeveler
, Torque, Condor, SGE

Batch systems


ARC CEs


ALICE
VOBoxes


No LCG/gLite CEs

Compute element flavors


Not actually needed by the VOs

No single entry point for job submission


SAM tests (modified for ARC
-
CE)


Aggregation via SGAS and via ATLAS “dashboard”


No aggregation (yet) in ALICE monitoring system

WLCG monitoring and accounting procedures

www.ndgf.org

Worker Nodes


Typically, same as CEs:
RedHat
, Ubuntu, Scientific Linux,
CentOS

Operating systems


No Grid middleware installed


Files staged in/out by the CE


Optimizes data movement => better overall efficiency

Managed entirely by local batch systems


Normally no inbound


Outbound connectivity is sometimes limited to certain IP ranges by
sysadmins
.


Connectivity

2008
-
06
-
25

10

www.ndgf.org

Storage elements


Pools distributed over
wide

area (from Norway to Slovenia)


Single entry point via
dCache

SRM at srm1.ndgf.org


We use tapes as well (NDGF introduced
ngstage

tool n ARC)

Distributed storage


Extended by NDGF for distributed storage


Uses GridFTPv2

Middleware:
dCache

1.8


dCache
-
admin: Ubuntu


pools


all the abovementioned systems (RHEL,
CentOS
, …)

Operating systems


Same way as any conventional
dCache

SE (own monitoring tools, SAM tests)


WLCG monitoring

2008
-
06
-
25

11

www.ndgf.org

VOBoxes


One
VOBox

for ATLAS, located at CERN (deals with
data management only)


7 individual
VOBoxes

for ALICE (at each ALICE CE)
plus one central that deals with data movement (
xrootd
-
dCache
)



ALICE
VOBoxes

effectively being ALICE CEs, see CE slide

2008
-
06
-
25

12

www.ndgf.org

FTS


Standard gLite FTS 2.0, stand
-
alone installation,
depends on gLite, needs Scientific Linux CERN
flavor


P
atched with NDGF GridFTPv2 patches


We’d love to have other FTS servers patched the
same way

2008
-
06
-
25

13

www.ndgf.org

Data Indexing (RLS and LFC)


Until last week, Globus RLS was used; now
switching to CERN LFC


LFC support is implemented in ARC by NDGF


Central stand
-
alone installation


LFC is integrated with ATLAS Data Management
system


Both RLS and LFC support
dCache

SRM

2008
-
06
-
25

14

www.ndgf.org

Site BDII


Not used by NDGF
-
Tier1 services, but needed by
external gLite services (monitoring, FTS)


Stand
-
alone standard gLite BDII


A back
-
up server is being installed for redundancy


2008
-
06
-
25

15

www.ndgf.org

Internal monitoring and
operations


Every

listed service is monitored by
Nagios


Custom tests


Even the Web site is monitored


Sends nagging SMS messages to
OoDs


Operator
-
on
-
Duty (
OoD
)


Office hours currently, will soon be extended to
include off
-
hours, on
-
call


All together, ~20 persons are involved in daily
operations


IRC used for live communications; weekly meetings


Critical (“Level
-
1”) services are installed at a
central location, accessible by
OoD

2008
-
06
-
25

16

www.ndgf.org

Relation to Tier2s



Ljubljana Tier2 is affiliated via ATLAS


Practically, extends Tier1


W.r.t
. data retention, has a “normal” Tier2 policy


Several Swiss sites extend Tier1 computing
-
wise


Several Nordic Tier2s are being formed


Largely extending Tier1
w.r.t
. computing power,
but not charged with data storing duties

2008
-
06
-
25

www.ndgf.org

17

Non
-
LHC use case:

MDR analysis


(Thanks Joel
Hedlund

for the material)


Medium
-
chain
Dehydrogenases
/

Reductases


Move hydrogen on/off
molecules


Members from all
kingdoms of life


8 member families


~1000 members (as of
2002...)

2008
-
06
-
25

18

www.ndgf.org

Material courtesy of Joel
Hedlund

State of MDR knowledge today


The evolutionary tree for
MDRs


In this illustration:


Bacteria are removed


Everything >= 40% identical
is also removed.


All sequences <= 250aa are
removed.


This is much more complex
than the researchers ever
thought


But there are tools to sort it
out


HMMER: software making
use of Hidden Markov
Models (HMM)


2008
-
06
-
25

19

www.ndgf.org

Material courtesy of Joel
Hedlund

User requirements


New data are arriving at exponential
rates, search becomes more and more
resource
-
consuming


Users want to have a tool that takes
data in and spits results out


Write an explanation on what's needed to
get that done: memory,
cpus
, disk,
programs, ...


Submit the job to the grid, the grid finds a
cluster that can run this, runs the job, and
gives back the results


If my comp breaks, the results wait for
me; if the cluster breaks, the grid finds
another cluster that can run the job for
me


Some are convinced this is the future of
computing


2008
-
06
-
25

20

www.ndgf.org

Material courtesy of Joel
Hedlund

XRSL Job Description

&

(executable=run_hmmer.sh)

(arguments="
adhbig
")

(
jobname
=
hmmer_mdr
)

(
stdout
=
std.out
)

(
stderr
=std.err)

(
gmlog
=
gridlog
)

(
cputime
=1000)

(memory=1000)

(disk=4)

(
runtimeenvironment
>=APPS/BIO/HMMER)

(
inputfiles
=


("
sprot.fasta.gz
" "srm://srm.ndgf.org/path/to/sprot.fasta.gz")


("
trembl.fasta.gz
" "srm://srm.ndgf.org/path/to/trembl.fasta.gz")


("adhbig.hmm" "")

)

(
outputfiles
=


("
adhbig.sprot.hmmer
" "")


("
adhbig.trembl.hmmer
" "")

)



Can be easily automated via e.g. a Python script

2008
-
06
-
25

21

www.ndgf.org

Material courtesy of Joel
Hedlund

Running it

$
cd

/my/project

$ ./mkjobs.py data/*.hmm

$ for
xrsl

in data/*.
xrsl
; do
ngsub

-
f $
xrsl
; done


...wait to complete...

$
ngstat

hmmer_mdr


$
mkdir

outputfiles

$
ngget

-
d
outputfiles

hmmer_mdr

$
mkdir

results

$ find
outputfiles
/
-
name '*.
hmmer
'
-
exec cp '{}' results/
\
;

2008
-
06
-
25

22

www.ndgf.org

Material courtesy of Joel
Hedlund

Benefits



It just works


It's fast


HMMer

is CPU intensive and threaded


DB is cached on the I/O node, uncompressed on the fly to
worker nodes when the job starts


Small output files


It's easy to use


One interface to learn


Good abstractions of unwanted complexity


Drawbacks: unknown

2008
-
06
-
25

23

www.ndgf.org

Material courtesy of Joel
Hedlund

Summary


A complex but smoothly working distributed facility is in place


Full operational and data management compatibility with WLCG


Work on interoperability with gLite


NDGF Tier1 often outcompetes other conventional CERN
Tier1s in efficiency and reliability


Good choice of middleware (ARC,
dCache
)


Redundancy


Extremely dedicated staff


Other user communities are getting more and more involved


Bioinformatics


CO2 sequestration project


National research activities

2008
-
06
-
25

24

www.ndgf.org