ExTENCI Science Results

colonteeSoftware and s/w Development

Nov 4, 2013 (3 years and 9 months ago)

113 views


ExTENCI Science Results



June 5, 2012






Introduction

Extending Science Through Enhanced National CyberInfrastructure

(ExTENCI) is a joint
project of the Open Science Grid (OSG) and TeraGrid/XSEDE, funded by the National
Science Foundation
1
.

The goal of the project is to:
Develop and provide production
quality enhancements to the
n
ational
c
yberInfrastructure
(CI)
that will enable
specific science applications to more easily use both OSG and
XSEDE
or broaden
access to a ca
pability to both
XSEDE

and OSG users.


This project is targeted at providing

new technologies to the CI
. E
ach of those technologies
was selected with a particular science domain that was
expected

to benefit. These science
groups were not
directly
funded t
o work with ExTENCI
:
work on their part
was
driven by
the perceived benefit to their
research
. As a result, engaging those scientists typically
required the new capability to be functional and that began happening about a year into the
project.
W
ith the
new ExTENCI
-
developed
technologies
substantially
available for use as
we near the
original
end of the project (July 31, 2012), we
present here

an assessment of
science applications that have benefited from
these
new ExTENCI capabilities.

There are 4 subp
rojects with the originally targeted sciences shown in the following table:

Technology


Targeted
Science(s)


Leader(s)


Distributed File System (Lustre
-
WAN)

CMS/ATLAS HEP

Ralph Roskies
/ Paul Avery

Virtual Machines

STAR & CMS

Carol Song / Sebastian
Goasguen
/ Miron Livny

Workflow & Client Tools

SCEC & Protein Folding

Daniel S. Katz / Mike Wilde

Job Submission Paradigms

Cactus Application

Shantenu Jha

T
his assessment will only note
the status of the project d
eliverables while
focus
ing on
the

metric of
science use.

The project
second annual report and
final report will include
assessments of the SOW deliverables, the enhanced cooperation between OSG

and
TG/XSEDE, and science use.

All four subprojects have now delivered
much

of

planned capabilities and are using science
applications to validate those capabilities.
So far, three
of the four have
had
science
applications using the ExTENCI capability.

Distributed File System



[
61

of
80

deliverables complete.]
CMS
,

LQCD
, and ATL
AS

science applications
have been
run for verification and performanc
e analysis,
but no
“production” science work is being performed

yet
.

Two areas are being explored for
later
production science work. (1) A Lustre
-
based storage server containing ~200 TB
will

be
deployed
in Fall 2012
for wide
-
area data sharing as part of the Florida SSERCA (Sunshine
State Research and Education Computing Alliance)
collaboration, involving six Florida
universities. (2)
The
CMS muon application

ha
s

been tested

using
data sto
red
at
the
University of Florida T2 facility

in the Lustre file system
.

Six CMS analysis programs
running simultaneously at UF, Florida State University, Florida International University,



1

See
http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=1007115

ExTENCI Science Results







Page

2

University of South Florida, Fermilab and Pittsburgh Supercomputer C
enter were able to
read the CMS data in parallel, achieving a total throughput 95% of the throughput obtained
with the clients running one at a time.

The analysis clients were deployed using VM
technologies developed elsewhere in ExTENCI.

Virtual Machines



[17 of 2
6

deliverables complete.]
CMS, STAR,
and NanoHub science
applications have been tested

in VMs
.

About 6000 hours of
CPU time has been used on the
Purdue Wispy cloud for computer science work by the ExTENCI collaboration. In addition,
NanoHub

users have used just over 3600 hours

since ExTENCI connected NanoH
ub to the
Wispy cloud.


The production science use is detailed in the following section.

One lesson
learned in this project is that it
has been

difficult to identify science applications i
n
the US that require
or that are now using Virtual Machines
.

Workflow & Client Tools



[10

of 13 deliverables complete.]
Various science applications
have been run and
continue
to be run using the new ExTENCI capabilities. These will be
detailed in the
Workflow & Client Tools Results
section

below
.

Job Submission Paradigms



[7 of 11 deliverables complete.]
The
BFAST (Genome
Matching)

science application was used to verify capability and performance of the new
adaptors for condor and glideinWMS
. As par
t of ExTENCI, the research prototype BigJob
was hardened and expanded to use OSG an
d put into production

service. As
a result of the
production availability of BigJob, five science projects consumed 3.25 Million SUs on
XSEDE
. One of these,
Molecular Biop
hysics
,

used resources at multiple XSEDE sites, which
is rare. The
production

science work
that directly resulted from ExTENCI capability is

detailed in the Job Submission Paradigms Results section
below
.

Virtual Machines

Results


Science Domain:

High Energy Physics

The primary physics task of STAR is to study the formation and characteristics of
the quark
-
gluon plasma (QGP), a state of matter believed to exist at sufficiently high
energy densities. Detecting and understanding the QGP
helps

us to

understand the
universe in the moments after the Big Bang, where the symmetries (and lack of
symmetries) of our surroundings were put into motion.

Scientist:

Jerome Lauret



Email:


jlauret@bnl.gov



Result:

Over the course of
a

month, over eighty
thousand tasks were executed

on
ExTENCI
-
supported cloud infrastructure
, generating more than twelve billion
events using PYTHIA, a proton simulation event generator.

A subset of the events
went through detector simulation using a GEANT3 based package.

Fi
nally, STAR
reconstruction software was used to simulate a trigger on the remaining events.

In
all, the simulation ran for over 400,000 CPU hours.

Nearly seven terabytes of results
were transferred back to Brookhaven National Laboratories for further stu
dy.

During the month the simulation took place, the CPU hours used amounted to an
ExTENCI Science Results







Page

3

expansion of approximately 25% over STAR’s average capacity. Furthermore, the
STAR group has stated that it would have been only able to devote fifty CPUs to this
task on it
s own farm, increasing the real time to completion by a factor of twenty.

Publications:


1.


Lauret, M. Walker, S. Goasguen, L. Stout, M. Fenn, J. Balewski, L. Hadju and K.
Keahey “When STAR meets the Clouds

Virtualization and Cloud computing
experiences” Jo
urnal of Physics, Vol. 331, Conference Series, 2011

2.

L. Stout, M. Fenn, M. Murphy, and
S. Goasguen
. "Scaling Virtual Organization
Clusters over a Wide Area Network using the Kestrel Workload Management
System." 4th International Workshop on Virtualization
Technologies in Distributed
Computing (VTDC 2010), Chicago, IL, June 2010.

3.

L. Abraham, M. Fenn, M. Murphy and
S. Goasguen

“Self
-
Provisioned Hybrid Clouds”
7
th

IEEE International Conference on Autonomic Computing, June 7
-
11, 2010, D.C,
USA

4.

M. Fenn, J. Laure
t and
S. Goasguen

“Contextualization in Practice: The Clemson
Experience” ACAT, Jaipur, India February 2010.

Resources Used:

OSG: 400,000 CPU hours


XSEDE: 0 SUs

Key ExTENCI capability for this application:
Onecloud, opennebula deployment
at Clemson, Kes
trel task management system, KVM

Why was
ExTENCI

capability useful?
The simulation would not have been
possible without the ExTENCI functionality at Clemson

that provided access to a
large set of VMs.


Workflow & Client Tools

Results


Science Domain
:

Bio
chemistry / Protein Science

Scientist:

Aashish Adhikari


Email:


aashish@uchicago.edu




Result:

The collaborating groups of Karl Freed and Tobin Sosnick

used OSG to
develop and validate new protein structure and folding pathway prediction
algorithms that start from the amino acid sequence, without the use of templates.
Their approach employs novel, compact molecular representations and innovative
“moves”
of the protein backbone to achieve accurate prediction with far less
computation than previous methods. One of their methods, called “loop modeling”,
which rebuilds local protein structures, was highly ranked in the international
CASP9 competition (
Critica
l Assessment of Protein Structure Prediction)
. The group
used OSG and XSEDE to evaluate
the
loop modeling algorithm on new proteins and
parameter spaces, to develop a new “tertiary fixing” algorithm, and to compare
these algorithm to other approaches.

ExTENCI Science Results







Page

4

Publ
ication
:


In final preparation for submission: A.N.Adhikari, Karl F. Freed and
Tobin R. Sosnick. “Using sequential stabilization to predict protein folding pathways
and structure without using homology, fragments or Gō models.”

Resources Used:

OSG:
110,00
0 CPU hours
XSEDE:

Some
SUs
, not
tracked

Key ExTENCI capability for this application:

Swift parallel scripting with Glide
-
in
Workload Management System (runs to date used a precursor
to

the Swift
-
GWMS
interface).

Why was ExTENCI capability
useful
?

1.

Made

it easier to express the application processing
.

2.

Made it possible to extend local resources with more CPU hours
.

3.

Made it easier for the scientist to run across many diverse resource providers
.


Science Domain:

Theoretical Chemistry

Scientist:

Glen
Hocky

Email:

hockyg@gmail.com

Result:

Glen Hocky of the Reichman
g
roup at Columbia is evaluating a new cavity
method for theoretical chemistry studies of how and why a liquid

becomes a glass.

This fundamental topic of statistical mechanics is described
as “the deepest and
most interesting unsolved problem in

solid state theory.”


OSG was used via Swift to
calculate from simulation what is known as the “mosaic length”, where particles are
simulated by molecular dynamics or Monte Carlo methods within cavit
ies having
amorphous boundary conditions.

This work has implications in many fields,
including biology, biophysics, and computer science.

The computations comprised approximately 30,000 independent tasks broken into
hundreds of thousands of jobs
.
They w
ere executed in parallel on resources
provided by the University of Chicago

(UC)
Computation Institute, (Beagle and
PADS clusters), the Open Science Grid, and XSEDE. These resources were used by
expressing the simulations in the Swift parallel scripting l
anguage.

Publication:


Glen M. Hocky, Thomas E. Markland, David R. Reichman. Growing point
-
to
-
set length
scale correlates with growing relaxation times in model supercooled liquids.

Physical Review Letters

108, 225506(2012)


June 1, 2012
.

http://prl.ap
s.org/abstract/PRL/v108/i22/e225506

Resources Used:

OSG:
~815,000 CPU hours

XSEDE:
~100,000
SUs

Key ExTENCI capability for this application:

Swift parallel scripting with Glide
-
in
Workload Management System (runs to date used a precursor of the Swift int
erface
to GWMS).

Why was
ExTENCI

capability
useful
?

1.

Made it easier to express the application processing

ExTENCI Science Results







Page

5

2.

Made it possible to extend local resources with more CPU hours

3.

OSG resources provided the lengthy run times needed by these jobs

4.

Made it easier for the

scientist to run across many diverse resource providers

5.

The n
ext campaign will require
integration of XSEDE HPC and OSG

HTC

resources (will need to run MPI apps integrated into script with serial apps).



ExTENCI Science Results







Page

6


Science Domain:

Earth Systems Science

Scientist
:

Joshua Elliott

Email:


jelliott@ci.uchicago.edu

Result:

The NSF RDCEP
2

project develops a large
-
scale integrated modeling
framework for decision makers in climate and energy policy.

The project is using
Swift to study land use, land cover, and the im
pacts of climate change on agriculture
and the global food supply using a crop systems model called DSSAT (“decision
support system

for agro
-
technology transfer”)
.

Benchmarks of this model were
performed on a simulation campaign measuring yield and climat
e impact for a
single crop (maize) across the US with daily weather data and climate model output
spanning 1981
-
2100 with 16 configuration
s of fertilizer
,
irrigation

and cultivar.
Initial results have been presented in an advisory board meeting of the RDC
EP
project.

Each simulation runs 125,000 DSSAT models using Swift.

The project has
recently moved these runs from the Be
agle Cray system to the UC

UC3 OSG campus
grid system

and
the OSG engage VO using Swift and the GWMS workload
management system. The R
DCEP project plans to rely mostly on OSG

and UC3

computing resources for their ongoing scientific production.

The RDCEP team is

participating in a fast
-
track research program
with

the
Agricultural Modeling Intercomparison and Improvement Project (AgMIP)
and the
Inter
-
Sectoral Impact Model Intercomparison P
roject (ISI
-
MIP). This

coordinat
ed
multi
-
model comparison will be based on
CMIP5 climate model data, which is only
now being released by m
ost climate modeling groups. The group will be using
OSG+UC3 via
Swift to develop
high
-
resolution gridded global impact results,
targeting
publications

by

January 2013

for inclusion in the IPCC AR
-
5 report.

Publication:

in development.

Resources Used:

OSG:
~30,000 CPU hours (UC3)
XSEDE:

Use desired and
pending. Use
d Beagle HPC resources (~50,000 hours)

as a precursor to Kraken.

Key ExTENCI capability for this application:

Swift; GWMS; flocking f
rom UC3 to
additional resources

Why was
ExTENCI

capability
useful
?

1.

Made it easier to express the application processing
.

2.

Made it possible to extend local resources with more CPU hours (will need
significant CPU to run global models; even higher resolution is desired, which
would increase needs to 36X those above)
.

3.

Made it easier for the scientist to run across many diverse r
esource providers
.






2

http://as102.http.sasm3.net/awardsearch/showAward.do?AwardNumber=0951576

ExTENCI Science Results







Page

7


Science Domain:

Biological science: sociology of science, collaboration network

Scientist:

Andrey Rhezhetsky

Email:

arzhetsk@medicine.bsd.uchicago.edu

Result:

PI Andrey Rzhetsky is
testing scripts that

use Swift for his project
“Assessing and Predicting Scientific Progress through Computational Language
Understanding

3
.

The project develops general methods relevant for students,
policy makers and scientists.

The goal is to generate dynamic high fideli
ty maps of
knowledge claims in chemistry and related fields such as pharmaceuticals and
toxicology.

Their initial use of Swift involves optimization of parameters for graph
-
based models of scientific collaboration.

Publication:

None so far

Resources Used
:

OSG:
~10,000
CPU hours

XSEDE:
<1,000
SUs

(to evaluate dual
grid usage)

A typical optimization run of this graph
-
based simulation is about 115,200 CPU
hours.

We expect to perform approximately 20 such simulations per year, requiring
2.3M CPU hours per

year.

Some large fraction of that would be from OSG (Estimate:
~1.5M hours with the remainder coming from the Beagle Cray system).

Key ExTENCI capability for this application:

Swift; GWMS; flocking from UC3 to
additional resources;

ability to mix MTC an
d HPC resources (using OpenMP on HPC
systems).

Why was
ExTENCI

capability
useful
?

1.

Made it easier to express the application processing
.

2.

Made it possible to extend loc
al resources with more CPU hours
.

3.

Made it easier for the scientist to run across many dive
rse resource providers
.

Job Submission Paradigms


Science Domain:

Data
-
Intensive Computational Biology (NGS Alignment)

Scientists:


Joohyn Kim (LSU/CCT) gateway coordinator

Chris Gissendanner (University of Louisiana at Monroe)

Erik Flemmington

(Tulane Cancer Center)


Result:

Tools developed as part of ExTENCI have been used for Next
-
Genergation
Sequencing (NGS) data analytics.

Two most prominent problems to which Pilot
-
Job
based approaches have been employed in NGS are RNA
-
Seq and ChIP
-
Seq.

With Dr.
Chris Gissendanner, Department of Basic Pharmaceutical Sciences, College of
Pharmacy, University of Louisiana at Monroe, a pipeline for ChIP
-
Seq has been



3

http://as102.http.sasm3.net/awardsearch/showAward.do?AwardNumber=0915730

ExTENCI Science Results







Page

8

developed.

For RNA
-
Seq, the fusion gene discovery which is important for cancer
research is
now available with the support of TopHat
-
Fusion.

Publication:


S
cience papers

not yet published
. The Computer Science papers are:

1.

A Luckow, M Santcroos, O Weidner, S Maddinenei, A Merzky and S Jha,

Towards
a Common Model for Pilot
-
Jobs

, Poster Paper,
Proceedings of the HPDC 2012
(Delft)

2.

P Mantha, A Luckow and S Jha, “Pilot
-
MapReduce: An Extensible a
nd Flexible
MapReduce Implemen
tation for Distributed Data”, Third International
Conference on MapReduce, Proceedings of the HPDC 2012 (Delft)

3.

P Mantha, J Ki
m, N Kim, A Luckow and S Jha, “Understanding MapReduce
-
based
Next
-
Generation Sequencing Alignment on Distributed Cyberinfrastructure”,
Emerging Computational Methods for the Life
-
Sciences, Proceedings of the HPDC
2012

4.

Thota, Mukherji, Fujioki, Bishop and J
ha, "Running Many MD Simulations on
Many Supercomputers", Accepted for XSEDE'12

5.

Romanus, Khamra, Mantha, Merzky, Bishop, Jha, “Anatomy of a Successful ECSS
Project”, Accepted for XSEDE'12

Resources Used:

OSG:
25,000 hours EGI: 25,000 Hours
XSEDE:
>100
,000
SUs

Key ExTENCI capability for this application:

SAGA
-
BigJob

Why was
ExTENCI

capability
useful
?

SAGA
-
BigJob enabled effective distribution
of tasks across resourc
es.

S
ummary

Given that ExTENCI was funded to enhance technologies
to

make it easier to use

the
National CyberInfrastructure for scientists, it
will

primarily be evaluated on the delivery of
those technologies. In this assessment, we used
the
criterion

of scientists
that
have been
taking advantage of these new capabilitie
s to execute science appli
cations on the National
CI
.

Given
this criterion
, ExTENCI has so far enabled
7

scientists, in
6

different domains, to run
their applications on both OSG and XSEDE resources. They have consumed
1,390
,000 CPU
hours on OSG and 251,0
00 SUs on XSEDE and other HPC resources. (Also, 25,000 hours on
EGI

and 3,600 hours on Purdue Wispy
.)
Eight
paper
s

have
been
published

and
another
three are
ready
for publication
.

S
everal papers
/
presentations have been
made on the
technology projects a
t OSG and
TG/
XSEDE conferences as well as
at
UCC 2011, OGF34, LUG
2012, CHEP

2012
.

As we complete work on the technology enhancements, we
plan for

additional use
by
science projects

and will update these results
in the project final report
.