ExTENCI Project Annual Report 2011-2012

tribecagamosisAI and Robotics

Nov 8, 2013 (3 years and 11 months ago)

185 views












ExTENCI Project

Annual Report

2011
-
2012

June 30,

2012



Extending Science
Through Enhanced National Cyberi
nfrastructure











Paul Avery

University of Florida



Principal Investigator

Ralph Roskies

Pittsburgh Supercomputing Center

Co
-
PI

NSF Grant 1007115

Funded by MPS & OCI


ExTENCI Project Annual Report 201
1
-
201
2



Table of Contents


1.

Summary and Participants

................................
................................
................................
...

3

1.1

People

................................
................................
................................
................................
.............

4

1.2

Partner Organizations

................................
................................
................................
.....................

5

1.3

Participants: Other Collaborators

................................
................................
................................
...

5

2.

Project Activities and Findings:

................................
................................
...........................

6

2.1

Project Activities

................................
................................
................................
............................

6

2.2

Project Findings

................................
................................
................................
............................

14

2.3

Training and Development

................................
................................
................................
...........

15

3.

Publications and Products

................................
................................
................................
...

16

3.1

Journal publications

................................
................................
................................
......................

16

3.2

Book(s) and/or other one time publication

................................
................................
...................

16

3.3

Web site or other Internet sites created

................................
................................
........................

16

3.4

Other specific products

................................
................................
................................
.................

17

4.

Contributions
................................
................................
................................
........................

17

4.1

Contributions within Discipline

................................
................................
................................
...

17

4.2

Contributions to Other Disciplines

................................
................................
...............................

17

4.3

Contributions to Education and Human Resources

................................
................................
......

19

4.4

Contribution to Resources f
or Research and Education

................................
...............................

19

4.5

Contributions Beyond Science and Engineering

................................
................................
..........

20

5.

Conference Proceedings

................................
................................
................................
......

20

6.

Special Requirements

................................
................................
................................
..........

22

Appendix A: Cyberinfrastructure Usage by ExTENCI

................................
..........................

22



ExTENCI Project Annual Report 201
1
-
201
2



Page
3

1.

Summary

and Participants

The
Extending Science
Through Enhanced National Cyberi
nfrastructure

(ExTENCI)
project
goal
is to d
evelop and provide production quality enh
ancements to the National Cyberi
nfrastructure
that will enable specific sc
ience applications to more easily use both
the Open Science Grid
(
OSG
)

and
TG/XSEDE
or broaden access to a capability to both
TG
/XSEDE

and OSG users.

This document is an interim annual report of the
second

year of the project
that is
funded by
MPS and OCI
.

ExTENCI is a combination of four technology components, each run as a
separate project with
one or more science projects targeted as the first user.
The components and sciences are:
Workflow & Client Tools



SCEC and Protein
Modeling
,
Distributed File
System



CMS/ATLAS
,
Virtual Machines



STAR and CMS
, and
Job Submission Paradigms



Cactus
Applications (EnKF and Coastal Modeling)
.

A
nother important benefit
of ExTENCI
has been to provide an opportunity for cooperation and
joint activity between OSG and
TG
/XSEDE
. Each of the projects has fostered sharing of
information, joint problem solving, and combined teams to better serve users across both
environments
,

and
thus
paved

the way to OSG becoming a Service Provider in XSEDE
; and,
beginning
this year, off
ering resources through the XSEDE allocation process
.

After 23

months, significant progress has been made by each of the projects.


A no
-
cost
extension has been granted through 7/31/13, so this is the 2
nd

annual report, not the final project
report.

Workfl
ow & Client Tools

continued work in

earthquake hazard predict
ion (from the SCEC
project) and branched out to add several new science applications in addition to the protein
structure work in the first year
.

For SCEC, we now have a single script to run the

SCEC
Cybershake processing on both OSG and XSEDE and have developed a new data management
interface the data handling/management capabilities of Swift. The SCEC Broadband code has
been installed, problems resolved, and results verified. A version of par
allel Broadband code in
VMs was run on Futuregrid as a precursor to run
ning

the "hanging wall" benchmark in one day.

Focus of the Swift workflow engine work shifted this year to supporting science applications in
protein science, theoretical chemistry, ea
rth systems science, and sociology of science. These
applications resulted in published papers and used over 1.1 million CPU hours on OSG and
XSEDE.

Distributed File System

updated software to the latest version of Lustre (2.1) and base OS
(CentOS 5.7 and

Scientific Linux 6.1).

FSU and FIU were incorporat
ed

into the Lustre network
and a
remote
Lustre Object Storage Server (OST) was added to make the first instance of a
distributed OST. The CMS, ATLAS, and LQCD applications were tested and the CernVM File

System was ported to the PSC distribution stack. The supported VM images no
w

include all
major VM systems (XEN, VirtualBox, VMWare, and KVM). Performance testing and
benchmarking w
ere

completed across the network. Four papers on this work were presented at
conferences as detailed in Section 5.

Virtual Machines

has created

'Elastic IP' and 'Security Group' services for leasing of public IP
addresses that are in production in Clemson's

OneCloud and has been tested by FutureGrid and
DESY in Germany. A Cloud Dashboard has been developed at Purdue that enables users to
start, interact with, and stop VMs via a browser. A study was completed and presented on the

ExTENCI Project Annual Report 201
1
-
201
2



Page
4

utility of using Cloud reso
urces for NanoHUB.

Work has begun on deploying VMs to
FutureGrid's Eucalyptus, Nimbus, and OpenStack services.

Job Submission Paradigms

has
completed, hardened and de
ployed the SAGA
-
Condor adaptor
and
tested it on OSG. The project has integrated the SAGA
-
Condor adaptor within the BigJob
Pilot
-
Job mechanism and integrated the Condor enabled BigJob within the DARE framework of
the Cactus Gateway. Support of Cactus
-
based Numerical Relativity and Computational Biology
application groups has begun.

1.1

People


Nam
e

Project Role

Paid

160
Hours

Institution

ExTENCI PIs





Paul Avery

PI

No

No

UF

Ralph Roskies

Co
-
PI

No

No

PSC

Dan
iel S.

Katz

Co
-
PI

(until 4/2012)
; Institution lead for UC; task
lead for SCEC part of Workflow & Client Tools
Project

Yes

No

UC

Project

Management





James Weichel

ExTENCI Project Management

Yes

Yes

UF

Senior Personnel





J Ray Scott

Distributed File System

Lead

Yes

Yes

PSC

Michael Wilde

Task lead for Protein Structure part

of
Workflow &
Client Tools
, Institution lead for UC as of
4/2012

Yes

No

A
NL

Shantenu Jha

Job Submission Paradigm

Lead

Yes

No

RU/LSU

Carol Song

Virtual Machines

Project Director

No

No

PU

Sebastien Goasguen

Virtual Machines

Project Director

Yes

No

CU

Miron Livny

Virtual Machines

Project Director

No

No

UW

Ruth
Pordes

Distributed File System; OSG Representative

Yes

No

F
RA

Gabriele Garzoglio

Distributed File System

FNAL Lead

Yes

No

F
RA

Jorge Rodriguez

Distributed File System Faculty

No

No

FIU

Jim Wilgenbusch

Distributed File System
Center Director

No

No

FSU

Daniel Majchrzak

Distributed File System
Center Director

No

No

USF

Technical Staff





Al
l
an Espinosa

Workflow & Client Tools

Graduate Student

SCEC

Yes

Yes

UC

Aashish Adhikari

Workflow & Client Tools
Graduate Student

Protein
Structure

Yes

Yes

UC

Glen
Hock
y

Workflow & Client Tools Graduate Student

Glass
Materials

No

No

UC

Marc Parisien

Workflow & Client Tools
Postdoc

Protein Structure

No

No

UC

Josephine Palencia

Distributed File System

Professional Staff

Yes

Yes

PSC

John Kochmar

Distributed File
System

Professional

Staff

Yes

Yes

PSC

Brian Johanson

Distributed File System
Professional
Staff

Yes

Yes

PSC

Dimitri Bouril
kov

Distributed File System
Professional
Staff

Yes

Yes

UF

Yu Fu

Distributed File System Professional Staff

Yes

Yes

UF

Yujun Wu

Distributed File System Professional Staff

Yes

Yes

UF

Mengxing Cheng

Distributed File System
Post Doc

Yes

Yes

UF

Dave Dykstra

Distributed File System
Professional
Staff

Yes

Yes

F
RA

Nirmal Seenu

Distributed File System
Professional
Staff

Yes

Yes

F
RA


ExTENCI Project Annual Report 201
1
-
201
2



Page
5

Alex Kulyavtsev

Distributed File System Professional Staff

No

No

F
RA

John Dilascio

Distributed File System Professional Staff

No

No

FIU

Donald Shrum

Distributed File System Professional Staff

No

No

FSU

Drew Oliver

Distributed File System Professional
Staff

No

No

USF

Preston Smith

Virtual Machines Professional Staff

Yes

Yes

PU

Alex Younts

Virtual Machines Professional Staff

Yes

Yes

PU

Stephen Harrell

Virtual Machines Professional Staff

No

No

PU

Fenping Hu

Virtual Machines Professional Staff

CMS
science

No

No

PU

Lance Stout

Virtual Machines Graduate Student

Yes

Yes

CU

Todd Miller

Virtual Machines Professional Staff

Development

Yes

Yes

UW

Jaime Frey

Virtual Machines Professional Staff

Development

Yes

Yes

UW

Todd Tannenbaum

Virtual Machines
Professional Staff

Architecture

Yes

No

UW

Brian Bockelman

Virtual Machines Professional Staff
-
CMS science

No

No

UNL

Siyuan Ma

Virtual Machines Graduate Student

Yes

Yes

FRA

Ole Weidner

Job Submission Paradigms Professional Staff

Yes

Yes

LSU/RU

Melissa
Romanus

Job Submission Paradigms
Graduate Student

Yes

Yes

R
U

Sharath Maddineni

Job Submission Paradigms Professional Staff

No

No

LSU

Andre Luckow

Job Submission Paradigms Professional Staff

Yes

No

LSU/RU

Pradeep Mantha

Job Submiss
ion Paradigms Graduate
Student

No

No

LSU


1.2

Partner Organizations

List of Project Organizations

1.

University of Chicago (UC)

2.

Clemson

University

(CU)

3.

Fermi
Research Alliance, LLC (FRA)

4.

University

of Florida (UF)

5.

Pittsburgh Supercomputing Center (PSC)

6.

Purdue

University

(PU)

7.

Rutgers
University (RU)

8.

University of Wisconsin, Madison

(UW)

1.3

Participants:

Other Collaborators

ExTENCI
collaborates

with additional orga
nizations that provide software and tools
and that are
users of new ExTENCI capabilities

1.

Brookhaven National Lab and STAR
Experiment


production computing using cloud
technology

2.

CERN LXCLOUD


Virtual Machine Image Catalog (VMIC), OpenNebula deployment

3.

Columbia University


Using Workflow & Client Tools work in material modeling for
glass

4.

Florida International University (FI
U)


CMS physics users of Distributed File System

5.

Florida State University

(FSU)


IT organization and CMS p
hysics users

of Distributed
File System

6.

University of Southern Florida (USF)


IT organization and users of Distributed File
System

7.

Louisiana State
University

(LSU)



Development for Job Submission Paradigms, Cactus
and users of Cactus


ExTENCI Project Annual Report 201
1
-
201
2



Page
6

8.

Universit
y of Nebraska, Lincoln (UNL)


T
esting Condor EC2 support with Magellan,
CMS VM design

9.

Red Hat


Condor collaborator for VM support

10.

Whamcloud


Lustre support

2.

Project
Activities and Findings:

2.1

Project
Activities

2.1.1

Project Activities of Workflow & Client Tools

In this period we enhanced the Swift pilot
-
job mechan
ism to be able to use the Glide
in Workload
Management System
(GlideinWMS)
supported by the OSG Engage VO
, in addition to its prior
mechanism that

sent jobs directly to Condor
-
G. This mechanism is typically able to launch
approximately 800 jobs in less than 15 minutes. We also performed significant reliability
improvements to Swift’s ability to stage data dir
ectly to a worker node local filesystem, and
added the ability to do this staging both with Swift’s own transfer protocol as well as with
GridFTP. This flexibility is helpful for running jobs under the wide range of
configurations that

users require.

The f
ollowing talks were developed and presented:

1.

D. S. Katz, "Many
-
Task Computing Tools for Multiscale Modeling," MAPPER All Hands
Meeting, London, UK, 2012.

2.

D. S. Katz, "Many
-
Task Computing Tools for Multi
-
Scale Modeling," RPI
-
NSF Workshop
on Multiscale Model
ing of Complex Data, Troy, New York, 2011.

In addition, a training presentation was made


see Section 2.3.1.


Science Domain: Earthquake simulation (SCEC)


We have split SCEC processing into XSEDE and OSG parts, and run Cybershake

processing
end
-
to
-
end on both infrastructures. This works as a demo of cross
-
grid application execution, but
is not yet practical, as cores are generally found on multiple sites, and SCEC's large data set
cannot be stored (long
-
term) on most OSG sites. Fu
ture work must enable this workflow to use
OSG’s storage elements, for which we need

to provide new data management capability

for
Swift. We have recently developed a new data management interface on the Swift worker no
de
that will make this possible.

We h
ave examined and report
ed

on alternatives for data movement in the contexts of
performance and ease of use. Further optimization can and will continue. Data transfers have
been aggregated into bundled jobs. We run jobs on OSG when we can amortize the data
transfer
cost through the amount of computing that needs to be done.

We reported on alternatives for task mapping, etc., in the contexts of performance and ease of
use.

Most mapping of task bundles is done automatically through Swift now, though there is s
ome
decision
-
making on what to run on OSG vs. XSEDE that is still somewhat manual. From the

ExTENCI Project Annual Report 201
1
-
201
2



Page
7

end
-
user point
-
of
-
view, a single script exists to run a Cybershake analysis using XSEDE and
OSG, which is quite simple. Further optimization opportunities exist.

Th
e currently sequential Broadband code was ported to Swift in order to parallelize it and to
enable it to run efficiently on both XSEDE and
OSG
. The goal of running the “hanging wall”
benchmark in one day has not yet been achieved, in part due to staff tran
sitions. We are
coordinating with SCEC to continue working with latest version of Broadband code.

We downloaded and installed the latest release 11.2.2 of SCEC Broadband code on the XSEDE
Ranger platform. This required the installation of numerous complex

supporting packages and
libraries. Minor issues in unit testing were addressed in consultation with SCEC. The acceptance
tests were successfully performed on Ranger. A number of tests showed divergence from desired
tolerance values. Preliminary probes sug
gest a compiler version mismatch could be the reason
for these failures. This is being addressed in consultation with SCEC. The tests were then
successfully run after applying broadband patches supplied by SCEC.

Hanging wall


processing
stages, inputs, ou
tputs and degrees of parallelism have been defined with SCEC.

SCEC supplied
the input data and a use
-
case input dataset.

Hanging wall


simulations were set up on Ra
nger,
and successfully tested. Sixteen

simulations have been successfully run in parallel

through
Swift.

We are building a version of parallel Broadband code
in
VMs to run on XSEDE Wis
py
systems
and potentially Amazon EC2. Broadband has been installed within a VM on Futuregrid. The unit
tests are now passing, and we are working on the acceptan
ce tests.


Science Domain:
Biochemistry / Protein Science

Resources Used:

OSG: 110,000 CPU hours XSEDE: Some SUs, not tracked

The collaborating groups of Karl Freed and Tobin Sosnick used OSG to develop and validate
new protein structure and folding pa
thway prediction algorithms that start from the amino acid
sequence, without the use of templates. Their approach employs novel, compact molecular
representations and innovative “moves” of the protein backbone to achieve accurate prediction
with far less c
omputation than previous methods. One of their methods, called “loop modeling”,
which rebuilds local protein structures, was highly ranked in the international CASP9
competition (Critical Assessment of Protein Structure Prediction). The group used OSG and
XSEDE to evaluate the loop modeling algorithm on new proteins and parameter spaces, to
develop a new “tertiary fixing” algorithm, and to compare these algorithm to other approaches.

Publication of this work is in final preparation for submission: A.N.Adhik
ari, Karl F. Freed and
Tobin R. Sosnick. “Using sequential stabilization to predict protein folding pathways and
structure without using homology, fragments or Gō models.”

Publication is in final preparation for earlier work by M. Parisien: Discovering RNA
-
protein
interactome using chemical context profiling of the RNA
-
protein interface. Marc Parisien,
George Perdrizet II, Corissa Lamphear, Carol A. Fierke, Ketan C. Maheshwari, Michael J.
Wilde, Tobin R. Sosnick, Tao Pan.


Science Domain:

Theoretical Chemi
stry

Resources Used:

OSG: ~815,000 CPU hours

XSEDE: ~100,000 SUs


ExTENCI Project Annual Report 201
1
-
201
2



Page
8

Glen Hocky of the Reichman group at Columbia is evaluating a new cavity method for
theoretical chemistry studies of how and why a liquid

becomes a glass. This fundamental topic
of statistic
al mechanics is described as “the deepest and most interesting unsolved problem in
solid state theory.” OSG was used via Swift to calculate from simulation what is known as the
“mosaic length”, where particles are simulated by molecular dynamics or Monte
Carlo methods
within cavities having amorphous boundary conditions. This work has implications in many
fields, including biology, biophysics, and computer science.


The computations comprised approximately 30,000 independent tasks broken into hundreds of

thousands of jobs. They were executed in parallel on resources provided by the University of
Chicago (UC) Computation Institute, (Beagle and PADS clusters), the Open Science Grid, and
XSEDE. These resources were used by expressing the simulations in the

Swift parallel scripting
language.

This work has been published
as noted in Section 3.1

(Hockey).


Science Domain:

Earth Systems Science

Resource used:

OSG: ~30,000 CPU hours (UC3) XSEDE: Use desired and pending.
Used Beagle HPC resources (~50,000
hours) as a precursor to Kraken.

Key ExTENCI capability for this application:

Swift; G
lidein
WMS; flocking from
UC3 to additional resources

The NSF RDCE
P

project (
award number 0951576)
develops a large
-
scale integrated modeling
framework for decision makers in climate and energy policy. The project is using Swift to study
land use, land cover, and the impacts of climate change on agriculture and the global food supply
using a crop syste
ms model called DSSAT (“decision support system for agro
-
technology
transfer”). Benchmarks of this model were performed on a simulation campaign measuring yield
and climate impact for a single crop (maize) across the US with daily weather data and climate

model output spanning 1981
-
2100 with 16 configurations of fertilizer, irrigation and cultivar.
Initial results have been presented in an advisory board meeting of the RDCEP project. Each
simulation runs 125,000 DSSAT models using Swift. The project has

recently moved these runs
from the Beagle Cray system to the UC UC3 OSG campus grid system and the OSG engage VO
using Swift and the GWMS workload management system. The RDCEP project plans to rely
mostly on OSG and UC3 computing resources for their ongoi
ng scientific production.

The RDCEP team is participating in a fast
-
track research program with the Agricultural
Modeling Intercomparison and Improvement Project (AgMIP) and the Inter
-
Sectoral Impact
Model Intercomparison Project (ISI
-
MIP). This coordinate
d multi
-
model comparison will be
based on CMIP5 climate model data, which is only now being released by most climate
modeling groups. The group will be using OSG+UC3 via Swift to develop high
-
resolution
gridded global impact results, targeting publications

by January 2013 for inclusion in the IPCC
AR
-
5 report.


Science Domain:

Biological science: sociology of science, collaboration network

Resources Used:

OSG: ~10,000 CPU hours
XSEDE: <1,000 SUs

Key ExTENCI capability for this application:

Swift; G
lide
in
WMS; flocking from
UC3 to additional resources; ability to mix MTC and HPC resources (using OpenMP on
HPC systems).


ExTENCI Project Annual Report 201
1
-
201
2



Page
9

NSF Co
-
PI Andrey Rzhetsky is testing scripts that use Swift for his project “Assessing and
Predicting Scientific Progress through Computat
ional Language Understanding” (NSF
award
number 0915730)
. The project develops general methods relevant for students, policy makers
and scientists. The goal is to generate dynamic high fidelity maps of knowledge claims in
chemistry and related fields su
ch as pharmaceuticals and toxicology. Their initial use of Swift
involves optimization of parameters for graph
-
based models of scientific collaboration.

A typical optimization run of this graph
-
based simulation is about 115,200 CPU hours. We
expect to pe
rform approximately 20 such simulations per year, requiring 2.3M CPU hours per
year. Some large fraction of that would be from OSG (Estimate: ~1.5M hours with the
remainder coming from the Beagle Cray system).

2.1.2

Project Activities

of

Distributed File System

The goal of this activity is to provide
Distributed file systems
operating across wide area
networks that simplify access to data in the OSG and TG
/XSEDE
. This distributed storage
mechanism will significantly improve access by U.S.
Tier 3
universities to

Large Hadron
Collider data.

Coordination and training activities:



The web site
https://wiki.psc.edu/twiki/viewauth/KerbLustre/ExTENCIProjectWithOSG

(requ
ires login/passwo
rd to access)
continues to be updated to provide a repository for
secure communications among all Distributed File Systems staff. Documentation for all
phases of the project is stored here. Major updates this year include results of extensive
application
s testing.



The e
xtenci
-
tech
@psc.edu

mailing list
continues to be used for communication among
staff.
Weekly status meeting

conference calls
continue

to help coordinate the efforts of
the staff setting up the file system and the staff who would be using it

for secure
communications. Notes from these meetings are available on the wiki.

Software, test facilities, and testing:

The effort for the past year was mainly focused on application testing. In addition, the packaging
for the Lustre software was update
d to include a new version of Lustre and to provide more
virtual machine (VM) configurations. New sites, Florida State University (FSU) and Florida
International University (FIU)
,

were incorporated into the testing and some of the project
hardware was upd
ated.

In the area of applications testing, the project performed benchmarking on local systems and
virtual machines provided by PSC at the PSC location in Pittsburgh. The CMS application
continues to be the primary test application. Testing of CMS client
s occurred at all of the
ExTEN
C
I sites: FNAL, FSU, FIU, and UF.

New for this year was the incorporation of LQCD into the LustreWAN testing. Tests of multiple
clients running LQCD were performed and showed good scaling as additional clients were
added. Al
so new this year wa
s testing the ATLAS application.

We chose

to use the CernVM
File System (CVMFS) in conjunction with ATLAS and Lustre to help overcome some
performance issues. CVMFS was ported into the PSC distribution stack. FNAL created
documentatio
n for CVMFS.


ExTENCI Project Annual Report 201
1
-
201
2



Page
10

To further understand performance issues, UF tested Lustre performance with the Cern ROOT
package.

During the past year a new version of Lustre, version 2.1, was announced by Whamcloud. The
PSC upgraded all of the packaging, including the VM
packages to include the new version. This
included getting the new version working with CentOS 5.7 and Scientific Linux 6.1. All clients
were upgraded to version 2.1.5.4. Interoperability testing was performed to determine version
compatibility during t
he upgrades.

The project was successful increasing the number of supported VM images. Images for the VM
systems XEN, VirtualBox, VMWare, and KVM were completed. This represents all major VM
systems.

The VM images contain Kerberos, Lustre, and the applica
tions as well as configuration files and
instructions.

Implementation and documentation was completed for Lustre Access Control Lists (ACL's).
Lustre quotas were enabled. A Lustre tunables document was developed for both clients and
servers.

In order to

better manage the user account structure, account synchronization between
the client sites and the UF server was developed and deployed.

To address concerns over stability of the Kerberized Lustre software, discussions were held at
the Lustre

User Group meeting with Whamcloud, the Naval Research Laboratory (NRL), who
have an interest in using Kerberized Lustre, and PSC, representing ExTENCI. PSC has created a
testing Kerberos realm to investigate the problems. Whamcloud has agreed to include

Kerberized Lustre testing in their mainstream testing code.

PSC continues to evaluate PKINIT for allowing keys and certificates to be exchanged between
organizations. PKINIT has been configured and tested on PSC VM test servers using a PSC
testbed kerber
os realm FROG.PSC.EDU.

During the course of the year, wide area network testing was conducted on a continuing basis to
monitor the network health and help diagnose performance issues. The University of Florida
conducted local tests to evaluate and improve

the performance of the Lustre servers at UF.
Tuning parameter settings were established and documented. These parameter settings are
included with the VM images.

Various pieces of the project hardware infrastructure were upgraded this year. UF receiv
ed new
networking hardware. FIU installed new networking hardware and a new server. FNAL made a
Lustre Object Storage Server (OST) available to the project. This is the first instance of a
distributed OST in the project.

During routine performance testi
ng, poor results at the UF server were eventually traced to a bad
power supply in a RAID shelf. Resolving this very obscure problem required extensive
diagnostic testing by the team.

The project web page for monitoring the sites was updated to include FSU

and FIU. The page is:
http://wanpage.extenci.org

All project activity is reviewed during weekly meetings hosted by PSC and documented on the
project private wiki at:
https://wiki.psc.edu/twiki/view/KerbLustre/ExTENCIProjectWithOSG
.
This site includes detailed documentation and results from all the application benchmarking.


ExTENCI Project Annual Report 201
1
-
201
2



Page
11

The project has written papers on this work and presented th
em at the CHEP12 and XSEDE12
conferences as listed in Section 5 Conference Proceedings.

2.1.3

Project Activities

of
Virtual Machines

The Virtual Machines project goals are to deploy consistent hypervisors on
large scale
OSG and
TG/XSEDE

resources and validate
that
science VMs can be executed using those resources.

It is
also adapting
Glidei
n
WMS technology to provide efficient pilot job startup of large numbers of
VM
s

across various cloud provider
s, including OSG, TG/XSEDE
, and Amazon EC2.
Additionally, the Vi
rtual M
achines effort will develop tools to enable cross
-
cloud management
of virtual machine images, reach out to potential user communities, and document our
experiences with this new technology.

Clemson:

The Clemson team has been working on the use of th
e OpenFlow protocol in its cloud computing
resources (OneCloud). The aim is to provide advanced networking services for users, in order to
configure VM instances and make them appear as seamlessly integrated as possible with the
user’s own resources. Initi
al work has been on ‘Elastic IP’

and ‘Security Group’, services named
by Amazon for the leasing of public IP address, its dynamic/late binding association with
running instances and dynamic firewall. These two functionalities are now in production in
Clems
on’s OneCloud. A user is able to start VMs, lease Clemson public IPs and associate them
on the fly with the instances. Documentation and screencasts are available
(
http://sites.google.com/site/cuonec
loud/
). This new capability

has been tested by Massimo
Canonico (FutureGrid user) and Owen Synge from DESY (Germany).

These two new functionalities are also being tested by
Mike Wilde’s group
(UC)
working on the
use of SWIFT
for

SCEC and
by STAR
.


Softwar
e developed has been contributed back into the
OpenNebula 3.4, released
in
early April.

Security Groups are been included in the code base and
release is planned for OpenNebula 3.5.

This work is also described in a blog entry on the
OpenNebula web site (
http://blog.opennebula.org/?p=2695
).

Clemson presented this work with Elastic IP and Security Groups at the First GENI Research
and Educational Experiment Workshop in Los Angeles, CA in conjunction with G
EC13.
Another paper
was

accepted at
the
VTDC workshop during HPDC in June 2012. Further work in
the use of Openflow in the cloud will result in more advanced network services, such as dynamic
load balancing, virtual private cloud and efficient data transfe
r between sites. In May of 2012,
the Clemson team presented its cloud computing work at the Computing in High En
ergy Physics
(CHEP) conference. See Section 5, Conference Proceedings.

Purdue:

As
an added Year 2 deliverable
, the Purdue team is developing Cl
oud Dashboard, a tool for
easier access to Wispy through a graphical user interface. Using the Dashboard, users can start,
stop and interact with the VMs running on Wispy in a browser.

An X11 version has been
completed, and the team is adapting it to an o
nline version that will work in a web browser,
within the Springboard HUB environment. The Purdue team automated the authentication
process on Wispy to make it easier for users to access. This authentication process has been
integrated into the Cloud Dashb
oard. The team also successfully integrated the U
W
’s VM
authoring and management tools (the early February version) into the Dashboard and tested it

ExTENCI Project Annual Report 201
1
-
201
2



Page
12

with Wispy.

The Purdue team is currently (July) in touch with FutureGrid and Nimbus teams to
demonstrate t
he tool.

During a visit by the ExTENCI project manager on Feb. 14, the Purdue team demonstrated the
prototype Cloud Dashboard using the integrated Wisconsin’s VM tool for staging VM images
into Wispy.

The team gave an overview of several nanoHUB

applications that run on Wispy and
the comparison with running on other resources (comparable performance for these
applications). Although the team demonstrated and shared some initial usage and charts of the
CMS VMs running at Purdue, a more general sol
ution for running the CMS VMs will be the use
of the GlideinWMS factory
. T
his work
was

on hold until the Fermilab GlideinW
MS team was
available to proceed
. The Purdue team has re
-
engaged with the CMS GlideinWMS team and has
made great progress integrating

Wispy and Nimbus clouds into the GlideinWMS infrastructure.

The Purdue team also worked with Clemson to get BOINC CernVMs (LHC work) running
successfully
on
Wispy.

During the
winter

of 2011, the Purdue team collaborated with the
NanoHUB.org science gatew
ay to study the utility of Cloud resources for the NanoHUB, and the
results were
published

at the 2011 IEEE Utility and Cl
oud Computing (UCC) conference (see
Section 5 for reference).

Purdue’s Wispy system was moved to a new data center in February as the
previous data center
housing Wispy was closed due to a power and cooling savings plan for IT on the West Lafayette
campus of Purdue University.

By the end of the summer, Wispy will undergo a hardware
upgrade to servers based on Intel's Sandy Bridge proces
sor architecture.

Wisconsin
:

The Wisconsin team completed support for staging and de
-
staging on Purdue's Wispy and
Clemson's OneCloud services in its VM management tool, and started support for control
operations on these services.


It also incorporated us
eful feedback from Purdue's experience
using the tool. The team had FutureGrid's Eucalyptus, Nimbus, and OpenStack services
approved as the final component of the corresponding work item. Wisconsin also helped to
define FutureGrid's requirements for a form
al ExTENCI project definition.


The OpenStack
integration has progressed to being approximately half
-
completed, and preliminary work was
started on the other two services. Additionally, the team has completed a draft of the cloud
informati
on services repor
t

and is in process of collecting feedback on the report from the VM
team.


The team has also begun to develop a survey of stakeholders to provide more focused
requirements and motivating scenarios to guide further development of the cloud information
serv
ice.
The deliverable to “d
emonstrate that a CMS VM can be authored and placed in a library
for different cloud services, and then distributed to and run in production volume at 5 different
TG and OSG sites using 3 cloud technologies, including the Purdue
c
loud and the FutureGrid
testbed”

is waiting on the completion of the GlideinWMS CMS VM.

The deliverable to “
Determine and document what information VOs require to decide where (on
which sites) to run th
eir VMs (and hence their jobs)"

has been drafted and
sent to some
stakeholders, but with little response.

The ExTENCI VM team has recently received a request from the STAR project which needs
massive resource for 5k job slots for 4 months for a near real
-
time data production (Run 12 data).
After some discuss
ion with the STAR team, it was determined that resources were not available
at sufficient scale to service this request.


ExTENCI Project Annual Report 201
1
-
201
2



Page
13

2.1.4

Project Activities

of Job Submission Paradigm
s

The primary project activities have been, (i) completion, hardening

and deployment of SA
GA
-
Condor adaptor, (ii) testing on
OSG
, (iii) Integration of SAGA
-
Condor adaptor within the
BigJob

Pilot
-
Job mechanism and integration of Condor enabled BigJob within

the DARE
framework of the Cactus Gateway, and (iv) support of

application groups
--

Cactu
s
-
based
n
umerical Relativity and

Computational Biology groups.

As a result of the production availability of BigJob, five science projects consumed 3.25 Million
SUs on XSEDE. One of these, Molecular Biophysics, used resources at multiple XSEDE sites,
whic
h is rare.

Other production science work that directly resulted from ExTENCI was:

Science Domain
: Data
-
Intensive Computational Biology (NGS Alignment)

Tools developed as part of ExTENCI have been used for Next
-
Genergation

Sequencing (NGS)
data analytics. Two most prominent problems to which Pilot
-
Job based approaches have been
employed in NGS are RNA
-
Seq and ChIP
-
Seq. With Dr. Chris Gissendanner, Department of
Basic Pharmaceutical Sciences, College of Pharmacy, Universit
y of Louisiana at Monroe, a
pipeline for ChIP
-
Seq has been developed. For RNA
-
Seq, the fusion gene discovery, important
for cancer research, is now available with the support of TopHat
-
Fusion.

2.1.5

Overall ExT
ENCI P
roject Evaluation of Progress

The ExTENCI pro
ject is being managed through the use of Statements of Work (SOW)
consisting of the goals, deliverables, staffing level, and funding that is agreed to by each funded
institution. These SOWs are in place for each of the eight funded partner institutions li
sted in
Section 1.2.

We have modified a few deliverables and added a few in the second year of the project.

Last
year we listed 119 deliverables and now t
here are a total of
12
3

deliverables for ExTENCI. We
expected to complete
111

of these
by 6/30/12
.
The status of these
11
1

deliverables is
characterized in the following table:

Project

Institution

Number of
Deliverables
Due by 6/30/12

#
100%
Done

#

75
-
99%
Done

#
<75%
Done

Distributed
File System

PSC

31

23

8


UF

22

22



FNAL

15

8

2

5

Virtual
Machines

Purdue

5

3

2


Clemson

6

3

1

2

UW

11

10


1

Workflow &
Client Tools

UC

12

10


2

Job Submission
Paradigms

LSU/Rutgers

9

7

1

1

Totals

All

11
1

86

14

11

% Complete



77%

13%

10%


ExTENCI Project Annual Report 201
1
-
201
2



Page
14

Overall, the project is behind schedule

but in better shape than in the last annual report

(now
90% are >75% complete vs. 72% last year)
.

We plan to complet
e the objectives over the next 5

months with the no
-
cost extension.


2.2

Project
Findings

2.2.1

Project Findings of Workflow & Client Tools

Finding
1: The Swift parallel scripting language can utilize the OSG GlideIn
WMS
to efficiently
get computing resources on Open Science Grid, and enable scientists to efficiently compute in a
location
-
independent manner with high
-
level script
s that specify their wo
rkloads.

Finding 2: Swift can perform efficient data staging directly from client storage systems to local
hard disk filesystems on OSG compute nodes. Efficiently leveraging OSG storage elements
(which are based on SRM and DCcache, and use clients like lcg
cp) should be possible in the
future with recent flexibility added to Swift by ExTENCI work.

Finding 3: The installation and verification of complex software stacks such as those required by
the SCEC Broadband application are still complex in the OSG envir
onment. These are much
easier for application groups to accomplish

on VM
-
equipped computed nodes.

Finding 4: Swift can efficiently utilize VM systems such as FutureGrid (which we have
demonstrated at XSEDE
-
2012), and can thus also leverage XSEDE resources
such as Wispy
(which we seek to demonstrate under current ExTENCI funding).

Finding 5: Swift scripts can use a variety of application attributes to route jobs with specific
requirements (such as dedicated parallel nodes for OpenMP or MPI applications) to s
pecific sites
(such as XSEDE Ranger).

2.2.2

Project Findings of Distributed File System

Finding 1: A Kerberized Lustre filesystem can be successfully deployed to Tier 3 universities
with access to data from Tier 2 sites
.

Finding 2: Distributed OST pools
can be
deployed across a WAN.

Finding 3: Kerberos can extend security to universities but different implementations and
security levels are problematic.

Finding 4: There was a lack of Kerberos test facilities and lack of testing of Kerberos from
Lustre developmen
t. These problems have been reduced by creation of a Kerberos test facility at
PSC and by getting agreement from Whamcloud to include Kerberos testing for their Lustre
releases.

Finding 5
: For large files, benchmarking has showed good scalability across t
he WAN for the
CMS and LQCD science applications.

2.2.3

Project Findings of Virtual Machines

Finding 1
: The NanoHUB.org science gateway has demonstrated that interactive
nanotechnology simulations can be run in a cloud computing environment.

Finding 2
:
The
cur
rent state of VM authoring and submission

has been researched and

a
document has been written summarizing the
results

(requires login/password).
Additional

ExTENCI Project Annual Report 201
1
-
201
2



Page
15

deliverables (relative to the original proposal) have been defined to simplify user access and use
of cloud environments:



A database catalog of supported cloud environments, submission endpoints, and
corresponding VM formats



A query tool for que
rying this database and authoring and submitting appropriate VM
image.



A virtual machine image usable by the CMS experiment that can be submitted to selected
cloud providers
. This image, while possible on a proof
-
of
-
concept scale, proves to be
unwieldy due

to the size and complexity of the software stack. For real
-
world use, we
have collaborated with GlideinWMS team to integrate with CMS submission paradigms.



A graphical client tool to easily upload, start, stop, manage, and access custom user
-
submitted VMs
.

Finding 3: A technique was developed to provide external Internet access from virtual machines
in a cloud sitting in a secure network.

2.2.4

Project
Findings

of Job Submission Paradigms

Finding 1
:

As part of this project, we have developed SAGA
-
Condor
adaptors.

The

outcome of
this phase of the project is that there now exists, a

single common standards
-
based approach to
using heterogeneous

infrastructure.


Finding 2:
The development and deployment of a SAGA
-
based Pilot
-
Job has resulted

in an
underst
a
n
ding of the limitations of "common abstractions" have.

Specifically, even though
SAGA
-
based Pilot
-
Jobs have the ability to

utilize OSG and XSEDE concurrently, the "usage
modes" supported on OSG

and XSEDE are different, and thus imposing implicit constrai
nts on
the

usability of the pilot
-
jobs.

This is not a case for removing or abandoning "common
abstractions"

but a reminder/lesson that the end
-
to
-
end usage of tools and

abstractions must be
considered, i.e., development, deployment and

execution phase.

2.3

Tr
aining and D
evelopment

2.3.1

Training and Development in

Workflow & Client Tools



D. S. Katz delivered training on

"ExTENCI: Usin
g multiple cyberinfrastructures
"
at the
MAPPER Seasonal School, London, UK, 2012.



Trained new developer D. Kelly in use of OSG, XSEDE,

GRAM, GridFTP, and Glide
-
in
workload management system.



RDCEP graduate student M. Glotter attended OSG Summer School 2012 to learn OSG
basics for execution of DSSAT workflows.



Graduate students G. Hocky and A. Adhikari and postdoc M. Parisien trained in u
sing
OSG, XSEDE, and Swift.

2.3.2

Training and Development in
Distributed File System



No training was planned in year 2.


ExTENCI Project Annual Report 201
1
-
201
2



Page
16

2.3.3

Training and Development in
Virtual Machines



No training was planned in year

2
.

2.3.4

Training and Development in
Job Submission Paradigms



Extensive and self
-
contained BigJob documentation and how to use

BigJob on OSG and
XSEDE



In conjunction with ECSS support (XSEDE) we developed and

delivered a BigJob
tutorial at XSEDE
12.

3.

Publications and Products

3.1

Journal publications

Publications Involving

ExTENCI work

1.

L. Stout, M. Walker, J. Lauret and S. Goasguen “Using Kestrel and XMPP to Run Scientific
Applications
in the Cloud” Submitted to the J
ournal of Grid Computing, May 2011.

2.

J. Lauret, M. Walker, S. Goasguen and L. Hadju “From Grids to Cloud the
STAR
Experience” Submitted to Journal of Physics, October 2010
.

3.

Glen M. Hocky, Thomas E. Markland, David R. Reichman.

Growing point
-
to
-
set length
scale correlates with growing relaxation times in model supercooled liquids



Physical

Review Letters 108, 225506(2012)


June 1, 2012.

http://prl.aps.org/abstract/PRL/v108/i22/e225506

3.2

Book(s) and/or other one time publicatio
n

No books or other publications are planned
.

3.3

Web
site or other Internet sites created

ExTENCI maintains a public website with basic project information:
https://sites.google.com/site/extenci

ExTENCI also uses a project website to track deliverables, s
tore documents, house a project
calendar, and manage/document ExTENCI project meetings. This site is available only to
members of the project:
https://sites.google.com/site/extenciproject/

3.3.1

Dist
ributed File System

The project created a Lustre WAN metric page showing the Lustre WAN servers and clients at:
http://quipu.psc.teragrid.org/wanpage/www/extenci.php

3.3.2

Virtual Machines

Cle
mson published the following web sites:



http://blog.opennebula.org/?p=2695



http://sites.google.com/site/cuonecloud


ExTENCI Project Annual Report 201
1
-
201
2



Page
17

The Springboard HUB (with cloud da
shboard
)



http://springboard.hubzero.org

3.3.3

Job Submission Paradigms

Existing
websites apply
and new websites/pages
were created:



http://saga.cct.lsu.edu/



http://en.wikipedia.org/wiki/SAGA_C%2B%2B_Reference_Implementation



http://faust.cct.lsu.edu/trac/saga



http://bit.ly/ksX3Pm

(Condor
-
Specific SAGA work items)

3.4

Other specific products

None in year 2.

4.

Contributions

4.1

Contributions within D
iscipline

Workflow & Client Tools:

ExTENCI support has contributed to improved algorithms for site
selection, data management, and resource acquisition within implicitly parallel scripting
languages.

Distributed File Systems:
The
Distributed File

System
s

project
is

contribut
ing

to Computer
Science by providing a demonstration use case of a distributed file system running in a wide area
network
s at sites separated by more than a thousand kilometers
. The project explores the issues
of multi
-
administrative domains. Solutions for
cross
-
site authentication have been explored

and
deployed. Based on extensive benchmarking across the WAN, knowledge has been developed
on tuning Lustre in a WAN environment.

Virtual Machines:


The Virtual

Machines project contributed
to the Computer Sci
ence discipline
by

publishing studies about the costs of cloud
vs.

grid computing (Purdue); and deployment of
cloud technology on batch clusters, distributing virtual machine images, cloud interoperability,
and the XMPP
-
based Kestrel cluster scheduler (Cle
mson).

An implementation of ‘Elastic IP’
and ‘Security Group’ services for leasing public IP addressed by VMs has been
deployed and
published.

Job Submission
Paradigms:


The intellectual contribution is not academic: it has led to practical

solutions/chan
ges which have resulted in pilot
-
jobs not being an

"inseparable part of the
infrastructure" but being a fundamental

abstraction which can be used to reason (either
algorithmically or via

system
-
performance information) about distribution.

As

is evidenced b
y the three sc
ience papers submitted to XSEDE
12

(of which two were accepted
at XSEDE), the use of BigJob as a

"programmable" extensible and interoperable pilot
-
job is
gaining

traction.

4.2

Contributions to Other Disciplines

In the
23

months since the start of the
ExTENCI

project
our work has contributed to
:


ExTENCI Project Annual Report 201
1
-
201
2



Page
18



Protein
Structure
:
ExTENCI
-
supported campaigns contributed to scientific results for
protein
-
RNA interaction prediction and theoretical studie
s of glassy material properties.
See
Section 3.1 Publications, Hockey.



Biochemistry / Protein Science
:
The collaborating groups of Karl Freed and Tobin
Sosnick used OSG to develop and validate new protein structure and folding pathway
prediction algorithms that start from the amino acid seque
nce, without the use of
templates.

One of their methods, called “loop modeling”, which rebuilds local protein
structures, was highly ranked in the international CASP9 competition (
Critical
Assessment of Protein Structure Prediction)
. The group used OSG an
d XSEDE to
evaluate the loop modeling algorithm on new proteins and parameter spaces, to develop a
new “tertiary fixing” algorithm, and to compare these algorithm to other approaches.

Two publications are nearing completion.



Earth System Science:
The NSF
RDCE project (award number 0951576) develops a
large
-
scale integrated modeling framework for decision makers in climate and energy
policy. The project is using Swift to study land use, land cover, and the impacts of
climate change on agriculture and the g
lobal food supply using a crop systems model
called DSSAT (“decision support system for agro
-
technology transfer”).
B
enchmarks of
this model were performed on a simulation campaign measuring yield and climate impact
for a single crop (maize) across the US

with daily weather data and climate model output
spanning 1981
-
2100 with 16 configurations of fertilizer, irrigation and cultivar. Initial
results have been presented in an advisory board meeting of the RDCEP project.



Biological science: sociology of s
cience, collaboration network
:
NSF Co
-
PI Andrey
Rzhetsky is testing scripts that use Swift for his project “Assessing and Predicting
Scientific Progress through Computational Language Understanding”


(NSF award
number 0915730). The project develops genera
l methods relevant for students, policy
makers and scientists. The goal is to generate dynamic high fidelity maps of knowledge
claims in chemistry and related fields such as pharmaceuticals and toxicology. Their
initial use of Swift involves optimization

of parameters for graph
-
based models of
scientific collaboration.



Data Intensive Computational Biology (NGS Alignment):
Tools developed as part of
ExTENCI have been used for Next
-
Genergation Sequencing (NGS) data analytics.

Two
most prominent problems t
o which Pilot
-
Job based approaches have been employed in
NGS are RNA
-
Seq and ChIP
-
Seq.

With Dr. Chris Gissendanner, Department of Basic
Pharmaceutical Sciences, College of Pharmacy, University of Louisiana at Monroe, a
pipeline for ChIP
-
Seq has been devel
oped.

For RNA
-
Seq, the fusion gene discovery
which is important for cancer research is now available with the support of TopHat
-
Fusion.



Earthquake Research:

We have shown that earthquake simulations can be partitioned to
the
best
-
suited

part of the cyber
infrastructure (HPC and HTC), but have not yet
completed any production simulations.



CMS
:
We are building the capability for providing access to any CMS data to T3 CMS
sites without mo
ving all the data to the site although only test
runs have been made.
W
e
are building
a CMS virtual machine image suitable for deployment on cloud facilities
.

ExTENCI Project Annual Report 201
1
-
201
2



Page
19

We are now

building
facilities to create, manage, distribute and run large numbers of
CMS
v
irtual
m
achines, but have not yet done production runs. (This is an added
deliverable to the ExTENCI project.
)



High Energy Physics
-

STAR:
The primary physics task of STAR is to study the
formation and characteristics of the quark
-
gluon plasma (QGP), a state of matter believed
to exist at sufficiently high energy densities. Ove
r the course of a month, over eighty
thousand tasks were executed on ExTENCI
-
supported cloud infrastructure, generating
more than twelve billion events using PYTHIA, a proton simulation event generator. A
subset of the events went through detector simulat
ion using a GEANT3 based package.
Finally, STAR reconstruction software was used to simulate a trigger on the remaining
events. In all, the simulation ran for over 400,000 CPU hours. Nearly seven terabytes of
results were transferred back to Brookhaven
National Laboratories for further study.
During the month the simulation took place, the CPU hours used amounted to an
expansion of approximately 25% over STAR’s average capacity. Furthermore, the STAR
group has stated that it would have been only able to

devote fifty CPUs to this task on its
own farm, increasing the real time to completion by a factor of twenty.



Nanotechnology: We provided computing resources to the NanoHUB.org science
gateway to enable user simulations and study the feasibility of utiliz
ing the cloud.

4.3

Contributions to Education and Human Resources

ExTENCI support was instrumental in the training of new Swift team member David Kelly,
postdoc Ketan Maheshwari (now at Cornell) and graduate student Allan Espinosa.

Jha

taught a course on Distributed Scientific Computing at LSU in Fall

201
1

(and will do so
again in 2012 Fall), as well as several

independent studies at Rutgers University. In each of these
classrooms

as well as research
-
credit courses, SAGA and BigJob were

introduced.

Also, as part
of these courses Jha devotes a 1.5 hour lecture to

introducing OSG and XSEDE and what kind of
computation each supports

as well as the software environment associated with these

infrastructure
s
.

4.4

Contribution to
Resources for
Res
earch

and
Education

Lustre servers have been set up at UF. Lustre client systems mounting the file system from those
servers have been set up at UF, F
RA
, and PSC. All sites are using Kerberos authenticated access
with the EXTENCI.ORG realm, which was cre
ated for this project.


The site
https://wiki.psc.edu/twiki/viewauth/KerbLustre/ExTENCIProjectWithOSG

(userid
required)
contains a documentation section with nearly twenty

articles on topics such as:

Setting up Servers and Clients



Rpm requirements




Checklist




Installing Kernel and Lustre RPMS




Setting up patchless clients on Scientific Linux




Merging keytab entries




Debugging


Client
-
specific



Single Realm, Clients on Different domain



ExTENCI Project Annual Report 201
1
-
201
2



Page
20



Mixed Realm Clients


Server
-
specific



Create keytabs




Changing lustre kerberos flavor


SLASH2



Client Cross
-
Realm
Testing




Lustre ACL




Lustre quota




Lustre ext4




Kerberized Data Transfers: GridFTP, GSI
-
scp, Kftp


Purdue University has deployed a cluster running the Nimbus software (“Wispy”) that is
available for cloud

users. Computer Science students from Marquette University have utilized
Wispy during laboratory work studying cloud computing.

Clemson University has deployed a VMIC that synchronizes with one in operation at CERN.

4.5

Contributions
B
eyond Science and E
ngine
ering

No contributions beyond science and engineering.

5.

Conference Proceedings

1.

Ket
an Maheshwari, Allan Espinosa,
Daniel Katz,
Michael Wilde,

Zhao Zhang,

Ian
Foster,

Scot
t Callaghan,

Phillip Maechling “
Job and Data Clustering for Aggregate Use of
Multi
ple P
roduction Cyberinfrastructures”
Workshop
on

Data Intensive Di
stributed
Computing (DIDC 2012)
i
n conjunction with HPDC 2012, June 18
-
22, Delft, Netherlands

2.

R. Wartel, T. Cass, B. Moreira, E. Roche, U. Schwickerath and S. Goasguen “Image
Distribution i
n Large Scale Cloud Providers” IEEE Second International Conference on
Cloud Computing Technology and Science (CloudCom),
Nov. 30
,

2010
-
Dec. 3
,

2010
.

3.

S. Goasguen “Interoperability in the Clouds” Computing Seminar, Fermi National
Laboratory, January 14th 2
011,

4.

S. Goasguen “Building IaaS Clouds for Scientific Applications”, Petascale Data Analytics on
Clouds: Trends, Challenges and Opportun
ities, ACM/IEEE Supercomputing C
onference,
November 14th, 2010, New Orleans, LA, USA.

5.

J. Lauret, M. Walker, S. Goasguen
“When STAR meets the Clouds: Virtualization and
Cloudified Grids Experience” Computing in High Energy and Nuclear Physics, CHEP 2010,
Taipei, Taiwan
.

6.

S. Goasguen, M. Guijaro, B. Moreira, E. Roche, U. Schwickerath, R. Silva and R. Wartel
“Integration of Vir
tual Machines in the Batch System at CERN” Computing in High Energy
and Nuclear Physics, CHEP 2010, Taipei, Taiwan
.

7.

A. Carlyle,

S. Harrell, and P. Smith "Cost
-
Effective HPC: The Community or the Cloud?"

IEEE Second International Conference on
Cloud Computi
ng Technology and Science
(CloudCom)
, pp.169
-
176, Nov. 30
,

2010
-
Dec. 3
,

2010
.


ExTENCI Project Annual Report 201
1
-
201
2



Page
21

8.

Allan Espinosa, Daniel Katz, Michael Wilde, Ian Foster, Scott Callaghan, and Phillip
Maechling
, “
Data
-
intensive CyberShake Computations on an Opportunistic
Cyberinfrastructure
,”
TG’11 Conference.

9.

G. Stabler, A. Rosen, K.C. Wang and S. Goasguen “OneCloud: Controlling the Network in
an Openflow Cloud” 13th GENI Engineering Conference, March 13th
-
15th 2012, Los
Angeles, CA

10.

Zentner, L.K.; Clark, S.M.; Smith, P.M.; Shivarajapura, S.;
Farnsworth, V.; Madhavan,
K.P.C.; Klimeck, G
,

"Practical Considerations in Cloud Utilization for the Science Gateway
nanoHUB.org," Utility and Cloud Computing (UCC), 2011 Fourth IEEE International
Conference on , vol., no., pp.287
-
292, 5
-
8 Dec. 2011

11.

S. Goa
sguen “Middleware Evolution: from Grids to Clouds, a non
-
HEP Perspective”;
Computing in High Energy Physics (CHEP), May 2012

12.

A Luckow, M Santcroos, O Weidner, S Maddinenei, A Merzky and S Jha,

Towards a
Common Model for Pilot
-
Jobs
”,

Poster Paper, Proceedi
ngs of the HPDC 2012 (Delft)

13.

P Mantha, A Luckow and S Jha, “Pilot
-
MapReduce: An Extensible and Flexible MapReduce
Implementation for Distributed Data”, Third International Conference on MapReduce,
Proceedings of the HPDC 2012 (Delft)

14.

P Mantha, J Kim, N Kim
, A Luckow and S Jha, “Understanding MapReduce
-
based Next
-
Generation Sequencing Alignment on Distributed Cyberinfrastructure”, Emerging
Computational Methods for the Life
-
Sciences, Proceedings of the HPDC 2012

15.

Thota, Mukherji, Fujioki, Bishop and Jha,
"Running Many MD Simulations on Many
Supercomputers", Accepted for XSEDE'12

16.

Romanus, Khamra, Mantha, Merzky, Bishop, Jha, “Anatomy of a Successful ECSS Project”,
Accepted for XSEDE'12

17.

A Luckow, M Santcroos, O Weidner, P

Mantha, A Merzky and S Jha, “P*

A Mo
del for Pilot
-
Abstractions

, submitted, 8th IEEE e
-
Science Conference, Chicago

18.

J. Palencia, R. Budden, K. Benninger, D. Bourilkov, P. Avery, M. Cheng, Y. Fu, B. Kim, D.
Dykstra, N. Seenu, J. Rodriguez, J. Dilascio, D. Shrum, J. Wilgenbusch, D. Oliver, D.
Majchzak, "Using Kerberized Lustre over the Wide Area Network for High Energy Physics
Data", Proceedings of the 2012 XSEDE Conference ACM ISBN: 978
-
1
-
60558
-
818
-
6,
Chicago, IL, USA

19.

J. Palencia, R. Budden, K. Benninger, D. Bourilkov, P. Avery, M. Cheng, Y
. Fu, B. Kim, D.
Dykstra, N. Seenu, J. Rodriguez, J. Dilascio, D. Shrum, J. Wilgenbusch, D. Oliver, D.
Majchzak, "Using Kerberized Lustre over the Wide Area Network for High Energy Physics
Data", Proceedings of the (LUG) Lustre User Group 2012, April 24
, 2012, Austin, TX, USA

20.


D. Bourilkov, P. Avery , M. Cheng, Y. Fu, B. Kim, J. Palencia, R. Budden, K. Benninger, J.
Rodriquez, D. Dykstra, N. Seenu, "Secure Wide Area Network Access to CMS Analysis
Data using the Lustre Filesystem," Computing in High Ene
rgy and Nuclear Physics (CHEP)
2012, New York, USA

21.

D. Bourilkov, P. Avery
, M. Cheng, Y. Fu, B. Kim, J. Palencia, R. Budden, K. Benninger,

Using Virtual Lustre Clients on the WAN for Analysis of Data from High Energy

ExTENCI Project Annual Report 201
1
-
201
2



Page
22

Experiments
,”


Computing in High Energy

and Nuclear Physics (CHEP) 2012, New York,
USA

22.

K. Maheshwari, A. Espinosa, D. S. Katz, M. Wilde, Z. Zhang, I. Foster, S. Callaghan, and P.
Maechling, "Job and Data Clustering for Aggregate Use of Multiple Production
Cyberinfrastructures," Proceedings of F
ifth International Workshop on Data Intensive
Distributed Computting (DIDC'12).

23.

A. Espinosa, D. S. Katz, M. Wilde, K. Maheshwari, I. Foster, S. Callaghan, and P.
Maechling, "Data
-
intensive CyberShake Computations on an Opportunistic
Cyberinfrastructure," T
eraGrid 2011, 2011.

24.

D.S. Katz, "ExTENCI: Extending Science Through Enhanced National CyberInfrastructure:
project status update," Open Grid Forum, 31st quarterly meeting, Academia Sinica Taipei,
Taiwan, March 21
-
25, 2011

25.

D.S. Katz, "ExTENCI: Extending Scie
nce Through Enhanced National CyberInfrastructure:
project status update," Open Grid Forum, 32st quarterly meeting, Salt Lake City, Utah USA,
July 15
-
18, 2011

26.

D.S. Katz, "ExTENCI: Extending Science Through Enhanced National CyberInfrastructure:
project sta
tus update," Open Grid Forum, 33rd quarterly meeting, Lyon, France, September
19
-
23, 2011

27.

D.S. Katz, "ExTENCI: Extending Science Through Enhanced National CyberInfrastructure:
project status update," Open Grid Forum, 34th quarterly meeting, Oxford, UK, Mar
ch 12
-
15,
2012

6.

Special Requirements


None.




ExTENCI Project Annual Report 201
1
-
201
2



Page
23

Appendix
A
: Cyberinfrastructure Usage by ExTENCI

These graphs illustrate the ExTENCI usage of the OSG and XSEDE over the last year.