Thoughts from the east, on research in the cloud

dizzyeyedfourwayInternet and Web Development

Nov 3, 2013 (3 years and 9 months ago)

66 views

NeCTAR Research Cloud Workshop
30 – 31 March 2011, Melbourne, Australia
Nick Jones
n.jones@auckland.ac.nz
www.eresearch.auckland.ac.nz
Thoughts from the east,
on research in the cloud
Genomics, Genetics,
Bioinformatics, Molecular
Modelling:
NZ Genomics Ltd
Maurice Wilkins Centre
Alan Wilson Centre
Virtual Institute of Statistical
Genetics
Wind Energy, Geothermal &
Minerals Exploration:
• GNS Exploration
• Institute for Earth Science and
Engineering
• Centre for Atmospheric
Research
Earthquakes, Tsunami,
Volcanoes:
• Natural Hazards Research
Platform
• DEVORA Auckland Volcanic
Field
• GeoNet
Invasive Species, Water / Land Use,
Emissions:
• Agricultural Greenhouse Gas
Centre
• Bio-Protection Research Centre
• National Climate Change Centre
Human Development, Bioengineering,
Social Statistics:
• National Research Centre for Growth and
Development
• Auckland Bioengineering Institute
• Liggins Institute
• Malaghan Institute
• Social Sciences Data Service
Nanotechnology and High
Technology Materials:
MacDiarmid Institute
Materials TRST
Roadmap of NZ e-Infrastructure
National eScience
Infrastructure
BeSTGRID – Compute / Data Services
NZ Genomics Ltd
BeSTGRID Federation
Tuakiri – Research Access Federation
Research Data
Infrastructure
KAREN Advanced Research Network
Weather / Atmosphere NIWA HPC – P575
BlueFern BlueGene/L
Genomics
Data
Identity
HPC
eScience
Network
NZs eResearch Infrastructures
HPC & Distributed Computing
BeSTGRID, since 2006
• $2.5M + $800k
• Largest national grid users in
Australasia

Built capability, established a
model for sharing resources
NZ eScience Infrastructure (NeSI)
• ~$50M ($27M Crown)
• 4 HPC centres across NZ
• Signing agreements now, running
for 4 years > review > ongoing…
NZ Genomics Ltd (NZGL)
• National genomics service
• 4 major partners
• Funded in 2008

Almost operational..
NZ Genomics Ltd
Centralised
Cloud (?)
~$3.5M
Services
federated
with NeSI
Science Users
Principal Investors
UJV
Procurement Planning
Operational management
Data & Grid middleware to manage
access
Outreach
NIWA
• People
• HPC capacity
• Storage
• Portion of facilities,
power, depreciation costs
Auckland
• People
• New cluster
• New storage
• Portion of facilities, power,
depreciation costs
Principal and Associate Investors
• Access through a “single front door”
• Specialised concentrations of capability at each institution
• Receive government coinvestment
• Capacity available to institutions reflects their level of investment
• Managed and metered access to resources, across all resource types
• Institutions’ access costs covered by investment
Services
Canterbury
• People
• HPC
• New storage
• Portion of facilities, power,
depreciation costs
Government
• Invest over 4 years, plus out-years
AgResearch / Otago
• People
• New cluster
• New storage
• Portion of facilities, power,
depreciation costs
Managed Access
Legend
Financial flows
Access
Research institutions (non investors)
• Access through a “single front door”
• Capacity scaled out from partners capabilities
• Managed and metered access to resources,
across all resource types
• Access fee initially calculated as partial costs of
metered use of resources and reviewed annually
Industry
• Access through a “single front door”
• Capacity scaled out from partners
capabilities
• Managed and metered access to resources,
across all resource types
• Access fee calculated as full costs of metered
use of resources
Governance
Principal Investors
• Transparent
• Collaborative
Cluster &
Services
HPC &
Services
HPC &
Services
Cluster &
Services
Board
(7 members)
Director
(Fulltime Position)
Team @
AgResearch
Team @
Auckland
Team @
Canterbury
Team @
NIWA
Strategic Advisory
Group
( 7 members)
Governance
Management & Services
4 Board nominated researchers
Chair (independent via UJV)
1 independent (via UJV & MoRST)
1 from each Principle Investor
1 nominated by Associate Investors
Chair
3 NZ researchers
3 offshore researchers
Management Team
Programme & Communications
Finance & Administration
User Advisory Group
Following roles are covered across all teams (in varying portions):
Manager / Technical Lead / Systems / Middleware / Applications
Director
Programme
Manager
Communications
Manager
Finance &
Administration
User Advisory Group
Management & Services
4 Board nominated researchers
Services
AgResearch
/Otago
Manager
Technical Lead
Applications
Middleware
Systems
Auckland
Manager
Technical Lead
Applications
Middleware
System
Canterbury
Manager
Technical Lead
Applications
Middleware
System
NIWA
Manager
Technical Lead
Applications
Middleware
System
SL
A
SL
A
SL
A
SL
A
Management
Core Functions
Land of the long white cloud..

Within NZGL and NeSI, we have reasonable
scale investments into virtualised
infrastructure
• Our aim is to build at least a federated cloud
platform, if not a single research cloud
(perhaps with mulitple availability zones)
Migrating users’ tools into the Cloud
Tools
Desktop/Server
*aaS
Cloud
Who are our users?
• Researchers
• HPC users
• System Administrators
• Application Developers
What are the funders expectations?
– Facebook for scientists?
– VM and development platforms
– Scaled out server infrastructure & development platform efficiencies and coordination
– National collaborative infrastructure
Needs and solutions depend on which
layer of the cloud you’re creating:
SaaS, PaaS, IaaS
*aaS Examples
Galaxy (Genetic Marker
Design)
– analysis modules

workflows
Biocommons
– multisite

sheep.biocommons.org.nz
(Genomics)
– analysis modules
– pipelines

Scale out analyses?

Self service

Customised/ templated
services
• Reuse of methods?
www.bestgrid.org
www.nesi.org.nz
technical.bestgrid.org
www.eresearch.org.nz
www.nzssds.org.nz
df.bestgrid.org
• Schedulers
• LRMS
IaaS: Compute
Elements
• Computational Libraries
• Compilers
• Grisu + Globus, VDT
PaaS:
Scriptable
Environments
• Finite Element, Fluid
Dynamics, Statistics
• Grisu clients
• GSI-SSH
• Web portals
SaaS:
Packaged
Applications
• Eucalyptus
• Nimbus
• Amazon EC2
• Storage Service
• File Service
IaaS
• Biocommons
• GenePattern
• Galaxy
• Drupal multisite website platform
• Shibboleth Federation Services
• iRODS
PaaS
• Sakai
• Bionimbus
• sheep.biocommons.org.nz, etc
• DataFabric
SaaS
Applications & Services HPC
+
Joining the Grid to the Cloud
Issues, early 2010:
• Maturity of solution
• Overhead of VM
provisioning
Cloud Workshop
18/02/2011

https://wiki.heprc.uvic.ca
/twiki/bin/view/Main/Clo
udWorkshop
Examines queue, launches
VMs on Nimbus, EC2,
Eucalyptus
Let users define the
computational image
Provide a library for
them to share,
discover, launch
GPU
GigE
Memory
Cores
Network
Storage
Clusters
Virtual
Labs
Services
Adhoc
servers
Databases
Websites
Portals
Research Cloud
GPU
Network
GigE
Memory
Cores
Storage
GPU
GigE
Memory
Cores
GPU
GigE
Memory
Cores
IaaS
PaaS
SaaS
Users Issues & Needs
What are their issues?
– Accessibility
– Usability
– Domain specificity
– Funding sources (opex, capex, none)
– Sustainability
What are their needs?
– Self service / on demand
– Applications Registry
• Ease of discovery
– Consultancy
• Migration, adaptation
– Mature Service Delivery
• Eucalyptus
• Nimbus
• Amazon EC2
IaaS
• Drupal multisite website platform
• Compute Platform
• Shibboleth Federation Services
• Biocommons
• GenePattern
• Galaxy
PaaS
• Genetic Marker Design
• Sakai
• Bionimbus
• sheep.biocommons, etc
• Grisu Compute Jobs
• DataFabric
• HPC Applications
SaaS
Operating Environments
Package
Maintenance
Systems
Management
Data
Networks
Node
Interconnects
Application/Service Development
Community
Building
Software
Engineering
Integration
Application/Service Administration
Content
Customisation
Community
Building
Derinkuyu Underground City, Cappadocia,
Turkey
• Eucalyptus
• Nimbus
• Amazon EC2
IaaS
• Drupal multisite website platform
• BeSTGRID Compute Platform
• Shibboleth Federation Services
• Biocommons
• GenePattern
• Galaxy
PaaS
• Genetic Marker Design
• Sakai
• Bionimbus
• sheep.biocommons, etc
• Grisu Compute Jobs
• DataFabric
SaaS
Operating Environments
Package
Maintenance
Systems
Management
Data
Networks
Node
Interconnects
Application/Service Development
Functional
Requirements
Software
Engineering
Integration
Application/Service Administration
Content
Customisation
Community
Development
eResearch Tools development
In the past, staff have been systems operators
and middleware developers

few business analysts and user centered designers
Who is driving product and service
development?
Who discovers needs, translates into eResearch
Tools?
Apps
Federated services
Consistent service
interfaces
University
Community
Institute
University
Community
Institute
National
Research
Cloud
Institutional services
Where are the users now?
Under their desks!
In institutional facilities
In externally supported communities
Migrating users to the Cloud
And, we want
them to join
us!
Migrating eResearch Tools
Monolithic systems aren’t compatible with HPC,
Grid, nor the Cloud
• Applications require scaling out, whether
databases, web based analysis engines, desktop
applications
Migration
eResearch
Tools
Development
Research
Cloud –
SaaS
Research
Cloud – PaaS
Research
Cloud - IaaS
Going the last mile?
Who is supporting the researcher,
to understand their needs, define
their requirements, and
implement their solutions?

Are there services and
applications either already in
place, or under development,
to meet needs, and fill
capacity created?
Who will cross the
boundaries between
these groups?
Researchers
Applications &
Services
developers
Platform
service
providers
Infrastructure
service
providers
Who are our users?
Who are the users?
– Researchers – technically savvy ones..
– HPC users.. maybe
–System Administrators
–Application Developers
“All researchers that need to use eresearch tools already
know how to and are doing so..”
“If you can’t write the code yourself, how can you trust
the results? Surely you must be able to write the code!?”
What do I talk about, when I talk about
Cloud?
Instead of the machinery:
VMs, hypervisors, schedulers,
block storage, virtual networks,
hosts, networks, firewalls
How about capabilities:
Scalability: Methods, Communities
Self Service
Sustainability (through preservation)
Where do we do this well?
Data?
• Preservation

Curation
HPC?
• Algorithms
• Methods
• Hardware optimisation techniques
When communicating to end
users, common vocabularies in
other communities aren’t about
the machinery, they’re higher
level…
What do I talk about, .. Scalability
What is the equivalent of algorithms and optimal approaches in the Cloud era?
The knowledge, skills, and methods to optimally use
the cloud?
How to be elastic (scale up/down, on demand)
How to manage failure
Multi-tenancy
Building communities, leveraging network effects
What do I talk about, .. Preservation
What are the research processes and
artefacts we are interacting with?
Methodologies..
enshrined as VMs?
e.g.
research codes, which take cuts and bruises to re-instantiate
= knowledge artefacts that have value and are scarce
What about workflows?
Shouldn’t we wait until we have enshrined these methods into
sophisticated workflows, and ontologies?
.. So we should throw away any method until it reaches a
predefined threshold of maturity? Because before then, it has no
value?
Oh, so all methods are important?
that’s up to the researcher to say.. Method construction
and archive, self service.
Ahh, so we’re close to being able to archive and preserve rich
research methods?
for those that enshrine their methods in software based
systems… YES!
Reuseable workflows and services for large
collaborative communities?
… like Virtual Organistions, they only suit
certain levels of scale and maturity
Individual research codes, workflows,
pipelines
=
research notebooks
for those who
efficiently enshrine their thoughts and
ideas in code
These are people we should support!
And, their experiments things we
should preserve..
What did we learn from the grid?
So..
…. what do we need to get
right,
to be successful?
Get the team right
Composition & Coordination of distributed team
essential
• Fully committed FTEs
• Cross site responsibilities, including shared
systems administration
• Strong leadership and programme
management
Strong guidance
Which resources are getting used, why?
• Prescriptive framework necessary to
ensure participation
… need a well
defined service model
Should we build new institutions?
Why do we want to build institutions?
Scale efficiencies
Sustainability
.. and reliable targeted accountable services
We often take
project funding
,
and aspire to build new institutions
Creating institutions is and will continue to be expensive and difficult..
Our institutions are long lived, and can absorb most technologies,
given care and enough time to mature
Really think about Coordination
Why do we need Government money?
Scale?
We’re partly seeing a coordination failure
(coordination of capital and resources)
It is the incentive to bring out institutions into
alignment..
Funding:
work with funders to align opex and capex. Crown
and coinvestors fund both in NZ, clarifying intent of
investments
Otherwise, whoever funds
the operating, eats the cake
Ensure KPIs incentivize sustainability
Incentives and obligations
… clarify expectations to coinvestors, sector
“Single Front Door”
Coordinate with broadest scale IT community
that has responsibility and capability

Research labs can be good, though often will
be introspective / focus on local solutions /
research outputs

Drive innovations into core IT Services groups;
strong coordination from top level sponsors
required
– Collaborate nationally and internationally
Usability really counts !!
Make it easy!
• Authentication … Authentication …
Authentication …
• Consistent User Interfaces?
1.Don’t get in my way
2.Don’t break the abstraction
3.Don’t give me middleware!!
Build communities
Is the aim to build aggregations of communities?
• Support national scale communities?
… but these may not exist
will need to build communities
(developers, end users, administrators, leaders)
Who will mediate within and between
communities
,
seek out commonalities
,
and
take ownership of developments?
Support the long tail
• There’s an endless list of applications and services, that individual
researchers need and want

We often focus on the large communities, common platforms, and
scalability
• To make a Research Cloud useful for the majority of the community, we
need to design for great diversity

We can do this incrementally
• We’re looking to learn, fast, what works for researchers, and what doesn’t
>>> we need agility, and very strong
engagement with users communities
Thanks
NeCTAR Research Cloud Workshop
30 – 31 March 2011, Melbourne, Australia
Nick Jones
n.jones@auckland.ac.nz
www.eresearch.auckland.ac.nz
Thoughts from the east,
on research in the cloud
Image Credits
• www.nzhistory.net.nz, the website of
NZHistory.net.nz, New Zealand history online, at
Manatū Taonga. Licensed by Manatū Taonga for
re-use under the Creative Commons Attribution-
Non-Commercial 3.0 New Zealand Licence.
• www.history.army.mil, Communications
Electronics 1962-1970, Department of the Army
• www.travelpod.com, Travel Blog, 7.08.2010,
http://www.travelpod.com/members/wbeardsl
Goals
Needs of users, as identified at this workshop
Policy implications
• Applications & Services that are in common national usage
• Enable provision of excellent cloud services to researchers

Nodes operating within a prescribed framework
Framework
• Consistent user interface
– to the range of applications and services running at the distributed nodes
• Applications and Services
– Selected by Panel
– Data analysis, visualisation, collaboration, security, application and service platforms, portals / interfaces to HPC and commercial
cloud providers
• Infrastructure as a Service
– Hosting Platform as a Service offerings for research communities
• Distributed Nodes
– Selected by panel
• Prescriptive operating model
– Lead Node will create & operate framework; monitor service delivery
• Access, interop, application migration, security, licensing, accounting, implementation practicalities, monitoring and maintenance
– Common layer that each node can interoperate with