The Cloud - the Future of Grid Computing?

lovingbangInternet και Εφαρμογές Web

3 Νοε 2013 (πριν από 4 χρόνια και 6 μέρες)

81 εμφανίσεις

The Cloud… the Future
of Grid Computing?

Steven Newhouse

Background

The Grid Challenge

Workflows of
Composable

Services

The Software Challenge

Microsoft’s HPC Solution

The Future of the Grid

Agenda

NORDUnet 2008

2


Degree in Aeronautical Engineering


The importance of computer modeling in design


PhD in Fluid Structure Interaction


Use parallel computing to increase simulation size


Developing an HPC community


Managing Imperial College’s HPC resources


Improving access to distributed resources


UK e
-
Science
programme

& Open Grid Forum


Move to Microsoft’s HPC team


These views are mine, not Microsoft’s!

About

me…

NORDUnet 2008

3


HPC technology has matured and come of age


30 years ago a small research activity


Now a critical resource in many industries


Commercial support from small and big
organisations


OS: Linux and Windows


Application Types: Compute & Data
-
intensive, SOA, …


Verticals: CAE, Bioinformatics, Oil & Gas, Finance, …


Next Phase: Distributed computing for the masses


Networking and computing resources continue to grow


Cluster Computing: The leading crest of the wave


Cloud Computing: A usable ‘Grid’ for the masses?


My thesis for today…

NORDUnet 2008

4


1970s: Walking across campus to submit jobs


Carrying a stack of punch cards!


1980s: Cross
-
campus networking


Distributed Computing. International networks


1990s: Limited home connectivity


56K modems to ADSL


2000s: Internet everywhere


Massive bandwidth to the home & to the mobile



Networking over

40 years

NORDUnet 2008

5


1970s: Time sharing mainframes


Perhaps a few machines per campus


1980s: Affordable Distributed Computing


Home & Office desktops. Research workstations


1990s: High Performance Computing


Vector processors, high
-
speed interconnects, COTS


2000s: The Grid


Enough connected computing to do interesting ‘stuff’


Computing over 40 years

NORDUnet 2008

6

Corporate Infrastructure

Storage

Clusters/Super Computers

High Speed networking

Engineers

Scientists

Information workers

Compilers

Debuggers

Specialized languages

Mainstream Technologies

Financial Analysts

Today’s Environment

NORDUnet 2008

7

Combined Infrastructure

Integrated Desktop and HPC
Environment

Unified Development
Environment

High Productivity Computing

… an integrated
infrastructure that
brings together
various distributed
resources…

… bridging different
users and resources
in different
administrative
domains …

… managing a
complex
distributed shared
infrastructure …

… effectively
exploiting diverse
resources (parallel
& multi
-
core)
through the same
simple API ……

Grid Challenge

Reward

… a collaborative
infrastructure
accessible from
your desktop…

… workflow is a
drag and drop
development
programming
environment…

NORDUnet 2008

8


Setting up trust relationships is hard


Exploiting different operating systems is hard


Exploiting the same operating system is hard


Dealing with different resources is hard


Dealing with network failures is hard


Programming such an environment is hard


Running such an environment is hard


Using such an environment is hard

Grid Computing is
VERY

hard

NORDUnet 2008

9


What do your users want to do?


They want to solve their own unique problem


But increasingly as part of a larger collaboration


What sort of problems do they want to solve?


Large scale simulations (traditional HPC)


Parametric Searches (naturally parallel!)


Why do they need additional resources?


Problem is too big for their own local resources


Resources are too
specialised

to be replicated


Let’s solve

an easier problem…

NORDUnet 2008

10


Services to access computing resources


As a unified unit for HPC


As a custom function for parametric searches


Services to store and access data


My data


Shared data


Services to collaborate


Mail, IM, Web 2.0…

Requires an infrastructure…

NORDUnet 2008

11

Organisation

A

Resources

Resource

Services

Standards

Organisation

C

Resources

Resource

Services

Standards

Organisation

B

Resources

Resource

Services


Cloud Services

Standards

Domain 2

Domain 1

Web

Applications

Web

Services

Web Infrastructure

Browser

Browser

Browser

Browser

Browser

The Grid

Web 2.0 / Portals / APIs

E
-
Science

The Grid

Healthcare

Enterprise

A

cloud of services

Standards

Standards

NORDUnet 2008

12


What is a Service?


Great approach to decomposing a complex systems


Isolated activity
composable

with other activities


What is a Cloud?


Stuff that is out there, somewhere….


Enabled by ubiquitous network connectivity


What is a Cloud Service?


Is it any different from any other service?


My definition: ‘A user oriented grid service’


As opposed to a provider oriented grid service


Are Cloud Services the future?

NORDUnet 2008

13

Design

Analyze

Concept / Goal
Setting


Design &

Pre
-
Processing

Simulate

Testing &/
Simulation

Result

Analysis

Post processing

End
-
to
-
End Services Workflow


W
orkflow

between local & remote services


Running applications locally


Using

remote services


Integrated audit, provenance, publication, …


Collaboration: IM, email, document
sharing, …


We are all becoming information workers!


NORDUnet 2008

14


North East Pacific Time
-
Series Undersea
Networked Experiment


The world’s first plate
-
scale
undersea observatory


From raw data to useable
data products


Data cleaning, analysis,
regridding
, interpolation


Support real time, on
-
demand visualization

Scientific Workflow:

NEPTUNE

NORDUnet 2008

15

Custom activities for oceanographers

In
-
browser workflow editing

Scientific Workflow:

NEPTUNE

NORDUnet 2008

16


Standalone Software (e.g. Traditional MS, OSS)


Your hardware & staff


Isolated Desktop installation


Networked workgroup licensing


Hosted Software (e.g. MS Online, EGEE)


Service Agreement


Not your hardware, software or staff


Cloud Services (e.g. MS Live, Amazon, Google)


Software as a service


Potential pay per use model


Software Everywhere

NORDUnet 2008

17


It can be….


MPICH in HPCS: Contributions back to community


OSS on HPCS: Key numerical libraries & apps ported


Codeplex

-

http://www.codeplex.com/


But the cost to maintain ALL software is still high


Is your mission critical OSS software supported?


Is it under active development?


Corporate contributions:
Redhat
, Sun, IBM, Microsoft, …


Volunteer contributions: You, academia, hobbyists, …


How is the software roadmap being developed?


Functionality, reliability, quality, portability, documentation



Is
OSS the solution?

NORDUnet

2008

18

An example: OMII
-
UK


Open Middleware Infrastructure Institute


Provide high quality OSS to UK academics


Problem:


UK e
-
Science (Grid) program relying on OSS base


Software quality became a barrier to productivity


Solution:


Outreach to gather requirements &
prioritorize


Provide effort to document, test & integrate software


Fund OSS developers to add specific functionality


Approximately: IBM support + UKP 12M over 5 years.


NORDUnet 2008

19

Innovation to Production


Software is a burden

to maintain: ~ 2/3 of costs


Cost frequently hidden in academia (PhD students!)


Non
-
core

activity in many
organisations


Industry

will

become engaged when it makes sense ($$$)


Defined markets


Defined requirements


Defined risk

Idea

Research

Prototype

Deploy

Adopt

Proliferate

Academia

Industry

NORDUnet 2008

20

Current Issues



HPC and IT data centers merging: isolated cluster
management



Developers can’t easily program for parallelism



Users don’t have broad access to the increase in processing
cores and data

How can Microsoft help?



Well positioned to mainstream integration of application
parallelism



Have already begun to enable parallelism broadly to the
developer community



Can expand the value of HPC by integrating productivity and
management tools

Microsoft Investments in HPC


Comprehensive software portfolio: Client, Server,
Management, Development, and Collaboration


Dedicated teams focused on Cluster Computing


Unified Parallel development through the Parallel Computing
Initiative

Why Microsoft in HPC?

NORDUnet 2008

21

Administrator

Application Developer

End
-

User


Integrated Turnkey HPC
Cluster Solution


Simplified Setup and
Deployment


Built
-
In Diagnostics


Efficient Cluster Utilization


Integrates with IT
Infrastructure and Policies


Integrated Tools for Parallel
Programming


Highly Productive Parallel
Programming Frameworks


Service
-
Oriented HPC
Applications


Support for Key HPC
Development Standards


Unix Application Migration



Seamless Integration with
Workstation Applications


Integration with Existing
Collaboration and Workflow
Solutions


Secure Job Execution and
Data Access


Windows HPC allows you to accomplish more, in less time, with reduced effort by
leveraging users existing skills and integrating with the tools they are already using.

Microsoft’s Productivity Vision for HPC

NORDUnet 2008

22

Systems
Management

Job
Scheduling

Networking

& MPI

Storage



New System Center UI



PowerShell for CLI Management


High Availability for Head Nodes


Windows Deployment Services


Diagnostics/Reporting


Support for Operations Manager



Support for SOA and WCF



Granular resource scheduling



Focus on Enterprise size clusters



New Job scheduling policies


Interoperability via HPC Profile



NetworkDirect (RDMA) for MPI



Improved Network
Configuration Wizard



Shared Memory MS
-
MPI for
multi
-
core



MS
-
MPI integrated with
Windows Event Tracing



Improved
iSCSI

SAN & parallel
file system Support in Win2008



Improved Server Message
Block ( SMB v2)



New 3
rd

party parallel system
file support for Windows



New Memory Cache Vendors

What’s New in
the

HPC Pack 2008

Parallel Programming

Integration into Visual Studio of
various compilers, debuggers,
profilers & tracing tools

NORDUnet 2008

23

NORDUnet 2008

24

Rich User Interface

+ CLI +
PowerShell

App.exe

App.exe

Service

(DLL)

Service

(DLL)

Service

(DLL)

Service

(DLL)

App.exe

App.exe

Engineering
Applications

Structural Analysis

Crash Simulation

Oil & Gas
Applications

Reservoir simulation

Seismic

Processing

Life Science
Applications

Structural Analysis

Crash Simulation

Financial Services

Portfolio analysis

Risk

analysis

Compliance

Actual

Excel

Pricing

Modeling

Interactive

Cluster
Applications

Your applications

here

Job Scheduler


Resource allocation

Process Launching

Resource usage tracking

Integrated MPI execution

Integrated Security

WCF Service Router


WS Virtual Endpoint Reference

Request load balancing

Integrated Service activation

Service life time management

Integrated WCF Tracing

V1 (focusing on Batch jobs)

V2 ( + support for Interactive jobs)

+

Scenario: Broaden Application Support

NORDUnet 2008

25

The HPC Basic Profile


An OGF interoperability standard
for batch job scheduler task
submission and management


Based on web services standards
(HTTP, XML, SOAP)

What is its value?


Enables integration of HPC
applications executing on
different platforms and
schedulers via web services
standards

What’s the Status?


Passed the public comment
period


Working on new extensions


Installed with HPCS 2008

Windows Cluster

Window Center

LinuxCenter

LSF / PBS / SGE

Linux, AIX
, Solaris

HPUX, Windows

Interoperability & Open Grid Forum

HPCBP

HPCBP

HPCBP

HPCBP

NORDUnet 2008

26


Compare with where HPC was 30 years ago…


MPI a key factor in driving application adoption


Commodity hardware in driving deployment


Grids have had a very successful 10 years…
but


Need for consolidation within the community


Define interfaces/standards for solved problems


Decide what you want to develop & what to
buy


Pragmatic approach to uptake in other domains


Not everyone wants to do HEP particle analysis


Where are the software/standard specifications?

Grids


where next?

NORDUnet 2008

27


It’s nothing new!


Software running remotely on a computer


Supporting users through various access methods


Thin client web browser (e.g. traditional server side)


Active thin client (e.g. smarter client


Silverlight
, AJAX,…)


Native fat clients (e.g. Office,
Sharepoint
,
Powershell
)


It’s a whole new paradigm!


Everything is done & stored on remote resources


Your (business) success depends 100% on someone else


Breaks the enterprise silo to enable collaboration

Moving

into the Cloud

NORDUnet 2008

28


The Cloud is a new perspective on the Grid


Focused on its use rather than its provision


Essential to move out of the pure academic space


Define your requirements through standards


Open Grid Forum, W3C, OASIS, …


Reuse not reinvent


Download latest beta and provide feedback!


http://www.microsoft.com/hpc



Windows HPC Community site


http://www.windowshpc.net



Contact: steven.newhouse@microsoft.com



Summary

NORDUnet 2008

29

©
2008 Microsoft
Corporation. All rights reserved.

This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

NORDUnet 2008

30