Space Weather Model Execution Platform (SWEMEP)

smilinggnawboneInternet και Εφαρμογές Web

4 Δεκ 2013 (πριν από 4 χρόνια και 28 μέρες)

102 εμφανίσεις

Space Weather Model Execution
Platform (SWEMEP)
A Platform as a Service (PAAS) for executing
Space Weather Models in the Cloud

ESAW, June 2013

Presenter: Simon Reid, RHEA
Overview
• Part of Wider Cloud Study:-
– Requirements definition and validation of a Cloud
Computing solution of general applicability to
programmes
1.Cloud Infrastructure (IaaS) requirements definition (CGI)
2.IaaS Solution Blueprint (The Server Labs)
3.Space Weather PaaS (RHEA)

• Space Weather Models
• Cloud Infrastructure / PaaS Concepts
• SWEMEP Services and Main Features
• Technology Stack for SWEMEP Solution
• High Level Architecture


2
ESAW 2013: Space Weather Model Execution Platform
What is the SWEMEP PaaS?
IaaS
Infrastructure as a Service
Virtual Computing Resources
PaaS SWEMEP Dedicated Code
PaaS -
Platform as a Service
PaaS standard technology and tools

SaaS -
Software as a Service
Space Weather Custom Application
Space Weather Applications are
implemented using the PaaS
toolkit. These can be
developed as SaaS or standalone
applications
PaaS provides standard means
to develop applications hosting
Space Weather Models.
This custom development can
benefit from use of standard
PaaS tools.
PaaS uses services of underlying
IaaS
IaaS provides access to the
virtual computing resources,
hosted in the private, public or
hybrid cloud
3
ESAW 2013: Space Weather Model Execution Platform
Space Weather Context
• Complex Sun-Earth
System
• Many independent
activities modelling
aspects of the system:-
– Solar Activity
– Coronal Mass Injection
– Shock, Solar Wind
– Solar Energetic Particles
– Radiation
– Magnetosphere
– Ionosphere
• Heavy / variable computing
resource needs
• Increasing need to ‘join’
models together
• Moving from research into
operational services (e.g.
SSA) – increased need for
reliability and security

4
ESAW 2013: Space Weather Model Execution Platform
Reference Space Weather Models (1)
Model Name

Summary
FROMAGE Magnetohydrodynamic (MHD) force-free model constructing the coronal magnetic field for the
entire Sun, using ground-based photospheric magnetographs as input.
AMRVAC Full compressional ideal MHD model, with added non-ideal effects, parallelized.
Used for coronal reconstruction (using magnetogram data as boundary conditions), 2D and 3D
solar wind models (polytropic and MHD), 2.5D and 3D CME initiation and early evolution
simulations, interaction with the background solar wind and the Earth’s magnetosphere.
iPIC-3D Particle-In-Cell C 3D parallel code simulating particle-dominated environments applied both to
magnetospheres and the interplanetary medium.
iPIC3D is based around massive parallelisation. The calculation can be spread across up to 16000
processors
SOLPENCO Particle propagation code providing proton flux and cumulative fluence profiles at 10 energy
levels for a set of gradual SEP events, for observers located at either 1.0 AU or 0.4 AU.
The code also gives the transit time and velocity of the shock from the Sun to the observer, the
maximum proton intensity (peak flux), and the total fluence of the SEP event computed from the
onset of the event up to the arrival of the associated transient CME-driven shock
GUMICS-4 Self-consistent 3D numerical MHD model developed to study the solar wind interaction with the
near Earth space environment . magnetospheric part is based on solving the ideal MHD
equations and its ionosphere part is based on solving the electrostatic current continuity
equation. Automatically refined Cartesian octogrid and temporal subcycling are used to speed up
the computation, parallelized using MPI, with dedicated load-balancing algorithms
5
ESAW 2013: Space Weather Model Execution Platform
Reference Space Weather Models (2)
Model Name Summary
BAS DGRBM 3D physics-based particle simulation of Earth's trapped electron radiation belt flux for
different solar wind and geomagnetic activity. Uses a time series of inputs, including satellite
data and geophysical indices such as Kp, to calculate changes in the relativistic electron flux
throughout the Earth's electron radiation belts.
SALAMMBO Computes instantaneous particle flux inside the radiation belts following a magnetic storm or
a Solar Energetic Particle event, to model the creation and dynamics of the radiation belts.
Also used for interpolating between actual measurements to increase resolution
CMAT2 3-D numerical coupled model of the Ionosphere-Neutral Atmosphere-Thermosphere system,
from 30km altitude (15 km in later version) up to the exosphere, and to 10,000km for the
ionosphere. Solving self-consistently the coupled equations of momentum, energy and
continuity for neutral particles and ions. Calculations are performed on a spherical grid with
different properties depending on the regime.
TRANSCAR Physics-based 1-D model for coupling the thermosphere and the ionosphere, describes the
ionosphere from two coupled approaches; fluid and particle. Based on an eight-moment
approximation of the Boltzmann equation using EUV solar flux, or particle precipitation as
inputs, also electron densities and temperatures computed by the fluid part.
DTM Semi-empirical thermosphere model used primarily for orbit computation, based around
density/drag computation. A call to the DTM subroutine gives a single density prediction
localized in time and space. The density prediction is a function of altitude, latitude, longitude,
time, mean and daily solar radio flux, and daily mean and 3-hourly Kp.
6
ESAW 2013: Space Weather Model Execution Platform
Example Outputs in Graphical Representation
ESAW 2013: Space Weather Model Execution Platform
SWEMEP Model Management
• In principle, these models may be linked to
represent the full space system.

• General Considerations for existing models:-
– Focus on research not operational results.
– No automatic assimilation of recent data.
– Varied methods for Graphical outputs
– Few common data structures, types,
calibration etc
– Mixed IPR, varied maintenance approaches
– Mainly Fortran, File I/O, IDL graphic output

• SWEMEP must consider
– Wrappers, Data Translators
– Parallel Processing support for Legacy & New
models.
– Standard environment for development of
future/advanced models
– External Interfaces for latest data

8
ESAW 2013: Space Weather Model Execution Platform
IaaS Blueprint

• Hybrid of Private and
Public Cloud
Infrastructures

• Common Approach to
governance, security,
billing, etc.

• Provisioning via
templates

• Common API and
Common tools

• Applicable document for
SWEMEP
9
ESAW 2013: Space Weather Model Execution Platform
Platform as a Service (PaaS)
• Services offered via CLI, API and MMI
• Application Development Kit
• Provisioning and Orchestration
• Monitoring & Management
• Load Balancing, High Availability
• Security
• Standard Platform Components

• Most well-known PaaS are intricately linked to specific Infrastructure
• AWS, Google App Engine, Azure, IBM Cloud
• A range of newer open(-ish) alternatives are emerging
• Cloud Foundry, OpenShift, Cloudify, RightScale
• Some wide variations in scope...changing emphasis....fast-moving

Selection will be dependent on final “ESA IaaS” specification


10
ESAW 2013: Space Weather Model Execution Platform
SWEMEP Services & Main Features
• Objectives
– Maximise Ease of Deployment
– Maximise use of cloud capabilities
• Static Provisioning
– Provisioning of Infrastructure (machines, clusters) for on reusable recipes
– Integration & provisioning of a model
– Configuration for combining models or to provide a higher level service
• Dynamic Orchestration
– APIs for programmed dynamic control of resources, parallel processing, load
balancing,...
• Execution Services
– Manage Model Execution and chaining, results display, etc.
• Security, Management and Monitoring, Reporting
, ...
• Development Toolkit
11
ESAW 2013: Space Weather Model Execution Platform
SWEMEP Solution Stack
12
ESAW 2013: Space Weather Model Execution Platform
SOLUTION LAYER Candidate Solutions
SAAS
Space Weather Specific Application (SAAS)


Space Weather Models

Solpenco, Transcar, Sheildose, GRAS, ....
PAAS
Space Weather PAAS API

SWEMEP ICD
Space Weather PAAS Software

SWE Model Factory, SWE Task Management, SWE Data Management
Business Process Engine

BEPL/BPML
Web Front End

AJAX, jQuery
Component Containers

Equinox (OSGi), Tuscany (SCA), Glassfish (EJB)
Service Integration Platform

ServiceMix, OpenESB, Jboss, Oracle ESB, Mule, swordfish, camel
Communication & Data Distribution

ActiveMQ, JORAM, ZeroMQ, RabbitMQ
Data Layer

PostgeSQL, NOSQL/HBASE, Oracle
Model Processing Layer

Hadoop, HBASE, Cassandra, Hive, etc, doozer
Orchestration Layer

Cloud Foundry, Stackato, Cloudify, OpenShift, RIghtScale, enStratius, Pivotal
IAAS
Provisioning Layer

Chef, Puppet, Juju
Mediation Layer

jClouds, EC2 API
Cloud Provider Layer

Private Cloud: OpenStack, OpenNebula,
Eucalyptus, Cloud Stack, Vcloud, .....
Public Cloud:
AWS, RackSpace,
CloudSigma, Azure, Google, Compute
Engine

Virtualisation Layer

VMWare, Xen, KVM
Database as a Service (DAAS)

MySQl, PostgreSQL, Oracle, SQL Server
Operating System

Linux ( SLES, Ubuntu, CentOS)
Windows 2008 Server

Physical Computing Layer (Network,
Storage, CPU)

HP, DELL, IBM, ..
OSGi, Apache ServiceMix, Fuse ESB
• Follows EGS-CC selection

• OSGi – provides a module and service
platform, component bundles for
deployment

• Apache ServiceMix is a ready
integrated set of open source
technologies for an integration
runtime platform.
• Karaf (Kernel Runtime)
• Camel (Routing and EIP)
• NMR (Message Router)
• ActiveMQ (Messaging)
• CXF (Web Services)
• ODE (BPEL engine)




13
ESAW 2013: Space Weather Model Execution Platform
Fuse ESB
• Fuse ESB
– RedHat ServiceMix distribution with
additional components, particularly
well suited to complex distributed
environment, including cloud
• Fuse Fabric
– Flexible dynamic provisioning suited
for highly distributed, dynamic
environment.
– High Availability dynamic service
provisioning with jClouds API for
dynamic access to IaaS
• Management Console
• IDE
– Flexible approach to provisioning
and routing



14
ESAW 2013: Space Weather Model Execution Platform
SWEMEP Development Toolkit
• Used by Development Team to create/configure SaaS from PaaS
– Integrate & Configure legacy models
– Develop New Models
• Eclipse with SWEMEP Plugins
– API library
– Model Wrapper Component
– Data Type Transformation
– Database / File I/O
– Web Forms & Graphical Library (J3D)
• Fuse IDE ( also an Eclipse Plugin )
– Effectively a Graphical Editor for Camel
– Drag and Drop model Endpoints and JMS
Queues etc component routing
• IaaS Orchestration & Provisioning Tools
– e.g. Cloud Foundry, Chef.
15
ESAW 2013: Space Weather Model Execution Platform
SWEMEP Business Process Automation
• Used by End Users to Configure Model Chains and Data Transforms


• Business Process Modelling
Notation (BPMN)....

• Business Process Execution
Language (BPEL)

• Apache ODE executes BPEL services
within Fuse ESB


ESB
Model
Model
Model
16
ESAW 2013: Space Weather Model Execution Platform
SWEMEP Model Parallel Processing
• MPI (Message Passing Interface)
– Widely used in research environment
– Used in several of our existing reference models
– Explicit programming of parallelism & data distribution interfaces needed
– Some code changes may be needed to run in SWEMEP

• OpenMP
– Suitable for multi-core architecture ( shared memory ).
– Explicit Parallelism - Implicit Data/Communication

• Supported by SWEMEP Cluster Compute Service
– Instantiate multiple machines as a Cluster
– Efficient communication
– Master/Slave Topology – Access master only.

17
ESAW 2013: Space Weather Model Execution Platform
Map-Reduce Model - Hadoop
• Straightforward Development
– Algorithm as Maps and Reductions
– No parallel programming, communication, ...
• Cloud compatible
– Many providers provide in-built support.
• Data Storage
– Distributed file system, efficient, insensitive to data
type
• Processing
– Suited for efficient distribution processing of batch
jobs over a computing network
• SWEMEP
– Existing models would need to be re-engineered to
benefit, but is considered for future models
implemented in SWEMEP.

18
ESAW 2013: Space Weather Model Execution Platform
SWEMEP Service Modality
• “System of Systems” architectural
approach.
• Scalable, decentralised, fully distributed.
• Flexible joining of public, private,
standalone SWEMEP clouds
• Service Model.
• Portal technology JSR-286 used to
combine front ends


SWEMEP
Service
Container
Mod
el
SWEMEP
Service
Container
Mod
el
Mod
el
Service
Container
Mod
el
Mod
el
Service
Container
Model
Service Container
Mode
l
SWEMEP
Service Container
Mode
l
Mode
l
Service Container
Mode
l
Mode
l
Service Container
Model
Service Client (Portal)
Service
Container
Model
19
ESAW 2013: Space Weather Model Execution Platform
Acknowledgements
• ESA
– Vicente Navarro; ESA Technical Officer
• CGI
– Daniel Novak; Project Manager
– Andreas Sisask; IaaS System Architect
– Mika Hallfast; IaaS Requirements Engineer
• The Server Labs
– Paul Parsons; IaaS Blueprints
• RHEA
– Gareth Lawrence, Space Weather Scientist
– Simon Reid, PaaS System Architect
– Luca Febo, System Engineer
– Doug Wiemer, Cyber Security Consultant


20
ESAW 2013: Space Weather Model Execution Platform