The Evolution of AWIPS - NOAA Earth System Research Laboratory

righteousgaggleΔιαχείριση Δεδομένων

31 Ιαν 2013 (πριν από 4 χρόνια και 5 μήνες)

399 εμφανίσεις

1

National Weather Service

Jason Tuell

Office of Science and Technology

The Evolution of AWIPS

NURO

5 September 2007

2

Overview


The Why and What of AWIPS Evolution


Objectives


Re
-
architecture Approach


Roadmap


What does AWIPS II mean to ensembles?


Summary

3

Why Change?


Case for change briefed to NWS Corporate Board


Nov 2004


AWIPS Present State Summary


Hardware


AWIPS hardware was in good shape


Communications Infrastructure


AWIPS communications infrastructure was in OK shape


Data


AWIPS Data was in need of improvements


Software


AWIPS software was in critical need of improvements


Costly software development, maintenance and inability to meet NWS and customer
needs


Corporate board direction to focus on addressing software shortcomings


Plan and requirements developed


Shaped portions of the AWIPS O&M re
-
compete activity

4

What is AWIPS Evolution?


AWIPS Evolution


A long
-
term project which delivers a modern, robust software infrastructure
that provides the foundation for future system level enhancements


AWIPS II


Implements a modern Services Oriented Architecture (SOA) infrastructure


First output of AWIPS Evolution and provides the foundation for all
subsequent improvements


AWIPS II Extended
-

Creation of a seamless weather enterprise
spanning NWS operations


Migration of NAWIPS into the AWIPS II SOA


Delivery of thin client


Support to the Weather Service Offices and Center Weather Support Units


Integration of “orphan” systems (e.g., Weather Even Simulator)


AWIPS II Enterprise Enhancements


Data Delivery Enhancements


“Smart push
-
smart pull” data access


Katrina satellite WAN back up


Integrated visual collaboration


Visualization Enhancements


Information Generation Enhancements

5

AWIPS Evolution

Objectives


Establish Service Oriented Architecture for AWIPS and NAWIPS


Create a seamless weather enterprise that supports all levels of NWS
operations from National Centers to WSOs


Build a common development environment that will be used by all
developers


Establish infrastructure for GIS integration


Enable access to data independent of its location, i.e., provide
access to data not resident locally at the WFO or RFC.


Provide infrastructure for real time graphical collaboration
between


WFOs, RFCs and centers for enhanced internal collaboration


Other NOAA entities and


Trusted partners, e.g., Emergency Managers


Implement a Common AWIPS visualization environment (CAVE)
used by all applications


Standardize generation of NWS products and services

6

AWIPS II

Re
-
Architecture Approach


Perform “black
-
box” conversion


Preserve existing functionality, look and feel on top of new
infrastructure


Thorough field validation and acceptance before
deployment


No loss of functionality


Deployed system current with deployed AWIPS capability (i.e., OB9)


Use open source projects
-

No proprietary code


JAVA and open source projects enable AWIPS II to be platform and
OS independent


No plans to move from Linux


ADE enables collaborative development


OS, Platform independence allows non
-
Linux based research to be
easily integrated into AWIPS II

7

AWIPS II

Features


AWIPS Development Environment (ADE)


Used by all AWIPS developers (National, Regional, & Local)


Developers concentrate on new capabilities, not re
-
implementing existing ones (i.e.
screen I/O, communications protocols, data access routines, logging routines, or
other previously developed capabilities)


Software can be developed on a variety of platforms


Robust infrastructure for improved software O&M


Use of plug
-
ins: visualization extensions; new data types and transforms


System level, remediation, core services reduce system complexity


Improved support for local requirements (e.g., local apps, scripts, plug
-
ins)


Common AWIPS Visualization Environment (CAVE)


Provides a common development and execution environment for AWIPS GUIs (e.g.
D2D, NMAP, GFE, etc.)


Ability to pan/zoom large data sets (Raster & Vector) with flexibility over data
rendering


GIS tools


Thin Client (Web Browser) enabled


Dynamic Load balancing


Processing dynamically allocated among available CPUs

8

AWIPS II

Roadmap

Migration

Strategy

2007

2008

2010

2009

2006

SW CTR
(AWIPS II)

AE OSIP Gates

2

3

4a

4b

=
Fiscal Year

= Calendar Year

MPLS

Meshed Topology


9/05/07

OBx

7

8

9

10

PIP

Analysis

Migration Planning

RTS

IRAD

ADE Development

“User” Functional Tests

Baseline Application Migration

8

8.3

ADE

Training

Field Ops Training

--

ITO, ESA

Deployment

OTE

C & A

Deployment Planning

Local App Migration

ADE Local App Training

New Release

Paradigm

O & M Transition

NWS New Capability
Development in ADE

OB 9 Dev & Test

Deployment

O & M Transition Prep & Coordination

ADE Training Development

OTE / Deployment Support

AWIPS II 1.0

Note: Task bar colors are

For speaker reference only

9

2007

2008

2009

2010

2011

2012

2013

2014

AWIPS Evolution

Roadmap

AWIPS II

Extended

=
Fiscal Year

=
Calendar Year


OTE / Deployment


AWIPS II

AWIPS II


Baseline Application Migration



Governance Model


NAWIPS Migration


Thin Client


WES Integration


AWIPS II

Enhancements


Data Delivery


IOC


FOC


Phase 1 Collaboration


Phase 2


Phase 3


Information Generation


IOC


FOC


Visualization


IOC


CHPS


10

AWIPS Evolution

Governance Model


What is it?


Governance model controls the development, test, integration,
configuration management, deployment and support of the new
system
--

both hardware and software


Why?


AWIPS II offers new levels of flexibility and extensibility


New rules needed to take advantage of system capabilities and also
define limits


Tension between unlimited modifications and ability to support the
system


Sample issues for consideration


Monolithic configurations no longer required
--

how do we manage
site specific configurations?


Plug ins can be down loaded and installed on demand


Scripting that modifies AWIPS menus, functions


Flexible Governance Model needed for limited open source
implementation

11

AWIPS II

What does it mean to ensembles?


NAWIPS migration


Same infrastructure supporting both NAWIPS and AWIPS
applications


Algorithms


Common library for meteorological and hydrological
calculations and tools


Data delivery


Smart push
-
smart pull may provide more flexibility for access
to large data sets


Visualization


Supports new tools and techniques for visualizing ensemble
data


Information Generation


New architecture for product generation and delivery will
allow tailored products capturing uncertainty

12

AWIPS II

Training


Strategic Training Plan developed


Training targeted for the following groups:


ESAs


ITOs


AWIPS and application focal points


Developers (both baseline and local)


NCF


SST


Training Organizations involved in planning,
developing and implementing courses

13

AWIPS II

Testing


Layered testing strategy


Different testing phases overlap


functionality tested multiple times in
different settings


ADE usage


Not formal testing, but expect to get feedback as developers use the ADE


Algorithm V&V




Assist algorithm choice


Verify port of algorithms


V&V


Task order validation


Executed by dev orgs in controlled setting


Performance testing


Evaluate system against known and developing
benchmarks


Acceptance testing


part of the task order acceptance process


Risk reduction


Side by side testing of new system by field personnel in a “lab” environment


Testing at limited number sites in manner that doesn’t impact operations


Operational Test & Evaluation (OT&E)


Formal testing of the entire system


Tests system interfaces, operations, support, training, etc.

14

AWIPS II

Risks and Challenges


Organizational challenges


Significant off contract workload on Government


Matrixed structure increases project complexity


Performance


Supporting the short fuse warning mission


Handling large global data sets


Schedule


Completing the migration and testing


Migration of local applications


Local applications outside the baseline


Not a Raytheon responsibility

15

Summary


New architecture defined and infrastructure delivered


ADE 1.0 delivered June 14, 2007


Application migration underway


Migration Plan delivered June 2007


AWIPS baseline migration to be completed FY09


Local applications migration planning in progress


Incremental approach will extend AWIPS II
architecture throughout NWS


AWIPS II and subsequent enhancements will provide
more flexible foundation for delivering products that
capture uncertainty

16

National Weather Service

Back Up

17

AWIPS II

Local Applications


Survey


completed August 31, 2007


4191 local applications identified


37% critical, 44 % important, 19% less important


Migration plan to address approach based on survey
results


Training requirements and approach to be refined based on
survey


Raytheon to provide sample migration and code samples for
approach


“Integrate with” vice “interface to”


Level of effort required uncertain


Raytheon estimate that 80% of local apps will be able to be
rewritten in Javascript, without extensive programming in the
ADE

18

AWIPS II

What gets us excited so far…


Dynamic load balancing


Failover handled automatically


Enables consideration of tailored hardware configurations


Mathematically intensive calculations handed off to the graphics card


Significant performance improvements


Progressive disclosure of all data


Imagery via quad tree tiling, grids and observations


Integrated thin client


Allows baseline solution to be extended to CWSUs, WSOs, and IMETs


Integrated drawing and graphical collaboration


Tools built into the infrastructure, implemented in 2011


Built in GIS via geotools library


Scripting level access to practically all system level services and
functions


LESS CODE


Potential order of magnitude reduction in amount of software with increase in
functionality

19

AWIPS II

GIS Capabilities


GIS functionality integrated into AWIPS II architecture via
GeoTools


GeoTools


Open Source Java code library


Provides OGC compliant methods for manipulation of geospatial
data


Equivalent to ESRI’s arc engine


AWIPS II will support JavaScript access to GeoTools library


ADE 1.0 supports limited access


Read, write shape files


Create GeoTiffs


Spatial queries of ingested data


Task Order 10 will deliver full JavaScript access


GeoTools does
not

make AWIPS a stand alone GIS system


Reference:


GeoTools home:

http://geotools.codehaus.org


GeoTools & Eclipse:
http://docs.codehaus.org/display/GEOT/Eclipse+Developers+Guide

20

AWIPS Evolution

Outcomes


Short
-
term (1
-
3 years)


Shorten transition of research to operations


Improve software O&M and technology refresh


Fewer DRs and TTs


Focus on hardening and productionizing for life cycle support


Minimize adverse impacts on operations from software and
hardware upgrades


Long
-
term (3
-
10 years)


Increase integration of AWIPS and National Center AWIPS


Improve performance and functionality of AWIPS


Improve collaboration at all levels of NWS operations


Increase access to all environmental data for decision making

21

AWIPS Evolution

Data Delivery


OSIP Project 05
-
040


Enables “smart push
-

smart pull” data delivery


Implements a discovery service within the SOA


Access to data not available locally


Freedom from the tyranny of the SBN


Enables consideration of new data delivery architecture


What data to you broadcast over SBN?


What data do you make available on servers?


Schedule


IWT starting Q4 2007 to define concept of operations and
operational requirements


IOC
-

2011
-

software implementation for remote data access


FOC 2012
-

enterprise configuration (servers, comms, etc.)
that enables remote data access

22

AWIPS Evolution

Collaboration


OSIP Project 05
-
041


Objective


Integrated graphical collaboration throughout the NWS
Weather Enterprise and beyond


Phase 1
-

Integrated collaboration between all levels of NWS
operations


Phase 2
-

Collaboration between NWS offices and other NOAA
entities


Phase 3
-
Collaboration between NWS offices and trusted
external partners, e.g., Emergency Managers


Schedule


Phase 1 IOC
-

2011


Phase 2 IOC
-

2012


Phase 2 IOC
-

2013

23

AWIPS Evolution

Information Generation & Visualization


OSIP Projects 05
-
042 (IG) and 05
-
021 (Vis)


Information Generation objective


Re
-
architect generation of all NWS products and services


Separation of content generation from formatting and
dissemination


Enable faster response to emerging customer demands


Visualization objective


Common user interface
-

standardize User Interfaces across
applications


3
-
D visualization


Improve user interfaces based on latest principles and
research


24

What is AWIPS?

NEXRAD

AWIPS Workstations and

Servers

GOES/POES/NPOESS

ASOS

NCEP

Models

Buoys,

River Gauges

Forecasts

Warnings

Advisories

Watches

AWIPS

Communications

Service provided to

3066 US Counties

24 hrs/day,

365 days/yr.

169 separate AWIPS systems at

137 geographical locations

~900 Workstations (total)

~1200 Servers (total)

25

Environment
Environment
Environment
Lightning
Detection
Cooperative
Observers
AWIPS
LOCAL EXTERNAL
USERS;
COMPUTER
SYSTEMS;
OBS PLATFORMS
AND NETWORKS
NESDIS
NWSTG
WALLOPS
CDA
GOES I-M, Polar
Orbi ter, and
Other Satel li te Data
- Satel l ite Product Gen
- Satel l ite Operati on
GOES
I-M
Spacecraft
Data Collection
Platforms (DCP)
Offi cial User
Products
USERS
USERS
PROFILER
MICROART
NEXRAD
(May Interface Locall y
at Si te or Centrall y
VIA NWSTG
Upper Ai r
Observations
Weather Radar
Products
Cl ass 4
NIDS
USERS
DCP
Data
GOES
Data
USERS
Forecaster
- Product Formul ati on
- Data Integrati on & Anal ysis
- Forecasti ng
- Servi ces Di ssemi nati on
GTS
Upper Ai r
Observi ng
(Services
Dissemination)
Surface
Observations
Worl d-Wide Data
Col l ection &
Di stributi on
- Numeri cal Predi ctions
- Large-scale
Forecaster Gui dance
Gateway
for Internati onal & Non-
NOAA Domesti c products
-Surface
Observi ng
Remote Surface
Observi ng
- Radi ance Measurements
- DCP Data Col lecti on &
Di stributi on
Satell i te
Di stributi on &
Processi ng
Atmospheri c
Profi les
Offi cial User
Products
- RADAR Refl ecti vi ty
Measurement
- RADAR Product Gen
NWWS
NWR
ASOS
POES
Spacecraft
POES
Data
Lightning
Data
Ocean
Obs
NCDC
(NCF)
MDCRS
Commercial Aircraft
Data
NCEP
Retrospecti ve
Users
Offi cial User
Products,
Observations, &
Graphi cs
NCF
-Archi vi ng
LDAD
USERS
FAA Data
Level 2 &
Level 3 Data
Data
- Cal ibrati on Data
Model i ng
- Interactive Data
Anal ysi s
POES
Data
NCEP
Gui dance,
Observations,
& Offi cal User
Products
Enterprise Architecture

Dissemination

Modeling & Product Generation

Observing

Systems Legend

* Note some systems perform multiple functions, e.g. LDAD disseminates information and receives observations

“AWIPS is the Integrating
Element of the
Modernized Weather
Service”

Figure 6
-
8 of

NOAA EA v 1.6

26

Architectural Compliance


Collaboration on Achieving Enterprise Solutions


AWIPS is the Integrating Element of the Modernized Weather Service


AWIPS Uses Enterprise Red Hat Linux support contract


AWIPS Contract supports Climate, Weather and Water, Transportation, and
extensible to Coastal and Ocean Resources


Reuse of existing assets


HazCollect uses AWIPS IT infrastructure for message handling/creation and
routing to dissemination systems


Shared use of NOAA
-
Net for Wide Area Network in progress


AWIPS provides NEXRAD visualization and data collection functions


AWIPS software used by other Government projects
-

RSA II, WES, FX
-
Net


Usage of Standards


Complies with statutory requirements


Information Quality (Section 515) & Accessibility (Section 508)


Government Paperwork Elimination Act (GPEA)


FISMA/NIST Standards and Guidance

27

AWIPS Evolution


An EA Perspective


EA perspective focuses on hydrological and
meteorological mission operations


Administrative functions not part of AWIPS


Supercomputing functions outside of AWIPS


AWIPS II compliant with current DMIT data standards


Some services, e.g., discovery service, not
implemented as part of AWIPS II


Will be implemented in the Data Delivery system enhancement


Exploring partnerships with other agencies within and
outside of NOAA


NESDIS and NPOES NDE


NASA SPORT and NASA Goddard

28

Open Source Projects


Used in ADE

OS Project

Functionality

ADE Role

ANT

Build scripting

Build system

Mule + Spring

Enterprise Service Bus + Container

Runtime Environment

ActiveMQ

Java Messaging System Broker

JMS Broker

Jibx

XML to Object Serialization

Canonical XML Message

GeoTools + JTS

GIS capabilites

GIS primitives

Tomcat

Web Server

Test Client Server

Baltik

Scalable Vector Graphics Tools

SVG tools

Ehcache

Event Driven Clusterable Cache

Cache Framework

Log4j

Java Logging API

Log manager

Jogl

Java API to OpenGL

CAVE rendering interface

Eclipse RCP

GUI plug
-
in based framework

CAVE framework

Eclipse IDE

Java Integrated Development

Development environment

MC4J Console

JMX Management Console

Remote management console

29

SOA Architecture


Logical Layered View

Layers Separated By Simple APIs

Client/Presentation Services

Platform Layer

Mission Services
Layer

Data Access Layer

Metadata

Index

Data Persistence

Store

Enterprise Service Bus
-

Communication

Security Services

/
Demilitarized Zone (DMZ)

Spatial

Index

Hydro Models

LAPS

FORTRAN/C/C++

Command Line

Programs

External Programs

JMX

<<Java>>

DataLayer

PostgreSQL

HDF5

<<abstract>>

BaseDao

Hibernate

<<Java>>

HDF5DataStore

HDF5 API

IngestSrv

PersistSrv

IndexSrv

ProductSrv

AdapterSrv

NotifySrv

SubscribeSrv

AutoBldSrv

PurgeSrv

Mbean

CAVE

StagingSrv

UtilitySrv

Localization

Store

30

AWIPS
-
II ADE High Level System Services


SOA Services Running in an ESB Container

IngestSrv

StagingSrv

PersistSrv

PurgeSrv

IndexSrv

ProductSrv

NotifySrv

AutoBldSrv

Collaboration

CAVE

Visualization

Client

CORE SOA Services

Mbean

Mbean

Mbean

Mbean

Mbean

Mbean

JMX Remote

Service Management

Client

METAR Plug In

Radar Plug In

Satellite Plug In

FileSystem

RDBMS via JDBC

Meta Data Index

Spatial Data Base

uEngine

Manage Subscription

Subscription

Notify

Data Rendering

CAVE Bundles

CAVE Procedures

Wx Drawing

Wx Warning

Data Interrogation

Enterprise Service Bus


HTTP, JMS, Virtual Memory, File Endpoints

AdapterSrv

Mbean

Exec Adapter

JNI Adapter

Radar All Tilts

ADE 1.0

uEngine

Services Independent of End Points

Enterprise Service Bus


HTTP, JMS, Virtual Memory, File Endpoints

HDF5 persistence

XMPP

Updated T06

Hydro Visualization

WAN Visible

GRIB Plug In

UtilitySrv

Mbean

Localization Data

GFE IFPServer

Color Tables

Maps and Topo