SENSOR TO USER

scacchicgardenSoftware and s/w Development

Dec 13, 2013 (4 years and 6 months ago)

198 views

Cooperative Agreement # NNS04AA78A

1

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

SENSOR TO USER


Progress Report

-

2
nd

Year (Q2
-
Q4)



Prepared by



Dr. Erick Malaret, ACT Corp. (703
-
742
-
094)

Email: malaret@actgate.com

Cooperative Agreement # NNS04AA78A

2

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Description and Objectives

Applied Coherent Technology Corporation (
ACT
), the Naval Research
Laboratory (
NRLSSC
) Oceanographic Division, and our
NOAA

partners,
are under a cooperative agreement with NASA to maximize the value of
NASA/EOS Earth science observations to decision makers related to
coastal, marine, and ocean environments.


ACT and NRLSSC are integrating real
-
time ocean systematic
measurements from NASA and NOAA satellites and available coastal
observations along with coastal ocean model output into an automated real
-
time database of ocean weather in the Gulf of Mexico.


This project focuses on monitoring coastal ocean properties and it is aimed
at providing new real
-
time support to NOAA efforts initially in the following
focus areas: Harmful Algal Blooms (HABs), Coastal Flooding /freshwater
discharge, and Hypoxia.


ACT in collaboration with NOAA will expand the project to other U.S. coastal
areas and add additional NASA,NOAA,DOD and foreign data sources.


Approach

The on
-
going effort is divided into an application component and a
and product research/development component.

•
The
Application Component

[
by ACT
]
–

provides an
architecture that addresses issues related to data ingestion,
processing, distribution, access, and decision support.
The key
Decision Support Server and Fusion Server will be installed at
NRL/SSC, followed by other servers at different NOAA sites.


•
The
Product Research/Development component

[by
NRLSSC]

–

provides state of the art oceanographic expertise and
research addressing issues related to generation of new
geophysical data products as part of a production pipe
-
line.

Schedule and Key Deliverables

Year 1:

Application Component

–

a) Req. Gathering, b) Install high end
HW/SW to ingest/process/serve data, c) enhance current Fusion
server & Decision Support Tool Technology.


Product R&D Component

-

….

Year 2:

Application Component

–

a)
Demonstrate widely available internet
access to all data levels, b) develop initial Hazardous Algal Bloom
product based on satellite/model

data in GIS and other compatible
formats, c) add new data streams (e.g. NPP VIIRS)


Product R&D Component

-

….

Year 3
-
5:
Application Component
–

a) Expand to other application areas,
geographic areas, and data sources, b) identify and customize
architecture to provide additional decision support tools, assist in
transition of application to operation.



Product R&D Component

-

….



Initial Implementation

2
nd

year Started May 2005

Note: Big Scope of Cooperative
Agreement Scope and participants
remains unchanged

Cooperative Agreement # NNS04AA78A

3

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

De


The next slides will concentrate on
Application Component progress in the
last 7 months



Cooperative Agreement # NNS04AA78A

4

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

WIPE

(server)

REACT

(CSC User)

Fusion Server at

NAVY/NRL
-
SSC

REACT

(NOS User)

REACT

(Field Analyst)

WIPE

(
servers
)

REACT

NOOA Fusion Servers at ACT



Web Browsers

(HTML and Applet Access)

SOAP

WSDL

HTTP/

CGI

Local

+Satellite Imagery

+MODELS

+ …

…

HAB Test
-
bed for NOAA: Present Network Configuration

NOAA/CW

•
MODIS data

NOAA/NDBC

•
Buoy data

FNMOC

•
Many models

OTHER

+in
-
situ obs.

+satellites Obs.

NOAA/CW

NOAA/NDBC

NOAA/NCEP

optional access


directly by REACT

NRL
-
SSC

•
MODIS OBS

•
MODELS

•
MODIS VAPs

FTP

&


HTML xfers

Final HAB Bulletin For Decision
Maker (user Type 3)

HAB bulleting creation Wizard

NOAA/NOS/CSC Analyst Team (user Type 2)

Data providers

(user Type 1)

Cooperative Agreement # NNS04AA78A

5

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Apply WIPE/REACT

To HAB’s needs by

NOAA

Cooperative Agreement # NNS04AA78A

6

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

DATA INGESTION PROGRESS

Cooperative Agreement # NNS04AA78A

7

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Cooperative Agreement # NNS04AA78A

8

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

•
Ingestion Data Aspects and Engines (continue)

–
NRL Model Data

•
Model data ingestion engines were enhanced in REACT to rapidly open any of the NRL
volumetric NetCDF files. The files can be open via the REACT interface.

•
Visualization tools were added to REACT to simplify the inspection of volumetric model
data.

–
Layer animation

–
File animation

–
Data inspection with row/col/depth proves

–
Interactive Intensity Mapping

–
NRL Level 2
-
4 HDF Products

•
Integrated the ability to read any of the sub
-
datasets included in the NRL HDF files.

•
The REACT interface provides a hierarchical file layout over the file system and the
HDF file

•
User can load desired set of sub
-
dataset layers simultaneously, and remap into desired
cartographic projection

•
Visualization capabilities similar to the model data were added.

Done: multiplicative and additive scaling constants are applied.


–
NOAA Coast Watch

•
After Meeting with Coast Watch it was clear that ALL NOAA/CoastWatch products are
being saved in CW HDF format.

•
Added capabilities to read/load all the sample data provided by CW to ACT

–
NOS Needed Data

•

Incorporated the Buoys data and filtering algorithms

•
In
-
situ observations can not be accessed automatically at this time

DATA INGESTION PROGRESS (continue)

Cooperative Agreement # NNS04AA78A

9

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Data Ingestion and Status of WIPE Server(s)

NRL/SSC

•
The WIPE server at NRLSSC has been working since October 05.

•
NRL/SSC has given security approval to the machine, hence it can be access over
the internet as a web server.

–
The web page is:
http://cesium.nrlssc.navy.mil


•
Met with NRL/SSC at ACT, and identified automatic process to populate the WIPE
server at NRL/SSC with both MODIS L2
-
L4 products and with MODEL data.

–
The advantage of the approach taken is that it does not replicate in any way Hark Disk
storage at NRL/SSC. It uses the same hard disk area that their DODS server is using.

•
Training on WIPE was done in November

–
ACT is presently incorporating the suggestion received from NRL




Basemap MODIS COMPOSITE NCOM TEMP

Cooperative Agreement # NNS04AA78A

10

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Cooperative Agreement # NNS04AA78A

11

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Cooperative Agreement # NNS04AA78A

12

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Data Ingestion and Status of WIPE Server(s) (continue)

NOAA CW Servers at ACT:

•
Working. Tested with Sample CW
-
HDF format data. All modes
working, except the Swath data sets.

•
Bow
-
tie projection problems in the MODIS L1B data.

–
To save computational time we were not using all the ground control
points in the MODIS data. This resulted in visible bow
-
tie projection
problems that were not being handled correctly.

–
Investigated used of third party SW


decided against it after
discussing with NRL/SSC that all that is needed is to use ALL the GCP
information.

–
Made significant modifications to the processing of Ground Control
Point to generate a properly corrected geo
-
referenced image.

–
The new approach is more computationally intensive, hence it takes
longer to generate a full area mosaic at full resolution.



5
-
10 minutes.


Cooperative Agreement # NNS04AA78A

13

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Status of HAB WIPE Server at ACT

•
Presently working on getting all the data automatically pushed into the WIPE server
at ACT.

•
Some of the cell count data is at FTP sites. Presently embedding at FTP library into
WIPE to simplify process of getting data from remote servers via FTP.



Data Ingestion and Status of WIPE Server(s) (continue)

Cooperative Agreement # NNS04AA78A

14

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

SOFTWARE DEVELOPMENT PROGRESS

Cooperative Agreement # NNS04AA78A

15

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

•
MSHELL

-

Image
Processing Interpreter
Language developed by
ACT. The language has a
high level syntax similar to
Matlab and IDL.


•
ProVIEW
-

Professional
Windows Environment for
Windows (ProVIEW).
ProVIEW has extensive
capabilities in image
processing, analysis, and
map generation
.


•
WIPE
–

WWW Information
Processing Environment.
Data Assimilation,
Processing, and distribution
system with batch and
interactive capabilities.

MSHELL Provides the Computational Engine Used by ProVIEW & WIPE

ProVIEW Provides Many of the Graphical Capabilities Used Under WIPE

WIPE’s building blocks have been used in multiple mission critical operations

Rapid Environmental Assessment Composition Tools (REACT)

has been implemented and continues under rapid development.


It captures elements of MSHELL+ProVIEW+WIPE

all under an easy to deploy stand alone application, and

portable GUI environment.



During 2
nd
-
3
rd

quarter continue expanding the tool space in REACT in
response to needs of the HAB needs


CORE SW Development Progress

Cooperative Agreement # NNS04AA78A

16

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

•
As part of our NASA Cooperative Agreement, ACT has been participating in the
Technology Infusion working group. The Technology Infusion has identified many
capability blocks required in systems to support the EOS needs.


•
REACT inherits key components of the ACT’s SW remote sensing SW tools

–
MSHELL

-

interpreter language with extensive capabilities for multidimensional data


manipulation

–
ProVIEW

-

graphical user interface around MSHELL

–
WIPE

-

provides fusion center architecture, with over 200 streams


•
ACT has been using REACT and WIPE as a platform/architecture to demonstrate
technology infusion from the OpenSource community, i.e.

–
Qt

–

provides a portable library in C++ that allows GUI and system level support (
A prototype
version of REACT has been implemented for LINUX and Windows.)


–
GDAL

–

the Geographic Data Abstraction Library is used by REACT to perform raster IO over
many geo
-
graphic data formats



new formats can be easily added following an OpenSource approach empowering developers
to rapidly contribute in the adaptation of new formats. GDAL has been extended to

support : CW
-
HDF, NRL
-
HDF, PCI
-
AUX (extended SRS support)

–
gSOAP

–

used to SOAP enable REACT


the
developer needs to primarily concentrate on the
core subroutine functionality.
CONTINUED EXPANDING THE SOAP INTERFACE

–
PROJ4

–

a cartographic library with support to most of the projections used by USGS

–
SQLITE
–

a built
-
in SQL engine

–
OpenGIS/OGC

–

describes protocols/methods used for the request of products (This is work in
progress)


CORE SW Development Progress (continue)

GDAL + PROJ4 +


GSOAP+…

MSHELL

(Interpreter+ scripts!)

REACT

Cooperative Agreement # NNS04AA78A

17

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

•
REACT is been configured to
provide Web Services

for geo
-
spatial data

–
Web Services Requirements are:

•
Available over the Internet

•
Not tied to any operating system or programming language

•
Self
-
describing via a common XML grammar

•
Discoverable via a simple find mechanism


•
REACT captures a WIPE like type of interface for accessing inventory queries and
generating mosaics, a set of five core functions are being added to the
REACT/MSHELL scripting language resulting in a
LARGE space of remote
computing capabilities/options
! (implemented using SOAP)

–
$result

= rmshell_execute($server, $scriptCommandList )

–
Data

= rmshell_getVar( $server, $variableName)

–
$result

= rmshell_putVar( $server, $variableName)

–
$result

= rmshell_getFile( $server, $inFile, $outFile )

–
$result

= rmshell_putFile( $server, $inFile, $outFile)

ADDITIONAL FUNCTIONS WERE ADDED TO HANDLE ASYNCHRONOUS REQUESTS

•
The above functionality can be exported to other applications via WSDL (an XML
grammar for specifying public interface to the web service)


GAVE A PRESENTATION TO THE ESIP FEDERATION MEETING IN SAN DIEGO, ON
WEB SERVICES USING WIPE&REACT.

CORE SW Development Progress (continue)

Cooperative Agreement # NNS04AA78A

18

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

REACT

REACT

REACT

WIPE

Script/algorithm or data

result or data

Script/algorithm or data

result or data

WIPE and REACT share the same processing engine

REACT is configured to handle local user and a small number of remote applications.

REACT can support remote applications as a WIPE
-
LITE system

Client side

(Field System)

Server side

(Field or Fusion Center System)

Web Enabled

Application

WIPE

Or

REACT

Script/algorithm or data

result or data

CORE SW Development Progress (continue)

Cooperative Agreement # NNS04AA78A

19

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

•
REACT can be seen as a WIPE
-
LITE system with more flexible visualization
capabilities, and network centric capabilities.

•
A
prototype windows version

of REACT can be downloaded from ACT, i.e.


http://www.actgate.com/reason



CORE SW Development Progress (continue)

Cooperative Agreement # NNS04AA78A

20

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

DECISION SUPPORT TOOLS PROGRESS

Cooperative Agreement # NNS04AA78A

21

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Decision Support Tools means different things to different users, e.g.

–
Data Provider


need coverage and cartographic
maps/mosaics with simple to use query engines

–
Analyst at Fusion Center


•
Visualization Tools, e.g animations of layers, volumetric vis.

•
Implementation of a Process in the form of a Wizard

•
Analysis: Multivariate Detection/Estimation/Prediction/…

–
High Level Decision Maker, e.g. Coastal Manager



GO/NoGo results with access to supporting data


•
ACT has been working during the last year in the above end
-
user
categories with requirements driven by the needs of:

–
NRL,

–
NOAA/CW

–
NOAA/NOS

–
NOAA/CSC

–
NOAA’s end
-
users

DECISION SUPPORT TOOLS …. The Challenge

Cooperative Agreement # NNS04AA78A

22

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

•
NOAA/NOS, NOAA/CSC, NASA/SSC, NOAA/NCDDC, and ACT met in February
-
05
to discuss the HAB Bulleting creation process and future architecture developments.


•
As the result of the meeting a number of concrete suggestions were provided by NOS
and CSC. (see following slide).

DECISION SUPPORT TOOLS PROGRESS (continue)

Cooperative Agreement # NNS04AA78A

23

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Development Operations Delivery

•
[done]Use WIPE for


new data stream


ingestion

•
[done]Easy Incorporation of


other geographies

•
[done]Mosaic Images

•
[done]Alternate look from


same data => flicker

•
[done]View data values
from multi
-
sensors at same


time

•
Skill assessment


match in
-
situ data and


imagery

•
[done] Output data

•
[done] Oriented geo
-
tiff

Requirements

•

[done] Save view settings

•

[done] Archive data used in view

•

[done] Access from multiple sites

•

[done] Use of consistent color tables


(use same tables as CW)

Outcome: NOAA/NOS/CSC requested ACT to demonstrate how WIPE/REACT could be
used reproduce current HAB Bulleting


creation process in REACT/WIPE

Possible Advantages:

•
Proposed approach could provide clear view of process and problems

•
Need web centric creation process (can be a browser or application)

•
Explore REACT/WIPE as next generation environment



DECISION SUPPORT TOOLS PROGRESS (continue)

Cooperative Agreement # NNS04AA78A

24

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Cooperative Agreement # NNS04AA78A

25

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Cooperative Agreement # NNS04AA78A

26

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Cooperative Agreement # NNS04AA78A

27

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Cooperative Agreement # NNS04AA78A

28

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Cooperative Agreement # NNS04AA78A

29

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

HTML output

+ PDF output

Cooperative Agreement # NNS04AA78A

30

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

SPEED Data Cube Generated
on the Fly from U&V

components

Cooperative Agreement # NNS04AA78A

31

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

EXAMPLE OF READING AND DISPLAYING: generic formats, NRL’s
MODEL DATA, MODIS L1B, CW Products, …

Cooperative Agreement # NNS04AA78A

32

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

EXAMPLE OF READING AND DISPLAYING NRL’s MODEL DATA

DATA

Probes

On

Active

Image

Cooperative Agreement # NNS04AA78A

33

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

EXAMPLE OF READING AND DISPLAYING NRL’s MODEL DATA

Metadata information

Easy

Access to

Any Layer,

Including

Volume Animation

Cooperative Agreement # NNS04AA78A

34

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

EXAMPLE OF NRL L2
-
L4

File System Navigation and

HDF Data Structure Navigation


Cooperative Agreement # NNS04AA78A

35

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Cooperative Agreement # NNS04AA78A

36

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Meetings/Training/Demos

–
Supported Different ESIP Federations Meetings

–
Gave Training on WIPE at NRL (2 days)

–
Explore Training on REACT (over the WEB)

•
For Example, Gave Presentation to CSC on REACT using
VNC


Cooperative Agreement # NNS04AA78A

37

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

ACT’s REASON

Web page Is up an running

http://www.actgate.com/reason


Provides:

•
Report Summaries

•
Actual Data Access

•
DST Demos Products

•
Tutorials

•
…



…LIVE DEMO …

Cooperative Agreement # NNS04AA78A

38

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

•
NRL Related

–
Work over WIPE enhancements/customization requested by NRL
-
SSC

•
Custom product generation

•
Virtual products on the fly…

•
Legend

•
…

–
Further Training to NRL on Use of WIPE/REACT

•
NOAA Related

–
Fine
-
tune HAB Wizard based on inputs from NOAA/NOS/CSC/…

–
Provide Training as needed to NOS and CSC

•
ESIP Federation Related

–
Demonstrate Web Services Capabilities of WIPE/REACT (SOAP/WSDL)

•
Improve interfaces with OGC and OpenDap

•
Demonstrate with ESIP Federation Web Services Chaining (using WIPE as a prototype)

•
Use REACT as a test case for Technology Infusion Process being Developed by the SEEDS Working
Group on Technology Infusion


•
Investigate interface WIPE within NOAA/CLASS Architecture


•
Start Investigation Next Year Target Application Areas Based on Team
Recommendations

Next Quarter Main Effort …

Cooperative Agreement # NNS04AA78A

39

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Summary


•
All ACT WIPE/REACT resources are accessible via ACT’s reason web page, i.e.

–
http://www.actgate.com/reason


–
On
-
line demos are being constructed to simplify end
-
user training


•
NRLSSC WIPE Server is actively receiving NRL data streams

–
NOAA can access all NRL data directly from WIPE/Applet interface or from WIPE/REACT
interface! (over 1,200 different fields populated daily)


•
REACT has matured to a state that NOAA/NOS/CSC can:

–
Test its use for HAB analysis tools

–
Test its use for HAB bulleting creation

–
ACT WILL WORK WITH THEM IN SIMPLIFYING USAGE OF TOOL


•
WIPE Axon Architecture for Decision Support Tools is working

–
Need testing and use …

Cooperative Agreement # NNS04AA78A

40

Applying NASA/EOS Data to Coastal Zone Management Application Developed from Integrated Analyses

SENSOR TO USER

Possible Items for Discussion


•
Are we heading (REACTing) in the right direction?


•
How can we help make the NRL products easier to evaluate by NOAA?


•
How can we make the emerging architecture more useful to NOAA?


•
How can we make the large volume of unique data more useful to a larger audience?


•
Can the data in the WIPE servers be accessed by non
-
team members?

–
The goal is to raise awareness of the many products available and the web
-
services infrastructure


•
Can we interface WIPE better with NOAA’s CLASS architecture?


…NRL and NOAA suggestions go here …