NOAA Environmental Information Service (NEIS)

addictedswimmingΤεχνίτη Νοημοσύνη και Ρομποτική

24 Οκτ 2013 (πριν από 3 χρόνια και 9 μήνες)

81 εμφανίσεις

NOAA Environmental

Information Service

(NEIS)

CIRA Seminar

August 22
nd
, 2012


Jebb

Q Stewart

Chris
MacDermaid

Randy Pierce

Jeff Smith


CIRA in collaboration with

NOAA/OAR/ESRL/GSD

Agenda

Overview

NEIS Back
-
end

NEIS Frontend

Demonstration

Future

Questions

ESRL Global Systems Division

Team

Program Manager


Jebb

Stewart


Research Team



Chris
MacDermaid



Woody Roberts*


Darien Davis*




Jeff Smith


Mike Romberg*



Paul Schultz*


Development Team



Julien

Lynge

**



Randy
Pierce




Eric
Hackathorn
*



Jeff Smith



Chris
MacDermaid



* Federal Employee


** Cooperative Institute for Research in Environmental Sciences (CIRES)












ESRL Global Systems Division

Background

GSD Director Discretionary Funds (DDF) Proposal


NEIS Requirement



Fast access to any data, any time, right now, on any platform.


NEIS Group Objectives


Build upon NEIS work in the past


Assess GSD capabilities pertinent to the concept of NEIS.


Evaluate existing, developmental technology, and solutions relevant
to concept.


Evaluate envisioned needs of primary users.


Provide recommendation of how to best use GSD resources to
contribute to a National capability.

ESRL Global Systems Division

Evaluation

Interviewed/Discussed concepts with variety of groups across
NOAA Line Offices. More than 15 individuals.


Questions Asked:

How are people using your data?

How are you using NOAA data yourself?

What tools/systems/services/standards are you using?

What related recommendations/lessons learned exist?

What obstacles or roadblocks have you encountered with
using/disseminating NOAA data?


Listened for items to help facilitate projects or help NOAA data
become more interoperable.


ESRL Global Systems Division

NEIS Findings


Existing metadata systems contain stale or obsolete information and are
not useful



Search for common term like wind returns hundreds of results.



Expertise required to understand what metadata you are looking for.


Many “Stovepipes” but no ability to combine/integrate data from different
disciplines (Ecological, Meteorological, Oceanographic, Fisheries, etc...)


Data sets are large and getting bigger. Future data sets will be difficult if
not impossible to move around network to consumer applications.


Cross data source data processing service does not exist. Some data
processing services exist but usually only for data available from that
particular service.


Feedback (broken links, data usage, etc…) to data providers and
consumers is non
-
existent.

Many more…





ESRL Global Systems Division





xkcd.com

THREDDS


LAS



ERDDAP

Google Earth


NASA World Wind

GeoPortal

MERMAid


REVERB


iRODS

Grads Data Server

OGC Services




More…


Lots of Great Technology Exists


Biological

Chemical

Physical

Biological, Chemical, and Physical data are all interrelated


Carbon
Tracker

FIM

Data Integration

Concept

Take advantage of existing technology where we can to quickly find and
access to data 100 of years in the past to 100 years in the future, from the
top of the atmosphere to the bottom of the ocean in a single application.

Framework provides capability to answer questions that require data from
different data sources.







Applying existing concepts to NOAA data.

Improving the User
Experience

How does industry make sense of
massive amounts of information?

NEIS Committee and

Group Involvement





ESRL Global Systems Division

NEIS Team is aware and members of many related organizations


We have active membership in a variety of committees and groups including:

Open Geospatial Consortium (OGC) Standards Committees

NOAA Environmental Data Management Committee (EDMC)

NOAA Data Management Integration Team (DMIT)

NOAA Unified Access Framework (UAF) group

NSF Earth Cube

NextGen
/CSS
-
Wx

Programs

FIM/WRF Modeling

GIS Committee

NOAA Climate Program Office (CPO) Data Interoperability Team


GSD Central Facility has vast experience in working with data and standards

NEIS Back
-
end

Randy Pierce

ESRL Global Systems Division


Automatically find (harvest) data from disparate data sets


Leverage existing metadata, standards, data sets, map servers


Search (think about Google, Kayak, and Amazon, except for
data)


Temporal


Geospatial


Text


Facets


REST API for search and for time
-
collated data retrieval


Continued

ESRL Global Systems Division

NEIS
-
core Back
-
end
Subsystem Goals

ESRL Global Systems Division

ESRL Global Systems Division

NEIS
-
core
Back
-
end
Subsystem Goals (Cont.)


Administrative interface for managing data sources, authorization,
etc.


Deploy on


Cloud servers


Local systems


Platform independence


Leverage existing code (Java)


Framework for future core services

Utilizing
Existing

Data Set
M
etadata



NOAA directive to use metadata standard ISO
-
1911


T
he
current "best practice" standard for geospatial
metadata. Metadata provides for (at minimum)

o
Data identification

o
Data extent

o
Spatial and Temporal Reference

o
Online resource location


NEIS
-
core
must Harvest from any
Geoportal

server
using the
Geoportal

REST API.

Where is the Data?

Geportal

server

ISO
19115

metadata

metadata

metadata

NEIS
-
core

Neiscore

Metadata

record

Neiscore

Metadata

record

Neiscore

Metadata

record

Search API

Data Retrieval API

TerraViz

Retrieves data

Directly

from server

3

NEISCORE
harvests from

Geoportal
, publishes NEISCORE

metadata
by
combining ISO

metadata
and
server specific

metadata

4
Search

and
retreive

data
urls

ISO

metadata

Specific

metadata

SOLR

2
Geoportal


knows about data ,

publishes
iso

19115 metadata

1
Data provider

Server publishes

specific metadata

metadata

metadata

metadata

metadata

metadata

Data Server

Data Server

Data Server

metadata


Process
creates
NEIS
-
core
Metadata Records


Retrieve ISO 19115 metadata records from a
Geoportal

Server,
parse each record for data identification elements like abstract,
text, tags, temporal extents, geographic extents.


Retrieve online resources metadata (like capabilities file for
WMS).


Combine subset of online resource metadata with elements from
the ISO 19115 metadata record to produce the
NEIS
-
core
metadata record


Store
NEIS
-
core
Metadata in the search subsystem as a
document.

Harvesting Metadata



Standalone enterprise search server with REST and Java API.


Index documents via Java API.


Query via Java API, producing XML, JSON, or binary results.


Advanced Full
-
Text Search Capabilities:


Facets


Declarative Analyzers


Full Document Storing (allows recreation)


Supports geospatial searching.


Optimized for High Volume Web Traffic


Continued

SOLR Search

Subsystem from Apache


Standards Based Open Interfaces


XML, JSON and HTTP


Scalability
-

Efficient Replication to other
Solr

Search Servers


Embeddable
-

in a Java web framework or application


Very fast and efficient searching
-

extends
Lucene

SOLR Search

Subsystem from Apache (Cont.)



High
-
productivity web framework for the Java platform


Re
-
uses Java technologies, Hibernate and Spring


Seamless mechanism for re
-
using existing java code


Powerful, consistent, and low maintenance persistence
framework (GORM)


Powerful view templates using GSP (Groovy Server Pages)


Continued

Grails Framework



Dynamic tag libraries


Customizable Ajax support


Complete development mode, including web server and automatic
reload of resources


Straightforward deployment model providing development, test, and
production environments, extendable to other custom environments if
needed


Provides for easily implemented REST services

Grails Framework

(cont.)


Better searching, perhaps natural language


Harvest more protocols
-

smarter harvesting


Analytics for server side processing


Scaling, multi
-
node services

Future
Core
S
ervices

NEIS Frontend

Jeff Smith

What is
TerraViz
?


3D visualization tool for Earth
datasets developed in conjunction
with NEIS


Developed in Unity, a popular 3D
game engine


Leverages the power of GPUs
(graphical processing units). For

example,
TerraViz

can load (and

render) 2.6 million polygons from the
FIM G9 global model and performs
quickly

What is
TerraViz
?


Since we control the source code,
we can customize
TerraViz

however
we wish and add any feature we
need


Develop once, then will run on
Windows, Mac, web browsers,
iPhones,
iPads
, Android devices,
and even game systems (Wii, Xbox
360, etc.)


Our source code is written in C# (a
natively compiled language very
similar to Java in syntax)


Unity Development


Unity is primarily used for game development, but groups in NOAA and
NASA are using it for data visualization


Developing code in a game engine is quite different than traditional
languages. Some challenges include:


Thinking in 3D space
--
rotations, coordinates, quaternions (
x,y,z,w
)


Considering frame rate
--
how many frames/sec can the code generate? If too
slow, the visualization will stutter


Camera perspective and movement


Creating game objects (e.g., a sphere for Earth)


Creating 3D meshes representing environmental objects (e.g. mountain
topography)


Applying textures to meshes (e.g. satellite imagery)


Understanding light
-
mapping, z
-
order,
shaders
, rendering, etc.

Why
TerraViz

and

not another tool?

Why not Google Earth?


Handles ~10,000
kml

polygons before slowing to a crawl


Closed system (Google Earth source code not available) so we can’t add needed
features


Google Earth (web) plugin works on Windows and Mac 32
-
bit browsers only (won’t run
on iPhones,
iPads
, or Android devices)

Why not NASA
WorldWind

/
iGlobe
?


WorldWind

(and
iGlobe

offshoot) is now open source (not sure how much NASA will
continue to support it)


Requires Java (so it won’t run on iPhones,
iPads
, Android devices, etc.) and Java
numerics

are significantly slower than
TerraViz

native
numerics

Why not Integrated Data Viewer (IDV)?


Open source but is one million lines of code (a lot to learn) and original two developers
have now left
Unidata


Requires Java (so it won’t run on iPhones,
iPads
, Android devices, etc.) and Java
numerics

are significantly slower than
TerraViz

native
numerics


Very feature rich application, but current user interface is very complicated



TerraViz

Search Screen

Facets




Keywords



Geographic extent



Date / Time Range



TerraViz

can load local data or search the NEIS backend (> 1100 datasets so far)

TerraViz

Collaboration





Annotation


Basic drawing
capabilities on top of the
globe (and any loaded
datasets)


Multiuser/Collaboration


real time screen sharing with
web cams




30

TerraViz

Datasets





TerraViz

can currently display


KML


FIM native grids


movies


images served up via web map
servers (e.g.
NetCDF
)

Earthquakes

NORAD satellites

FIM global weather model

GOCE
gravimetry

31

TerraViz

Maps / Terrain





Progressive
disclosure of terrain
(as you zoom in you
see higher resolution
satellite imagery)


3D topography


Ability to fly to
locations on Earth

32

TerraViz

Multiple Datasets

Time wheel





Can load multiple
datasets over the
globe and change
the transparency
between them


Time wheel
shows when data
is available for
each dataset and
can handle
different time
intervals (e.g. one
dataset could be
daily and another
monthly)

TimeWheel


Drag it up or
down to
move
forward or
backward
through time

TerraViz

Future Work


Support more data types


Native support for
NetCDF

and GRIB2

(don’t rely on web map servers to

create images)


Implement more of the KML stack


Improve analysis tools (point probe,

transect, averaging, graphs and charts)


Possibly switch to TOAST map projection

(like World Wide Telescope) or another map projection for improved
accuracy at extreme latitudes


Improve stability and error handling

Demonstration

Future

Build new Graphical User Interface system integrating and leveraging
new and emerging technologies to meet NEIS goal ‘any data, any
location, any platform, now’



Perform processing within cloud environment and with high speed
connectivity to data sources, taking advantage of large processing
power within clouds.



Send graphics and server side processed/rendered/streamed data to
GUI, improving bandwidth utilization.



Take advantage of fast networking to make remote requests and
processing appear like local application.



Similar to how the new Amazon Silk Browser works.


Looking to the Future

ESRL Global Systems Division

Improve existing Metadata services making information useful.



Improve searching by understanding what data means and
providing improved filtering capability similar to how airline or hotel
type search engines work.


Incentivize/encourage people to use proper Metadata.


Create a Metadata Dashboard to:

o
Gather information from users on relevance, ratings, usage patterns, search
key words used (
ie

crowd sourcing).


This information can be used to determine what users are searching for,
how they are finding data, what data they are not finding, etc...

o
Provide feedback on broken links, service uptime

o
Provide feedback on adherence to standards (Services, Metadata).

o
Provide feedback to users of similar or related data sets.

Looking to the Future

Part 2

ESRL Global Systems Division

Data are ever increasing in size.



New Polar Orbiting NPP data (~ 4 TB / day)


GOES
-
R


New Global Forecast Models, rapidly increasing in size.


We want to:



Minimize data we transfer.


Avoid data duplication.


Make data appear local to users.


Allow users to collaborate.


Dealing with Big Data

ESRL Global Systems Division

NOAA Vision
-

2020


Most of NOAA's environmental data will be accessible via web
services and will be described by good metadata


A central and intelligent search engine, that has harvested the
metadata of NOAA's datasets, will exist to find data


NEIS server side analytics/processing will enable
subsetting
,
aggregating,
diff'ing
, and other operations to be performed on data


Data will be efficiently streamed to a variety of platforms and clients,
including mobile devices


TerraViz

will be a general purpose client running on multiple
platforms capable of searching for NEIS data and visualizing it in
novel ways


Framework is extensible, other groups can add knowledge and
services.

39

ESRL Global Systems Division

Impacts




NOAA data ready for action. Services model facilitates agile response to
events. Services can be combined or reused quickly


Any data available through NEIS system can be operated on or
combined with other data. Integrated standardized formats and access.


New and Existing systems have access to wide variety of NOAA data.
Any new data added, easy incorporated with minimal to no changes
required.



ESRL Global Systems Division

Next Steps




Continue presentations and discussions with community.


Continue to look for partnerships and collaborations to help build out
concepts.


NEIS is listed in NOAA FY 12 Annual Guidance Memo (AGM)

“Define requirements for a NOAA Earth Information System (NEIS) to
provide integrated access to a wide range of NOAA and external
environmental data sets.”





ESRL Global Systems Division

Questions?

Jebb.Q.Stewart@noaa.gov

http://www.esrl.noaa.gov/neis

Backup Slides


Interviews






Person

Role/Responsibility

Michele Jacobi,

NOS ORR/ERMA.

Just finished OAR detail
with
GoogleOceans
,
part of DWH ERMA response team

Karen Sender

NMFS inPort

Katie Fisher

NOS Center for Operational Oceanographic Products and
ServicesHarmful Algae Bloom POC

Dan
Pisut

NESDIS NVL

Chris Ortiz

Space Weather Center

Charley Alexander

IOOS

Tiffany Vance

PMEL

HPCC

Glen Rutledge

NOMADS

Ted Haberman

NGDC

Tony Lavoi

CMSP, CSC, NOAA GIS

Tim Haverland

NMFS GIS group

Cecelia DeLuca, OAR

OAR

NESII

Steve Hankin

PMEL

John McDonough

OER

Mike Little

NASA

Cathy Smith

OAR/PSD

Jeff de La Beaujardiere

NESDIS/OSD/TPIO

also
DMIT