A Data Access Framework

fizzlargeSecurity

Nov 3, 2013 (3 years and 9 months ago)

68 views

A Data Access Framework
for ESMF Model Outputs

Roland Schweitzer


Steve Hankin

Jonathan Callahan

Kevin O’Brien

Ansley Manke

20 July 2005

2005 ESMF Workshop


2

Some Motivation

Model Components produce …




Model Runs which produce …




Data and Metadata which



should/could produce …




A data portal which allows user to:


Search, browse, compare, validate and
analyze results …




w
hich in turn helps produce
insight

20 July 2005

2005 ESMF Workshop


3

Some More Motivation


ESMF and the ESCurator manage
components, runs and output


LAS is a candidate to provide a data
portal

20 July 2005

2005 ESMF Workshop


4

The Live Access Server (LAS)

A highly configurable Web server designed to provide
flexible access to geo
-
referenced scientific data

20 July 2005

2005 ESMF Workshop


5

LAS


LAS is an information broker that:


ingests descriptions of data


data sets


variables


(optional reshuffling of above into new categories)


domains: 1d
-
4d


special constraints (ensemble index)


rendering hints (smoothing, palettes)


presents the descriptions to the user in a
coherent fashion


asks other software to perform tasks for the
user


and gives the results to the user



20 July 2005

2005 ESMF Workshop


6

Architecture Components

User

Interface

Server (UIS)

Product

Server
(PS)

Backend
Server (BES)

Ferret

Data

(OPeNDAP)

Server

(FDS)





Backend
Server (BES)

Data

20 July 2005

2005 ESMF Workshop


7

LAS Offers


Independence from underlying data


Uniform interface for browse
(visualize), compare, validate and
analyze results for many data types


Easy way to make connections to
metadata and documentation


Customizable user interface

20 July 2005

2005 ESMF Workshop


8

Example (Multiple Hierarchies)

20 July 2005

2005 ESMF Workshop


9

Example (Collapse Hierarchies)

Experimental Development Server

20 July 2005

2005 ESMF Workshop


10

Example (Custom Headers)

The NVODS Server (http://www.ferret.noaa.gov/nvods)

20 July 2005

2005 ESMF Workshop


11

Example (Adopt a Drifter)

20 July 2005

2005 ESMF Workshop


12

Return to the Motivation

Model Components produce …




Model Runs which produce …




Data and Metadata which



should/could produce …




A data portal which allows user to:


Search, browse, compare, validate and
analyze results …




w
hich in turn helps produce
insight

20 July 2005

2005 ESMF Workshop


13

Data Browsing

20 July 2005

2005 ESMF Workshop


14

Analysis and Comparison


Compute
differences


Can define new
variables that are
the result of some
analysis operation
(done in delayed
mode so only data
that is needed is
computed)

20 July 2005

2005 ESMF Workshop


15

Analysis and Comparison


Can use FDS to define regriddings (or
other analysis) to allow more complex
comparisons


20 July 2005

2005 ESMF Workshop


16

Validation


Use FDS to create samplings of in
situ observations which can be
compared to model data


(This is a planned for a future
release.)


20 July 2005

2005 ESMF Workshop


17

Detailed Analysis


Can jump to local tool (Matlab, IDL,
Ferret, etc.) by requesting scripts for
that application


Scripts encode access to remote LAS
data via FDS


Can take advantage of FDS analysis
to compute some data reductions
and transfer only the result to the
local application

20 July 2005

2005 ESMF Workshop


18

Additional Flexibility beyond
Configuration


Use LAS framework with different
backend engine (new Product Server
is designed to make this as easy as
possible)


Web
-
services interfaces


To Product Server from other clients


Will allow implementation of “well
-
known” service like WMS


20 July 2005

2005 ESMF Workshop


19

Architecture Components

User

Interface

Server (UIS)

Product

Server
(PS)

Backend
Server (BES)

Ferret

Data

(OPeNDAP)

Server

(FDS)





Backend
Server (BES)

Data

20 July 2005

2005 ESMF Workshop


20

Access Control and the GRID


The new architecture makes participating as a
GRID partner easier


Web services
-
based authentication schemes can
be used by the Product Server and Backend
Server to get authentication information (if not
supplied by a browser certificate).


Implemented as another “filter” in a already
-
existing mechanism for filtering on IP
-
address,
number of accesses, etc.


Configuration metadata can be augmented to
identify allowed access “roles” by dataset and/or
variable (and maybe by output product).

20 July 2005

2005 ESMF Workshop


21

Return to the Motivation

Model Components produce …




Model Runs which produce …




Data and Metadata which



should/could produce …




A data portal which allows user to:


Search, browse, compare, validate and
analyze results …




w
hich in turn helps produce
insight

20 July 2005

2005 ESMF Workshop


22

More Information


www.ferret.noaa.gov


Subscribe to the LAS User’s mailing
list

20 July 2005

2005 ESMF Workshop


23

The End


20 July 2005

2005 ESMF Workshop


24

browser


UI (java/script)

gif,netCDF,etc.

servers












UIS

PS

LAS XML
configuration
metadata

browser

interface

BES

HTTP Request

(later SOAP)

scripts &/or

commands

products

HTTP Response

data

LAS Architecture with OPeNDAP

Legacy

OPeNDAP

OPeNDAP

Server

browser


OPeNDAP

Server

BES

BES

Legacy

OPeNDAP

UIS = User
Interface Server

PS = Product
Server

BES = Backend
Server

FDS = Ferret
Data Server

SOAP

addXML

FDS








20 July 2005

2005 ESMF Workshop


25

Architecture Components


User Interface Server (UIS)


keeps
browser up
-
to
-
date with relevant
interface widgets


Product Server (PS)


takes users
requests and returns results


Backend Server (BES)


runs legacy
application to fulfill request


Ferret Data Server (FDS)


General
OPeNDAP server for LAS data

20 July 2005

2005 ESMF Workshop


26

The UI Server Architecture


User Interface Server


Installation process translates LAS
configuration into User Interface Widgets need
to present datasets, variables, geo
-
spatial and
time extents in HTML


Allows dataset collection to be easily organized
into any hierarchies the installer chooses


Communicates requests to the Product Server
and returns responses (images, files, data
tables, etc.) to the Web Browser


Many other customization features

20 July 2005

2005 ESMF Workshop


27

The Product Server


Product Server


Has knowledge of the LAS configuration


Makes requests from one or more Back
-
end Servers


Fully implemented Web
-
services interface will be
available to client writers


Can interact with a series of Backend
-
servers to fulfill a
single request from the user client


Back
-
end server


Can produce product itself

or


Interact with a single legacy application (like CDAT or
Ferret) to produce product


Requests are communicated via Request and
Response XML that is exchanged via SOAP.

20 July 2005

2005 ESMF Workshop


28

FDS (Ferret Data Server)


FDS is an OPeNDAP server


Automatically installed with LAS


Therefore every gridded LAS dataset
is served via OPeNDAP


Has extended analysis capabilities
beyond basic OPeNDAP servers