sccc_presentation_JCothran

scacchicgardenΛογισμικό & κατασκευή λογ/κού

13 Δεκ 2013 (πριν από 3 χρόνια και 8 μήνες)

63 εμφανίσεις

Open Solutions to Regional
Observing Systems

Outline

Recent near real
-
time in
-
situ observations are aggregated to a
‘Xenia’ schema relational database(RDB) via its import XML
format ‘ObsKML’ schema. From this database of recent
observations a variety of file formats, web services and
applications can be driven. By suggesting a minimally
common observation oriented XML and RDB schema,
developed scripts and products can benefit from and build
around these shared schema. For more info contact
jeremy.cothran@gmail.com


Keywords

Scripting languages
Perl PHP

Relational Database
Sqlite Postgres

Mapping Engine/Interface
Mapserver OpenLayers TileCache

OpenGeospatial Consortium(OGC) services
WMS/WFS/SOS

Google Earth/Maps
KML/KMZ

Google Code/Analytics


Problems Xenia intended to address


Grants for research instrumentation which will be collecting
observation data while lacking a data management/sharing
component beyond basic file storage


Low
-
volume data(< 100,000 records per hour) in
-
situ
observational platforms or system arrays (e.g. 1 to 1000
platforms collecting 10
-
20 observations per hour) collecting
data at any geographic scale (local,regional,national,etc)


Bridging the gap between raw data collection and the
organization and sharing of data using previously developed
products, services and standards(leveraging earlier work
against new data providers)



Fostering a standardization of products and services via a
common openly shared technical infrastructure(common
database schema and product support scripts)


Problems Xenia not intended to address

High
-
volume data (millions of records per
hour) such as gridded model outputs, hf
radar, etc. High
-
volume data problems at
this time are better addressed using
traditional file processing techniques
where data management can suggest
output file formats(such as images,
shapefiles, etc) and metadata that are
conducive to search and usage needs.


Xenia Relational Database Schema

Hierarchy = organization
-
>platform
-
>sensor
-
>multi_obs(observation data)


ObsKML XML schema for import/export

Data flows


general

Data flows


ObsKML

ObsKML (Observations KML)


A simple XML encoding of

observation metadata

associated with a KML

Placemark.


Default XML import/export

format for Xenia database

Instances.


Postgres or Sqlite database paths

Xenia data organized by time interval

Latest

past several hours

New Data

Recent

0
-
6 weeks

Archival

3+ weeks to 1
-
2 years

Possibly table separated by year,month,etc

Archival file

1
-
2+ years

Files separated by product/year/month

Julian weekly divided database files(sqlite) of all
observations are available


http://tinyurl.com/6ctz66


Xenia aggregation, replication, redundancy

Xenia

A,B,C,D,E,F

Xenia

A,B,C

Xenia

D,E,F

Xenia

A

Xenia

B

Xenia

C

Xenia

D

Xenia

E

Xenia

F

Xenia Backup

A,B,C,D,E,F

Xenia Backup

D,E,F

Overlapping/redundant systems could aggregate/replicate via
SQL or XML


Flow/event monitoring

Around 10,000 observations are collected each hour from a variety of data
providers. The data flows are monitored for both successful import and
towards provided report metrics or notifications when problems or
events occur.

Sensor count graph showing
the hourly number of
successful sensor imports
per provider

More detailed
platform/observation table
showing daily counts color
coded to highlight specific
sensor issues

Applications


National Weather Service

Latest regional platform observations from Xenia database are styled to
html tables and presented in fixed map interface alongside NWS
forecast warnings/advisories and other map layers of interest

http://forecast.weather.gov/mwp/


Applications


SECOORA interactive map

Latest regional observation maps are accessed via WMS(Web Mapping
Service) and merged into an OpenLayers(browser javascript) map
interface. Observation database requests are supported via WMS
GFI(Get Feature Info)

http://secoora.org/maps/dynamic

Applications
-

CarolinasRCOOS

Map image layers supported via MapServer WMS with increased map
response time via TileCache which tiles and caches image requests for
repeated reference. Site feature info supported via Xenia database
instance styled output html table accessed via WMS GFI.

http://carolinasrcoos.org

KML/KMZ


Latest obs by platform

KML (Keyhole Markup Language) which is the XML format used to visualize data
in Google Earth/Maps and potentially other globes/maps with KML support
such as NASA WorldWind and ESRI ArcExplorer

Latest observation data organized by all observations per platform


http://tinyurl.com/664wtx

KML/KMZ


latest obs by obsType

KML (Keyhole Markup Language) which is the XML format used to visualize data
in Google Earth/Maps and potentially other globes/maps with KML support
such as NASA WorldWind and ESRI ArcExplorer

Latest observation data organized by observation type for all platforms


http://tinyurl.com/664wtx

All observations carry


* observation type(obsType)


* unit of measure(uom)


* measurement value

obsType+uom = measurement
type(m_type)

Color styled low to high value
blue/green/red

Development/documentation support

Google Code provides very simple, free browser based access to the
basic project support tools such as a subversion(svn)
code/documentation sharing/versioning, wiki documentation and issue
tickets.

Google Analytics provides free browser based access to website usage
statistics via a javascript project token included in webpage access.

Lessons Learned


Web GIS interface: php/mapscript
-
> openlayers/javascript/ajax,
tilecache


Latest in
-
situ obs: RDB


sensor/obs hard
-
coded/table approach
-
> obs
non
-
specific infrastructure(xenia,obskml)


Sensor/observation specific metadata: loosely link a variety of more
specific metadata files/schemas to common infrastructure


Hourly gridded data: RDB quickscat, model layers
-
>
ogr/sql+shapefiles


Full RDB to file
-
based RDB: postgresql+postgis
-
> sqlite+ogr,
sqlitegeo?


Self hosted to third party hosted: third party website hosting for static
content/context, common processing/storage. Google code/analytics
for basic shared development support/documentation.


ROA(Resource Oriented Architecture)/REST style simpler declarative
alternatives to complex query SOA(Service Oriented Architecture)

File formats and web services


File formats (CSV,shapefile,netCDF,KML/KMZ,Sqlite DB)


http://secoora.org/data


Free desktop analysis tools

MapWindows(shapefile/GIS)
http://www.mapwindow.com


ncBrowse(netCDF)
http://www.epic.noaa.gov/java/ncBrowse


ODV(ctd/csv)
http://odv.awi.de

Sqlite DB
http://sqlite.org




Web services (OGC WMS/WFS/SOS)



http://secoora.org/data

http://code.google.com/p/xenia/wiki/XeniaSOS


Credits

2
C.Calloway,
2
J.Cleary,
1
J.Cothran,
4
J.Donovan,

3
J.Dorton,
1
M.Fletcher,C.Galvarino,
1
S.King,

2
S.Haines,
3
L.Leonard,
1
D.Porter,
3
X.Qi,

1
D.Ramage,
2
H.Seim,
4
V.Subramanian,
1
S.Walker,
4
R.Weisberg


1
University of South Carolina,
2
University of North
Carolina at Chapel Hill,
3
University of North Carolina at
Wilmington,
4
University of South Florida