Using Python to implement a Hydrologic Information System for Texas

scacchicgardenΛογισμικό & κατασκευή λογ/κού

13 Δεκ 2013 (πριν από 3 χρόνια και 6 μήνες)

86 εμφανίσεις

Using Python to implement
a Hydrologic Information
System for Texas

James Seppi

(CRWR, University of Texas at Austin)

Dharhas Pothina

(Texas Water Development Board)

Ernest To

(Espey Consultants, Inc)

Tim Whiteaker

(
CRWR, University of Texas at Austin
)

Andrew Wilson

(Texas Water Development Board)


Texas Water Development Board


Provides
leadership

and
scientific expertise

for
development and conservation of
water


Accumulate and publish
large
-
scale State water

databases

in a form accessible as
WaterOneFlow

web
services.



Create a Texas
-
centric HIS
Water Data for Texas:


Partnerships

with Federal and Texas
agencies

to
obtain data


Inherits National HIS

technology


Adds Texas
-
specific services

to


provide ease of use and a level of


quality control

Why a Texas
-
specific HIS?


National HIS technology available but uptake low among
Texas agencies


lack of motivation


lack of time/resources


lack of experience with the technology



National HIS doesn’t quite do everything TWDB needs.


Parameter mapping


Unit Conversions


Web Viewer


i.e.

learn swahili

Put your data in an
ODM database

original paradigm

this didn’t get much traction…


Data is usually already in the format the
organization needs.


Changing formats/delivery mechanisms is a
pain.


ODM DB Schema doesn’t always fit the needs of
data collecting agencies.


Resources. Resources. Resources.


Overall response: yawn


new paradigm… make it easy


Form partnerships to obtain data


Provide technical know
-
how and support


Provide multiple paths to serve data


Provide a useful product with branding that is
available to the general public can use.


Scheduled updates to datasets



i.e. we have trained
swahili

translators
available please speak to us

1. Existing WaterML
compliant web
services within
Texas

Four Components of

2. Translation
services to
convert non
-
WaterML data
sources into
WaterML

4. Access Tools for
data discovery
and download:

GEMSS

(TNRIS hosted map based
web interface)

HydroExcel,


HydroDesktop,


HydroGET…

3. WDFT Central
Registry service


Limited
Parameter Set


List of Texas
HIS data
sources


Unit
conversions

WaterML

compliant

web

services

Why python?


Built
-
in cross
-
platform compatibility


Availability of a massive variety of scientific
libraries/modules


Amazing community support. Better than
commercial vendors


$FREE!$

Python @TWDB



pINT
-

P
ython
IN
strument
T
oolkit


SWIS
-

S
urface
W
ater
I
nformation
S
ystem


WOFpy
-

Py
thon
W
ater
O
ne
F
low Wrapper


pyHIS
-

P
ython
HIS

Client


pyHAT
-

P
ython
H
ydrographic
A
nalysis
T
oolkit


pyselfe/pug


hydrodynamic model analysis &
visualization

WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
WOFpy concept

Native

database

ODDataloader

SSIS scripts

SQL views

Manual loading, etc.

WaterML

Generic

ODwebservices


Significant learning curve


Data persists in ODM under Microsoft SQL server


i.e., not in native format and environment


“When is the next update?”


“What if data is collected in real
-
time?”

Data in ODM

Old paradigm
(i.e.

ODM db as intermediate)

WaterML

for GetValues

Old paradigm
(i.e.

ODM db as intermediate)

Native

metadata

WaterML

for GetSites,

GetVariables,

etc.


Real
-
time
data on
web

Metadata in ODM

Customized webscraper web service

Generic

ODwebservices

Loading scripts

Old paradigm
(ODM db as intermediate)

Native

metadata

Real
-
time
data on
web

WaterML

for GetValues

WaterML

for GetSites,

GetVariables,

etc.


Metadata in ODM

Customized webscraper web service

Generic

ODwebservices

Loading scripts

WaterML


WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
New paradigm
(i.e.

single framework)

Single framework


Dynamically map between native database
schema and WaterML


Allow data to persist in native database


Accomodate different database environments
(MSSQL, SQLite, etc)


Handle both static and real
-
time data


Dispatch and manage WaterOneFlow services
in both SOAP


and REST


Open
-
source and fully


customizable

WaterML


WOFpy architecture (backend


data access)

Native

metadata

Real
-
time
data on
web

Native

database

Data persists
in source’s
native db
environment

Source
-
specific

DAOs

(Data Access

Objects)

Real
-
time
data on
web


WOF module assembles
WaterML
elements

to WaterML
responses
.


Implements a
reduced

ODM
data model (‘
Model
’) for
mapping to WaterML
elements
.


Model

is also mapped to the
native db of the source by
source
-
specific

DAOs
.

WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
Model

contains classes
common

to both
WaterML

and
ODM
.


DAOs

define object
-
relational mapping (
ORM
)
from native schema to
Model
.


Contains methods for translating user
arguments into queries to native db.


The python module,
SQLalchemy
, supports
ORMs and can accommodate different db
environments (e.g. MSSQL, SQLite, etc)

(WOF module)

Native
metadata
WaterML
Real
-
time
data on
web
Native
database
Source
-
specific
DAOs
(Data Access
Objects)
Real
-
time
data on
web
WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
Native
metadata
WaterML
Real
-
time
data on
web
Native
database
Source
-
specific
DAOs
(Data Access
Objects)
Real
-
time
data on
web
WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
Backend (WOF module)


The python libraries provide capabilities
to serve

REST

and
SOAP
. Both types
access the
same backend.


WOFpy

web services can be
accessed

by
CUAHSI HIS clients

just like any
other
WaterOneflow web service
.

WOFpy_flask

-
Defines the REST
methods for
WaterOneFlow
using
Flask

-
Creates URLs and
views for both
REST and SOAP
web services.

WOFpy_soap

-
Defines the SOAP
methods for
WaterOneFlow
using
SOAPlib

SOAP

web
service

REST

web
service

Runserver_all.py

-
Management script

-
Launches both webservices using
Werkzeug
.DispatcherMiddleware

WOFpy architecture (frontend
-

webservices)

WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
WOFpy demo

Next steps


Release
WOFpy

and other python modules as
open
-
source

software


Developmental version on GITHub (search for
WOFpy
)



Continue to
add datasets

to


Add parameters to
Texas Registry


Conduct training
workshops
.


Develop
how
-
to’s

for putting data into the system
and also retrieving data. WDfT System will be live
by end of summer

WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services
WOF
py
WOFpy

a
py
thon wrapper for
W
ater
O
ne
F
low
Services

A
Texas
-
specific HIS

hosted by
the Texas Water Development
Board


Employs
partnerships

with
Federal and Texas agencies to
share water data


Inherits

National HIS technology


Provides

additional tools and
services

to provide ease of use
and a level of quality control for
partners and clients


(e.g.
WOFpy
)



A fully customizable, open
-
source framework written in
Python
to publish
WaterOneFlow web services


User can maintain their data in
their
original database

format


Mapping to waterML is defined
by user
-
customized
Data
Access Objects

(DAOs)


Tutorials and documentation
coming soon!



Acknowledgments:

James Seppi

WOFpy creator