HDMC_VSO - Virtual Solar Observatory - NASA

handslustyInternet and Web Development

Dec 14, 2013 (3 years and 9 months ago)

97 views

VSO Status Update


HDMC Meeting

9 June 2008


Vision


To allow solar physicists to identify and
search for data even if they don't know it
exists.


Make it easier for them to get to the data
whether they knew where it was before
or not.

Why VSO?


Why not CDAWeb?


Will they take > 2TB/day of data?


Why not FTP and VSPO:


Do you want to scan > 70k files per day to
determine which ones meet your needs?


Every time you perform a search?

(these numbers are just for SDO)

How to accomplish:


Standardized API for querying &
requesting data using a consistent
model at each provider




System to allow UIs to query a single
API and have the queries distributed &
responses aggregated.

Current Status


Data Providers : 14


Observatories : 26






Instruments : 55


Data Sets : 180?

... dating back to 1915

Current Status (SPASE terms)


Repositories : 19


Observatories : 26


Observatory Groups : 3


Instruments : 66+


Instrument Groups : ~10


Numerical Data : ~180 products


Display Data : (not tracked directly)


Granules : ? millions

Data Sets Planned:


In progress:


Hinode/SOT






Huairou


NRL EIT & LASCO


TIMED/SEE


Planned:


Hinode/EIS


SMART


SDO

... and catalogs

SPASE use:


um ... no?

Current Interfaces / APIs


VSO SOAP API (finding & ordering):


Web
-
based interface


3 custom VSO
-
derived interfaces (NSO, SAO, SSC)


IDL
-
based interface


Dominic Zarro's IDL commands in SSW


Perl command line interface


AstroGrid


Registry API


Catalog 'Display' API for HelioViewer

Interfaces / APIs



In progress:


IDL client for Registry API


More complex Catalogs API(s)


Planned:


Cart API


and UI in IDL


Catalog APIs for HELM


access via IDL


SPASE compliant registry API


or at least SPASE descriptions of holdings

Data Coverage:


Incomplete.


Measurement method:


People are still asking us to add data sets.

Goals:


Prepare for the flood of SDO data


(and other recent missions: STEREO,
Hinode )

Plans for the Future:


Catalogs


Summary Rows


Looking to the work of others

Empowering new searches!

We’re doing some work under the hood!

Moving beyond time


An event or feature driven approach


Science catalogs


LASCO CME, GOES X
-
ray Flare, NOAA Active Region Number, Coronal
Holes, Bright points etc.


Science queries

One of the drivers for this


SDO


So much data need to move to feature / event driven approach,


But useful now …

Give me all of these!

Actually I really want all

XRT jets observed on the

limb with coincident SOT

and EIS data.


Would have made things

much easier if I could have

performed this query!

Science Queries

How do we know what queries to support?


We get asked!


“It would be nice if … / I’d really like to be able to …”

Coincident observations


Observations of same features in multiple instruments within
specified time window.

Cadence


Find an hours worth of observations where there are images every
minute. (Helioseismology)

DEM Datasets


Find XRT datasets suitable for DEM analysis. (Closely spaced
observations, number of different filters/wavelengths). EIS data
would be a bonus!
?

Science Queries

Investigating EIT dimming regions as proxies for CMEs



Combine GOES Flare List + LASCO CME list to generate list of on disk flares
without associated CMEs. Compare list to catalog of EIT dimming regions. Do
spatial comparison. Generate list of EIT dimming regions whose properties are
observed to be associated with CME list.


Currently waiting on EIT dimming region catalog! Can do the first part to at least
generate list of EIT images that should be analyzed.


Support driven by science questions!

Conclusions


We are moving from just data discovery in a certain
time range, to finding datasets of interest that fit a
number of scientific criteria.


Searches for features and events and in the process
blurring the distinction between data and catalogs.


SDO will require these new approaches.


But it’s going to return a LOT of data.

Plans : Summary Rows


VSO data providers can return a record
summarizing multiple data granules


Data Providers can define their own groupings, UIs can
decide to expand them.


Summary products still work like regular products


Data providers serve a tarball or data cube in response
to a data transfer request


UIs that don’t understand summary products can still
retrieve the collection


Changing over to a new JavaScript toolkit to
provide AJAX summary row expansion

Plans : Look to Others


Catalogs


Bobby Candey's HELM project (VxO funded)


Peter Fox's SESDI project (NASA funded)


Related IVOA & astronomy standards


Presentation / UI


Use existing JavaScript toolkits


ExtJS and GWT

... still requires customization to serve our needs, but gives
head start

Plans : Look to Others (2)


Massive amounts of data:


How libraries handle interlibrary loan


How libraries handle 'de
-
duplicating' of
records


Functional Requirements of Bibliographic
Records

… is going to require significant changes at the data
providers.

The Future of VSO


We are moving from just data discovery in a
certain time range, to finding datasets of
interest that fit a number of scientific criteria.


Searches for features and events and in the
process blurring the distinction between data
and catalogs.


SDO will require these new approaches.

http://virtualsolar.org/