The NVO DataScope: Internals

whooploafΛογισμικό & κατασκευή λογ/κού

13 Δεκ 2013 (πριν από 3 χρόνια και 7 μήνες)

77 εμφανίσεις

Sept. 16 2004

NVO Summer School

1

The NVO DataScope: Internals

Tom McGlynn

NASA/GSFC

T
HE

US N
ATIONAL

V
IRTUAL
O
BSERVATORY

Sept. 16 2004

NVO Summer School

2

Summary



DataScope as an example VO application


Issues in building DataScope


Sept. 16 2004

NVO Summer School

3

Overall Organization

CacheManager

DataScope

VO Registry

VO Resources

User

Sends request to
DataScope

Is this request
cached?

Yes: Just send
results back to
user

No: find
resources to
query

RequestManager

Queue the requests
and start them off

Start telling
the user about
requests that
have returned

Cache the results
as they come in

Sept. 16 2004

NVO Summer School

4

Perl VO Objects: Services

Service

URLAggregater

Scanner

Cone

SIAP

PositionalService

Tarrer

OASIS

Aladin

HeasarcResolver

Resolver

JHURegistry

Registry

HeasarcScanner

Services

Not easily usable in other
applications


Portable object

Sept. 16 2004

NVO Summer School

5

Perl VO Library: Display

Display

Criterion

State

VOTableRenderer

ServiceFilter

Sept. 16 2004

NVO Summer School

6

Perl VO Library: Control

CacheManager

RequestManager

CacheableRequest

Sept. 16 2004

NVO Summer School

7

Simple Galaxy Checker


Look for all resources with subject of
galaxy (or galaxies) and see if there are
any matches at a given position.


Run service at:
http://skyview.gsfc.nasa.gov/tam/gTest.html


Fix bug in SOAP…

This is where VO library was
placed

Going to use these

Convert user position to standard
decimal J2000 RA/Dec

Errors and defaults

Create objects

Query registry

Loop over matches

Make a cone search
service and indicate
where to search

Display the results. Note how we use eval which allows us to
recover from a parsing error if the VOTable is malformed.

Query the cone search and get
the VOTable

We only look for galaxies at the
beginning of the subject. We
might use “%Galax%” to match
anywhere.

Sept. 16 2004

NVO Summer School

9

How does DataScope use these
objects?


Main DataScope page is basically a
HeasarcScanner which gets the data and a
ServiceFilter which shows the results.


Cone Search and SIAP results are shown using a
VOTableRenderer


A few custom filters which have knowledge not in the
VO registries are used to make links to ADS, MAST and
HEASARC archives


The URLAggregater handles most of the details
of getting information to OASIS and Aladin


Single file services (VOPlot, VOStat) currently
use custom code at CGI script level.


Sept. 16 2004

NVO Summer School

10

DataScope: Registry Issues


Registry is the most rapidly evolving element of the VO


SOAP interface to registry


SOAP::Lite used in Perl


One local fix to library to retrieve multiple string fields properly.


Setup may be non
-
trivial


How do we select resources of interest?


Restrictions on ServiceType


Cone or SIAP


Restrictions on ContentLevel


Should we use positional information about service?


Not currently. The data is too ill
-
formed.


Do we need to cache registry queries?


Takes several seconds, so not needed for DataScope


Organizing resources


Use ServiceType, Subject and Content fields


DataScope organization is just one of many possible ideas


Redundant and superseded resources


Currently minimal information on these


Priority of resources


Currently no information


More work needed if we are going to handle 10
3

to 10
4

resources


No information on analysis services in registry


Hardwired into top level DataScope routines

Sept. 16 2004

NVO Summer School

11

DataScope: Cone and SIAP issues


Search radius:


SIAP implicitly handles overlapping regions but Cone search does not. May need to increase
size of region by ServiceSize to get all relevant rows from cone search services.


Failed queries:


Should we retry?


Overloading provider sites


Don’t want to spit hundreds of queries to the same site simultaneously


DataScope sorts resources by destination URL and only sends one query to a given host at a time.


Non
-
compliant services


Cone searches returning multiple tables


Cone searches without position columns


SIAP services with improper UCD columns


But these are useful sites! (ADS, VizieR, XMM
-
Newton)


Need to cache SIAP results before they expire


How do we handle deferred SIAP results? Don’t have any now.


SIAP can have both quicklook and FITS files but the relationship between them is
not defined.


A registry entry may describe only some of the results returned by a SIAP request
(using ID_MAIN column == ServiceName convention)


Look for multiple registry entries with identical ServiceURL’s


When will these services be available as Web services?

Sept. 16 2004

NVO Summer School

12

DataScope: Analysis software
issues


Currently two types of `analysis’ tools, some
handle only one file at a time (VOPlot, TopCat,
DS9,…) others take a set of URLs (Aladin,
OASIS).


URL Aggregating services may run over GET
limits or limits on number of URLs that can be
used.


Single file tools may have more capabilities in
non
-
applet modes


Tools are not described in registry so knowledge
of them is built into DataScope itself.

Sept. 16 2004

NVO Summer School

13

DataScope caching issues


Not necessarily a VO issue


Limits loads on data providers


Enhances response time


Data may go out of date.


Failed queries may now be available


SIAP results may expire if left on provider


Some services (e.g., Aladin) cannot currently
handle FTP URLs (though this should change
with latest versions of Java). SIAP (and SSAP)
URLs may need to be cached to be available to
such services.

Sept. 16 2004

NVO Summer School

14

Miscellaneous


SkyNode’s not accessible through
DataScope.


Cone search probably provides appropriate
functionality for DataScope.


Are all positional SkyNode tables available
through cone search?


Sept. 16 2004

NVO Summer School

15

HEASARC Perl library distribution


Very preliminary distribution with limited
documentation and testing for the non
-
VOTable modules.


Available at:


http://skyview.gsfc.nasa.gov/vo/volib.0.1.tar or volib.0.1.zip


Installation documentation at
http://skyview.gsfc.nasa.gov/vo/volibinstall.doc

Simple command line script
to invoke an SIAP service and
download all of the
referenced files.