SDMX Open Source Software

martencrushInternet και Εφαρμογές Web

8 Δεκ 2013 (πριν από 3 χρόνια και 9 μήνες)

98 εμφανίσεις

SDMX Open Source Software

Erik van Ingen

FAO/D4Science

23 April 2011

Eurostat Luxembourg

www.d4science.eu

D4Science
-
II project
|


www.d4science.eu

2

SDMX Open Source Software

23 April 2011

Content


OpenSDMX is used in


FAO/TechCDR


FAO/Fisheries (
www.fao.org/figis/sdmx
)


EU/D4Science
(
http://opensdmxdevelopers.wikispaces.com/PisaCNR2011January
)


Looking at ECB & FED


Why OpenSDMX?


OpenSDMX pillars


OpenSDMX Development Environment & Technical Architecture


OpenSDMX Communication channels


Synergy analyses MT, Eurostat & OS


JAXB and XMLBeans

23 March 2011, Luxembourg

SDMX open source software

www.d4science.eu

3

TechCDR :Corporate Data Repository

TechCDR is a platform of services

Provide external access to FAO digital assets

Search and access statistics, maps, feature layers, documents,
imagery, etc.

External APIs

Provide internal services to Analytical Field Systems

Workspace management, ETL uploads, security, publishing,
notification and update alerts

23 March 2011, Luxembourg

SDMX open source software

www.d4science.eu

4

Staging Area

TechCDR Platform

Structured

Data Repository

Multidimensional

Data Model

(DWH)

Content

Repository

Statistical

Data

Services

Content

Management

Services

(Alfresco)

GIS Services

(Geoservers)

JBoss ESB

ETL Jobs

(Talend OS)

Data Analysis

Services

(
Pentaho)

Working System

Services

Working

System

Metadata

Manager

Portlets, Javascript, Flex components

data layer

service layer

front
-
end layer

Infrastructural

Services

(Security, log,

cache, etc)

Portlet container

Metadata

Services

Working

System

RDBMS

Metadata

Repository

Statistical

Data

Repository

Business

Services

Integration

Services

Custom

Applications

SDMX

Services

Ontologies

Manager

23 March 2011, Luxembourg

SDMX open source software

www.d4science.eu

5

TechCDR use of OpenSDMX

TechCDR publishes multidimensional data and artifacts in
SDMX

OpenSDMX features:

OpenSDMX repository services obtain datasets from OLAP
database

OpenSDMX services dynamically map data in OLAP cubes
to SDMX datasets

Codelists, Codes, DSDs are published

23 March 2011, Luxembourg

SDMX open source software

www.d4science.eu

6

TechCDR SDMX Services

OpenSDMX services exposed via

Restful API following SDMX 2.1 guidelines

TechCDR Custom

Interfaces to OpenSDMX services

23 March 2011, Luxembourg

SDMX open source software

www.d4science.eu

7

TechCDR SDMX Services

Relational

Data

Services

XMLA

data layer

SDMX

Services

Olap4j

REST

Controller

service layer

Multidimensional

Data Model

(DWH)

Metadata

Repository

Statistical

Data

Repository

Data Analysis

Services

(
Pentaho)

front
-
end layer

23 March 2011, Luxembourg

SDMX open source software

www.d4science.eu

8

OpenSDMX and D4Science

SDMX open source software

23 March 2011, Luxembourg

Codelist

uploader

www.d4science.eu

9

Looking at the


the ECB SDMX visualisation framework:

Federal Reserve Bank of New York and European Central Bank



SDMX open source software

23 March 2011, Luxembourg

Why Open Source:


Rapid, robust development


剥摵捥c 楮獴楴畴楯湡氠捯獴c


䵯牥⁴桯牯r杨 捯c攠牥癩vw


䉥瑴敲⁣e摥⁳異灯牴


Logistics and License


ECB, NY FED, BOC


䍯摥 桯獴敤 慮搠d畢汩捬c 慶a楬慢汥 潮
䝯G杬攠䍯摥


却物捴⁣S浭楴⁰潬楣i 慮搠捨c湧攠
浡m慧敭e湴n灲p捥c畲us


BSD license


Roles


䱥慤敲獨楰 牯r敳e⡐牯(散琠䵡e慧敲Ⱐ
偲潪散琠䅤癩獯爠慮搠偲d橥捴c䅲捨A瑥捴t


䍯浭楴瑥牳㨠瑨牥攠楮摩癩v畡汳


䍯湴物n畴u牳㨠捯c瑲楢t瑩潮猠慲a⁷敬捯浥
晲潭⁡汬


啳U牳㨠瑨t⁣潤攠楳⁡癡楬慢汥 瑯⁡tl


www.d4science.eu

10

SDMX Open Source Software

23 April 2011

Why OpenSDMX?


A platform was needed in order to serve three different
projects


The platform needed to be open source in a full fledged
manner.


FAO and D4Science needed to have a good governance
position in the open source project.



23 March 2011, Luxembourg

SDMX open source software

www.d4science.eu

11

SDMX Open Source Software

23 April 2011

OpenSDMX pillars


Implementation of version 2.1 REST webservice
specification


Supporting (only) version 2.1 (still migrating yet from
version 2.0)


Focus on core SDMX Artefacts (Codelist, DSD, Generic and
Compact Dataset)


Integratable in every context through adapters


Not necessarily having its own persistence layer, maybe
added in the future




23 March 2011, Luxembourg

SDMX open source software

www.d4science.eu

12

OpenSDMX Development Environment


Building with Maven


Continuous Integration with Continuum


Code Metrics with Sonar & FindBugs


Artifact server with Artifactory


SCM with Subversion on SourceForge


Documentation on Wikispaces
(
http://opensdmxdevelopers.wikispaces.com/)


Eclipse IDE mostly used amongst developers


Automatic deployment on Tomcat


Integration testing


Unit Testing



SDMX open source software

23 March 2011, Luxembourg

www.d4science.eu

13

OpenSDMX Technical Architecture


Spring for dependency injection, REST and caching


JAXB as the Java binding for XML


EHCache for caching


Java 6




SDMX open source software

23 March 2011, Luxembourg

www.d4science.eu

14

OpenSDMX Communication channels









https://sourceforge.net/p/opensdmx/home/

http://opensdmx.wikispaces.com/

http://opensdmxdevelopers.wikispaces.com/



SDMX open source software

23 March 2011, Luxembourg

www.d4science.eu

15

Synergy analyses MT, Eurostat & OS

SDMX open source software

23 March 2011, Luxembourg

Synergy Level

MT
-
OS

MT
-
ES

ES
-
OS

Mean in %

Java 6

Development Environment

*Maven

*Continuous Integration

*Unit Testing

*Integration Testing

*Code Metrics

Technical Architecture

*Spring

*Caching

*XML Binding

*Persistence

*Layering and interfaces

Functional Architecture

Open Source Organised

License

www.d4science.eu

16

JAXB and XMLBeans I

Metadata Technology uses XMLBeans, OpenSDMX uses
JAXB. Eurostat uses a proprietary XML binding.


XMLBeans


Email traffic from 18/02/2011
-
18/03/2012 is 15 (10
developermail, 10 usermail)


Last release 2.5.0 was on 14
-
12
-
2009


JAXB


Email traffic from 18/02/2011
-
18/03/2012 is 70 usermail


Last release 2.2.3.1 was on 16/01/2011



SDMX open source software

23 March 2011, Luxembourg

www.d4science.eu

17

JAXB and XMLBeans II

Observations:


Spring supports Castor, Jaxb, Jibx and XmlBeans


XmlBeans is less active than JAXB


JAXB is natively part of Java


SDMX open source software

23 March 2011, Luxembourg

www.d4science.eu

18

SDMX Open Source Software


23 April 2011

END

Erik van Ingen

FAO/D4Science

End of Presentation