CARBONES portal description Simplified version V0

hotdogrelishnoseΛογισμικό & κατασκευή λογ/κού

4 Νοε 2013 (πριν από 3 χρόνια και 9 μήνες)

349 εμφανίσεις




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

1
/
36













CARBONES portal description

Simplified version V0


Change History


Date

Author

Comment

11
th

Mar 2010

Mark Jackson

First Draft

14
th

Mar 2010

Mark Jackson

Review comments
(Chris Johnson, Christine McHugh)

30
th

Mar 2010

Mark Jackson

Comments by Noveltis, LSCE

30
th

Mar 2010

Mark Jackson

Reinstate analysis of engineering solutions





Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

2
/
36



Contents

1

Summary

................................
................................
................................
............................

3

2

Introduction

................................
................................
................................
.........................

5

3

Description of V0 portal website

................................
................................
.........................

7

3.1

Content

................................
................................
................................
.....................

7

3.2

Web mapping

................................
................................
................................
...........

7

4

Analysis of engineering solutions

................................
................................
.....................

10

4.1

Purpose of analysis of engineering solutions

................................
.........................

10

4.2

Summary of conclusions

................................
................................
........................

10

4.3

Technologies discussed

................................
................................
.........................

11

4.4

Case study

................................
................................
................................
.............

12

4.5

Market pen
etration

................................
................................
................................
.

12

4.6

INSPIRE compliance

................................
................................
..............................

13

4.7

Cost

................................
................................
................................
........................

13

4.8

Development support

................................
................................
.............................

14

4.9

Product longevity

................................
................................
................................
....

14

4.10

Technical implications of requirements

................................
................................
..

15

4.10.1

Operating system

................................
................................
................................
............

15

4.10.2

Programming language

................................
................................
................................
...

15

4.10.3

Graphs and charts

................................
................................
................................
...........

15

4.10.4

Metadata

................................
................................
................................
.........................

16

4.10.5

Download service

................................
................................
................................
............

16

4.10.6

Discovery service

................................
................................
................................
............

17

4.10
.7

Data format

................................
................................
................................
.....................

17

4.10.8

Interactive maps

................................
................................
................................
..............

18

4.10.9

Content management system

................................
................................
.........................

19

4.10.10

Post
-
processing

................................
................................
................................
..........

20

4.10
.11

Database

................................
................................
................................
....................

20

5

Portal architecture

................................
................................
................................
............

21

5.1

Post
-
processing
................................
................................
................................
......

21

5.1.1

Inputs

................................
................................
................................
..............................

21

5.1.2

Post
-
proce
ssing specification

................................
................................
..........................

23

5.1.3

Outputs

................................
................................
................................
...........................

27

5.2

Portal server and portal web application

................................
................................

28

5.2.1

Inputs

................................
................................
................................
..............................

28

5.2.2

Outputs

................................
................................
................................
...........................

28

6

Further development after V0

................................
................................
..........................

30

6.1

Development for the first annual meeting

................................
..............................

30

6.2

Development after the first annual meeting

................................
...........................

30

7

Test plan for the V0 portal

................................
................................
................................

33

7.1

Post
-
processing
................................
................................
................................
......

33

7.2

Test plan for web portal

................................
................................
..........................

35

8

Test report for the V0 portal

................................
................................
.............................

36

8.1

Post
-
processing
................................
................................
................................
......

36

8.2

Test

report for web portal

................................
................................
.......................

36





Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

3
/
36







1

Summary


This report describes deliverable D520.1 in the CARBONES project: the V0 version of the
CARBONES
web portal.


The
deliverable D520.1 comprises:



The source code for V0 of the portal, including the post
-
processing and the web
portal.



A live portal.

o

The portal web pages are at
www.carbones.eu

o

The portal web
-
mapping is at
http://carbones.eu:8080/Carbones
, to which a
link exists on the CARBONES home page. Figure 1 shows an image of the
portal web mapping. The web
-
mapping includes highly interactive maps with
zoom and pan functionality si
milar to that on Google Maps. It presents the
“raw” data set without post
-
processing as well as the results of post
-
processing, including temporal filtering.



Detailed description of the portal (in this document), including:

o

summary of architecture

o

plans fo
r further development

o

test plan and test report for the V0 portal



Figure
1
. The portal web mapping





Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

4
/
36



The V0 portal uses 2 years of sample CCDAS output data provided by LSCE, under action
PM2
-
02

1
. These data were provided so

that data were available in time for the V0 post
-
processing and portal. These preliminary data were provided so that data were available in
time for the V0 post
-
processing of the portal. These sample data are from 1 year of:



simulations done with the glob
al version of the ORCHIDEE model (without
assimilation) for the biogenic fluxes and stocks of C



Takahashi climatology for the net C ocean flux



GFED (Global Fire Emissions Database version 2) for biomass burning



from EDGAR database for anthropogenic fossil
fuel emissions


The data for the second year are the same as for the first year. The complete 20
-
year data
set, D510.1, is scheduled to be delivered at the same time as the V0 portal and, therefore,
could not be used in the V0 portal.


The objectives for V
0 portal have been met and the portal is available for use displaying maps
of the preliminary data.





1

Action 2 Progress meeting 2, 27
th

Oct 2010




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

5
/
36







2

Introduction

The task WP520 covers the development of a tailored, user
-
friendly interface, allowing
climate modellers, other science communities, local o
r regional communities and the general
public to understand and easily visualize the living carbon cycle over the past 20 years. The
users of the CARBONES system will use the system via a web site in which spatial data will
be displayed on a dynamic map, s
imilar to Google Maps. It has been developed so that users
can select the output according to area, time period and output variable.

Web map elements
are implemented as a thin client compatible with all major browsers without requiring any
plug
-
in or speci
fic setting (although JavaScript is required to view the maps).


This document describes the first version of the web portal (V0), deliverable D520.1 from
WP520, due at month 12. The Description of Work (DoW) mandates a second version of the
web portal (V1
), deliverable D520.2, due at month 30.


The definition of the web portal has been developed through gathering and analysing user
requirements. Various partners and others have been involved (NOV, LSCE, TASF, METO,
CUAG, AEAT). The conclusions have been ca
ptured in deliverables (D120.1, D120.2, D210.1,
D220.1, and D720.1).


Figure
2

shows an overview of the entire CARBONES system, showing how the portal fits
into the system.


The post
-
processing step is required to transform the output of the CCDAS and other sources
of data (validation data) into C
-
cycle relevant diagnostics for clim
ate users and policy makers.
The post
-
processing is performed off
-
line at CERC. The results of the post
-
processing are
presented through the web portal. Website users will be able to view the results of different
post
-
processing options through the web int
erface, but they will not be able to repeat the
actual post
-
processing calculations through the web interface.




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

6
/
36



Figure
2
. CARBONES system overview (data flow)




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

7
/
36







3

Description of V0 portal website

3.1

Content

The content of the portal at
www.carbones.eu

explains the project objectives and lists the
project partners. It gives an overview of the project, explaining the motivation, objectives,
product characteristics and plan
ned dissemination of the products. There is a news section
which is updated to give details of project progress. The website provides an email address
so that the public or other interested parties can contact the project team with any queries
they may hav
e.

3.2

Web mapping

The portal web
-
mapping is demonstrated at
http://carbones.eu:8080/Carbones

to which a link
exists on the CARBONES home page. The web
-
mapping includes highly interactive maps
with zoom and pan
functionality similar to that available in Google Maps. It presents the “raw”
data set without post
-
processing as well as the results of post
-
processing, including temporal
filtering. Flux and stock maps are presented.


The controls in the web application
are shown in
Figure
3
. To use the web application:



the user selects the variable to display from the tree control at the left. Only one
variable can be

displayed at a time, because the data cover the whole globe.



the user selects the temporal averaging period for the data from the drop
-
down on
the top left.
The options include “3
-
hourly”, “Day”, “Month”, “Year” and “Period”.
Different options are
available depending on the variable selected by the user in the
tree control.



the user selects the temporal filtering from the drop
-
down at the centre top. The
options are “Mean”, “Smooth trend”, “Smooth curve” and “Recentred parameter”.
Different options
are available depending on the variable selected by the user in the
tree control.



the user selects a date and time for which the data will be displayed using the
controls at the top right. The user can either select a date
-
time directly using the
calendar
control and the drop
-
down for the times, or, the user can step forwards and
backwards through the dates and times in order using the (arrows) buttons. The
interval between the dates and times depends on the temporal averaging period that
has been chosen, a
s shown in
Table
1
.



the user can select a region from the “Regions” tree control. This zooms the map to
the selected region and also hides areas outsi
de the selected region.




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

8
/
36





Temporal
averaging
period

Interval between
time steps

Example time steps

3
-
hourly

3 hours

1 Jan 1990 (00:00, 3:00, 6:00, 9:00, 12:00, 15:00, 18:00, 21:00),

2 Jan 1990 (00:00, 3:00, 6:00, 9:00, 12:00, 15:00, 18:00, 21:00),


=
䑡y
=
ㄠNay
=
1 Jan 1990, 2 Jan 1990, 3 Jan 1990, …
=
䵯湴h
=
ㄠN慬e湤慲潮瑨
=
Jan 1990, Feb 1990, Mar 1990, …
=
v敡r
=
ㄠye慲
=
ㄹ㤰ⰠI9㤱
=
m敲e潤
=
=
l湬y= 潮e= s整e 潦= 摡t愬a av敲慧e搠 ov敲e 瑨攠 wh潬攠 p敲楯搮= 啳敲e
睯畬搠d潴ob攠ebl攠瑯tch潯s攠eiff敲敮琠eim攠e瑥ts
=
Table
1
. Temporal averaging periods and corresponding time steps


The web mapping application displays the variable values as a colour
-
shaded map. Continent
and country outlines are overlaid on the map to assist the user in relating the
colour
-
shaded
map to particular regions of the globe. The colour legend and units used in the map displayed
are given at the bottom left of the web display.


The map controls allow the user to pan and zoom the map freely. Clicking the globe resets
the view

to show the entire world. Note that the user can also pan the map by clicking and
dragging with the
mouse, and can zoom the map with the mouse wheel (if their computer has
a mouse wheel). The user can also view particular regions of interest by selecting
them from
the pre
-
defined list in the regions controls.


The web mapping application will be compatible with the following browsers:



Internet Explorer 7



Internet Explorer 8



Firefox 3



Safari 4



Chrome 10.


Internet Explorer 6 is not supported. There are well
-
known difficulties in supporting this
browser for
any website. The development effort required would not be justified by Internet
Explorer 6’s usage share, which is small and rapidly decreasing (under 4% in Europe
2
).
Microsoft has recently begun a public campaign
3

to persuade all users to move away from
I
nternet Explorer 6
.




2

Data across all operating systems, dated 28 February 2011, cited on
ie6countdown.com


3

The campaign home is the Microsoft website
ie6countdown.com









Figure
3
. The portal web mapping


user controls

Tree control allows
the user to select a
variable

to display in
the map.

Legend

gives the
colour shading
and units used in
the map to display
the variable.

Time period drop
-
down allows
the user to select the
time
period for averaging
. The
options are “3
-
hourly”, “Day”,
“Month”, “Year” and “Period”.

Filtering drop
-
down allows the
user to select the
temporal
filtering
. The options are “Mean”,
“Smooth trend”, “Smooth curve”
and “Recentred parameter”.

Calendar control

and hour drop
-
down allow the user to choose
the date and time to display. The
buttons allow the user to
step
forwards and backwards

through the time steps.

Map co
ntrols

allow the user to
pan and zoom the
map.

Regions

tree
control allows the
user to view
specific regions of
the world.

Position indicator

shows the latitude
and longitude of the
current mouse
position






Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

10
/
36



4

Analysis of engineering solutions

4.1

Purpose of analysis of engineering solutions


4.1.1

Post
-
processing of CCDAS output

The post
-
processing calculations presented some special considerations. The development
of the scientific specification of the post
-
processing calculation required
iteration between
LSCE, Noveltis and CERC that is ongoing. This presented a risk to the project timelines. To
mitigate this risk, the solution for the V0 post
-
processing is based on a commercial package
(IDL) and proprietary code (CCGVU). This mitigates th
e timeline risk for several reasons:
these packages have been proven in practice in similar projects; and LSCE, Noveltis and
CERC had staff members available with experience with IDL. We may study the use of
alternative tools for the V1 portal.


4.1.2

Web

portal

This remainder of section 4 explains the analysis and selection of engineering solutions that
have been used to create an INSPIRE
-
compliant portal for the CARBONES project.


The analysis concentrates on two approaches: a solution based on commercia
l packages and
an open source solution.



The commercial vendor solution uses the ESRI ArcGIS Server product to illustrate
this option.



The open source solution is based on MapServer, a platform developed in the mid
-
1990s at the University of Minnesota with
support from NASA. Use of MapServer
requires, in addition, the use of a number of other free, open source packages.


The following criteria have been used in the analysis:



Baseline functionality: INSPIRE
-
compliant discovery services, view services and
dow
nload services; web portal features such as interactive maps and interactive
graphs.



Ease of implementation: including the capability to implement new services by
developing an API/SDK.



Licence cost and support costs.



Market penetration. In general, the la
rger the user base, the less the risks associated
with adopting the technology and of the product being retired or not supported in the
near future. A large user base generally means better support from the supplier as
well as the wider (user) community.


4.2

Summary of conclusions

From our research which is reported in sections 4.3
-
4.10, we are confident that both the
commercial and open source solutions can provide the functionality required for the project.





Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

11
/
36






With little separating the two options in terms o
f functionality, there is little justification for the
extra cost of the ESRI solution (a minimum of £14,275 over the 3 years of the CARBONES
project, with an ongoing annual support cost of £1,600 after the end of the project).
Therefore, an open source s
olution was selected to create the portal. The open
-
source
technologies listed in the table have been identified as suitable for use in the portal and have
been used in the V0 portal. The specific choice of individual technologies remains open and
may chan
ge for the V1 portal.


One of the strengths of the project architecture is that the components communicate using
standardised communication protocols, making it possible to change the implementation of
individual components to different technologies with
minimum effect on the system as a whole.
In particular, the use of OGC standards such as CS
-
W, WCS and WMS would make it
possible to change to a different GIS server package such as ArcGIS Server either during the
project, or after the project is completed
. The technology analysis indicates that this will not
be necessary, but the system architecture minimises risk by leaving the possibility open.


Functionality

Technology Recommendation

Discovery services and metadata (CS
-
W)

GeoNetwork

Download service
(WCS)

MapServer

View service (WMS)

MapServer

Server operating system

Linux

Web server

Apache / Apache Tomcat

Programming language

Java

Content Management System

Alfresco

Interactive maps in the portal

OpenLayers JavaScript API

Interactive graphs and

charts in the portal

JFreeChart

Post
-
processing

IDL

Table
2
. Selected technologies recommendations for V0 of the web portal


GENESIS (GENeric European Sustainable Information Space for Environment), an EU 7
th

Framework project, will provide INSPIRE
-
compliant Web services, portal components and
toolkits. CARBONES will use GENESIS products, where appropriate and according to
availability and ease of implementation, to provide CARBONES functionality. The GENESIS

products will be assessed for suitability in CARBONES after the V0 portal.


4.3

Technologies discussed

This analysis concentrates on two approaches to achieve an INSPIRE
-
compliant web
mapping solution: a commercial vendor solution and an open source solution.




The commercial solution used in this document to illustrate this option is ESRI’s
ArcGIS Server product. Much of the information used in this document is based on



Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

12
/
36



version 9.3.1, but we also include changes addressed in the recently released
version 10.
The ArcGIS Server package is capable of fulfilling many of the
functionality requirements (WMS, WCS, CS
-
W, and JavaScript API).




The open source solution considered is based on MapServer, although other options
4

were researched. MapServer is a GIS server
platform developed in the 1990s at the
University of Minnesota. The portal would also require a number of other free open
source applications, e.g. GeoNetwork and OpenLayers.


One of the advantages of ESRI is that it potentially supplies a single product t
o fulfil all of our
obligations, whereas the open source solution would depend on a several different products.
However, the required open source products are commonly used together, so integration
problems should not present a significant risk.


4.4

Case stu
dy

This analysis will make use of the experiences of the British Geological Society (BGS), who
are active developers of INSPIRE
-
compliant web mapping solutions. BGS have used both
ESRI and MapServer in their work, including work on their own Geology of Br
itain viewer
5

(using ESRI’s ArcGIS Server) and an international project, OneGeology
6
,
7

(using
MapServer
8
). BGS have also taken a lead in bringing INSPIRE and MapServer together,
with their work on Open Geoscience and in producing their “cookbook”
9
. The co
okbook is a
set of instructions on implementing INSPIRE with MapServer that was produced in
conjunction with the Scottish government.


4.5

Market penetration

In general, the risk associated with adopting a technology is lower if there are a large number
of use
rs. This reduces the risk of the product being retired, and it generally means better
support, from the supplier as well as the wider community.


ESRI have the largest GIS user base in the world
10

(over 100,000 organisations in 2000).
There are many prove
n implementations of ArcGIS Server available, including INSPIRE@EC,
Croatia’s Multipurpose Spatial Information System Geoportal Interface, etc.


There are many proven MapServer implementations, such as the OneGeology site mentioned
above. It is difficult
to precisely quantify the size of MapServer’s user base. There are no
central records of its usage because it is open source software. However, there are hundreds
of messages each month on the mailing lists, and each year at the popular FOSS4G “Free
and Op
en Source Software for Geospatial” international conference, MapServer is the subject
of many of the presentations and workshops
11
.





4

Geoserver was seen as the main competitor to MapServer. ncWMS was also considered.

5

http://maps.bgs.ac.uk/geologyviewer_google/googleviewer.html

6

http://portal.onegeology.org/

7

Cookbook on setting up a WMS server using MapServer and ESRI:
http://www.o
negeology.org/wmsCookbook/1_1.html

8

Example implementation and templates:
ftp:
//ftp.bgs.ac.uk/pubload/OneGeology/OneGeology
-
WMS
-
exemplar
-
data.zip

9

http://www.scotland.gov.uk/Publications/2010/05/06161701/22

10

http://www.esriuk.com/services/survey210509/

11

http://foss4g.org





Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

13
/
36






4.6

INSPIRE compliance

ESRI participated in the prototype edition of the INSPIRE geo
-
Portal
12
, were part of the
consortium awarde
d the INSPIRE@EC work
13

and are committed to providing INSPIRE
-
compliant software. ESRI
-
Europe are an SDIC (Spatial Data Interest Community) involved
in INSPIRE. BGS raised concerns of ArcGIS Server 9.3 failing to deliver an INSPIRE
-
compliant solution
14
, bu
t in a private communication to CERC BGS said “The latest version
(v10) of ArcGIS Server really moved ESRI’s support of OGC standards forward and enabled
us to provide [WMS] 1.3.0 services”.


OneGeology


Europe is an example of an INSPIRE
-
compliant portal

developed with
MapServer and GeoNetwork. OneGeology
-

Europe are an SDIC involved in INSPIRE, and
the Project Coordinator for OneGeology are BGS. BGS themselves are an LMO (Legally
Mandated Organisation) involved in INSPIRE. While BGS (and others) conti
nue to develop
INSPIRE
-
compliant solutions using MapServer and GeoNetwork, we can be confident that
these packages will adhere to INSPIRE directives.


Both ArcGIS Server and MapServer/GeoNetwork are capable of providing an INSPIRE
-
compliant solution for th
e CARBONES portal.

4.7

Cost

There will be no software costs for the end user in using the CARBONES portal, although
there is the possibility of charges to cover data download costs. The following figures are
the costs to the CARBONES project of the portal sof
tware.


CERC have identified free open source options for all required elements in this development.


ESRI is a commercial vendor and adopting their solutions incurs costs for development
licenses, server licenses and maintenance. The pricing structure is

shown in Table 3.


Item

Details

Annual Cost

Purchase Cost

Development Tools

1 EDN (ESRI Developer Network)
license, including ArcGIS Server

£995

£0

ArcGIS Server
License and Annual
Maintenance

Workgroup

Edition

from £1,600

from £6,490

Enterprise
Edition

from £6,490

from £25,960

Enterprise ASP Edition

about £13,000

about £50,000

ArcGIS GeoPortal
Extension


Awaiting details

Awaiting details

Total

(Workgroup Edition)

At least £2,595

At least £6,490

Table
3
. Cost
implications of ESRI ArcGIS Server





12

http://www.inspire.esri.com/gos

13

http://news.gislounge.com/2009/02/european
-
commission
-
awards
-
inspireec
-
contract
-
to
-
esri
-
technology
-
based
-
consortium/

14

http://location.defra.gov.uk/wp
-
content/uploads/2009/12/BGS
-
Data
-
Providers
-
Seminar
-
20100323
-
v1
-
0.pdf




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

14
/
36



The ArcGIS Server Workgroup license should be sufficient for the CARBONES portal. The
Workgroup license comes with the limitation of using a file
-
based database (SQL Server
Express) and using a maximum of four processor
cores. The Workgroup license can only be
used if the portal website does not charge the end user and does not show paid
advertisements, otherwise we would need to upgrade to the Enterprise license.


It is not clear whether an ArcGIS Server license could b
e transferred from CERC to another
organisation at the end of the project. CERC pursued this with ESRI but were unable to
obtain a definite reply.


The total cost of the ESRI solution over the three years of the CARBONES project would be
at least £6,490 fo
r the server license and £6,185 for three years of annual support costs
15
.
After the project there would be ongoing maintenance costs of at least £1,600 per year. There
may be additional costs for the GeoPortal extension in order to provide INSPIRE
-
complian
t
services. CERC are awaiting a definite statement from ESRI on whether this extension is
necessary to achieve INSPIRE compliance in the CARBONES portal. These costs are based
on a portal website which does not charge end
-
users or show paid advertisements.


The cost of the ESRI solution means we have to find definite advantages over the open
source solution to justify this expenditure.


4.8

Development support

ArcGIS Server’s user forums
16

are widely used, with new posts and published code
examples
17

every day. ESRI also provide support, although there are cost implications.


MapServer comes with good documentation, although following their installation guide
requires a certain level of web server familiarity. BGS and the Scottish Government have
cr
eated a “cookbook”, a set of instructions on creating INSPIRE
-
compliant solutions using
MapServer.


Both options have good documentation and active “community” forums.


4.9

Product longevity

From CERC’s experience of ESRI’s desktop development environment, Arc
View, there is a
history of ESRI replacing software or development languages, without providing an easy
upgrade path. In the last 10 years ESRI have retired at least three programming languages
(Avenue, VBA and VB6) and one product line (the GIS server pr
oduct ArcInfo).


The open source alternatives, MapServer, GeoNetwork and OpenLayers, would be more
likely to evolve rather than being replaced. The potential concern is that the packages
themselves could be overtaken by new open source solutions, reducing

the level of support
for the older products.


However, all the technologies have good customer bases and there is no immediate threat to
any of them. Although there are no guarantees in this area of rapidly
-
evolving technologies,



15

Three years of EDN license and 2 y
ears of maintenance (the first 12 months maintenance is covered by the initial
purchase)

16

http://forums.arcgis.com/

17

http://resources.esri.com/a
rcgisserver/apis/javascript/arcgis/index.cfm?fa=codegallery&page=2




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

15
/
36






we can be relatively con
fident that whichever path we choose there will be a supported
product for the lifetime of the project and beyond. Furthermore, one of the strengths of the
INSPIRE architecture is that the use of open standards makes it relatively straightforward to
replac
e individual components if the need arises.


4.10

Technical implications of requirements

4.10.1

Operating system

The two most popular and best supported operating systems for servers are Microsoft
Windows and Linux. Whilst the majority of desktop machines
18

and genera
l servers
19

run
Windows (which has the built
-
in Internet Information Services web server), the majority of
web servers
20

run Linux (using Apache as the web server). Linux is also regarded to be as
reliable and secure as Windows.


There are a wide range of

scientific packages available on Linux for manipulation of netCDF,
which may be useful in the post
-
processing. Not all of these packages are available for
Windows.


ArcGIS Server
21

and MapServer can both run on Windows or Linux, and we have selected
Linux

for the web server.

4.10.2

Programming language

The chosen development language must be able to create and consume SOAP web services,
provide multi
-
lingual pages (i.e. must have full Unicode support), offer database connectivity
and have the ability to interact
with the geo
-
server.


Java is a cross
-
platform language which can work on both operating systems and is a very
widely
-
used language for web server development. Both MapServer and ArcGIS Server offer
good support for Java development.


On the client side,
we will use JavaScript to program the interactive maps.

4.10.3

Graphs and charts

There are a number of free open source Java graph and chart libraries, e.g. JFreeChart.
JFreeChart provides a number of professional looking graph styles, includes line graphs,
scat
ter plots, multiple
-
axis, overlaid, box & whisker and candlestick graphs (similar to the
Carbon
-
Tracker combination bar chart and scatter plots with error bars
22
).


The graphs are dynamically generated on the server. This makes it possible to allow the user

to “zoom” to different ranges on the axes, or select different series for inclusion in the graphs.


It’s difficult to precisely quantify the user base for JFreeChart, since it is free open source
software. There are at least 20 commercial products availab
le that make use of JFreeChart
23
,



18

http://www.w3schools.com/browsers/browsers_os.asp

19

http://www.zdnet.com/blog/microsoft/behind
-
the
-
idc
-
data
-
windows
-
still
-
no
-
1
-
in
-
server
-
operating
-
systems/5408

20

https://secure1.securitys
pace.com/s_survey/data/201007/index.html

21

http://wikis.esri.com/wiki/display/ag93bsr/ArcGIS+Server+Supported+Platforms

22

http://www.esrl.noaa.gov/gmd/ccgg/carbontracker/fluxtimeseries.php

23

http://www.jfree.org/jfreechart/users.html




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

16
/
36



which gives confidence that there is a very large number of sites using the JFreeChart library
or one of the commercial libraries that incorporate JFreeChart.


Neither ArcGIS Server nor MapServer offer native methods of cre
ating and publishing graphs.
Therefore the choice of charting software is independent from the choice of GIS server.

4.10.4

Metadata

INSPIRE’s approach to metadata is hampered by some confusion surrounding various ISO
standards/schemas (19115, 19119 and 19139).

These schemas are defined at two different
locations: the ISO repository for official standards and the Open Geospatial Consortium
Schema Repository. Unfortunately, the schemas available at these two sites differ because
they use different versions of Ge
ography Markup Language (GML).


Max Craglia of the Joint Research Centre of the European Community stated “As a result, the
INSPIRE implementing rules are short and only say what functionalities are required, leaving
the detailed implementation to nonbindi
ng guideline documents. This has its drawbacks, as
we cannot guarantee that everyone will use the guidelines and that interoperability will be
achieved immediately. On the other hand, experience has shown that we are still making
small adjustments to the g
uidelines for metadata two years after their approval.”
24

Within
CARBONES, however, we can ensure all partners will adhere to the same guidelines.


We believe that the developers of ESRI ArcGIS, MapServer and GeoNetwork are committed
to support metadata sta
ndards used in INSPIRE.

4.10.5

Download service

The INSPIRE documentation is not definitive regarding what is and what is not allowed in a
download service. INSPIRE’s draft technical guidance document
25

covers two types of
download service:


1.

Downloading a pre
-
def
ined dataset

or part of a pre
-
defined dataset via HTTP GET
request.

2.

Direct access download services

including a query capability.


Both types must have a metadata record and be discoverable using an INSPIRE
-
compliant
discovery service.


In the context of
the CARBONES portal we would propose two download services:


1.

Download pre
-
defined datasets.

This would be direct downloads of post
-
processed CCDAS output in netCDF. INSPIRE’s guidelines for encoding of spatial
data document
26

mentions netCDF as a format for coverage
-
based data, particularly
in meteorological themes. The OGC has recently proposed netCDF for adoption as
an OGC standard (using the Climate and Forecast convention)
27

2.

Direct access download services

via WCS, the most

appropriate OGC web service
for direct access of grid coverage data. The draft INSPRE technical guidance
mentions WCS.




24

http://www.esri.com/news/arcnews/spring10articles/building
-
inspire.html

25

http://inspire.jrc.ec.europa.eu/reports/ImplementingRules/network/Draft_Technical_Guidance_Download_Services_v
1.0.pdf

26

http://inspire.jrc.ec.europa.eu/reports/ImplementingRules/DataSpecifications/D2.7_v3.0.pdf

27

http://www.opengeospatial.org/standards/requests/71





Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

17
/
36







The specific needs of the end users may generate further requirements for additional
download services, besides the services mandated b
y INSPIRE. These requirements will
develop through the ongoing consultations with the users through the Climate Users Advisory
Group. We do not anticipate any implications for the technology selection. However if
necessary, there will be an opportunity to
iterate in the later stages of the project, after the V0
portal.


Both the open source and commercial options would use the same technology for the pre
-
defined download option. The netCDF files would be hosted on a web server and the HTTP
URL will be made
available through a discovery service.


ESRI provide support for an INSPIRE
-
compliant download service using their GeoPortal
Extension add
-
in, or con terra’s sdi.suite
28

(including the “INSPIRE download service
extension for ArcGIS Server”). ESRI UK recomm
end con terra’s product over ESRI’s own
extension, but have also raised the possibility of providing INSPIRE
-
compliant services
without additional extensions. CERC pursued this with ESRI but were unable to obtain clear
guidance.


MapServer can provide an I
NSPIRE
-
compliant WCS download service.

4.10.6

Discovery service

INSPIRE requires us to provide a compliant CS
-
W service.


There is uncertainty about what would be required to create an INSPIRE
-
compliant CS
-
W
service using ESRI’s ArcGIS Server. ESRI’s websites su
ggest their GeoPortal Extension,
which supports a number of protocols including CS
-
W, and supports a number of profiles
including INSPIRE CSW 2.0.2 AP ISO
29
. However, ESRI UK have stated that GeoPortal
Extension is complicated, and instead they recommend u
sing con terra’s sdi.suite. ESRI UK
have also mentioned that it might be possible to provide an INSPIRE
-
compliant CS
-
W service
without any extensions. CERC pursued these issues with ESRI, but were unable to obtain
clear guidance and a firm price quotation
.


MapServer can publish a variety of OWS (WMS, WFS, WCS, etc) from which a CS
-
W can
harvest metadata and publish for discovery, but cannot act as a CS
-
W. GeoNetwork is the
most popular open source CS
-
W package. It offers OGC compliance: in fact OGC cite
GeoNetwork as the reference implementation for the CS
-
W standard
30
. GeoNetwork is used
by large organisations
31
, including the UK government, the United Nations and the European
Space Agency. The UK government is sponsoring ongoing development of GeoNetwork
.


Both ESRI and GeoNetwork provide well
-
supported INSPIRE
-
compliant CS
-
W discovery
services.

4.10.7

Data format

The main input for the CARBONES portal will be provided in netCDF format, a standard
format for scientific array
-
based data, which can contain metadat
a as well as data. It would



28

http://www.isotc211.org/WorkshopSoutham
pton/Workshop_presentations/WS_09_Paredes_Hahn.pdf

29

http://webhelp.esri.com/geoportal_extension/9.3.1/index.htm#hrvst_use_tool.htm

30

http://cite.opengeospatial.org/reference


31

http://www.geosur.info/geosur/contents/GeoNetwork_and_ESRI_GIS_Port
al_Toolkit_Comparison.pdf

http://metheny.esrin.esa.int/geonetwork/srv/en/main.home





Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

18
/
36



be desirable to be able to use this format directly in our solution rather than converting,
although conversion remains an option, particularly as some post
-
processing of the output will
be necessary so the conversion routines
will exist.


Both MapServer and ArcGIS Server can read netCDF files to produce WMS and WCS,
although MapServer only supports certain formats of netCDF and they advise “for stability it
may be necessary to convert the files into GeoTiff format”
32
.


In the V0

portal we have found it necessary to follow this advice and convert the netCDF files
into GeoTiff format, using the free open source package gdal_translate, in order to publish
WMS using MapServer. As part of the further development of the portal beyond V
0, we will
investigate using the netCDF directly from MapServer. See section
6.2

below.


There appears to be a problem using the temporal element
33

o
f netCDF in ArcGIS Server,
but in general, the ArcGIS netCDF support seems to be slightly more robust than
MapServer’s.


As well as producing WMS and WCS services, both ArcGIS Server and MapServer can
consume external services in WMS and WCS. This may be
useful in consuming data from
other services, for example MACC and GeoLand2.

4.10.8

Interactive maps

An essential part of the CARBONES web portal is the interactive maps. The functionality
required is summarised below:




Support for different internet browsers inc
luding Internet Explorer, Safari and Firefox.



Display different base mapping: proprietary free maps such as Microsoft’s Bing or
Google’s layers, and map layers available through WMS.



Basic navigation: zoom and pan functionality



More advanced navigation: ov
erview maps, place name searches



Interaction with CARBONES data, including clicking to show individual data or links to
time series graphs.



Temporal data support through a calendar control, time selection control, etc.



Support for animations would be desir
able but is not available in any standard web
mapping API at the present time. There are several possible methods for including
animations in the portal. All of these options would use different technologies from the
WMS/JavaScript API maps and would not a
llow the user to zoom and pan when
viewing the animations. It is not necessary to make the final selection of animation
technology now.

o

JavaScript could be used to animate static images by manipulating the
browser Document Object Model, as in CarboScope
34
.
This would not allow
the end user to download the animations for offline use.

o

Animations could be created in QuickTime movie format, as in
CarbonTracker
35
. Animated GIF files or Flash animations are other common
methods of presenting animations in web pages
.

o

The Google Earth API could be used to embed Google Earth animations into
the web pages
36
.




32

http://mapserver.org/ogc/wcs_format.html

33
http://forums.arcgis.com/threads/6086
-
Using
-
data
-
from
-
netCDF
-
with
-
ArcGIS
-
Server
-
10?highlight=netCDF

34

http://www.carboscope.eu


35

http://www.esrl.noaa.gov/gmd/ccgg/carbontracker/weather_movie.html


36

http://code.google.com/apis/earth/





Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

19
/
36








Support for printing.


ArcGIS Server provides client
-
side mapping utilising JavaScript, based on AJAX. ArcGIS
Server provides a wizard to enable development of a f
ront
-
end with zooming/panning, an
overview map, searches, data interaction, editing, printing, queries and geoprocessing
37
.
This can be customised by using the ArcGIS API for JavaScript. ArcGIS Server 10 also
adds a time
-
slider to deal with temporal data.

Support for Web Services includes KML,
WMS, WCS, WFS and WFS
-
T (Advanced only). REST and SOAP access are included with
all editions
38
. Base mapping can be added using any ArcGIS Server cached map service, or
an ArcGIS Online base map
39
. ArcGIS Server can

provide a client
-
side mapping utility
fulfilling all of our requirements.


The open source mapping JavaScript API under consideration is OpenLayers
40
.
OpenLayers supports KML, WFS and WMS, but WCS is currently not supported. However it
would be possible
for MapServer to consume any external WCS on the server, and publish it
as a WMS into the JavaScript API. This would allow us to achieve all the functionality needed
for the CARBONES portal. Time
-
slider controls have also been used in OpenLayers
41
.
Examples

of web map portals created with OpenLayers include the Sustainable Energy
Authority of Ireland’s geothermal maps at
http://maps.seai.ie/geothermal


One element of displaying maps that is not yet covered is ma
p tile caching, which is
sometimes used to improve performance. There are open source tile cache products such as
Meta Carta Labs Tile Cache. These may be used in development of the portal if it proves
necessary. There are no fundamental implications on th
e basic choice of technologies.

4.10.9

Content management system

A web content management system (CMS) is a software system which provides website
authoring and administration tools designed to allow users with little knowledge of web
programming languages or mar
kup languages to create and manage the site's content with
relative ease.


A CMS would allow some CARBONES partners to edit the portal web pages directly, without
having to go through CERC. We require a system which can offer the following features.




Multilingual support



Security



Allow web designers without programming skills to create templates that define the
appearance of web pages



Ability to create custom modules through a programming API:

o

for graphs generated on the server

o

for our interactive maps


There are a number of free open source CMS systems available fulfilling our requirements,
for example Alfresco. These could be used with ArcGIS or MapServer: there would be no
direct interaction between the GIS server and the CMS.




37

http://www.esri.com/library/brochures/pdfs/arcgis
-
server
-
functionality
-
matrix.pdf

38

http://www.es
riuk.com/products/showproduct.asp?prodid=26&groupid=2&activetab=1&mode=keyfeatures

39

http://resources.esri.com/help/9.3/arcgisserver/apis/javascript/arcgis/help/jshelp_start.htm#jshelp/overview_api.htm

40

http://openlayers.org/

41

http://www.mail
-
archive.com/dev@openlayers.org/msg03685.html




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

20
/
36



4.10.10

Post
-
processing

It wil
l be necessary to post
-
process the CCDAS output data, including the carbon flux and
carbon store data. This post
-
processing will require tools capable of manipulating the
netCDF files. Possible tools include:




Commercial packages like MatLab, IDL or ArcGI
S Server’s built
-
in tools;



Free open source tools like Ferret, the Climate Data Analysis Tools (CDAT), the
NCAR Command Language (NCL) and the netCDF operators (NCO).



The temporal filtering package CCGVU developed at NOAA. This is proprietary.


The post
-
pr
ocessing calculations presented some special considerations. The development
of the scientific specification of the post
-
processing calculation has required considerable
iteration between CERC, LSCE and Noveltis, and is still continuing now. Therefore, to
mitigate the risk presented to the V0 deliverable timeline, the solution for the post
-
processing
in V0 is based on a commercial package (IDL 8.0.1) and a proprietary temporal filtering
package developed by NOAA, CCGVU. Both these packages have been proven
in practise in
similar projects at LSCE; and CERC, Noveltis and LSCE had staff available with experience
in IDL.

We may study the use of open
-
source replacements for the V1 portal (see the future
development section below).

4.10.11

Database

The CARBONES portal wil
l store some data in a relational database system, for instance the
post
-
processing output data for display on graphs and charts. The full output will not be
loaded into a relational database, since using the post
-
processed output files directly offers
mor
e flexibility.


The Workgroup license for ESRI ArcGIS Server would restrict the selection to Microsoft’s free
product SQL Server Express. SQL Server Express is a file
-
based database, designed for
small datasets, with limits
42

on CPU support, memory usage, d
atabase size and
administration tools. It would be adequate for the CARBONES portal.


The free open source solution would be to use MySQL or PostGIS: PostGIS is based on
PostGreSQL. Both of these are robust and widely used and can support larger databases

than SQL Server Express. The advantage of PostGIS is its support for spatial extensions,
which offer more power in manipulation of geographic data than MySQL.


SQL Server Express, MySQL and PostGIS are all suitable for the CARBONES portal and
cost
-
free.

In the V0 portal we have used PostGIS to store spatial data: some of the tile indices required
to serve WMS
-
T with MapServer.



42

http://www.itproportal.com/portal/news/article/2010/7/3/limitations
-
sql
-
server
-
express/




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

21
/
36







5

Portal architecture

This section describes the CARBONES portal architecture, building on D220.1
-
D230.1.


Figure
2

showed an overview of the CARBONES system data f
low. This indicates how the
post
-
processing

and the
web portal

fit into the overall CARBONES system. The later
sectio
ns of this document discuss these two components in more detail.

5.1

Post
-
processing

The post
-
processing component is run off
-
line at CERC. The post
-
processing algorithms will
be developed in IDL. These algorithms will make use of the C and IDL temporal filter
ing
programs CCGCRV and CCG_CCGU.PRO, which belongs to NOAA.

5.1.1

Inputs

The input for the post
-
processing component will be netCDF files containing the output from
CCDAS. The format is specified in D220.1 section 4. The PM2
-
02 deliverable to be displayed
in th
e V0 portal contains only 2 years of data.


Outstanding issues with the netCDF formatting are:

-

compliance with the Climate and Forecast (CF) conventions for netCDF due to the
data packing method used

-

use of the CMIP
-
5 conventions for variables that are not

yet included in CF (minutes
of progress meeting 2).


Both issues are to be resolved after the V0 portal.


The format of the netCDF files was originally described in deliverable D220.1.
Correspondence and discussions between CERC, Noveltis and LSCE have de
veloped a
more detailed description.


The sample 2 years dataset (PM2
-
02) supplied by LSCE/Noveltis consists of 10 carbon flux
files and 4 stock files, as shown in Table 4. A further 4 variables have been derived from
those supplied. The derived variables
are shown in Table 5.




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

22
/
36




No.

Type

Variable name

Symbol

Units

1

Land C flux

Net ecosystem flux

Fnee

kgC/m
2
/hr

2

Land C flux

Photosynthetic flux

Fgpp

kgC/m
2
/hr

3

Land C flux

Respiration flux

Fresp

kgC/m
2
/hr

4

Land C flux

Forest net ecosystem flux

Fnee_for

kgC/m
2
/hr

5

Land C flux

Grass net ecosystem flux

Fnee_gras

kgC/m
2
/hr

6

Land C flux

Crop net ecosystem flux

Fnee_crop

kgC/m
2
/hr

7

Land C flux

Biomass burning flux

Fbbur

kgC/m
2
/hr

8

Land C flux

Fossil fuel emission

Ffos

kgC/m
2
/hr

9

Ocean C flux

Net
ocean flux

Foce

kgC/m
2
/hr

10

Land C stocks

Total above ground biomass

Babove

kgC/m
2

11

Land C stocks

Total below ground biomass

Bbelow

kgC/m
2

12

Land C stocks

Leaf Area Index (equivalent to leaf
carbon stock)

LAI

m
2
/m
2

13

Land C stocks

Soil organic
carbon

SOC

kgC/m
2

14

Land

Latent heat flux

Fle

W/m
2

Table
4
. Carbon flux and stock files to be supplied by LSCE/Noveltis for V0


No.

Type

Variable name

Symbol

Derived equation

Units

15

Land & ocean
C flux

Natural carbon fluxes

Fnatural

Fnee + Foce + Fbbur

kgC/m
2
/hr

16

Land & ocean
C flux

Total carbon fluxes

Ftot

Fnee+Foce+Ffos+Fbbur

kgC/m
2
/hr

17

Land C flux

Total photosynthesis
and respiration fluxes

Fgppresp

Fgpp + Fresp

kgC/m
2
/hr

18

Land C stocks

Total biomass

Btot

Babove +

Bbelow

kgC/m
2

Table
5
. Carbon flux and stock files to be derived by CERC for V0

The netCDF files supplied from CCDAS are ‘packed’ or compressed. This reduces the
physical size of the file through a linear transformation to map the

original 32
-
bit floating point
values to a 16
-
bit integer value. To unpack the netCDF file, the following calculation is
performed:


Unpacked data = (Packed data x scale factor) + Offset


All values of 0 in the packed file indicate that the value is not a
vailable (e.g. for a forest net
ecosystem flux value at a grid point that is in the ocean) and do not correspond to a valid,
packed value of zero.


It remains to be resolved whether the ‘packing’ is compliant with the CF conventions.


NetCDF3 format has be
en used. Migration toward netCDF4 will be considered in later stages
of the project.





Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

23
/
36






The data supplied comprise:



Longitude

-
179.5° to 179.5° in 1° steps



Title = longitude



Units = ‘degrees_east’




Latitude

-
89.5° to 89.5° in 1° steps



Title = latitude



Unit
s = ‘degrees_north’




Time


Decimal date format in years. For the year 1990, this will range
from 1990.000171232877 to 1990.999828767123 in steps of 3 hours i.e. 2920 points
in a non
-
leap year. The times are the mid
-
points of the 3
-
hour periods.



Title = t
ime counter



Units = ‘decimal date’




Date array

An array with a row for each time point. The year column will contain the year (e.g.
1990), the month column will take a value between 1 and 12 and the day column will
contain a value between 1 to
28/29/30/31. The hour column will be either 1, 4, 7,
10, 13, 16, 19, 22 and the minute column will be 30. Therefore, a column in the date
array that reads:

1990, 4, 27, 4, 30

will correspond to 4.30 am on the 27
th

April, 1990.



Title =
date array yy/mm/jj/hh/mm



Units




Variable

Gridded data for the specific variable values. There is a value for
each longitude and latitude point. There is a data grid for each time step.


5.1.2

Post
-
processing specification

CERC will post
-
process the data provid
ed by LSCE, temporally and spatially aggregating
them as shown in
Figure
4
.





Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

24
/
36




Fig
ure
4
. Post
-
processing to be performed by

CERC. See
Table
6

for definitions of P, T
and A.





Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

25
/
36






For the 2
-
year dataset, the time filtering CCGVU program fits the data to a polynomial of
degree 2,

with
3

harmonics. The residuals are filtered in the frequency space using a low
-
pass filter of 667 days and a high
-
pass filter of 80 days. The outputs that are then passed
on to the CARBONES web portal are: the smoothed trend (T); smoothed curve (S); re
-
centred parameter A. To calculate A the period mean is subtracted from the original, raw,
data.


The output values from the temporal filtering process are summarised in Table 6.


CARBONES post
-
processing
output parameter

Description

Input values, P

The
original input data (flux, stock)

Smoothed trend, T

The polynomial fit plus the long term filter of the residuals

Smoothed curve, SC

The fitted function plus the short term filter of the residuals

Re
-
centred parameter, A

The original data minus the
period mean.

Re
-
centred smoothed curve,
A_SC

The smoothed curve (SC) minus the period mean. A_SC
is not included in V0 of the portal.

Table
6
. Output parameters from the CCGVU_PRO program


The spatial aggregation uses region masks provided by LSCE. There are 2 netCDF files for
the region masks, namely
regions_mask.nc

and
regions_mask_extend.nc
. In both files,
there will be 52 regions, with the difference being that the ‘extended’ masks hav
e expanded
boundaries where there are land
-
only or ocean
-
only regions. The 52 regions are shown in
Table
7
. These regions are also used in the portal

web mapping “regions” control. The
expanded boundaries of the regions_mask_extend.nc include adjacent ocean/land grid boxes
of 1x1 degree. The region file selection will depend on the region type. The file
regions_mask.nc

is used for regions that are lan
d and ocean, and
regions_mask_extend.nc

is
used for all other regions.


A third set of region masks was supplied to CERC by LSCE for use by the web portal in
creating maps of regions. This set masks the ocean regions for land only quantities and the
land r
egions for ocean only quantities. It is said to be more general than
regions_mask.nc

and
regions_mask_extend.nc

and does not rely on the CCDAS particular definition of land/sea
regions.


The precise treatment of missing data (NaN values) in the original da
ta remains to be
resolved. CERC, Noveltis and LSCE are working together to specify this treatment.




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

26
/
36





Region name

Land/

ocean

Region name

Land/

ocean

Global

Both

Southern sub
-
antarctic ocean

Ocean

Northern land and ocean

Both

Austral ocean

Ocean

Tropical land and ocean

Both

Boreal North America

Land

Southern land and ocean

Both

Temperate North America

Land

Global land

Land

Canada

Land

Northern land

Land

USA

Land

Tropical land

Land

Northern Europe

Land

Southern land

Land

Central mid
-
Europe

Land

Tundra

Land

Eastern Europe

Land

North America

Land

Mediterranean Europe

Land

Europe

Land

Boreal Asia

Land

North Asia

Land

Temperate Asia

Land

Tropical Asia

Land

Boreal Asian forest

Land

Southern America

Land

Boreal Asian steppe

Land

Africa

Land

China
-
Korea
-
Japan
-
Mongolia

Land

Australia and New Zealand

Land

Arabia

Land

Global ocean

Ocean

South Asia

Land

Northern ocean

Ocean

South east Asia

Land

Tropical ocean

Ocean

Equatorial Asia

Land

Southern ocean

Ocean

Australia

Land

North Atlantic
ocean

Ocean

North Africa savannah

Land

North Pacific ocean

Ocean

Equatorial African forest

Land

Tropical Atlantic ocean

Ocean

South Africa

Land

Tropical Pacific ocean

Ocean

Tropical South American forest

Land

Tropical Indian ocean

Ocean

Tropical South
American savannah

Land

Southern sub
-
tropical ocean

Ocean

South America temperate

Land

Table
7
. The regions for the spatial aggregation and the portal web mapping




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

27
/
36








5.1.3

Outputs

The output parameter
-
averaging time combinations that
will be passed on to the portal are
shown by the white boxes in Tables 8 and 9, for maps and time series respectively. A_SC is
the re
-
centred value of the smoothed curve and is not included in V0 of the portal.


Averaging
time

C flux and latent heat flux

C stocks including LAI

P

T

SC

A

A_SC

P

T

SC

A_P

A_SC

3
-
hour











Daily











Monthly











Annual











20yr











Table
8
. Parameter
-
averaging time combinations used for maps in the web portal


The
gridded data for maps have been created in netCDF files in an identical format to the
supplied CCDAS data, except that the files are not packed. These files are the input to the
portal server for the generation of maps and direct
-
access data services throu
gh WMS and
WCS. They will be made available for direct download by the end
-
user through the web
portal but the V0 data sets have not been made available for download as they contain only
preliminary data.


Averaging
time

C flux and latent heat flux

C
stocks including LAI

P

T

SC

A_P

A_SC

P

T

SC

A_P

A_SC

3
-
hour











Daily











Monthly











Annual











20yr











Table
9
. Parameter
-
averaging time combinations used for time series (regional
aggregates) in the web portal


The 1
-
D time series resulting from the spatial aggregation and temporal filtering have been
created in a simple ASCII CSV (comma separated variable) format. These are not in V0 but
will be made available in V1 of the portal.
These data will be loaded into a database for use
by the web portal server to create graphs. We do not anticipate making the whole post
-



Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

28
/
36



processing output CSV files available for direct download by the user: instead the web portal
server will provide a CSV
download service that queries the data in the database.


For V1 there will be animations, created off
-
line at CERC during the post
-
processing that can
be displayed on the web portal. Possible formats include KML/KMZ (for display in Google
Earth), Flash or
QuickTime. Some of these formats are proprietary, closed, formats but they
have the advantage of wide user acceptance for deploying multimedia on the web. The end
user of the CARBONES animations would not be required to purchase special software.
There are

alternative open standards but they have not seen significant user uptake. This
area of the architecture remains open, and will be determined at a later time.

5.2

Portal server and portal web application

Figure 5

shows the detailed architecture for the portal

web server and web application.

The custom code on the portal web server has been developed in Java and the web
application in JavaScript / HTML / CSS. Some third party components have also been used.
The choice of components is listed in the analysis of
engineering solutions, section 4.2.

5.2.1

Inputs

The inputs for the portal web server are the outputs of the post
-
processing, and the CCDAS
output. Both are described above.

5.2.2

Outputs

We are implementing the following web services from the portal web server.



Disco
very services: access to metadata served by OGC CS
-
W.



View services: maps and map legends served by OGC WMS. These will be used
by the CARBONES portal web application and will also be available for use in
other services, e.g. other geoportals or other GMES

services.



Download services

o

Downloading a pre
-
defined dataset. Download of the netCDF files created
in the post
-
processing and by CCDAS

o

Direct access data services: direct programmatic access to the gridded
data in the netCDF files (post
-
processing output

and CCDAS output),
served by OGC WCS.


The portal web mapping application is a JavaScript application which runs in the user’s web
browser when they visit the
www.carbones.eu

website. The maps are displayed using
Ope
nLayers, a JavaScript client for OGC WMS, connected to the OGC WMS service from the
portal web server. Dojo, a well
-
known open source JavaScript prototyping library, has been
used for some of the controls in the web mapping application.


The graphs will be

generated dynamically on the server using JFreeChart. The server
-
side
code will access the post
-
processing output from a database on the server.


The download of graph data will use a dynamic service on the web server, which will query
the data from the d
atabase.



Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

29
/
36











Portal Server

Other core
services

Scientific users,
policy makers

General public

HTML

CSS

JavaScript

Images

Maps

WMS


Maps, direct
access to
data

WMS, WCS

CCDAS output, post
-
processing output


CMS content
Data for graphs

netCDF

Metadata

JFreeChart

MapServer

Download
data files

netCDF

GeoNetwork

Metadata

CS
-
W

Interactive maps and
graphs

CMS

Chart

Module

Module

Map Module

Web browser

carbones.eu

OpenLayers

Custom code

Dojo

Maps,

data,

metadata

WMS, WCS,


CS
-
W, netCDF

Figure
5
.
CARBONES web portal (data flow)




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

30
/
36




6

Further development after V0

6.1

Development for the first annual meeting

Development of the web portal will

continue after the delivery of the V0 version. Here is a list
of the features to be implemented, in priority order. We hope these features will be completed
by the Annual Meeting in May 2011.


Post
-
processing

1.

Complete the specification of the spatial
aggregation, including treatment of missing
values in calculations (NaN).

Post
-
process and integrate the
20
-
year dataset PM3
-
04

2.

LSCE are to deliver 20
-
year fluxes to CERC by 15th of March
43
. These fluxes will
include: ORCHIDEE + oceanic fluxes for 20 years.

The biospheric CO2 fluxes will
be optimized based on the PYVAR assimilation system. CERC will work to post
-
process this data and integrate it into the portal by the Annual Meeting
44
. It is
unlikely to be possible to integrate all the data by the Annual mee
ting.

Interactive time series graphs for stocks and fluxes

3.

Line and bar graphs with labelled axes, and legends identifying the series

4.

Ability for user to change the ranges on the axes

5.

Download underlying graph data as CSV

Further features for the web
mapping.

6.

User
-
controlled transparency for the flux and stock maps, with satellite imagery as
background maps. We will offer the user a choice of background maps. There will
be some foreground layers to be shown as transparent overlays on top of the
CARBONE
S data layers, e.g. continent outlines and country outlines. We will
offer the user the facility to turn off these overlays.

Animations

7.

Animations of the changes to carbon fluxes and stocks using the monthly values.
This will not be done in the interactive

maps but separately, e.g. with an animated
GIF, QuickTime, Flash or Google Earth. The user should be able to download the
movie (D230.1 section 4.3.2.2) which would not be possible if the animations
were hosted in the interactive maps.


6.2

Development after
the first annual meeting

There will be
another phase of requirements analysis and design

after the first annual
meeting. This will take into account various inputs.



Partner feedback on the V0 portal



EU reviewer feedback on the V0 portal



Second CUAG meeting

(D120.3 due end of September 2011)



AEAT refined market analysis (updated D720.3 due end of March 2011)


Most of these inputs should be
available in May 2011 for the first annual meeting
. It may
be possible to begin the discussions at this point.





43

Action PM3
-
04 from progress meeting 3, 1
st

February 2011

44

Action PM3
-
06 from progress meeting 3, 1
st

February 2011




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

31
/
36






We also
propose
another version of the portal, V0.5
, to be delivered at the end of
December 2011
. This version is not required by the DoW but ensures progress made is
publicly available before the deliverables required by the DoW, V1 being required at month 30.




V
0.5 will display the first CARBONES 20 years data set with assimilation of data (due
to be made available by
LSCE at the annual meeting
45
)
.



Creating the V0.5 portal will allow some feedback from the further requirements
analysis phase to be realised in a li
ve portal. This will help with further discussions
about features and priorities for the final portal, V1.


Here is a list of features to be considered for the V0.5 portal


Post
-
process and integrate the 20
-
year assimilated dataset (PM3
-
07) to be made avai
lable by
LSCE at the first annual meeting.

8.

After the annual meeting. CERC will begin work to
post
-
process this data and
integrate it into the portal.


The following features have been agreed but have not yet been prioritised for order of
implementation.
They may be left until V1.

9.

INSPIRE compliant services (CSW, WMS, WCS).

a.

INSPIRE compliant WMS and WCS can be provided simply by configuring
MapServer appropriately.

b.

INSPIRE compliant CSW will be provided by a cataloguing component.
CERC’s current selection
is GeoNetwork but there is a possibility that we
may use the GENESIS cataloguing toolbox. This selection will be made after
the annual meeting.

10.

Use a Content Management System so that website is easier to edit (for CERC and
some project partners)

11.

Website i
s to be translatable

12.

Portal to present analysis of sensitivity of regional carbon fluxes to climate change.
This feature needs to be defined more precisely.

13.

Portal to present comparisons between model fluxes and other similar products. This
feature needs t
o be defined more precisely.

14.

Portal to present uncertainties trend detection and vulnerability detection, sensible
heat fluxes, and other aspects that are only available in the V1 CCDAS output
D510.2. See D230.1 for details. This feature needs to be define
d more precisely.


The features listed below are speculative and may be implemented in the V1 portal, or may
not be implemented at all. We have not yet decided the priority of these features, nor
assessed their practicability. Later discussions with projec
t partners and others may suggest
additional desirable features that are not listed here.


15.

Click on a point on the map to obtain:

a.

the numeric value at the point

b.

a time series graph for the point

16.

Overview map

17.

Printing maps / saving map images / creation of
PDF

18.

Printing graphs / saving graph images / creation of PDF

19.

Export maps to Google Earth




45

Action PM3
-
07

from progress meeting 3, 1
st

February 2011




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

32
/
36



20.

Add/remove data series to graphs interactively

21.

Place name search in the maps

22.

User
-
defined region of interest for spatial aggregation (may require post
-
processing
to be
performed dynamically


impractical?)

23.

Download data for the map currently being viewed (i.e. dynamically create a small
netCDF file containing a subset of the data).

24.

In the V0 portal we have found it necessary to convert the netCDF files into GeoTiff
forma
t, using the free open source package gdal_translate, in order to publish
WMS using MapServer. We will investigate using the netCDF directly from
MapServer. This is likely to require the netCDF files to follow the Climate and
Forecast conventions. We may m
ake use of the open
-
source ADAGUC
modifications to GDAL and MapServer
46
.



46

http://trac.osgeo.org/gdal/wiki/ADAGUC




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

33
/
36







7

Test plan for the V0 portal

7.1

Post
-
processing

Each stage of the IDL post
-
processing programme

was tested and the outcome of each
testing stage recorded.


The code was temporally tested

for one real grid point (110, 43) or (108, 56) and one missing
data (‘NaN’) grid point (66,121) for the following variables:


Variable

Variable type (Original/derived; land/ocean)

Fgpp

Original, land C flux

Foce

Original, ocean C flux

Babove

Original,
land C stocks

Fnatural

Derived, land & ocean C flux

Ftot

Derived, land & ocean C flux

Table
10
. Variables used for testing


For the spatial aggregation, the code was tested using one mask and one time point. Manual
spot checks
were performed to ensure that the data values were read correctly from the input
netCDF file.


Note that in the testing files, there were slight differences in values due to IDL and C program
rounding errors. The percentage difference was used to determine

whether values were a
match or not rather than an absolute difference.


The IDL code includes checks to ensure the data are in the expected format. If the data do
not fit the expected format the programme stops with an error message.


a.

Checking that the
variable data are read in correctly from netCDF files:

-

the dimension of the longitude should be 360

-

the dimension of the latitude should be 180

-

the dimension of the time variable should be 5840 (corresponding to 2 years of 3
hourly data)

-

the dimension of
the variable should be 3 (i.e. that the variable has a value for each
longitude, latitude and time point)

-

the date dimension is as expected

-

the sum of the first half of the year entries in the date array should equal 1990

-

the sum of the second of the year
entries in the date array should equal 1991

-

the months, days, hours and seconds in the date array, should start at 01:30 on 1
st

January 1990, be at 3 hourly intervals and contain 8 values per day




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

34
/
36




b.

Checking that zero's are replaced by NaN's (missing data)

-

the number of real values added to the number of ‘NaN’ values after unpacking the
data should equal the unpacked number of data values

-

the number of zeros in the packed data should equal the number of ‘NaNs’ after
unpacking


Testing was carried out at each

of the stages in the IDL program listed in Table 11.


Number

Description of IDL post
-
processing step tested

1

Mask data read in correctly from netCDF files

2

Raw derived data as expected

3

Spatial aggregation of 3 hourly data

4

Time averaging: daily
means

5

Spatial aggregation of daily mean data

6

Time averaging: monthly means

7

Spatial aggregation of monthly mean data

8

Time averaging: yearly means

9

Spatial aggregation of yearly mean data

10

Time averaging: period mean

11

Calculation of daily

A (‘re
J
centred mean’)
=

=
p灡tial⁡杧r敧慴a潮=潦⁤=ily⁁
=

=
qim攠ev敲慧i湧㨠W潮瑨ty敡渠A
=

=
p灡tial⁡杧r敧慴a潮=潦=m潮瑨ty⁁
=

=
qim攠ev敲慧i湧㨠ye慲ay敡渠n
=

=
p灡tial⁡杧r敧慴a潮=潦=y敡rly=A
=

=
qem灯r慬⁦il瑥物n朠gf⁤=ily=m敡湳
=

=
p灡tial=
a杧r敧慴a潮=潦⁤=ily⁴牥湤
=

=
p灡tial⁡杧r敧慴a潮=潦⁤=ily⁓䌠Csm潯瑨t搠d畲u攩
=

=
䑥aim慬⁤=瑥⁴=s瑩湧
=

=
tri瑩湧⁴=攠湥瑃䑆⁦il敳
ra睟潵琩
=
Table
11
. Program stages tested




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

35
/
36







7.2

Test plan for web portal

The following steps have
been followed to test the web portal. The test steps will be repeated
with different browsers in order to test the browser compatibility of the website.


To test the content pages:

1.

Visit each of the content pages from
www.carbones.eu

by clicking the links from the
home page, and check the appearance. The appearance should be checked with the
different accessibility options selected: normal font, large font, very large font, and
high contrast.

a.

Home page

b.

Project page

c.

Par
tners page

d.

Contact details


To test the web mapping application:

1.

Load the web mapping application page. Check that the controls appear correctly.
The tree view list of variables should appear, with the variables expanded.
“Photosynthetic flux” should be se
lected. The map and legend should appear for
photosynthetic flux and the whole world should be visible. The drop
-
down selections
should be “3
-
hourly” and “Mean”, and the date
-
time selection should be the first
timestep 1/1/1990 00:00
-
3:00

2.

Select different
variables from the tree control. Check that the map and legend
change. Check that the tree control only permits one variable to be selected at a time
(“radio button” behaviour).

3.

Select different options from the temporal averaging period drop
-
down and the
temporal filtering drop
-
down. Check that the map and legend change when these
selections are made. Check that the hour drop
-
down is disabled (greyed out) except
when the temporal averaging period is “3
-
hourly”. Select different variables in the tree
and ch
eck that the available options in the temporal averaging period and temporal
filtering drop
-
down change appropriately.

4.

Select different time steps from the calendar control / hour drop
-
down and check that
the map and legend change appropriately. Check that

the selection is restricted to the
valid range of dates and times: 1/1/1990 to 31/12/1991 (with the 2
-
year data set).

5.

Select different regions from the region tree control and check that the map zooms to
the correct extent and that areas outside the selec
ted region are hidden.

6.

Test the use of the map controls.

a.

Use the mouse to click the left, right, up and down arrows, and check that the
map view is panned.

b.

Use the mouse to click the plus and minus buttons and check that the map
view is zoomed in and zoome
d out.

c.

Use the mouse to click the globe and check that the map view is returned to
the full extent (the whole world should be visible)

d.

Use the mouse to pan the map by clicking the mouse button, and then
holding the button down while dragging the mouse.

e.

Use

the mouse wheel button (if available on the computer) to zoom the map
in and out by scrolling the mouse wheel button.




Deliverable D520.1


Ref

CARBONES
-
D520.1
-
REP
-
CERC
-
015
-
01
-
00

Date

31/03/2011

Page

36
/
36




8

Test report for the V0 portal

8.1

Post
-
processing

Stages 1
-
21 described in Table 11 were tested. The results were as expected with the
fol
lowing comments or exceptions.


[3] Spatial aggregation of 3 hourly data

For derived variables such as Fgppresp, the spatial aggregation tests give the correct results
when the constituent variables have NaNs at the same spatial location. However, when NaN
s
occur at different locations the sum of the spatially aggregated variables
cannot
equal the
spatially aggregated derived variable. This is due to the way the derived variables are
created i.e. if one of the constituent variables has an NaN, then the der
ived variable is set as
NaN. The treatment of NaN is a matter still being discussed between the CARBONES
partners.


8.2

Test report for web portal

The testing steps described in the test plan were carried out in the web browsers listed in
Table 12. The web app
lication behaved correctly in all cases.


Browser

Version

Operating system

Internet Explorer

7

Windows XP

Internet Explorer

8

Windows Vista

Firefox

3.5

Windows XP

Chrome

10

Windows XP

Safari

4

Macintosh OS X (Snow Leopard)

Table
12
. Web browsers used for testing the web application