Status Report December 2012

wattlexanaduΛογισμικό & κατασκευή λογ/κού

31 Οκτ 2013 (πριν από 3 χρόνια και 9 μήνες)

426 εμφανίσεις

LROSE

Status Report

2012
-
12
-
05


1



NCAR/EOL

Earth Observing Laboratory



LROSE

Lidar Radar Open Software Environment



Status Report


December

2012





Mike Dixon
, Wen
-
Chau Lee,

Steve Cohn, Bill Brown


2012
-
12
-
05




LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

2

Lidar Radar Open Software Environment

LROSE

Status Report


December

201
2

1

Community feedback

on the white paper

1.1

Soliciting feedback

In order to solicit feedback from our user community, we assembled an email list of users likely
to have an opinion on the white paper. On September 18, the white paper was sent out to this list.
The

initial list was aimed mainly at the radar community.

To

better reach the community of profiler and lidar users, on October 11 we followed up with a
second email to a different set of users
.

The outgoing emails, and the responses received, are included

in their entirety at the en
d of this
document, in section 5
.

1.2


Summary of
responses

1.2.1

Positive

responses

For the most part, the responses are very positive. Clearly there are many users in the community
who do not have the resources to develop their own soft
ware, and are very interested in an
improved set of
community
-
supported
tools to help them with their rese
arch. This will
enhance

efficiency and reduce

time wasted through inadequate tools.

There also seems to be a consensus that NCAR is
the appropriate or
ganization to lead

this
development.

1.2.2

Suggestions for alterations,
additions
, cautions

The following table lists points raised by some of the reviewers, and how we propose to address
those concerns.


Response from:

Discussion points

Status in LROSE


Nick P
otts, NOAA

Please include support in Radx for the P3
native format data
.

This will be done. We already hav
e
code in soloii to handle the P3 data.
This will
be used as the basis for P3
support in the Radx package
.

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

3

Response from:

Discussion points

Status in LROSE


Paul Hein, CSU

Documentation
is
essential
.

The c
ode base must be maintained and
supported
.

Include issue tracking
.

Ensure careful updating of meta
-
data during
processing
.

These are all excellent points.

We have already set up a Jira site for
issue tracking.

We are aware of the need
for
documentat
ion and maintenance an
d
will ensure
that
this is
incorporated
into
the project from the start.

Daniel Michaelson, SMHI

Ensure support for Opera HDF5 format.

Algorithms should be modular


i.e. able to
be used either in isolation or in conjunction
with oth
er procedures.

Adoption of common formats will lead to
improved international collaboration.

Support for Opera HDF5 is
specifically mentioned in the white
paper.

This will be added to the Radx
package in due course.

Daehyun Kim, Columbia

Good documentatio
n is essential.

Agreed.

Chidong Xhang, Miami

Data assimilation into models is an important
end product of such a software system.

Quantitative statistical analysis (climatology)
would also be a valuable outcome.

These are good points and the system
will b
e

designed to facilitate such
outcomes.

Michael Bell, Hawaii

C
are
is needed
in how software i
s shared.
There is a requirement

for informal as well as
formal sharing. GitHub is seen as a good
model for open sharing


NCAR need not
necessarily host the code

repository

Agreed. The white paper specifically
addresses informal sharing in section
2.2.

Courtney Schumacher,
Texas A&M

A code repository populated by PIs would be
helpful.

We can consider how best to include
code from various sour
ces.

Frederic Fabry,

McGill

D
on’t ignore NOAA and DOE/ARM
.

Don’t ignore spectral data and fixed pointing
data
.

We are already actively collaborating
with both NOAA and ARM on
LROSE (see section 3).

We will make sure the system can
support spectral (time series) data
and fixed
-
pointing instruments.

Laura Bianco, NOAA/PSD

Take care with QC considerations. QC
algorithms vary widely depending on the type
of instrument.

This is an excellent point, and will
require careful consideration.

Clearly
QC is one of the more tricky aspect
s
of t he s ys t em, and will vary
cons iderably bet ween ins t rument s.
We will probably rely on s pecialis t
us ers for t he development of QC
procedures for new ins t rument s.

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

4

Response from:

Discussion points

Status in LROSE


Wayne Hocking, Ontario

Wayne does not have the flexibility to be able
to contribute to the

collaboration effort, and is
therefore
wary of getting involved.

This points out that we need to be
careful how we communicate the
aims of the project. It is not
necessary that everyone contribute,
only that the development is
somewhat shared with the
com
munity. We do not want to
discourage use of the software just
because the user does have the
resources to contribute.

Everyone is
welcome to use
LROSE
.

Scott McLaughlin, DeTech

Scott raised the issue of open source vs.
proprietary code. He asks whether we

have
plans on how to include support for
commercial products.

These are good points. Our principal
user base is the resear
ch community.
We are
proposing

a strict

open source
model, freely shared on the web.
Commercial entities will be welcome
to use the c
ode and to contribute if
they wish. Many commercial entities
support the open source model as
good for their business. We will not,
however, include commercially
-
protected code in the system.


2

NSF Community Workshop on Radar Technologies

This NSF workshop

was held at the NCAR Center Green campus from
27


29 November.

One of the sessions covered ‘Radar data and software analysis tools’. This was facilitated by
Sandra Yuter of North Carolina State University and Wen
-
Chau Lee of EOL/RSF. The focus of
this se
ssion was not LROSE per se, but rather a broad look at software needs in the community.

Sandra broke the participants into small groups of 6 to 8 people, each of which was asked to list
their highest

priority software needs. These lists
we
re posted on the

walls and
the participants
were

then asked to ‘vote’ (with colored stickers) for the items they deemed most important, from
both the personal research and community point of view.

Sandra will prepare a detailed report on this session and it is not appropr
iate to preempt that in
this document. However, it was clear from the votes cast that open source community software of
the type proposed by LROSE was a popular notion with the workshop participants. When asked
for a show of support, the workshop participa
nts appeared to be very positive about the LROSE
proposal.

A poster on LROSE was prepared for the workshop. The poster may be found on
-
line at:


LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

5

3

Visit
to EOL
by Scott Collis from DOE/ARM

Scott Collis came out to Boulder 2 days before the NSF workshop, and
spent the
Mon
day before
the workshop with EOL personnel. The main topic of discussion was the need for close
collaboration with EOL on the development of the
planned
suite of community software tools.

Scott and his group have developed an extensive radar a
nalysis support library in Python.
This
will be an excellent starting point for some of the functionality required by LROSE. Scott
demonstrated some of the functionality to EOL engineers, and we have already taken steps to
include use of this package in a
Python version of solo.

Scott’s group has hired a soft
ware engineer to take over the P
ython work, to make it more formal
from a development point of view,
the aim being
bett
er documentation and maintainability
.

It is clear that collaboration with DOE/ARM o
n LROSE will be essential to avoid duplication of
effort and make the most efficient use of the resources at our disposal.

4

Progress since September
.

4.1

Solo3

Nancy Rehak has made significant progress on solo3


the 64
-
bit replacement for soloii.

The GUI compo
nents have been upgraded to the latest C++ versions of the GTK toolkit. This was
time
ly work because some of the
older GTK components are no longer supported.

The I/O layer now uses the Radx library for accessing the data files. This is a significant step
towards full support for CfRadial files natively in solo3.

A number of bug fixes were made and the latest beta
-
release version of solo3 seems to be quite
stable.

Solo3 can now be built and run on Mac OSX platforms.

It compiles natively on 64
-
bit platforms.

4.2

Emerald

Greg Meymaris (RAL) has made significant progress on Emerald (
eMERALD = Matlab
Environment for RAdar and LiDar
).

This is a Matlab
toolkit

that will have similar functionality to
solo, but which will be much more extensible because it is in Matlab
and therefore the user can
modify it to
take advantage of

Matlab
-
support functions.

Emerald has an data access layer, that allows reading and writing of CfRadial files. This makes
use of the Matlab support for handling NetCDF
-
4 files.

On top of this is bui
lt a
radar data viewer
that supports PPI
-
type plots. This will be extended to
support RHIs and BSCAN type plots in the future.

One big advantage of Emerald over solo is
its

portability


it will run on any platform supported
by Matlab


i.e. Windows, Mac O
SX and LINUX.

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

6

4.3

PySolo

Xin Pan,
our

student from CU, has been working on a Python version of solo.
Like
Emerald, this
will also have the advantage of cross
-
platform portability. And additional advantage is that
Python is a free package, so that students who
do not have access to a Matlab license will be able
to use pysolo instead.

Joe VanAndel has kindly agreed to mentor Xin in her work on pysolo. Joe has worked in Python
for many years, and is therefor
e in a good position to make a positive contribution to t
his effort
.

Scott Collis contributed significantly to the pysolo effort during his one
-
day visit
. He was able to
point u
s to a Python package that makes the rendering of polar d
ata much more efficient.

4.4

Cartesian transformation

applications

Good progress ha
s also been made in the transformation of data from polar to Cartesian
coordinates. The Radx2Grid application now replaces most of the functionality of SPRINT. Jay
Miller has been testing this application, and comparing the results to those produced by SPR
INT
and CEDRIC. It appears that Radx2Gr
id does a good job of closely

reproducing the SPRINT
results.

The next step will be to incorporate the REORDER functionality into Radx2Grid. A start has
been made on this, but more work is required.

5

Community feedback

on

LROSE white paper

5.1

Radar community

5.1.1

Outgoing email

text

On September 18 the white paper was circulated to a list of people considered likely to be
inte
rested in the proposed ideas. The email sent out was as follows:

To: the radar/lidar research software
user community


NCAR/EOL has submitted a white paper to the National Science Foundation (NSF) proposing an
extensive upgrade to the suite of NCAR
-
supported software tools available to the radar/lidar user
community. A new software framework is proposed th
at would enable the community to jointly
develop and upgrade the software suite in the future. NSF has asked us to solicit comments on the
proposal from the user community.


The white paper is attached, and a copy of the executive summary is also included

at the end of
this email.


If you are interested in using NCAR
-
developed radar/lidar software in the future, we ask that you
review the proposal, and provide us with comments and feedback.


NSF is organizing a radar workshop, to be held at NCAR in Bould
er, from 27
-
29 November this
LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

7

year. We would like to present a summary of your comments at that workshop.


Therefore, if possible, please respond with your comments within 4 weeks by mid
-
October.


Thank you


Kind regards


Mike Dixon

5.1.2

Email list

The white

paper was sent to the following

email list:


a.seed@bom.gov.au Alan Seed BOM Australia

andrew.vandermerwe@weathersa.co.za Andrew VanDerMerwe SA Weather Service

brodzik@uw.edu Stacy Brodzik U. Wash.

chandra@engr.colostate.edu Chandra CSU/CHILL

chris.fairal
l@noaa.gov Chris Fairall NOAA

chris.weiss@ttu.edu Chris Weiss

chuck.Long@pnnl.gov Chuck Long PNNL

conrad.Ziegler@noaa.gov Corad Ziegler NOAA

cschu@tamu.edu Courtney Schumacher TAMU

czhang@rsmas.miami.edu Chidong Zhang U. Miami

daehyunk@gmail.com Daehyun Ki
m Lamont
-
Doherty

daniel.michelson@smhi.se BaltRad Radar Network Sweden

david.p.jorgensen@noaa.gov Dave Jorgensen NOAA NSSL

dhence@atmos.washington.edu Deanna Hence NASA GSFC

dparsons@ou.edu David Parsons OU

e.ebert@bom.gov.au Beth Ebert BOM Australia

ed.zi
pser@utah.edu Ed Zipser

frank.marks@noaa.gov Frank Marks NOAA

hanlei@ouc.edu.cn Han Lei Ocean University China

hblue@ou.edu Howie Bluestein OU

hcbarnes@uw.edu Hannah Barnes U. Wash.

heymsfield@agnes.gsfc.nasa.gov Gerry Heymsfield NASA

houze@uw.edu Bob Hou
ze U. Wash.

j.peter@bom.gov.au Justin Peter BOM Australia

jdavison@atmos.uiuc.edu Jennifer Davison U. Ill.
-
Champain
-
Urbana

jgeorge@engr.colostate.edu Jim George CSU/CHILL

jtrapp@purdue.edu Jeff Trapp Purdue

jwurman@cswr.org Josh Wurman CSWR

kakosiba@cswr.o
rg Karen Kosiba CSWR

karel.deWaal@weathersa.co.za Karel DeWaal SA Weather Service

katsu@jamstec.go.jp Masaki Katsumata JAMSTEC

kevin@nsstc.uah.edu Kevin Knupp UAH

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

8

liz@atmos.colostate.edu Elizabeth Thompson CSU

mdeng2@uwyo.edu Min Deng U. Wyoming

mmbell@nps
.edu Michael Bell, University of Hawaii

murakami@ipmet.unesp.br Jaqueline Murakami IPMET Brazil

mzuluaga@uw.edu Manuel Zuluaga U. Wash.

nick.guy@noaa.gov Nick Guy NOAA NSSL

nolan.Atkins@lsc.vsc.edu Nolan Atkins

oue@rain.hyarc.nagoya
-
u.ac.jp Mariko Oue Nago
ya University Japan

p.purdam@bom.gov.au Phil Purdam BOM Australia

paskenrw@slu.edu Robert Pasken

pat@chill.colostate.edu Pat Kennedy CSU/CHILL

paul.Smith@sdsmt.edu Paul Smith

paul.joe@ec.gc.ca Paul Joe Environment Canada

r.potts@bom.gov.au Rod Potts BOM Au
stralia

rauber@atmos.uiuc.edu Bob Rauber

rodi@uwyo.edu Alfred Rodi U. Wyoming

roelof.burger@wits.ac.za Wits University South Africa

rutledge@atmos.colostate.edu Steve Rutledge CSU

sally.mcfarlane@pnl.gov Sally McFarlane PNNL

samson.hagos@pnl.gov Samson Hag
os PNNL

schen@rsmas.miami.edu Shuyi Chen U. Miami

scollis@anl.gov Scott Collis ARM ANL

sdeszoek@coas.oregonstate.edu Simon DeSzoeke OSU

she.Feng@pnnl.gov Zhe Feng PNNL

spowell3@uw.edu Scott Powell U. Wash.

syuter@gmail.com Sandy Yuter NCSU

tlang@radarmet.a
tmos.colostate.edu Tim Lang CSU

walt.petersen@nasa.gov Walt Petersen NASA

yrichardson@psu.edu Yvette Richardson PSU

david.b.wolff@nasa.gov

David Wolff NASA

5.1.3

Response from: Jeff Snyder,
U
niversity of Oklahoma

jsnyder@ou.edu

Mike,

I read through the radar software request for comment, and, unofficially speaking on behalf of our radar
group (“our” being Howie Bluestein’s radar group), I think the project will ma
ke our research m
ore efficient.

Below is our standard workflow when w
e work with data collected by
the mobile radars from UMass:



Archive in UF



Convert UF to DORADE SWP format for diplay and data editing (velocity dealiasing, clutter removal,
etc.) using Solo II



Convert SWP

to netCDF (using xltrsii) for modification in Matlab (attenuation correction, KDP
estimation, etc.)

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

9



Convert back to UF, then convert back to SWP for display

Starting in 2011, we began to use CfRadial as the “archive” format for data fro
m our new mobile ra
dar
(RaXPol).
The primary advantage of the NetCDF
-
based CfRadial convention is that such data can be used
very readily with our existing Matlab programs/algorithms (it helps too that it’s volume based, which
makes
data management easier).
We still rely upo
n Solo II for display, radial velocity dealiasing, and gate
-
to
-
gate
or ray
-
to
-
ray editing, which means I still end up having to convert data back and forth between CfRadial and
SWP. Your RadXConvert utility significantly reduces the “hassle factor”, but it

still would be easier if we
could display CfRa
dial data natively in Solo II.

From the request for comment document, it looks like there *
is
* a new version o
f Solo that supports
CfRadial.
Are you looking for beta testers for this softw
are?


If so (or even
if not), I’d love to test it out.
All of
the data we collected this past year is currently in both CfRadial and SWP formats since we don’t have
access to a nice disp
lay program for CfRadial data.
If we could display CfRadial data in Solo, it would make
my
life a lot easier in terms of file forma
t and file version management.

I look forward to any further developments, and I really think that standardizing radar archive formats to
facilitate data sharing and algorithm development is greatly needed. Thanks!

J
eff Snyder

University of Oklahoma Meteorology PhD Student

http://www.tornadocentral.com

5.1.4

Response from:
Nick Guy, NOAA

nick.guy@noaa.gov

Hi Mike,

The document looks good. There is no mention of supporting the P3 rad
ar format moving forward. We
spoke a little about this in Seattle. The solo translator supports

the format (I believe Dick Oye

put this in for
Dave Jorgensen as a favor), but Radx does not as of yet. You mentioned that there was some mystery
code that
was left out of Radx at first that may be associated with the distinct P3 format.

I received the Sigmet manual that discusses the data format and wanted to pass along to your group as a
reference. Is there any way this could also be included in Radx in
the future?

Also, a while back we corresponded about getting Radx to work on MacOSX. At that time you were going to
try an install on a Mac machine. Have you had time to follow this up?

Thanks,

Nick Guy

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

10

5.1.5

Response from:
Paul Hein, Colorado State Universi
ty

hein@atmos.colostate.edu

Hi Mike,

I am emailing you directly with some comments from the CSU Radar Meteorology Group (headed by Steve
Rutledge) and from CSU
-
CHILL on the comm
unity software tools proposal.
The comments
are from various
individuals.
I have collected and combined the comments in the points below.


1) We are very encouraged to see NCAR taking the lead on developing new ve
rsions of the 'core' software.
New software is needed.
With the new radar
format CfRadial, we need software that can use it.


2) We do expect NCAR to take a lead role in developing and
maintaining the core software.
Our concern is
that the software will continue to be maintained and developed over the long term.


3) That is wh
y we feel strongly that it is very important that the larger community is involve
d in the software
development.
It should be easy for members of the community to take part in the ongoin
g development of
the software.
Being transparent in the development and

maintenance of

the core software would help.
Multiple points of view can help improve the software and keep it up to date.


4) There should be an obvious and defined way that bug reports are submitted and
tracked for the core
programs.
(It would be good
if community applications fo
llowed the same standard too).
Something lik
e
Bugzilla would be good.


5)
Setting up a website with forums and mailing lists would help encourage the community to take

part in
testing and debugging the core programs.
The larger

community should be a part of the development of the
core software so that the burden is not t
otally on the NCAR developers.
Also forums can be a place where
questions about the software could be answered by the community.


6) Detai
led documentation is c
ritical.
Online documentation should contain many simple
-
to
-
more
-
complex
software usage examples along with th
e resultant output plots, etc.
The original hardcopy manuals for
NCAR Graphics were good examples of this.


7) Documentation is also important fo
r community applications

as well as the core software.
The libraries
with high level language stubs should be documented, so that we can develop our own software in the
languages of our choosing (python, idl, matlab, ncl, etc.).


8) Also it should be easy

to update the meta data at all stages of the data manipula
tions.
Specifics of
calibration adjustments, thresholding applied, identification of unfolding algorithms used, etc. should ride
along with the da
ta sets as they are processed.
It would help keep t
rack of what is done so that one's results
could be reproducible.


To summarize, we are in favor of the proposal and we hope the above comment
s can help you with your
task.
Thank you for your work on this.

Paul

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

11

5.1.6

Response from:
Daniel Michaelson, SMHI, Sw
eden

daniel.michelson@smhi.se

Hej Mike et al.,


Thanks for including me on your list. As project manager for BALTRAD, I have some BALTRAD
-
centric
feedback in the following.


In Europe, BALTRAD has taken the

lead in the community
-
based development of a software platform for
exchanging and processing weather radar data. This got underway in 2009 with funding from the EU's Baltic
Sea Region Programme supporting the national meteorological services in the region
. This external funding
runs out at the end of 2013. Our scope is more limited than that in your proposal; we have targeted data
from scanning weather radars running at X, C, and S bands. We have also limited our scope in terms of
which data are relevant t
o us; in our case, we have targeted polar (spherical coordinates) data that are
already collected into complete scans (sweeps) at the radar site as our starting point. This means we have
not made accommodation for time series (I/Q) data.


Our software is
licensed according to the LGPL, where the developers retain the IPR to their contributions. I
can highly recommend this approach, or, if too many developers/organizations make this unwieldy, the so
-
called Fiduciary Licence Agreement with NCAR(?) as the sin
gle entity.


Providing analysis tools and interfaces for free high
-
level languages like Python, as you have proposed, is
also the way to go.


In Europe, the modern standard file format is ODIM_H5 (HDF5), agreed
-
upon by 30 countries in
EUMETNET OPERA. I'm

pleased to see that you have included the ability to convert to/from such data.
Doing so is a prerequisite for enabling interaction and interfacing between systems.


I would like to encourage increased interaction between North American and European comm
unities when it
comes to software development of this kind. Differences in scope and technical implementation will
inevitably be different, but the global community stands to benefit if open development initiatives like
BALTRAD and your Community software
can develop data processing algorithms in a way that facilitates
their integration and access. With this in mind, I would like to see that some of the algorithms found in Table
1 are made available relatively independently from the rest of the core suite s
uch that they can become
more easily accessible for integration with our "toolbox" product
-
generation framework. Similarly, we hope
that you might find some of our processing algorithms appealing too, such that you might want to integrate
them into your co
re or community suite.


The implication for increased interaction/cooperation of this kind is that the end users, e.g. NWP, stand to
benefit from increasingly harmonized datasets as data become increasingly available globally.


A small note on the defini
tion of HDF5 in the glossary: HDF5 comes from the HDF Group which is a non
-
profit organization based in Champaign, Illinois.
http://hdfgroup.org/



best, >
-
D

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

12

5.1.7

Response from:
Daehyun Kim,
Lamont
-
Doherty Earth Observatory

of Columbia
University

daehyunk@gmail.com


Dear Mike,

I've went through the paper, it is well
-
written. I like the basic idea and fully support it as a radar
-
analysis
beginner. One thing that might worth considerin
g to add to the paper is documentation. I think on/offlin
e
documentation of the software

is extremely important, and could aff
ect how widely the software is

used.
Maybe wiki
-
type online documentation can be used for the community ap
plication parts of the s
oftware
. I
believe you're aware of this, and it is implicitly included in your plan, but it would be great if you could
explicitly include that point in the abst
ract and the section 1.6 Goals.

Thank you for your ef
forts on this, I appreciate it.

Best,

Daeh
yun

5.1.8

Response from:
Chidong Xhang, University of Miami

czhang@rsmas.miami.edu


Hi Mike and Wen
-
Chau,

I glanced through the white paper, and here are comments
from a radar
-
illiterate person:

It appears that the p
roposed software tools are mainly for users from the radar community. Which

is fine.
But the ability of the proposed software to transform data to common Cartesian format will draw many users
outside the traditional radar community, such as modelers and da
ta diagnosticians.

This can be a good
selling point, but it does not come across obviously in the white

paper.

Figs. 2
-
5 give

an impression that display is the main end product of the proposed software. I'm sure this is
one of the most frequently ways of
using radar data. But quantitative statistics

can be another product.
(Some of them are listed in Table 1?) Maybe this can be highlighted by an additional box in the figures that
precedes the display box in the arrow flow. I guess these user defined statis
tics can be what you prefer to a
s
community developed modules.

These are what I have now. Thank you for trying to make
radar data more user friendly.

My students can definitely benefit from this. Wish you a p
roductive workshop in November.

Cheers,

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

13

Chido
ng

5.1.9

Response from:
Rod Potts, Bureau of Meteorology, Australia

R.Potts@bom.gov.au

Mike

Thank you for your invitation to

comment on the Lrose proposal.
We recently had a meeting to discuss the
proposal and the following comments come from this.

These comment
s are foc
ussed on radar data processing.

The Bureau values the good working relations with NCAR across a range of activities and will continue to
support these. This includes activities in radar meteorology.

In this regard the o
bjectives presented in the
LROSE

proposal are supported.

NCAR applications currently used in the Bureau include



SPRINT
-

research



CEDRIC
-

research



TITAN
-

research and operations

Applications have also been written here to convert 3D
-
Rapic data to
UF and the Lassen format to UF.

Th
e Bureau has a network on 75 operational weather radars and 2 research radars across the country with
a combination of C
-
band and S
-
band radars. In the last few years there has been a process underway to
significantly upgrade a number of radars in the netw
ork.

Raw radar data from the Bureau's radar network is transmitted in 3d
-
rapic format and it is likely th
is will
continue for some time.

The Bureau operates Nowcast Application Servers that process radar data from more than 20 radars to
generate nowcast pr
oducts for downstream applications. The Nowcast Application Servers run TITAN and
WDSS (providing TS parameters) and rainfields (QPE and precipitation nowcasting application).

The rainfields application generates output

products in CF
-
NetCDF4 format.

The
Bureau is currently working on the Strategic Radar Enhancement Project (SREP) with the aim of
assimilating radar data into high resolution rapid update NWP.

For SREP a data format based on the Opera Data Information Model (ODIM) (HDF5) has been adopted and

it is proposed that the ODIM HDF5 format will be used

for post
-
processed radar data.

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

14

As previously discussed Daniel Michelson from SMHI will be visiting the Bureau in early December and joint
discussions on data formats and other issues are proposed when
you visit the Bureau.

This will include potential areas for more collabor
ation in radar data processing.

Regards

Rod

5.1.10

Response from:
Jaqueline Murakami, IPMET,
University of Sao Paulo,
Brazil

murakami@ipmet.unesp.br

Dear Mike

The new software proposal is ve
ry good, I've sent
it
to our p
eople and the feedback was very

positive since
the main goal is building a suite of software tools to facilitate the manipulation and visualization of radar and
wind profiler data for use in operation, research and education.

IPMet has been using radar software since the 80's decade both for research purposes as for real time
operation. We have already worked with Sprint, Cedric, PPI
-
mmm, Soloii for radar data processing and we
have used the display software Zebra, Gempak and I
DV. Since 2005 we are mainly using TITAN and
Rview/CIDD displays for weather nowcasting and post
-
analysis studies, which we consider a very complete
and useful radar software

suite
. We have Sodar and Lidar sensors running proprietary software mostly for
re
al time visualization. Next year, with the creation of the B
.
S
.

in Meteorology course in our university, we
will be giving support to students and teachers of this course too. Thus, we have interest in the software for
several purposes.

Forecasters and re
searchers at IPMet are used to work with the software R, Grads, MatLab/SciLab, NCL
and NetCDF is the most used data format. The idea of having NetCDF as the software main format and the
facility to develop radar/lidar algorithms i
n these high level languag
es led them to be very

interested and
excited. All the worry on supporting open and standardized data formats are very ap
p
ropriate.

The three types of data structure for time series instruments (IQ time series, Radial and Cartesian) with their
specific set

of algorithms are an organized way of working with radar/lidar data that would help increasing
the development of algorithms for each data representation format.

We've appreciated the list of the core suite algorithms presented in the document, it include
s many TITAN
applications and others of our interest. The core services cover from data translators to data integration and
visualization, which are
mostly
complete.

Here, forecasters and researchers usually need the support of the IT group to have
the dat
a available for
use in
radar software because it usually requires converting data in a specific format, writing scripts or
running applications in linux environment, being aware of files and data directories. There is a great
expectation of having a user
-
f
riendly interface for setting up the environment and running applications to
enhance data accessibility.

We believe the proposed approach for the development of the software, which provides a core suite of
algorithms and encourages community collaboration,

is a very good solution to build a high quality software
system
which will be constantly improved and updated. It is good to know the software intend to make use
of already done work and it can grow in a modular way, therefore saving time and money.

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

15

Congr
atulations, we are looking forward to using the new software.

Jaqueline

IPMet/Unesp
-
Brazil

5.1.11

Response from:
Michael Bell,
University of Hawai'i at Manoa

mmbell@hawaii.edu

Aloha EOL friends,

Thank you for sending me the white paper. I support the proposal, an
d concur that NCAR/EOL is the right
organization to lead the improvement of our community software tools. My primary comment is in regards to
NCAR's role as a facilitator for code sharing. The white paper suggests "informal" and "formal" sharing
processes,

but in my experience "informal" code sharing does not happen as often as we would like, and
many students would be reluctant to submit their code for formal peer review. However, I'm sure there are
many worthwhile scripts and modifications to NCAR softwar
e that are developed by students and
researchers that never leave their desktop.


I believe that if open source sharing were available that was not as "formal", but was managed by NCAR as
a facilitator (not a software maintainer) then the incentive to info
rmally share code would be greater. This
would improve transparency in our methodologies, and hopefully improve research efficiency as well. For
more complex algorithms that meet certain coding standards, NCAR could officially conduct a formal peer
review
or include it in the core suite.

I see two models for this already existing and working well. The first is the Matlab File Exchange: anyone
can upload a
file, and the community decides which tools are of value
through tagging and ratings. The
second is Git
Hub: open source projects can be easily uploaded for free, with community feedback and code
modification strongly encouraged. GitHub now hosts over 4 million repositories, many of which are scientific
software.

I don't think NCAR necessarily needs to host
the code
--

MFE and GitHub can do that
--

but if NCAR could
provide a forum for people to submit links and encourage radar and lidar tool sharing that would be great. I
would participate in such a forum and would encourage my students to do the same.

I loo
k forward to further discussion with you on this topic at the Radar Workshop.

Minor comments:

p. 13: Should be "lightning" not "lighting"

p. 15: Do you have a reference for the convective/stratiform partitioning scheme you plan to use?

p. 22: Is the intent

to modify soloii to handle CfRadial, or solo3?

Regards,

Michael

5.1.12

Response from:
David B. Wolff
,
NASA Wallops Flight Facility

david.b.wolff@nasa.gov

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

16

Hi Mike,

A colleague sent me a link to a google docs set up by NCAR that listed some current/pl
anned radar s
oftware
packages.
I am contacting you for two differen
t reasons regarding this list.
First, I am very interested in
learning more about the radx2grid and radarcartmerge programs you are working on, and I would like to add
some NASA software to that list th
at might be useful to other users.

For the past 24+ years, I have led the TRMM Ground Validation group at NASA GSFC and we have used
Sprint/Reorder f
or our operational processing.
I am well aware of the strengths/weaknesses of both
packages and would love
to get more information on your progress and would happily offer to serve as
beta
tester if you are willing.

As the NASA software I mentioned, there are two packages: RSL and RSL_in_IDL

NASA's Radar Software Library (RSL) is an open
-
source C
-
based library
for ingesting, analyzing and
outputting radar data in a number

of common radar data formats.
The library is based upon a hierarchical
structure of radars, volumes (fields), sweeps, rays and gauges (range). Input formats

include: UF, WSR
-
88D, SIGMET.
Output

format: UF.

RSL is available at:
http://trmm
-
fc.gsfc.nasa.gov/trmm_gv/software/rsl/index.html

RSL_in_IDL is an IDL based verso n of RSL that is written in Interactive Data Langua
ge (IDL). RSL_in_IDL
provides the same input and output formats as IDL but also provides a more scientist
-
friendly interface for
ingesting, analyzing and plotting radar data. RSL_in_IDL is open source, but requires an IDL license to
execute.

RSL_in_IDL is
available at:
http://trmm
-
fc.gsfc.nasa.gov/trmm_gv/software/rsl_in_idl/RSL_in_IDL.html

Thanks and I look forward to hearing from you.

Best regards,

David

5.1.13

Response from:

Courtney Schumacher
, Texas A&M

cschu@tamu.edu

Hi Mike,

Sorry not to respond sooner, but I hope you've gotten good feedback since I think this is a great idea. It
would be esp. useful to researchers like me that don't have large staffs, but have a fair num
ber of students
doing a lot of radar processing (i.e., it would benefit students greatly). In the bigger scheme of things, I've
suggested to DOE ARM that a code repository populated by PIs would be of great benefit to the ASR
community so that ARM itself c
ould focus on providing quality controlled, calibrated data in netcdf format
(which is surprisingly hard to come by on their data archive). I was also at a DOE/European center
workshop recently and the Europeans (from a number of institutes and universitie
s) thought a code
repository would be good for international collaborations and standardization. In that sense, EOL is far
ahead of the game and could be used as a template. It would be even better if this was cross agency so that
DOE wouldn't recreate the

wheel, but I'm not sure how that would work. Hopefully you can have those
discussions with some of the DOE folks that will be at the workshop. I won't be there in perso
n
, but I'll be
calling in from Texas!

Courtney

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

17

P.S
. You should add vertically
-
integrate
d ice (VII), which is good for lightning studies, and echo tops (unless
that is already in Titan?) to your Table 1

5.2

Profiler community

5.2.1

Outgoing email

On October 11 2012, the following email was sent to the email addresses listed:

Dear Wind Profiler and Lida
r community members,

A wide variety of software is used to examine radar, wind profiler, and lidar data for QC,
analysis and visualization. NCAR's Earth Observing Lab (EOL) is b
eginning an effort to improve
it
s own software tools, as well as invite the c
ommunity to contribute to a software tools library.
Some software is very specialized or proprietary to particular instruments, but there are many
procedures and tools that could be shared across the community. Sharing these tools would make
comparing an
d sharing of data much easier, reduce us all reinventing tools, and even spur
collaboration.

As discussed at the recent ISTP (International Symposium on Tropospheric Profiling) conference,
Europe has made good progress in linking together common tools in
the diverse European wind
profiler and lidar communities. For example, for wind profilers, work for the EUCOS E
-
WINPROF network has enabled the routine assimilation of profiler data into numerical models,
with subsequent improvements in forecasts. For l
idars, the EARLINET community has enabled
coordinated monitoring of events such as volcanic ash plumes. Other communities (e.g, notably
numerical modelers) have been able to significantly advance their work by similar sharing of
tools and procedures. We e
nvision that this proposal, along with other community efforts, will
assist the North American radar, profiler and lidar communities to make similar connections and
advances.

We have attached a draft white paper that outlines ideas to start this effort, a
n effort that we hope
will involve much of the community. We would very much appreciate hearing your comments on
the idea and on ways to make it stronger and ensure it is broadly relevant. This request for
comment is specifically aimed toward the profiler
and lidar communities, and our colleagues
have sent a similar request to the weather radar community. If possible we would appreciate
comments by the end of October.

Thank you and best regards,

Steve Cohn and Bill Brown

5.2.2

List of email addresses:

C. David

Whiteman
dave.whiteman@utah.edu

joe.young@utah.edu

johnson@atmos.colostate.edu

"Billings, Brian J."
bjbillings@stcloudstate.edu

Christopher Williams
christopher.williams@Colorado.EDU

james.m.wilczak@noaa.gov

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

18

laura.bianco@noaa.gov

Alan Brewer
alan.brewer@noaa.gov

michael.hardesty@noaa.gov

Phillip Chilson
chilson@ou.edu

rpalmer@ou.edu

(reply came from Boon Leng Cheong)

rlcoulter@anl.gov

ecampos@anl.gov

bbdemoz@howard.edu

frederic.fabry@mcgill.ca

whocking@uwo.ca

Scott McLaughlin
scott.mclaughlin@detect
-
in
c.com

kevin.knupp@nsstc.uah.edu

dirk.knugmann@metoffice.gov.uk

5.2.3

Response from:
Fréd
éric Fabry,
McGill University, Montreal, Canada

frederic.fabry@mcgill.
ca

(This primarily addresses scanning radar issues)

Hi Ste
ve and Bill,

We do not use NCAR
-
designed software here, primarily because we use radars very differently than the rest
of the community. So I guess I am the wrong person to

make comments. But let me

try.

On one end, yes, it would be very nice if there was a common base set of tools to view and do some basic
processing of radar data. On the other end, one doesn't do much research with those anymore, at least not
very good one. So yes as
an
educational

tool and to spot things that attract your attention. And _maybe_ it
can serve as a base to foster further development. I know I struggled putting refractivity on SPol because its
real
-
time environment did not allow
us
to easily access the needed data, and

a complex roundabout route
had to be set
-
up to read the data and to output products. Hopefully, the new set
-
up will allow th
e process to
be a lot cleaner.

Also:



You seem to be ignoring the two elephants in the room: NOAA, and ARM. I understand that the
e
ffort is driven by NCAR's renewal of its software suite, but there is considerable software and
know
-
how that have been developed for WSR
-
88D and ARM that could be
nefit the community as a
whole.



And I regret that spectral data and fixed pointing systems se
em to be mostly ignored in that
initiative, but understand that those instruments tend to be used by climate
-
related communities,
while NSF facilities with their short
-
term deployments serve more th
e weather side of the business.

All that being writ
ten, go
od luck in your efforts!

Cheers,

Frederic

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

19


5.2.4

Response from: Boon Leng Cheong,
ARRC,
U
niversity of

Oklahoma (Bob Palmer at
OU asked him to reply to us)

boonleng@ou.edu

Hi, Steve,

I just finished our PX
-
1000 radar (
http
://arrc.ou.edu/px1000
) this year and have been collecting some nice
datasets.
We are p
retty happy with the system.

I finished reading the

proposal and I really love it!

I gathered most of the effort is
to
ease users (professors,
researchers, scientists or

students) for their algorithm development and tests. I have been through a lot of
situations like this and the biggest overhead, as you put it in the proposal, is spending time developing the
interface to ingest the data. So, I really like the proposed ef
fort.

I would suggest developing a set of templates
as
one of the major goals, e.g, reading in raw time
-
series data
and generate a simple CfRadial Moments; or reading in a CfRadial moment and generate another CfRadial
Moment, in various languages. That way
, a developer has something that is ready to go. He/she needs only
modify the core signal processing component. The display would automatically be compatible since the
template is designed that way. I saw recommended data flow for various algorithms but pe
rhaps a set of
working

examples would even be better!

That's my take. Good luck!

5.2.5

Response from: Laura Bianco, NOAA/PSD

laura.bianco@noaa.gov

Dear Steve and Bill,

Thanks for sharing the document. It is very well written and I agree with you that we need to
share tools to
efficiently and easily manipulate and visualize wind profiler data. I have developed a lot of software myself,
that I will be willing to share.


My comments:


The acronym LAOF is introduced after the word LAOF is used.

In Table 1 you might w
ant to add a column specifying the instrument benefiting the algorithm.

I am more confident about easily sharing software to manipulate and visualize data; I am a little bit more
con
cerned about the possibility of
sharing software for QC
of
the data, as th
ey might be very site/condition
dependent. For this reason, it might be worth mentioning how you envision solving the fact that this QC
software is not often generally valid for all sites (i.e. some decontamination algorithm might need some
tuning from the

operator depending on th
e severity of the contamination.
)

What will you do to evaluate the submitted algorithms
?

(i.e. some decontamination algorithm
s

might have
been tested on
some geographical areas only
); will you rely on previously published works onl
y?

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

20

As you mentioned, some software is very specialized or proprietary to particular instruments. What will you
do in this case? Will you still allow the development of similar software in your library?

I think this is a very important step in our community

and I really hope it gets approved.

Good luck,

Laura.

5.2.6

Response from: Edwin Campos, Argonne National Lab

ecampos@mcs.anl.gov

Dear Steve and Bill,

I like the proposed idea of NCAR hosting a software sharing/repository for the community, and will be happy
to

collaborate with some of my IDL code for rea
ding raw LAP
-
XM profiler files.

In addition, I can share some of my profiler analysis IDL code, addressing those algorithms requiring more
effo
rt as per your Table 1 (p.14).

For example, I am now developing for

DOE
-
ARM a new Boundary
-
Layer
-
depth algorithm (~90% complete)
using their 91
5 MHz profilers.

Another example is my algorithm for profiler precipitation/clear
-
air signal identification (Campos et al. 2007b,
Radio Sci.), which uses VHF rad
ar and could be ad
apted to UHF.

In any case, I will try to attend the NSF Community Radar Workshop, next month at Boulder, where we can
chat further.

Regards,

Edwin.

5.2.7

Response from: Richard Johnson, Colorado State University

johnson@atmos.colostate.edu

Hi Steve and Bill,

I think your overall concept is a good one. Supporting NCAR to develop core software, making it available
to the user community, and allowing them to contribute improvements and specialized applicati
ons woul
d
benefit both parties.

Personally, I have only been involved with the wind profiler (ISS) as a PI. Improvements in the visualization
and analysis software, both for real
-
time use in campaigns and post
-
analysis, would be very helpful for
understanding wha
t we are seeing in the field as it is happening and afterwards. The data we have used in
these campaigns (e.g., the ISS data for TOGA COARE and NAME, see
http://johnson.atmos.colostate.edu/publications/refereed.html for publications using those data) has
been of
enormous value and the proposed enhance
ments will make it even better.

Dick

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

21

5.2.8

Response from: Wayne Hocking, University of Western Ontario

whocking@uwo.ca

Hi Steve, Bill

I have been through this before with the
skiymet meteor radar
-

the people I worked with assured me that it
would take none of my
time, so I handed them

some software. The questions were un
-
ending
-

'"what does
this do?", why do you do this? e
tc
-
it took up so much time...
despite assurances to t
he contrary. In addition,
all my software is designed for
experimentation, so does not work well with t
he idea of a robust
package...
there are zillions of bits I have left with quirky looking statements that are designed to remind me
to do extra stuff in t
he future... and my newer radars use quite different protocols to the norm,. including real
-
time deconvolutions
-

I am just one guy, and I cannot let myself go
through this nightmare again..
sorry, if I
had a fulltime programmer who could deal with it, jus
t maybe .. bu
t for now, please count me out.

thanks

Wayne

Follow
-
up from Wayne:

Hi Steve, Bill

Displays is one area I would not really be able to help w
ith
-
you can see my examples at:

http://www.yorku.ca/oqnet/

-

they are functional, and work for me, but they are no competitor for MatLab or
Python or IDL! But I use them because I wrote them all myself (right down to generating the individual
lettering) so they are free to me, and easy to edit
-

but I wrote them in

the 1980's', and have just stuck with
them
--
certainly not something I could imagine others being interested in!

As for "add
-
ons"
-

I am deviating further from the norm with my new radar design
-

I can finally build a
complete radar with my own hands and a

com
mer
c
i
al wave
-
form generator, digit
i
zer and transmitter
-

which
gives me huge control and really brings prices down
-

a complete profiler capable of reaching 14 km altitude,
plus with a boundary layer spaced antenna system, now costs under $500K fully i
nstalled
--

but it has
required me taking

some interesting diversions, including completely

new forms of pulse
-
coding ...
and I am
still working on optimizing these (Barker and Compl codes are being replaced, but I an still learning what
works).. so my work

probably does not mesh well with current processes..

G
ood luck with your project

B
est wishes

Wayne

5.2.9

Response from: Scott McLaughlin, DeTech Inc

scott.mclaughlin@detect
-
inc.com

Steve,

Here are some comm
ents:

Sodar is mentioned a few times, and there are a lot of sodar installs and manufacturers. Obviously much of
the display and processing software would apply to this as well.

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

22

Perhaps you are saying this already, but my recommendation is that any effort
be broken into its very
specific independent parts. For example, 'data formats' should be their own independent initiative. Also,
some sort of model for a database extensible to time series, spectra, moments, winds and other products
could be useful, espec
ially as a method to allow various algorithms (from various parties) to process data.
For example, if someone wrote time series to a database in the XXX format, anyone could perform the
follow
-
on processing using any language etc. Display software is also
its own thing (independent of how,
who, where it was collected). Each of these efforts would be on its own timeline etc. Developing and
maintaining comprehensive ICDs is some
thing that is very much needed.

We have deployed our software to over 20 sites (si
nce 2004). Some of the software has undergone various
certifications etc. and is under configuration management. It is unlikely that many operational users will want
any chan
ges to processes, displays etc.

You do not mention DeTect in the write
-
up. I know
we are relative new, but we are just down the road...

Many commercial vendors employ what amounts to proprietary techniques. I am not sure how this initiative
will deal with this. Maybe just through the Open Source model?

Overall it is unclear if you inten
d this to be purely for researchers or more widespread. How does this apply
to commercial products? It is almost guaranteed that at some point we will sell a radar to a user who wants
the data in formats potentially originating from this initiative. How wi
ll this work exactly?

Our software was written in LabVIEW.

This is not mentioned in here.
While LabVIEW is not the best
software for the later processing, it is very good for the hardware interface.

That our (DeTect) software (and other commercial entities
) already exits and is deployed with operational
users is not really addressed in this write
-
up. Again, can you elucidate what your ideas/plans are for
commercial entities and their customers?

The original email was a BCC addressed to the community at larg
e. Are your plans to create a governing or
advisory body, or to have period meetings? Will the BCC turn i
nto a CC so we know who is who?

Thanks and Regards,

Scott

Follow
-
up conversation from Scott McLaughlin, DeTech Inc:

Steve,

The commercial/proprietary s
ide is an issue, and that maybe can be addressed in a case
-
by
-
case basis. I
am not sure how
to share some of the algorithms
we have implemented without risking

a competitor using it
to their
advantage
---
and it would be odd for sure
to know you are bidding
against
your own work. Any ideas
there? For Windows
versus Open Source Linux, there
are huge numbers of users, which almost ma
ke it a
moot point, but for the
meteorological world, the numbers are so

small, it is a different deal.
Maybe some
proprietary ite
ms could b
e handled via license fees with
non
-
compete clauses or something.

I agree in regard to the data and res
earchers. Too much data is just
'falling on the floor' as it were. This is
somewhat related to the Network
of Networks, in trying to get a wide

arra
y of users to produce something
greater than the sum. There is significant oppo
rtunity there, but someone
has to show the value and gain
trust and le
ad the way. And some commercial
users will want some money for the minor c
hanges they might
need on th
eir
software (e.g., the addition of a high speed
data link for time series). You
guys are well
positioned for bringing m
any people together
---
but since
different people have different motivation
s, you will
have to develop the
Henry Kissinger within!

LROSE

Status Report

2012
-
12
-
05

NCAR/EOL

23

Take C
are,

Scott

5.2.10

Response from: Brian Billings, Saint Cloud State University

bjbillings@stcloudstate.edu

Sorry for the late reply. I'm still coming back from a bad cold which started late
-
Tuesday.

I read through

the white paper, and I don't really have much to add. In the software class, we were able to
use IDV to read in the MISS data either straightaway or with a minimal amount of NCO manipulation. I see
IDV is in the core software suite (along with xprof), so
everything looks okay to me.

Brian