Earth Observing Laboratory
Lidar Radar Open Software Environment
Steve Cohn, Bill Brown
Lidar Radar Open Software Environment
on the white paper
In order to solicit feedback from our user community, we assembled an email list of users likely
to have an opinion on the white paper. On September 18, the white paper was sent out to this list.
initial list was aimed mainly at the radar community.
better reach the community of profiler and lidar users, on October 11 we followed up with a
second email to a different set of users
The outgoing emails, and the responses received, are included
in their entirety at the en
d of this
document, in section 5
For the most part, the responses are very positive. Clearly there are many users in the community
who do not have the resources to develop their own soft
ware, and are very interested in an
improved set of
tools to help them with their rese
arch. This will
efficiency and reduce
time wasted through inadequate tools.
There also seems to be a consensus that NCAR is
the appropriate or
ganization to lead
Suggestions for alterations,
The following table lists points raised by some of the reviewers, and how we propose to address
Status in LROSE
Please include support in Radx for the P3
native format data
This will be done. We already hav
code in soloii to handle the P3 data.
be used as the basis for P3
support in the Radx package
Status in LROSE
Paul Hein, CSU
ode base must be maintained and
Include issue tracking
Ensure careful updating of meta
These are all excellent points.
We have already set up a Jira site for
We are aware of the need
ion and maintenance an
the project from the start.
Daniel Michaelson, SMHI
Ensure support for Opera HDF5 format.
Algorithms should be modular
i.e. able to
be used either in isolation or in conjunction
Adoption of common formats will lead to
improved international collaboration.
Support for Opera HDF5 is
specifically mentioned in the white
This will be added to the Radx
package in due course.
Daehyun Kim, Columbia
n is essential.
Chidong Xhang, Miami
Data assimilation into models is an important
end product of such a software system.
Quantitative statistical analysis (climatology)
would also be a valuable outcome.
These are good points and the system
designed to facilitate such
Michael Bell, Hawaii
in how software i
There is a requirement
for informal as well as
formal sharing. GitHub is seen as a good
model for open sharing
NCAR need not
necessarily host the code
Agreed. The white paper specifically
addresses informal sharing in section
A code repository populated by PIs would be
We can consider how best to include
code from various sour
on’t ignore NOAA and DOE/ARM
Don’t ignore spectral data and fixed pointing
We are already actively collaborating
with both NOAA and ARM on
LROSE (see section 3).
We will make sure the system can
support spectral (time series) data
Laura Bianco, NOAA/PSD
Take care with QC considerations. QC
algorithms vary widely depending on the type
This is an excellent point, and will
require careful consideration.
QC is one of the more tricky aspect
of t he s ys t em, and will vary
cons iderably bet ween ins t rument s.
We will probably rely on s pecialis t
us ers for t he development of QC
procedures for new ins t rument s.
Status in LROSE
Wayne Hocking, Ontario
Wayne does not have the flexibility to be able
to contribute to the
collaboration effort, and is
wary of getting involved.
This points out that we need to be
careful how we communicate the
aims of the project. It is not
necessary that everyone contribute,
only that the development is
somewhat shared with the
munity. We do not want to
discourage use of the software just
because the user does have the
resources to contribute.
welcome to use
Scott McLaughlin, DeTech
Scott raised the issue of open source vs.
proprietary code. He asks whether we
plans on how to include support for
These are good points. Our principal
user base is the resear
model, freely shared on the web.
Commercial entities will be welcome
to use the c
ode and to contribute if
they wish. Many commercial entities
support the open source model as
good for their business. We will not,
however, include commercially
protected code in the system.
NSF Community Workshop on Radar Technologies
This NSF workshop
was held at the NCAR Center Green campus from
One of the sessions covered ‘Radar data and software analysis tools’. This was facilitated by
Sandra Yuter of North Carolina State University and Wen
Chau Lee of EOL/RSF. The focus of
ssion was not LROSE per se, but rather a broad look at software needs in the community.
Sandra broke the participants into small groups of 6 to 8 people, each of which was asked to list
priority software needs. These lists
re posted on the
then asked to ‘vote’ (with colored stickers) for the items they deemed most important, from
both the personal research and community point of view.
Sandra will prepare a detailed report on this session and it is not appropr
iate to preempt that in
this document. However, it was clear from the votes cast that open source community software of
the type proposed by LROSE was a popular notion with the workshop participants. When asked
for a show of support, the workshop participa
nts appeared to be very positive about the LROSE
A poster on LROSE was prepared for the workshop. The poster may be found on
by Scott Collis from DOE/ARM
Scott Collis came out to Boulder 2 days before the NSF workshop, and
the workshop with EOL personnel. The main topic of discussion was the need for close
collaboration with EOL on the development of the
suite of community software tools.
Scott and his group have developed an extensive radar a
nalysis support library in Python.
will be an excellent starting point for some of the functionality required by LROSE. Scott
demonstrated some of the functionality to EOL engineers, and we have already taken steps to
include use of this package in a
Python version of solo.
Scott’s group has hired a soft
ware engineer to take over the P
ython work, to make it more formal
from a development point of view,
the aim being
er documentation and maintainability
It is clear that collaboration with DOE/ARM o
n LROSE will be essential to avoid duplication of
effort and make the most efficient use of the resources at our disposal.
Progress since September
Nancy Rehak has made significant progress on solo3
bit replacement for soloii.
The GUI compo
nents have been upgraded to the latest C++ versions of the GTK toolkit. This was
ly work because some of the
older GTK components are no longer supported.
The I/O layer now uses the Radx library for accessing the data files. This is a significant step
towards full support for CfRadial files natively in solo3.
A number of bug fixes were made and the latest beta
release version of solo3 seems to be quite
Solo3 can now be built and run on Mac OSX platforms.
It compiles natively on 64
Greg Meymaris (RAL) has made significant progress on Emerald (
eMERALD = Matlab
Environment for RAdar and LiDar
This is a Matlab
that will have similar functionality to
solo, but which will be much more extensible because it is in Matlab
and therefore the user can
modify it to
take advantage of
Emerald has an data access layer, that allows reading and writing of CfRadial files. This makes
use of the Matlab support for handling NetCDF
On top of this is bui
radar data viewer
that supports PPI
type plots. This will be extended to
support RHIs and BSCAN type plots in the future.
One big advantage of Emerald over solo is
it will run on any platform supported
i.e. Windows, Mac O
SX and LINUX.
student from CU, has been working on a Python version of solo.
will also have the advantage of cross
platform portability. And additional advantage is that
Python is a free package, so that students who
do not have access to a Matlab license will be able
to use pysolo instead.
Joe VanAndel has kindly agreed to mentor Xin in her work on pysolo. Joe has worked in Python
for many years, and is therefor
e in a good position to make a positive contribution to t
Scott Collis contributed significantly to the pysolo effort during his one
. He was able to
s to a Python package that makes the rendering of polar d
ata much more efficient.
Good progress ha
s also been made in the transformation of data from polar to Cartesian
coordinates. The Radx2Grid application now replaces most of the functionality of SPRINT. Jay
Miller has been testing this application, and comparing the results to those produced by SPR
and CEDRIC. It appears that Radx2Gr
id does a good job of closely
reproducing the SPRINT
The next step will be to incorporate the REORDER functionality into Radx2Grid. A start has
been made on this, but more work is required.
LROSE white paper
On September 18 the white paper was circulated to a list of people considered likely to be
rested in the proposed ideas. The email sent out was as follows:
To: the radar/lidar research software
NCAR/EOL has submitted a white paper to the National Science Foundation (NSF) proposing an
extensive upgrade to the suite of NCAR
supported software tools available to the radar/lidar user
community. A new software framework is proposed th
at would enable the community to jointly
develop and upgrade the software suite in the future. NSF has asked us to solicit comments on the
proposal from the user community.
The white paper is attached, and a copy of the executive summary is also included
at the end of
If you are interested in using NCAR
developed radar/lidar software in the future, we ask that you
review the proposal, and provide us with comments and feedback.
NSF is organizing a radar workshop, to be held at NCAR in Bould
er, from 27
29 November this
year. We would like to present a summary of your comments at that workshop.
Therefore, if possible, please respond with your comments within 4 weeks by mid
paper was sent to the following
firstname.lastname@example.org Alan Seed BOM Australia
email@example.com Andrew VanDerMerwe SA Weather Service
firstname.lastname@example.org Stacy Brodzik U. Wash.
email@example.com Chandra CSU/CHILL
firstname.lastname@example.org Chris Fairall NOAA
email@example.com Chris Weiss
chuck.Long@pnnl.gov Chuck Long PNNL
conrad.Ziegler@noaa.gov Corad Ziegler NOAA
firstname.lastname@example.org Courtney Schumacher TAMU
email@example.com Chidong Zhang U. Miami
firstname.lastname@example.org Daehyun Ki
email@example.com BaltRad Radar Network Sweden
firstname.lastname@example.org Dave Jorgensen NOAA NSSL
email@example.com Deanna Hence NASA GSFC
firstname.lastname@example.org David Parsons OU
email@example.com Beth Ebert BOM Australia
firstname.lastname@example.org Ed Zipser
email@example.com Frank Marks NOAA
firstname.lastname@example.org Han Lei Ocean University China
email@example.com Howie Bluestein OU
firstname.lastname@example.org Hannah Barnes U. Wash.
email@example.com Gerry Heymsfield NASA
firstname.lastname@example.org Bob Hou
ze U. Wash.
email@example.com Justin Peter BOM Australia
firstname.lastname@example.org Jennifer Davison U. Ill.
email@example.com Jim George CSU/CHILL
firstname.lastname@example.org Jeff Trapp Purdue
email@example.com Josh Wurman CSWR
rg Karen Kosiba CSWR
karel.deWaal@weathersa.co.za Karel DeWaal SA Weather Service
firstname.lastname@example.org Masaki Katsumata JAMSTEC
email@example.com Kevin Knupp UAH
firstname.lastname@example.org Elizabeth Thompson CSU
email@example.com Min Deng U. Wyoming
.edu Michael Bell, University of Hawaii
firstname.lastname@example.org Jaqueline Murakami IPMET Brazil
email@example.com Manuel Zuluaga U. Wash.
firstname.lastname@example.org Nick Guy NOAA NSSL
nolan.Atkins@lsc.vsc.edu Nolan Atkins
u.ac.jp Mariko Oue Nago
ya University Japan
email@example.com Phil Purdam BOM Australia
firstname.lastname@example.org Robert Pasken
email@example.com Pat Kennedy CSU/CHILL
paul.Smith@sdsmt.edu Paul Smith
firstname.lastname@example.org Paul Joe Environment Canada
email@example.com Rod Potts BOM Au
firstname.lastname@example.org Bob Rauber
email@example.com Alfred Rodi U. Wyoming
firstname.lastname@example.org Wits University South Africa
email@example.com Steve Rutledge CSU
firstname.lastname@example.org Sally McFarlane PNNL
email@example.com Samson Hag
firstname.lastname@example.org Shuyi Chen U. Miami
email@example.com Scott Collis ARM ANL
firstname.lastname@example.org Simon DeSzoeke OSU
she.Feng@pnnl.gov Zhe Feng PNNL
email@example.com Scott Powell U. Wash.
firstname.lastname@example.org Sandy Yuter NCSU
tmos.colostate.edu Tim Lang CSU
email@example.com Walt Petersen NASA
firstname.lastname@example.org Yvette Richardson PSU
David Wolff NASA
Response from: Jeff Snyder,
niversity of Oklahoma
I read through the radar software request for comment, and, unofficially speaking on behalf of our radar
group (“our” being Howie Bluestein’s radar group), I think the project will ma
ke our research m
Below is our standard workflow when w
e work with data collected by
the mobile radars from UMass:
Archive in UF
Convert UF to DORADE SWP format for diplay and data editing (velocity dealiasing, clutter removal,
etc.) using Solo II
to netCDF (using xltrsii) for modification in Matlab (attenuation correction, KDP
Convert back to UF, then convert back to SWP for display
Starting in 2011, we began to use CfRadial as the “archive” format for data fro
m our new mobile ra
The primary advantage of the NetCDF
based CfRadial convention is that such data can be used
very readily with our existing Matlab programs/algorithms (it helps too that it’s volume based, which
data management easier).
We still rely upo
n Solo II for display, radial velocity dealiasing, and gate
ray editing, which means I still end up having to convert data back and forth between CfRadial and
SWP. Your RadXConvert utility significantly reduces the “hassle factor”, but it
still would be easier if we
could display CfRa
dial data natively in Solo II.
From the request for comment document, it looks like there *
* a new version o
f Solo that supports
Are you looking for beta testers for this softw
If so (or even
if not), I’d love to test it out.
the data we collected this past year is currently in both CfRadial and SWP formats since we don’t have
access to a nice disp
lay program for CfRadial data.
If we could display CfRadial data in Solo, it would make
life a lot easier in terms of file forma
t and file version management.
I look forward to any further developments, and I really think that standardizing radar archive formats to
facilitate data sharing and algorithm development is greatly needed. Thanks!
University of Oklahoma Meteorology PhD Student
Nick Guy, NOAA
The document looks good. There is no mention of supporting the P3 rad
ar format moving forward. We
spoke a little about this in Seattle. The solo translator supports
the format (I believe Dick Oye
put this in for
Dave Jorgensen as a favor), but Radx does not as of yet. You mentioned that there was some mystery
was left out of Radx at first that may be associated with the distinct P3 format.
I received the Sigmet manual that discusses the data format and wanted to pass along to your group as a
reference. Is there any way this could also be included in Radx in
Also, a while back we corresponded about getting Radx to work on MacOSX. At that time you were going to
try an install on a Mac machine. Have you had time to follow this up?
Paul Hein, Colorado State Universi
I am emailing you directly with some comments from the CSU Radar Meteorology Group (headed by Steve
Rutledge) and from CSU
CHILL on the comm
unity software tools proposal.
are from various
I have collected and combined the comments in the points below.
1) We are very encouraged to see NCAR taking the lead on developing new ve
rsions of the 'core' software.
New software is needed.
With the new radar
format CfRadial, we need software that can use it.
2) We do expect NCAR to take a lead role in developing and
maintaining the core software.
Our concern is
that the software will continue to be maintained and developed over the long term.
3) That is wh
y we feel strongly that it is very important that the larger community is involve
d in the software
It should be easy for members of the community to take part in the ongoin
g development of
Being transparent in the development and
the core software would help.
Multiple points of view can help improve the software and keep it up to date.
4) There should be an obvious and defined way that bug reports are submitted and
tracked for the core
(It would be good
if community applications fo
llowed the same standard too).
Bugzilla would be good.
Setting up a website with forums and mailing lists would help encourage the community to take
testing and debugging the core programs.
community should be a part of the development of the
core software so that the burden is not t
otally on the NCAR developers.
Also forums can be a place where
questions about the software could be answered by the community.
led documentation is c
Online documentation should contain many simple
software usage examples along with th
e resultant output plots, etc.
The original hardcopy manuals for
NCAR Graphics were good examples of this.
7) Documentation is also important fo
r community applications
as well as the core software.
with high level language stubs should be documented, so that we can develop our own software in the
languages of our choosing (python, idl, matlab, ncl, etc.).
8) Also it should be easy
to update the meta data at all stages of the data manipula
calibration adjustments, thresholding applied, identification of unfolding algorithms used, etc. should ride
along with the da
ta sets as they are processed.
It would help keep t
rack of what is done so that one's results
could be reproducible.
To summarize, we are in favor of the proposal and we hope the above comment
s can help you with your
Thank you for your work on this.
Daniel Michaelson, SMHI, Sw
Hej Mike et al.,
Thanks for including me on your list. As project manager for BALTRAD, I have some BALTRAD
feedback in the following.
In Europe, BALTRAD has taken the
lead in the community
based development of a software platform for
exchanging and processing weather radar data. This got underway in 2009 with funding from the EU's Baltic
Sea Region Programme supporting the national meteorological services in the region
. This external funding
runs out at the end of 2013. Our scope is more limited than that in your proposal; we have targeted data
from scanning weather radars running at X, C, and S bands. We have also limited our scope in terms of
which data are relevant t
o us; in our case, we have targeted polar (spherical coordinates) data that are
already collected into complete scans (sweeps) at the radar site as our starting point. This means we have
not made accommodation for time series (I/Q) data.
Our software is
licensed according to the LGPL, where the developers retain the IPR to their contributions. I
can highly recommend this approach, or, if too many developers/organizations make this unwieldy, the so
called Fiduciary Licence Agreement with NCAR(?) as the sin
Providing analysis tools and interfaces for free high
level languages like Python, as you have proposed, is
also the way to go.
In Europe, the modern standard file format is ODIM_H5 (HDF5), agreed
upon by 30 countries in
EUMETNET OPERA. I'm
pleased to see that you have included the ability to convert to/from such data.
Doing so is a prerequisite for enabling interaction and interfacing between systems.
I would like to encourage increased interaction between North American and European comm
unities when it
comes to software development of this kind. Differences in scope and technical implementation will
inevitably be different, but the global community stands to benefit if open development initiatives like
BALTRAD and your Community software
can develop data processing algorithms in a way that facilitates
their integration and access. With this in mind, I would like to see that some of the algorithms found in Table
1 are made available relatively independently from the rest of the core suite s
uch that they can become
more easily accessible for integration with our "toolbox" product
generation framework. Similarly, we hope
that you might find some of our processing algorithms appealing too, such that you might want to integrate
them into your co
re or community suite.
The implication for increased interaction/cooperation of this kind is that the end users, e.g. NWP, stand to
benefit from increasingly harmonized datasets as data become increasingly available globally.
A small note on the defini
tion of HDF5 in the glossary: HDF5 comes from the HDF Group which is a non
profit organization based in Champaign, Illinois.
Doherty Earth Observatory
I've went through the paper, it is well
written. I like the basic idea and fully support it as a radar
beginner. One thing that might worth considerin
g to add to the paper is documentation. I think on/offlin
documentation of the software
is extremely important, and could aff
ect how widely the software is
type online documentation can be used for the community ap
plication parts of the s
believe you're aware of this, and it is implicitly included in your plan, but it would be great if you could
explicitly include that point in the abst
ract and the section 1.6 Goals.
Thank you for your ef
forts on this, I appreciate it.
Chidong Xhang, University of Miami
Hi Mike and Wen
I glanced through the white paper, and here are comments
from a radar
It appears that the p
roposed software tools are mainly for users from the radar community. Which
But the ability of the proposed software to transform data to common Cartesian format will draw many users
outside the traditional radar community, such as modelers and da
This can be a good
selling point, but it does not come across obviously in the white
an impression that display is the main end product of the proposed software. I'm sure this is
one of the most frequently ways of
using radar data. But quantitative statistics
can be another product.
(Some of them are listed in Table 1?) Maybe this can be highlighted by an additional box in the figures that
precedes the display box in the arrow flow. I guess these user defined statis
tics can be what you prefer to a
community developed modules.
These are what I have now. Thank you for trying to make
radar data more user friendly.
My students can definitely benefit from this. Wish you a p
roductive workshop in November.
Rod Potts, Bureau of Meteorology, Australia
Thank you for your invitation to
comment on the Lrose proposal.
We recently had a meeting to discuss the
proposal and the following comments come from this.
s are foc
ussed on radar data processing.
The Bureau values the good working relations with NCAR across a range of activities and will continue to
support these. This includes activities in radar meteorology.
In this regard the o
bjectives presented in the
proposal are supported.
NCAR applications currently used in the Bureau include
research and operations
Applications have also been written here to convert 3D
Rapic data to
UF and the Lassen format to UF.
e Bureau has a network on 75 operational weather radars and 2 research radars across the country with
a combination of C
band and S
band radars. In the last few years there has been a process underway to
significantly upgrade a number of radars in the netw
Raw radar data from the Bureau's radar network is transmitted in 3d
rapic format and it is likely th
continue for some time.
The Bureau operates Nowcast Application Servers that process radar data from more than 20 radars to
generate nowcast pr
oducts for downstream applications. The Nowcast Application Servers run TITAN and
WDSS (providing TS parameters) and rainfields (QPE and precipitation nowcasting application).
The rainfields application generates output
products in CF
Bureau is currently working on the Strategic Radar Enhancement Project (SREP) with the aim of
assimilating radar data into high resolution rapid update NWP.
For SREP a data format based on the Opera Data Information Model (ODIM) (HDF5) has been adopted and
it is proposed that the ODIM HDF5 format will be used
processed radar data.
As previously discussed Daniel Michelson from SMHI will be visiting the Bureau in early December and joint
discussions on data formats and other issues are proposed when
you visit the Bureau.
This will include potential areas for more collabor
ation in radar data processing.
Jaqueline Murakami, IPMET,
University of Sao Paulo,
The new software proposal is ve
ry good, I've sent
to our p
eople and the feedback was very
the main goal is building a suite of software tools to facilitate the manipulation and visualization of radar and
wind profiler data for use in operation, research and education.
IPMet has been using radar software since the 80's decade both for research purposes as for real time
operation. We have already worked with Sprint, Cedric, PPI
mmm, Soloii for radar data processing and we
have used the display software Zebra, Gempak and I
DV. Since 2005 we are mainly using TITAN and
Rview/CIDD displays for weather nowcasting and post
analysis studies, which we consider a very complete
and useful radar software
. We have Sodar and Lidar sensors running proprietary software mostly for
al time visualization. Next year, with the creation of the B
in Meteorology course in our university, we
will be giving support to students and teachers of this course too. Thus, we have interest in the software for
Forecasters and re
searchers at IPMet are used to work with the software R, Grads, MatLab/SciLab, NCL
and NetCDF is the most used data format. The idea of having NetCDF as the software main format and the
facility to develop radar/lidar algorithms i
n these high level languag
es led them to be very
excited. All the worry on supporting open and standardized data formats are very ap
The three types of data structure for time series instruments (IQ time series, Radial and Cartesian) with their
of algorithms are an organized way of working with radar/lidar data that would help increasing
the development of algorithms for each data representation format.
We've appreciated the list of the core suite algorithms presented in the document, it include
s many TITAN
applications and others of our interest. The core services cover from data translators to data integration and
visualization, which are
Here, forecasters and researchers usually need the support of the IT group to have
a available for
radar software because it usually requires converting data in a specific format, writing scripts or
running applications in linux environment, being aware of files and data directories. There is a great
expectation of having a user
riendly interface for setting up the environment and running applications to
enhance data accessibility.
We believe the proposed approach for the development of the software, which provides a core suite of
algorithms and encourages community collaboration,
is a very good solution to build a high quality software
which will be constantly improved and updated. It is good to know the software intend to make use
of already done work and it can grow in a modular way, therefore saving time and money.
atulations, we are looking forward to using the new software.
University of Hawai'i at Manoa
Aloha EOL friends,
Thank you for sending me the white paper. I support the proposal, an
d concur that NCAR/EOL is the right
organization to lead the improvement of our community software tools. My primary comment is in regards to
NCAR's role as a facilitator for code sharing. The white paper suggests "informal" and "formal" sharing
but in my experience "informal" code sharing does not happen as often as we would like, and
many students would be reluctant to submit their code for formal peer review. However, I'm sure there are
many worthwhile scripts and modifications to NCAR softwar
e that are developed by students and
researchers that never leave their desktop.
I believe that if open source sharing were available that was not as "formal", but was managed by NCAR as
a facilitator (not a software maintainer) then the incentive to info
rmally share code would be greater. This
would improve transparency in our methodologies, and hopefully improve research efficiency as well. For
more complex algorithms that meet certain coding standards, NCAR could officially conduct a formal peer
or include it in the core suite.
I see two models for this already existing and working well. The first is the Matlab File Exchange: anyone
can upload a
file, and the community decides which tools are of value
through tagging and ratings. The
second is Git
Hub: open source projects can be easily uploaded for free, with community feedback and code
modification strongly encouraged. GitHub now hosts over 4 million repositories, many of which are scientific
I don't think NCAR necessarily needs to host
MFE and GitHub can do that
but if NCAR could
provide a forum for people to submit links and encourage radar and lidar tool sharing that would be great. I
would participate in such a forum and would encourage my students to do the same.
k forward to further discussion with you on this topic at the Radar Workshop.
p. 13: Should be "lightning" not "lighting"
p. 15: Do you have a reference for the convective/stratiform partitioning scheme you plan to use?
p. 22: Is the intent
to modify soloii to handle CfRadial, or solo3?
David B. Wolff
NASA Wallops Flight Facility
A colleague sent me a link to a google docs set up by NCAR that listed some current/pl
anned radar s
I am contacting you for two differen
t reasons regarding this list.
First, I am very interested in
learning more about the radx2grid and radarcartmerge programs you are working on, and I would like to add
some NASA software to that list th
at might be useful to other users.
For the past 24+ years, I have led the TRMM Ground Validation group at NASA GSFC and we have used
or our operational processing.
I am well aware of the strengths/weaknesses of both
packages and would love
to get more information on your progress and would happily offer to serve as
tester if you are willing.
As the NASA software I mentioned, there are two packages: RSL and RSL_in_IDL
NASA's Radar Software Library (RSL) is an open
for ingesting, analyzing and
outputting radar data in a number
of common radar data formats.
The library is based upon a hierarchical
structure of radars, volumes (fields), sweeps, rays and gauges (range). Input formats
include: UF, WSR
RSL is available at:
RSL_in_IDL is an IDL based verso n of RSL that is written in Interactive Data Langua
ge (IDL). RSL_in_IDL
provides the same input and output formats as IDL but also provides a more scientist
friendly interface for
ingesting, analyzing and plotting radar data. RSL_in_IDL is open source, but requires an IDL license to
Thanks and I look forward to hearing from you.
, Texas A&M
Sorry not to respond sooner, but I hope you've gotten good feedback since I think this is a great idea. It
would be esp. useful to researchers like me that don't have large staffs, but have a fair num
ber of students
doing a lot of radar processing (i.e., it would benefit students greatly). In the bigger scheme of things, I've
suggested to DOE ARM that a code repository populated by PIs would be of great benefit to the ASR
community so that ARM itself c
ould focus on providing quality controlled, calibrated data in netcdf format
(which is surprisingly hard to come by on their data archive). I was also at a DOE/European center
workshop recently and the Europeans (from a number of institutes and universitie
s) thought a code
repository would be good for international collaborations and standardization. In that sense, EOL is far
ahead of the game and could be used as a template. It would be even better if this was cross agency so that
DOE wouldn't recreate the
wheel, but I'm not sure how that would work. Hopefully you can have those
discussions with some of the DOE folks that will be at the workshop. I won't be there in perso
, but I'll be
calling in from Texas!
. You should add vertically
d ice (VII), which is good for lightning studies, and echo tops (unless
that is already in Titan?) to your Table 1
On October 11 2012, the following email was sent to the email addresses listed:
Dear Wind Profiler and Lida
r community members,
A wide variety of software is used to examine radar, wind profiler, and lidar data for QC,
analysis and visualization. NCAR's Earth Observing Lab (EOL) is b
eginning an effort to improve
s own software tools, as well as invite the c
ommunity to contribute to a software tools library.
Some software is very specialized or proprietary to particular instruments, but there are many
procedures and tools that could be shared across the community. Sharing these tools would make
d sharing of data much easier, reduce us all reinventing tools, and even spur
As discussed at the recent ISTP (International Symposium on Tropospheric Profiling) conference,
Europe has made good progress in linking together common tools in
the diverse European wind
profiler and lidar communities. For example, for wind profilers, work for the EUCOS E
WINPROF network has enabled the routine assimilation of profiler data into numerical models,
with subsequent improvements in forecasts. For l
idars, the EARLINET community has enabled
coordinated monitoring of events such as volcanic ash plumes. Other communities (e.g, notably
numerical modelers) have been able to significantly advance their work by similar sharing of
tools and procedures. We e
nvision that this proposal, along with other community efforts, will
assist the North American radar, profiler and lidar communities to make similar connections and
We have attached a draft white paper that outlines ideas to start this effort, a
n effort that we hope
will involve much of the community. We would very much appreciate hearing your comments on
the idea and on ways to make it stronger and ensure it is broadly relevant. This request for
comment is specifically aimed toward the profiler
and lidar communities, and our colleagues
have sent a similar request to the weather radar community. If possible we would appreciate
comments by the end of October.
Thank you and best regards,
Steve Cohn and Bill Brown
List of email addresses:
"Billings, Brian J."
(reply came from Boon Leng Cheong)
McGill University, Montreal, Canada
(This primarily addresses scanning radar issues)
ve and Bill,
We do not use NCAR
designed software here, primarily because we use radars very differently than the rest
of the community. So I guess I am the wrong person to
make comments. But let me
On one end, yes, it would be very nice if there was a common base set of tools to view and do some basic
processing of radar data. On the other end, one doesn't do much research with those anymore, at least not
very good one. So yes as
tool and to spot things that attract your attention. And _maybe_ it
can serve as a base to foster further development. I know I struggled putting refractivity on SPol because its
time environment did not allow
to easily access the needed data, and
a complex roundabout route
had to be set
up to read the data and to output products. Hopefully, the new set
up will allow th
e process to
be a lot cleaner.
You seem to be ignoring the two elephants in the room: NOAA, and ARM. I understand that the
ffort is driven by NCAR's renewal of its software suite, but there is considerable software and
how that have been developed for WSR
88D and ARM that could be
nefit the community as a
And I regret that spectral data and fixed pointing systems se
em to be mostly ignored in that
initiative, but understand that those instruments tend to be used by climate
while NSF facilities with their short
term deployments serve more th
e weather side of the business.
All that being writ
od luck in your efforts!
Response from: Boon Leng Cheong,
Oklahoma (Bob Palmer at
OU asked him to reply to us)
I just finished our PX
1000 radar (
) this year and have been collecting some nice
We are p
retty happy with the system.
I finished reading the
proposal and I really love it!
I gathered most of the effort is
ease users (professors,
researchers, scientists or
students) for their algorithm development and tests. I have been through a lot of
situations like this and the biggest overhead, as you put it in the proposal, is spending time developing the
interface to ingest the data. So, I really like the proposed ef
I would suggest developing a set of templates
one of the major goals, e.g, reading in raw time
and generate a simple CfRadial Moments; or reading in a CfRadial moment and generate another CfRadial
Moment, in various languages. That way
, a developer has something that is ready to go. He/she needs only
modify the core signal processing component. The display would automatically be compatible since the
template is designed that way. I saw recommended data flow for various algorithms but pe
rhaps a set of
examples would even be better!
That's my take. Good luck!
Response from: Laura Bianco, NOAA/PSD
Dear Steve and Bill,
Thanks for sharing the document. It is very well written and I agree with you that we need to
share tools to
efficiently and easily manipulate and visualize wind profiler data. I have developed a lot of software myself,
that I will be willing to share.
The acronym LAOF is introduced after the word LAOF is used.
In Table 1 you might w
ant to add a column specifying the instrument benefiting the algorithm.
I am more confident about easily sharing software to manipulate and visualize data; I am a little bit more
cerned about the possibility of
sharing software for QC
the data, as th
ey might be very site/condition
dependent. For this reason, it might be worth mentioning how you envision solving the fact that this QC
software is not often generally valid for all sites (i.e. some decontamination algorithm might need some
tuning from the
operator depending on th
e severity of the contamination.
What will you do to evaluate the submitted algorithms
(i.e. some decontamination algorithm
been tested on
some geographical areas only
); will you rely on previously published works onl
As you mentioned, some software is very specialized or proprietary to particular instruments. What will you
do in this case? Will you still allow the development of similar software in your library?
I think this is a very important step in our community
and I really hope it gets approved.
Response from: Edwin Campos, Argonne National Lab
Dear Steve and Bill,
I like the proposed idea of NCAR hosting a software sharing/repository for the community, and will be happy
collaborate with some of my IDL code for rea
ding raw LAP
XM profiler files.
In addition, I can share some of my profiler analysis IDL code, addressing those algorithms requiring more
rt as per your Table 1 (p.14).
For example, I am now developing for
ARM a new Boundary
depth algorithm (~90% complete)
using their 91
5 MHz profilers.
Another example is my algorithm for profiler precipitation/clear
air signal identification (Campos et al. 2007b,
Radio Sci.), which uses VHF rad
ar and could be ad
apted to UHF.
In any case, I will try to attend the NSF Community Radar Workshop, next month at Boulder, where we can
Response from: Richard Johnson, Colorado State University
Hi Steve and Bill,
I think your overall concept is a good one. Supporting NCAR to develop core software, making it available
to the user community, and allowing them to contribute improvements and specialized applicati
benefit both parties.
Personally, I have only been involved with the wind profiler (ISS) as a PI. Improvements in the visualization
and analysis software, both for real
time use in campaigns and post
analysis, would be very helpful for
t we are seeing in the field as it is happening and afterwards. The data we have used in
these campaigns (e.g., the ISS data for TOGA COARE and NAME, see
http://johnson.atmos.colostate.edu/publications/refereed.html for publications using those data) has
enormous value and the proposed enhance
ments will make it even better.
Response from: Wayne Hocking, University of Western Ontario
Hi Steve, Bill
I have been through this before with the
skiymet meteor radar
the people I worked with assured me that it
would take none of my
time, so I handed them
some software. The questions were un
this do?", why do you do this? e
it took up so much time...
despite assurances to t
he contrary. In addition,
all my software is designed for
experimentation, so does not work well with t
he idea of a robust
there are zillions of bits I have left with quirky looking statements that are designed to remind me
to do extra stuff in t
he future... and my newer radars use quite different protocols to the norm,. including real
I am just one guy, and I cannot let myself go
through this nightmare again..
sorry, if I
had a fulltime programmer who could deal with it, jus
t maybe .. bu
t for now, please count me out.
up from Wayne:
Hi Steve, Bill
Displays is one area I would not really be able to help w
you can see my examples at:
they are functional, and work for me, but they are no competitor for MatLab or
Python or IDL! But I use them because I wrote them all myself (right down to generating the individual
lettering) so they are free to me, and easy to edit
but I wrote them in
the 1980's', and have just stuck with
certainly not something I could imagine others being interested in!
As for "add
I am deviating further from the norm with my new radar design
I can finally build a
complete radar with my own hands and a
form generator, digit
zer and transmitter
gives me huge control and really brings prices down
a complete profiler capable of reaching 14 km altitude,
plus with a boundary layer spaced antenna system, now costs under $500K fully i
but it has
required me taking
some interesting diversions, including completely
new forms of pulse
and I am
still working on optimizing these (Barker and Compl codes are being replaced, but I an still learning what
works).. so my work
probably does not mesh well with current processes..
ood luck with your project
Response from: Scott McLaughlin, DeTech Inc
Here are some comm
Sodar is mentioned a few times, and there are a lot of sodar installs and manufacturers. Obviously much of
the display and processing software would apply to this as well.
Perhaps you are saying this already, but my recommendation is that any effort
be broken into its very
specific independent parts. For example, 'data formats' should be their own independent initiative. Also,
some sort of model for a database extensible to time series, spectra, moments, winds and other products
could be useful, espec
ially as a method to allow various algorithms (from various parties) to process data.
For example, if someone wrote time series to a database in the XXX format, anyone could perform the
on processing using any language etc. Display software is also
its own thing (independent of how,
who, where it was collected). Each of these efforts would be on its own timeline etc. Developing and
maintaining comprehensive ICDs is some
thing that is very much needed.
We have deployed our software to over 20 sites (si
nce 2004). Some of the software has undergone various
certifications etc. and is under configuration management. It is unlikely that many operational users will want
ges to processes, displays etc.
You do not mention DeTect in the write
up. I know
we are relative new, but we are just down the road...
Many commercial vendors employ what amounts to proprietary techniques. I am not sure how this initiative
will deal with this. Maybe just through the Open Source model?
Overall it is unclear if you inten
d this to be purely for researchers or more widespread. How does this apply
to commercial products? It is almost guaranteed that at some point we will sell a radar to a user who wants
the data in formats potentially originating from this initiative. How wi
ll this work exactly?
Our software was written in LabVIEW.
This is not mentioned in here.
While LabVIEW is not the best
software for the later processing, it is very good for the hardware interface.
That our (DeTect) software (and other commercial entities
) already exits and is deployed with operational
users is not really addressed in this write
up. Again, can you elucidate what your ideas/plans are for
commercial entities and their customers?
The original email was a BCC addressed to the community at larg
e. Are your plans to create a governing or
advisory body, or to have period meetings? Will the BCC turn i
nto a CC so we know who is who?
Thanks and Regards,
up conversation from Scott McLaughlin, DeTech Inc:
The commercial/proprietary s
ide is an issue, and that maybe can be addressed in a case
case basis. I
am not sure how
to share some of the algorithms
we have implemented without risking
a competitor using it
and it would be odd for sure
to know you are bidding
your own work. Any ideas
there? For Windows
versus Open Source Linux, there
are huge numbers of users, which almost ma
ke it a
moot point, but for the
meteorological world, the numbers are so
small, it is a different deal.
ms could b
e handled via license fees with
compete clauses or something.
I agree in regard to the data and res
earchers. Too much data is just
'falling on the floor' as it were. This is
somewhat related to the Network
of Networks, in trying to get a wide
y of users to produce something
greater than the sum. There is significant oppo
rtunity there, but someone
has to show the value and gain
trust and le
ad the way. And some commercial
users will want some money for the minor c
hanges they might
need on th
software (e.g., the addition of a high speed
data link for time series). You
guys are well
positioned for bringing m
any people together
different people have different motivation
s, you will
have to develop the
Henry Kissinger within!
Response from: Brian Billings, Saint Cloud State University
Sorry for the late reply. I'm still coming back from a bad cold which started late
I read through
the white paper, and I don't really have much to add. In the software class, we were able to
use IDV to read in the MISS data either straightaway or with a minimal amount of NCO manipulation. I see
IDV is in the core software suite (along with xprof), so
everything looks okay to me.