ehr-technical-scorecard.docx

shrubberystatuesqueData Management

Dec 1, 2012 (4 years and 4 months ago)

311 views







Technical Evaluation of Open Source
EHR frameworks




Prepared for:

National Cancer Institute


Center for Bioinformatics





2115 E. Jefferson Street

Rockville
,
MD

August 3
, 2009





PAGE
2

T a b l e o f Co n t e n t s

Executive Summary

................................
................................
................................
.......

4

Scorecard

................................
................................
................................
.....................

5

Analysis

................................
................................
................................
.......................

8

Metric Explanations

................................
................................
................................
......

11







PAGE
3






























PAGE
4

E x e c u t i v e Su mma r y

The purpose of this evaluation
i
s to analyze open source Electronic Health
Record applications
for their technical merits. This evaluation does not consider any of the features included in the
applications
. Instead, it

focus
es

only on their code, design and overall project characteristics.

OpenEHR
is

the b
est overall open source
project

because
it has a working build
,
is

clearly
documented,

and it

include
s

functioning unit tests. OpenEHR, though,
does not

include a client
application.
OpenEHR

is

attempting to coordinate an effort to construct a client. Until a client
has been buil
t and included in the distribution, OpenEHR seems incomplete when compared to
every other option.

TolvenEHR

is a relatively small (~70k LOC), well factored (highest CCN = 69) codebase. It
also requires PostgreSQL or Oracle, whereas all others (except the .
NET project, Caisis) work
with MySQL.

TolvenEHR was the only application
not

to receive

red flags

and
the evaluation
team

found

no

serious concerns

with it
. There
are

areas of minor concern
, such as
the amount
of duplication and
failures to adhere

to good
coding standards



on
occasion
. Those areas may
be easily overcome with some well planned refactoring.
It includes good testing for key areas
of the system. Other areas of the application do require more extensive automated testing.
Finally, the build for
the application does

n
o
t work as delivered by the development team.
The
development team

is aware of this problem

and plan to fix
how the software builds

in the
future. Any work extending the application should start by making the build pass.

OpenMRS gets
third mention because it shares many of the positive c
haracteristics as the first
two


notably, a small codebase (~72k LOC), working unit tests and a working build.

However, it
’s the
second most complex code base (in terms of cyclomatic complexity)

of tho
se
applications considered.
Extending this application would be
more effective if we first
examined

the problem areas
in order
to implement a better solution.

Caisis has some positive qualities,
such as

a relatively small codebase (~120,000 lines of
code)
that has been factored well (Highest CCN = 30). However, it includes only minimal
testing and no automated build script. Many
.NET

projects do not use command line
automation to the same extent as the Java community, but such automation would be highly
des
irable if ongoing active development takes place.

Caisis is the only .NET project included in
this
evaluation;

therefore it

i
s worth mentioning that it would requir
e a Windows deployment
platform, whereas all others would be portable across operating syste
ms.

Both PHP projects, OpenEMR and ClearHealth/MirrorMed, present significant risks. Both
codebases are very large (~350k and ~240k
,

respectively) but
these projects

do not employ
good architectural practices
such as

using multiple application layers to en
sure maintainability.
ClearHealth includes some automated testing, while OpenEMR does not. Finally, both projects
suffer from code duplication at around 30% code duplicat
ion

each.

Oscar demonstrates many poor qualities; large codebase (~290 LOC), confused architecture
(part
three
-
tier, part two
-
tier), high complexity (456 highest CCN), and poor coding standards
(968 basic violations). To complicate the project
further, there are no automated unit tests.
Oscar appears to be suffering from entropy as a project, which
might be expected from a
project with an

inception date during 2001.






PAGE
5

Sc o r e c a r d







PAGE
6








PAGE
7

Figure
1

-

Relative sizes of the applications and their problem areas. Note that
bigger is not necessarily better
.






PAGE
8

An a l y s i s

The purpose of this evaluation was to analyze open source Electronic Health Record
applications for their technical merits only.

Some appli
cations that were initially included in the
evaluation were excluded almost immediately based on leading indicators
to
deem

these
applications un
suitable.
These suitability factors

are included at the end of this section for
reference.

We evaluated the
fol
lowing applications in detail. Each was scrutinized as thoroughly as
possible for the characteristics outlined in the scorecard. Some technical limitations precluded
gathering all measurements for all applications
(
e.g.
,

calculating PHP application cycloma
tic
complexity does

n
o
t have the same tool support as other languages
)
.

In addition to the details from the scorecard, t
he following key findings were outlined as a part
of this evaluation
:

OpenMRS



Very active codebase.



Easy to open in Eclipse
.



Builds out
of the box.



Includes a fair amount of testing (40% coverage).



All tests pass out of the box.



Compile time deprecation warnings.



Ant file would be
nefit from an expert review, but, in its current state, it is
manageable
.
Standard targets. Rational dependenci
es.



Uses
PreparedStatement

almost everywhere. Conscious of SQL injection risks in code
comments
.



Developers aware of and f
ixing XSS risks, as per check
-
in

comments.



Entire package for HL7 support (
org.openmrs.hl7.*
)

Oscar



Very active codebase.



Has an Ant
script. Not at the root of the project.



Confusing repository structure.



Does NOT build out of the box.



Confusing module structure.



Some JDK 1.4, because using now reserved word "enum" in source. Easy enough to
rename from "enum" to "params" which would be
more self
-
documenting.



Some Ant, some Maven
.



Cyclomatic complexity good overall, but some very concerning outliers (456 highest).



Some use of
JUnit 4
.



Most recent r
elease on July 21, 2009
.



Looks like a
three
-
tier app, but might be mostly two
-
tier (JSPs and

DB).



SQL Injection Risk: String concatenation used for SQL statements.



No mention of
SQL injection or XSS on website or in the code.



HL7 listed among features that "you were afraid to ask about" on April 2009
datasheet.






PAGE
9

Open EHR



No GUI.



Heavy focus on the

specification; the code is a "reference implementation" of the
specification
.



Uses Maven.



Maven build works out of the box.



A fair amount of testing included.



All tests pass out of the box.



Low cyclomatic complexity outliers (40 highest).



Reasonably sized

codebase
.



No use of
java.sql.*

at all. No database. No SQL injection risk.



No client, therefore
the
risk of XSS is not applicable.



Regarding HL7 "they don't generally address EHR requirements (they are essentially a
new approach to doing messaging)."



Have

an entire list of standards that do apply to their effort.

1



It's clear that this application takes its decision to conform to standards very seriously,
as would be expected by their overall approach to application development.

Tolven



Everything has an An
t build.xml
.



Does NOT build out of the box


some configuration required.



Build system factored for reuse.



Lots of plugins/plugin architecture
.



Use of EJB3
.



Uses customized version of JBoss for app server
.



Docs call for PostgreSQL or Oracle. No mention of
MySQL
.



Mobile device integration
.



Tests for tolvenEJB, tolvenCommon & pluginFramework modules.



No use of
java.sql.*

at all. Uses EJB3. No
t

high
SQL injection risk.



Specifically address SQL Injection and XSS concerns on the website
.
2



Conformance to HL7
standards.

3

OpenEMR



Huge PHP codebase.



Wordy documentation.



No prescribed architecture: implicit "PHP" style architecture.



No unit testing
, though some libraries include tests.



Appears to escape input before concatenating SQL strings. Using standard SQL
l
ibraries.



Seeking volunteers to help with web security, specifically mention XSS and SQL
injection
.
4



SQL Injection discussed explicitly in
OpenEMR’s

board of directors meeting in January
2009
.
5




1

http://www.openehr.org/releases/1.0.1/html/architecture/overview/Output/standards.html#1141279

2

http://www.tolven.org/architecture/briefs/websecurity.html

3

http://www.tolven.org/faq/faqstandards.html

4

http://www.oemr.org/modules/wiwimod/index.php?page=OpenEmrSecurity

5

http://www.oemr.org/modules/wiwimod/index.php?page=AgendaMinutes11&back=WiwiHome






PAGE
10



Code includes a specific
PHP

function to clean for XSS.



Conformance to HL7 standards
6
.

ClearHealth/MirrorMed



Huge PHP codebase.



Big spike in downloads (from 250 to 300 downloads/month) in June '08 and June '09



Had a release (2.3) on July 17
, 2009



Switched to using GIT source code management



No development docum
entation on website, though advertised.



No prescribed architecture: implicit "PHP" style architecture.



Appears to escape input before concatenating SQL strings. Using standard SQL
libraries.



No mention on website

or in code

of XSS awareness.

Should be thor
oughly tested,
especially for XSS vulnerabilities.



Conformance to HL7 reported
.
7

Caisis



Not as open to outside contribution
.



No build file
.



Minimal ad hoc testing
.



Good cyclomatic complexity
.



Use of string concatenat
ion for SQL commands. Some SQL injection

risk.



No mention of XSS attacks on their website.



Some use of HTML encoding, but not prevalent. Some XSS risk.



No HL7 mention on main site or on wiki
.


The following applications were excluded from the bulk of this evaluation. Specific reasons for
their
exclusion are listed as follows.

Torch

The last source code commit date was more than
five

years ago (Feb 2004).

I
t was considered
by the evaluation team
to be a "dead" project.

Ultimate EMR

The last source code commit date was more than
four

years ago (Au
g 2005).
It was
considered by the evaluation team to be a "dead" project.

Indivio Health

The last source code commit date was
close to three y
ears ago (Sep 2006). It was considered
by the evaluation team to be a "dead" project.

Free Med

The last source cod
e commit date was between 2 and 3 years ago (May 2007). It was
considered by the evaluation team to be a "dead" project.




6

http://www.oemr.org/modules/wiwimod/index.php?page=TechnicalAspectsOpenEmr

7

http://www.clear
-
health.com/content/blogcategory/0/38/4/16/






PAGE
11

GnuMED

The application source consists of a combination of Shell scripts (.sh) and Python,
which is
considered
to be

non
-
standard by th
e evaluation team. It was excluded on this basis.

PrimaCare

The application was not freely available on the
Internet
. The evaluation team sent an email to
the parties listed on the website, but received no response. Since the application was
unavailable an
d the team was unresponsive, the application was excluded.

Me t r i c E x pl a n a t i o n
s

The evaluation team considered the overall health of the applications being considered.
We
evaluated

t
he health using a number of measures, some objective and others subjective. The
experience of the evaluation team with other successful open source projects played a part in
this evaluation. The following sections describe each of the measures used in this

evaluation
with some commentary on what constitutes a "good score" versus a warning sign or
an
unacceptable score.

Open Source

Applications may be distributed in many forms. Open Source strictly means that the
application's source code is available to any
one with rights to the application. Many
commercial applications are distributed with their corresponding source, but
they
require users
to pay a fee for their use. Technically, such an application would
still
be considered "open
source".

However, in commo
n usage, the term "open source" goes further to also mean that the
application
may be used
free of charge. The distinction between commercial open source and
free open source, although important, is

n
o
t relevant to this evaluation.

Open source in the context of this evaluation means both free for use and that the source
code is made freely available.
These qualities were instrumental in selecting the applications
included in this evaluation.
See the License section for more detail on

legal restrictions to
usage.


Source Last Updated

One sign of a good open source project is the amount of time since the source was last
updated.
Because the barrier to entry is so low, m
any open source projects are initiated. But
most open source project
s do not continue long enough to become stable applications

(i.e.
,
they are
appropriate for enterprise use
)
. If the source has

n
o
t been updated in months then
the application developers (usually volunteers) may have lost interest. Given multiple choices,
t
his evaluation prefers applications with very recent dates for source updates.

Downloads

The number of times an application has been downloaded may be used as a measure of its
adoption, and therefore success, in the marketplace. Popular, stable
,

open source projects
tend to have a high number of downloads.






PAGE
12

Unfortunately, many open source applications do

n
o
t make this metric readily available.
Common open source hosting sites,
such as

http://sourceforge.net
,

provide this statistic.
Many of the applications included in this evaluation are not using an open source hosting
infrastructure,
such as

SourceForge, instead preferring to self
-
host. As such, the downloads
metric should be considered when available, but
i
t should
not
have

a heavy bearing on the
final decision making process.

License

Open source licenses come in many forms. The choice of licensing may have far reaching
impacts on any extension and inclusion. As a
n

area that has ever evolving precedent cases
,
consulting a legal specialist may be warranted. The following serves as a brief description of
the various licenses encountered in this evaluation.

GPL

The GNU
General
Public License
8

has been widely used in open source software. It has been
termed "vira
l" because all derivative works are required
to distribute

under the same license
without any modification to the license. Derivative works are usually interpreted broadly to
include "linking". In languages
such as

Java that may be interpreted to mean that

any
application that compiles against a GPL library would be considered a derivative work.
Note
:

may be interpr
eted

, in the sentence above,
being the critical phrase as the community is
divided on the issue
9
.

In general, if one does

n
o
t mind their software being distributed under the GPL use of other
GPL libraries and source then this is a non
-
issue. However, if one wishes to distribute their
application and source under a different license, then using GPL software should be used with
c
aution and
with
good legal advice.

LGPL

The GNU Lesser General Public License
10

provides more permissiveness
with respect to

linking
by other applications. Other applications that link to libraries released under the LGPL are not
considered to be derivative

works and therefore may be distributed under any chosen license.
Derivative works must allow "
modification for the customer's own use and reverse engineering
f
or debugging such modifications."
11

In general, the LGPL
is

preferred for most enterprises becaus
e they are free
choose their own
license
. However, if the intention
were

to extend an open source application, not just use it,
then the extensions would almost certainly be considered derivative works
for both GPL and
LGPL
. Both would require the derivati
ve work to be distributed as open source applications,
but the LGPL would allow for applying a more liberal license to the extensions.




8

http://www.gnu.org/copyleft/gpl.html

9

http://en.wikipedia.org/wiki/GNU_General_Public_License

10

http://www.gnu.or
g/copyleft/lesser.html

11

http://en.wikipedia.org/wiki/GNU_Lesser_General_Public_License






PAGE
13

Mozill a Public

As the name implies, the Mozilla Public License
12

was created at Netscape (and Mozilla
Foundation) and wide
ly used for
its

products. It requires that source code copied and/or
changed must remain under the MPL
.
13

However, the source may be incorporated into a larger
work without impacting the larger work's licensing terms.

As this license allows for broad usage
of open source code without forcing itself on derivative
works, enterprises may use source distributed under this license without the risks associated
with GPL and LGPL
.

Mozill a Tri
-
Li cense

The Mozilla Tri
-
License
14

simply allows the user to cho
o
se one of t
hree licenses; MPL, GPL or
LGPL. This allows for the use of source within a variety of other applications without concern
about licensing.

Year started

Open source application longevity may be used as a measure of its stability. Typically, the
longer the
application has been in existence, the greater the dedication of the development
team. The application has had time for bugs to be discovered and fixed. The team has had
time to stabilize the architecture and feature
s
.


Newer projects may also have many po
sitive benefits. The teams tend to be very passionate,
especially if they are building an application with many competitors. It usually indicates that
the other applications are specifically lacking in some manner that they wish to overcome.

However, if a

project has not been in
maintained

for at least a year, it should be considered
suspect and used only by early adopters willing to invest time and energy in helping to
improve the application.

Architecture Type

Application architectures come in many forms
, depending heavily on the year of their inception
and
as thinking in computer science evolved. Most web applications, though, fall into two
categories


two tier and three

tier.

Two
-
tier applications usually include a database (or some more rudimentary
persistence
mech
anism
-
like files) and web pages that include business logic. Including business logic in
the web pages

(php or jsp files are typical)
-

makes it difficult to maintain the application

because it is difficult to isolate changes and test the c
ode
. Separating the "presentation" and
"business" logic is usually preferred.

Three
-
tier applications separate the code into database, business and presentation layers. This
architecture is the de facto standard for contemporary web applications. Popular f
rameworks
have been built to help facilitate this layering,
using

Model
-
View
-
Controller frameworks
such
as

Struts.




12

http://www.mozilla.org/MPL/

13

http://en.wikipedia.org/
wiki/Mozilla_Public_License

14

http://www.mozilla.org/MPL/






PAGE
14

The third architectural variation encountered in this evaluation was Service Oriented
Architecture (SOA). SOA applications typically use a
th
ree
-
tier

layering, while further dividing
their business layer into a set of "services". The services are exposed through a non
-
language
specific mechanism, for example SOAP, to provide broad access to the functionality in a cross
-
platform fashion.

SOA app
lications have been under development, probably since the first network applications
were written, but particularly since the development of
HTTP
-
based communication protocols.
They are particularly applicable in applications that provide general services

that may be called
by lots of external callers across disparate platforms. However, SOA has been hyped greatly
and overused in situations that do

n
o
t warrant the added cost.

Choosing SOA as the
architecture of choice should be weighed heavily and considere
d carefully.

Programming Languages

The choice of programming language depends mostly on the skills available within the
developer base in place at any enterprise. Some languages and frameworks purport to be
better at niche applications,
such as

Ruby on Rails or Grails. However, the de facto standard
for web applications in the enterprise is presently Java or .Net.

Several
of the
applications evaluated used PHP as their primary language. PHP works very well
for dynamic websites. It does include e
xtensions for database connectivity, network
communication and image processing.

However, as a general enterprise application
development platform, it has

n
o
t gained the wide acceptance of the other two. That means
that
the tooling, supplementary libraries

and established best practices do

n
o
t exist to the
same extent. PHP applications also te
nd to lean toward two
-
tier architectures, though nothing
in the language insists upon it.

Database

All the applications in this evaluation make use of a
mainstream

rel
ational database,
which is
considered standard practice in the industry. The choice of which database best suits an
enterprise's needs depends mostly on the existing infrastructure investment. Many enterprises
treat databases as a shared service centrally
managed and hosted. Under such conditions,
database administrators are reluctant to incorporate additional technologies.

Operating System

Except in the .NET community, most applications are designed and implemented in a cross
-
operating system language.
Bot
h
Java and PHP are portable across most operating system
s

in
use today. Th
is

makes the operating system choice a non
-
issue.

However, some .NET applications require Windows servers. If an enterprise uses Linux servers
exclusively, the addition of Windows se
rvers may be a significant barrier to entry.

Client

The "client" is the application that an application's end user uses to interact with the
application. Today, most applications use "web clients

,

which

means that users may access
the application with any compliant web browser. This makes upgrading applications a
relatively trivial matter, as the only software that must be changed resides on the server.





PAGE
15

Additional clients may
still
exist,
such as

iPhone
application
s, handheld accessible websites
and so on.


Lines of Code

T
he more lines of code

in an application the more complex it tends to be.

This measure is

n
o
t
absolute;

well
-
factored applications tend to have more lines of code than poorly factored one
s.

A well
-
factored application

usually consists of small, easily understandable chunks of code
(methods, classes, functions) with little repetition. A poorly factored

application

has big chunks
of code

that are not self
-
documenting

and so t
hey tend to be t
erse and brittle.

Factoring aside, lines of code should be weighed along with every other metric to gauge an
application's maintainability and extensibility. Preference should be given to applications with
fewer lines of code, all other factors being equal
.

Cyclomatic Complexity
Number
(CCN)

Cyclomatic complexity measures the number of paths through a method/function.

Every
method has at least one path
.

I
ntroducing a single conditional expression increases this
measurement to two. For example:

public
String

evaluateMyValue
() {


if (myValue == 10) {
return "The value is 10."
; }


return "The value is not 10.";

}

The method in the example

has two paths: o
ne path where
myValue

equals 10 and one where
it does

no
t.
The second path makes this method more difficul
t to understand than if it had a
single path. Although this method is trivial, the point is illustrated better by a method with 456
paths, as is the case with
the
Oscar

framework
.

Beyond understanding, CCN has an impact on testing. In order to minimally te
st a method, a
separate test should exist for each path.
Testing a method with a CCN greater than 10 quickly
becomes intractable. Therefore
,

we
recommend that any method with a CCN over 10 should
be refactored into simpler more succinct methods.

This is a
key metric when considering overall application quality. CCN should be tracked
closely throughout the application development lifecycle. An application with a high % of
methods above 10 shows signs of poor factoring and probably poor test coverage.
Applica
tions

with single methods that have very high CCN

(> 100) show signs of questionable development
standards. If the CCN issues are isolated they may be quickly refactored. However, if they are
widespread, the issue could be an endemic warning sign.

Duplicat
ion

Duplic
ated code introduces complexity

w
hile

reducing maintainability.

Copy
and

paste coding
,

whereby new functionality is added by copying a similar existing piece of code and making
small changes
,

duplicates bug opportunities. If a bug exists in the original functionality, the
new functionality will also contain that bug. When modifications are required
of

the original
functionality, they will also be required
of

the new functionality.






PAGE
16

A better alt
ernative would be
the use of well
-
accepted
object
-
oriented principles, or at least
extracting shared libraries.
Frequently
,

developers invoke the DRY principle


Don't Repeat
Yourself.
In such
a
fashion, code to perform a particular function should only ex
ist once in the
system.

A

high
percentage

of duplicate
d

code should be considered a significant warning sign.

Coding Standard Violations

Following coding standards results in a better application

because it will be easier to
understand, easier to maintain and less prone to bugs. There are lots of coding standards
defined by the industry. Some are as simple as syntactic standards such as the proper location
for a closing curly brace,
such as

those
used by the Sun development team
.
15

Others are
related to common programming mistakes, such as including empty catch blocks in Java
exception code.

Teams should define a programming syntactic standard and adhere to it. This evaluation
,

however
,

focuse
s

on
the latter type of coding standard violations
.
16

Syntactic adherence can
be corrected relatively simply, whereas programmatic errors can be significantly more difficult
to correct. Therefore
,

preference should be given to applications with fewer coding stan
dard
violations.

Unit Tests

Automated unit tests should be considered standard practice. The prevalence of the xUnit
17

testing frameworks makes it simple for developers writing in
nearly
any language to create
repeatable, independent unit tests.

Open source applications, especially, should include good
unit tests for all functionality
in order
to serve as a safeguard for developers new to the
application. The tests also serve to demonstrate the expected use of the classes in the
application.

Pref
erence should be given to applications that include unit tests with their source. Further
preference should be given to those application
s

that include unit tests that adequately cover
the entire application.

Because of the difficulty of getting tests runn
ing against the evaluated
applications, this evaluation does

n
o
t include code coverage numbers. Only a few of the
applications included tests, and almost all did

n
o
t include anywhere near
ly

enough to cover all
the functionality.

Build System

A good "build"

includes everything necessary to compile, package, test and deploy an
application. Most development platforms include standardized methods for scripting such
builds, notabl
y

Ant, Maven and Nant. A good open source project will include a file at the root
o
f the project that conforms to one of these standards. Upon initial checkout of the application
from the source control system, new developers should be able to run the default "target" with
zero configuration.




15

http://java.sun.com/docs/codeconv/html/CodeConvTOC.doc.html

16

http://pmd.sourceforge.net/rules/basic.html

17

http://en.wikipedia.org/wiki/XUnit






PAGE
17

SQL Injection Risks

SQL injection
18

poses a ri
sk to applications whereby hackers trick the application into
executing destructive (or subversive)
SQL

statements. A common example is dropping an
application table, unbeknownst to the application. If a hacker types the following into a web
application fo
rm field:

1;DROP TABLE users

And the underlying source code does the following:

statement = "select * from data where id=" + form_value";"

The resulting SQL statement could potentially have disastrous side effects

(e.g.,
dropping the
users table
)
.

The comm
on solution

to this
problem

is

the

use
of
parameterized statements with well
-
typed
bind variables, instead of simple string substitution.

The
applications under evaluation w
ere
scrutinized, manually, for typical SQL injection vulnerabilities.

One key indicator used was
whether or not the development team took steps that were obvious attempts to limit SQL
injection vulnerabilities.
If a risk has been identified during this evaluation, it means that
further more extensive testing should be conduc
ted to find (and fix) the problem areas.
Obviously, preference should be given to applications that take steps to
limit these
vulnerabilities.

Cross Site Scripting Risks

Cross site scripting
19

attacks take advantage of client side scripting vulnerabilities
to trick
users into giving away personal information

(usually without the
m

realizing it)
. Applications
that make use of client side scripting, like JavaScript,
may expose their users to cross
-
site
scripting attacks. Also, if applications allow users to inp
ut data containing html syntax, they
need to take special precautions to prevent malicious
cross
-
site

scripting attacks.

The common solution to this problem is to escape all
HTML

input by users in form fields. Or
better yet, do

n
o
t allow users to input htm
l in form fields.

The
applications under evaluation
w
ere scrutinized, manually, for typical
cross
-
site

scripting vulnerabilities.
If a risk has been
identified during this evaluation, it means that further, more extensive testing should be
conducted to fin
d (and fix) the problem areas.
Obviously, preference should be given to
applications that take steps to limit these vulnerabilities.





18

http://en.wikipedia.org/wiki/Sql_injection

19

http://en.wikipedia.org/wiki/Cross_site_scripting