Thai Nguyen HMIS Evaluation 20120718 - PHAD

rabidwestvirginiaΔίκτυα και Επικοινωνίες

26 Οκτ 2013 (πριν από 3 χρόνια και 5 μήνες)

109 εμφανίσεις




Gobee

Group and HISP

AP4 HMIS
P
roject
E
valuation

Improving P
rimary Health Care

in Thai Nguyen
Province:



Interventions to Improve Data, Evidence and Capacity

Arthur Heywood and Mahad Ibrahim

6/29/2012


CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







2

of
24


Table of Contents

Executive summary
................................
................................
................................
.........................

3

Introduction

................................
................................
................................
................................
.....

5

The Vietnam HMIS

................................
................................
................................
..................

5

AP4 Project

................................
................................
................................
..............................

6

Evaluation Activit
ies

................................
................................
................................
........................

6

Project Status

................................
................................
................................
................................
..

6

Implementation

................................
................................
................................
............................

7

Achievements of the project

................................
................................
................................
....

7

In
frastructure

................................
................................
................................
............................

7

Human Resources

................................
................................
................................
..........................

7

Training

................................
................................
................................
................................
...........

7

Software training

................................
................................
................................
.........................

8

Data use training

................................
................................
................................
.........................

8

Software

................................
................................
................................
................................
..........

9

Software Requirements and Specifications
................................
................................
................

9

Benefits
................................
................................
................................
................................
......

10

Challenges
................................
................................
................................
................................
.

10

Privacy & Security

................................
................................
................................
.....................

10

Assessment of Software Provider
................................
................................
.............................

11

Linking Commune HMIS with District Hospital HMIS

................................
...............................

12

Software Development Process

................................
................................
...............................

12

Moving towards mHealth
................................
................................
................................
...........

12

Making the Software Open Source

................................
................................
...........................

13

Information for Health Management

................................
................................
.............................

13

CHC data use training

................................
................................
................................
...........

14

Eviden
ce based unit

................................
................................
................................
..............

14

Research
................................
................................
................................
................................

14

Conclusions

................................
................................
................................
...........................

15

Recommendations

................................
................................
................................
........................

16

1

By AP4 conclusion (September 30th)

................................
................................
...............

16

2

Next phase

................................
................................
................................
.........................

17

MoH Alignment

................................
................................
................................
......................

18

Project management
................................
................................
................................
..............

18

Software

................................
................................
................................
................................
.

19

Data use
................................
................................
................................
................................
.

19

Capacity Building

................................
................................
................................
...................

19

Appendix A
: Terms of Reference for evaluation

................................
................................
.........

20

Annex B: Evaluation Activities

................................
................................
................................
......

23

Appendix C: Questions for evaluation team

................................
................................
................

24


CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







3

of
24


Executive summary


The computerization of health information in Thai Nguyen is yielding concrete benefits for
patients, health workers, and administrators. There are substantial reductio
ns in the time it
takes staff to input patient records, submit health insurance claims, manage drug inventories,
and provide district health directors with monthly reports. Many of these benefits are not directly
visible because the Thai Nguyen Health Dep
artment have chosen a cautious approach to the
transition to digital records and maintain both the paper and computer
-
based HMIS.


The project has provided a digital infrastructure for the HMIS and has strengthened the base of
computer knowledge from whi
ch to build modernization efforts. Each CHC has at least one
computer and one staff member trained in basic computer skills. Many CHCs now have high
-
speed access to the Internet. Data is being collected faster and more accurately than ever
before and a
store of digital data is being built that can potentially serve as the basis for disease
surveillance, administrative management and evidence
-
based decision making.


Achieving these results has not been a smooth process, as is to be expected in any major
computerization effort. The software company has developed software that achieves most of the
major requirements of the ministry. In choosing a cloud
-
based computing
approach, the
software company has built something that, as it improves, can potentially be used on a larger
scale. The main risk to the cloud
-
based approach is dealing with locations with limited or no
Internet access or major disruptions in power. The
software company developed an offline
mode to deal with these issues, but it still needs to achieve the robustness needed for mission
critical applications.


The medical examination and population modules in version 1 of the HMIS have been modest
successe
s in the current implementation with most complaints focused on persistent bugs and
slow application performance. Most of these issues have been addressed in version 2 of the
software, which has yet to be fully implemented in Thai Nguyen. At present, modu
les for

all
major Commune Health Centre functions have been developed, but only two are mature
enough for widespread use.


S
hort
-
term recommendations are made to improve software
functionality

and to prepare and document the code for open release

before th
e end of the
current project
.


The project has started the process of
capacity development

to use HMIS data for planning,
policy and management at CHC level in one district. A training needs assessment has been
done, curricula have been developed for CH
C and district staff and a training of trainers
curriculum has been developed.


The planned model of integrated
Evidence
-
Based Planning
, Policy and Management (EBPM)
has been tested at CHCs in one district, with training done by Hanoi Medical University.
Training
of trainers has not been conducted, mainly due to a lack of clarity in the roles and
responsibilities in using information at the various levels of the health system
.

Research to test
the efficacy and sustainability of EBPM has not been conducted,


The evaluation team feels that continuation of the project will result in a model of evidence
-
based decision
-
making that will be of national value in Vietnam and recommends a follow
-
on
project to continue development of the project’s aims.


CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







4

of
24


The
follow o
n project

should be focused on national scalability, be closely aligned to the
National Health Development Strategic Plan (HSDP) and work in collaboration with both the
provincial management and the national partners HMIS technical working group. Data use
approaches and research should support national processes to streamline existing indicators
and simplify tools as well as clarifying roles and responsibilities for institutionalised data use at
all levels.


Capacity development for analysis, interpretatio
n and use of information will need to be done at
all levels and a sustainable system for ongoing HMIS support and training developed.

The software should fit the vision of a national data warehouse and have the capacity to support
all phases of the inform
ation cycle (collection, processing, analysis, dissemination and use) by
providing flexible and appropriate tools to support evidence based decision making processes.

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







5

of
24


Introduction

Arthur Heywood (HISP) and Mahad Ibrahim (Gobee Group)
evaluated the AP4 proj
ect in Thai
Nguyen
for the Population Council (PC) Vietnam
between June 25


29,

2012
. The evaluation
was conducted

according to the terms of reference in Annex A.


The objectives of the evaluation were

three
-
fold,



1.

To determine the extent to which the
goals and objectives of the project have been
accomplished.

2.

To consider and discuss the likelihood that continuation along the lines currently
undertaken will result in a model of evidence
-
based decision
-
making that will be of
national value in Vietnam.

3.

To

recommend appropriate directions that might be taken in a follow
-
on project to
continue development of the project’s aims.

The Vietnam HMIS

As part of the National Strategy for the Protection, Care and Improvement of People’s Health for
2011
-
2020 and Vis
ion to 2030
1

the following achievements of the HMIS were noted country
-
wide
:



1.

Health information work has shown considerable improvements and many legal policies
in health information have been developed

2.

Many channels of information collection are
explored

3.

There are diverse information sources


Difficulties and challenges

included
:


1.

Information is not up
-
to
-
date

2.

Quality of information is limited

3.

There is no national policy, orientation and health information system development plan.

4.

Information of
some areas is not available

5.

Information sources do not have
a
dissemination mechanism, so it is not easy to access

6.

Limited knowledge in data analysis, assessment and forecasting


The Vietnamese Ministry of Health (MoH) runs t
he Vietnam Health Management In
formation
system (HMIS) through three separate and fragmented directorates:



Directorate of Finance and Planning (DFP) in charge of the HMIS,



Directorate of Science and Training (DST) in charge of Technology



Medical Services Administration (MSA) in charg
e of Hospitals.


The HMIS is currently paper based and is driven by 11 logbooks and focuses mainly on
reporting of raw data on activities to the higher level and monitoring of centrally set performance
targets. HMIS responsibilities of each level are to r
eport to the level above using either paper or
excel spreadsheets, depending on the program.




1

National Strategy for the Protection, Care and Improvement

of People’s Health for the 2011
-
2020 period and Vision to 2030

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







6

of
24


The Minister is committed to developing one national HMIS and is driving the computerisation of
the system. She is in the process of setting up an Administration for Technical Administration,
which will incorporate the functions of the three directorates a
bove.

AP4 Project

As a response to these challenges, the AP4 project (2009
-

2012) was set up to digitise the
paper based forms in Thai Nguyen province, with the aim of strengthening the primary health
care system by developing the culture and practice of

using evidence to guide health policy,
planning and management and to contribute to global knowledge and experience on the
implementation and evaluation of evidence
-
based approaches.


Specific objectives

were to

1.

D
evelop and maintain an effective and sust
ainable HMIS model in the province,
especially at commune level.

2.

D
evelop a critical mass of skills, capacities and motivation within TNHD leadership and
staff to analyze and use HMIS and related data as evidence for planning, policy and
management.

3.

W
ithin
TNUMP faculty, to teach and conduct research using Evidence
-
Based Planning,
Policy and Management (EBPM).

4.

T
est the efficacy and sustainability of an integrated, province
-
led model of EBPM.


Achievement of these objectives would reduce staff workload, impro
ve accuracy of data and
develop capacity for use of information for evidence based decision
-
making and policymaking.
In addition the project was to research the relationship between information and evidence.

Evaluation Activities

The evaluators interviewed

key informants in Hanoi and Thai Nguyen

including representatives
from Atlantic Philanthropies, Pathfinder International, HSP, Hanoi Medical University, and
staff
at
the P
rovince

health department
,
D
istrict

health offices
,
District
hospitals
,

and
Commune
health centers

(See annex B)
.


T
he software was assessed and the outputs of the project were
examined in order to answer the questions posed in annex C.

Project Status

The project, which started in March 20
09, is implemented by PC all 181

CHCs
in the Thai

Nyugen province
with the cooperation of the provincial health department (PHD) IT team.


The HMIS software has been developed for P
opulation
C
ouncil

by the company
-

HSP
. The
computerization project has been operating
since April

2010
. The first 18 month
s were spent on
software development by HSP. Since December 2011, the focus has been on installing the
software at the CHC level and refining the software based on operational experiences.


To date all 7 proposed modules have been developed, but only two
modules are in use at the
CHCs,

1.

Medical examination module
,

which captures the required data from each patient
encounter. The module also manages claims for patients with health insurance and drug
inventories.

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







7

of
24


2.

The population module allows CHC staff to find information about patients within their
commune.

Implementation

The a
chievements of the project

are,



The software has the capability of generating reports and logbooks as required by the
MoH system, with a fo
cus on information needed to improve patient care at the
commune level.



The system requires only skills and capabilities that already exist, or can easily be
developed, at local levels


that is, it does not require much statistical or epidemiological
educ
ation or much IT sophistication at the CHC level.



The software is on
-
line, with an off
-
line capability necessary for the situation.



The software's ability to link medical examination, drug management and health
insurance is valuable and important.



The po
pulation module permits important perspectives on client care locally and at
higher levels.



There is training and development on utilization of data at all levels which, if continued,
has considerable promise.



Testing a computerised system in one province
should generate experience that will be
of value to Vietnam.


All CHCs currently have Version 1 of the HMIS installed. Four CHCs have version 2 installed.
Only one CHC (Ba Hang) is using the digital HMIS without the paper system, while all others
are usi
ng both the paper and digital HMIS in parallel.

Infrastructure

Each CHC has at least one computer connected to the software. 137 out of
18
1

CHCs have a
second computer, but the software is not installed on it.

Human Resources

Population Council manages

the project

with technical support from HSP and
capacity building
support from
HMU. The PHD IT unit supports the software rollout and CHC staff are the end
users of the system. There are
5

dedicated IT staff and 4 part time EBU staff at the province
and
one part
-
time IT staff in each district who could be used for project support.

Training
2

Capacity building is at the core of the AP4 project, with Objective

2 being
,

“To develop a
critical mass of skill, capacities and motivation within Thai Nguyen Health

Service
leadership and staff to analyze and use the HMIS and related data as evidence for
planning, policy and management.”




2

Interim Report on Training in Thai Nguyen, April 2012

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







8

of
24


Population Council managed training
with support in research from PHAD
,

TNUMP

and ISMS
.
HSP provided training and training mater
ials for the software.

Hanoi Medical University (HMU)
led the training in data use and ICD10 coding.

Software training

362 CHC staff from all 18
1

CHCs were trained between March and September 2011
. This

mean
s

that all CHCs have at least two staff members that have attended training courses in
basic computer skills (two days) and on the digital HMIS (three days). Teaching methods for
CHC staff to use the computerized HMIS are adequate and curricula exist for CH
C and district
staff computer skills and software training, as well as data use for CHCs. A user manual has
been provided to each CHC. With staff turnover, in many cases only one trained staff member
was present at the CHCs visited, though CHC staff were
encouraged to teach others.


Provincial
staff

were

trained in
January 2011

and district
staff

were trained
in
October 201
1.

District staff
are

involved in supporting the HMIS software rollout
and
use
, but it is not their
principle responsibility.


There

is no consistent software support mechanism in operation and no clear guidelines on how
to access support if needed. The software does not have any online help options
, but each CHC
has been provided a physical software operation manual.
CHC staff curren
tly use four different
support options depending the nature of the problem, responsiveness, and access
:





Contact district IT staff



Contact province IT staff



Contact Population Council (
La
n
)



Remote support directly with HSP via Team Viewer application

Dat
a use training

Hanoi Medical University (HMU) has done a training needs assessment, developed a data use
curriculum (3 days) and an ICD 10 curriculum (3 days). ICD
-
10 and data utilization training are
at the same stage; with one pilot
having been conducted

at the commune level (Dai Tu) and
another for selected district and province staff. In Dai Tu district
,

all CHCs have attended a data
use training run by HMU.


ICD 10 training was well received and an abbreviated (300
-
code) ICD 10 coding is in use in all

CHCs
.
Training on data use to improve health care have been limited by the fact that
the
primary data output format is aggregated data for submission up the chain. The data is
available, but not easily accessible to the layperson.
The situation is further

complicated by the
fact that
co
mmune and district health staff

are not expected to use data for evidence
-
based
decision
-
making. Their job description is to be accountable to targets set by the Ministry of
Health.


Training of trainers for district and pr
ovincial level staff to use the data for planning and
management is planned but has not happened. TNUMP has expressed interest in both pre
-
service and in
-
service training in data use but this is not planned until the system matures
further.

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







9

of
24


Software

The ap
proach to create a web
-
based HMIS is a good one that is consistent with the direction of
modern software development and offers a robust, flexible platform to build upon
.

The platform
in modern terms is called
cloud computing
or
software as a service
. Wit
hin this approach,



Deployment is as simple as providing a url and user account information



Software updates can be easily migrated across a large client base



Both data entry screens and report formats can be easily changed to accommodate
policy changes.



A

central repository of data offers numerous benefits outside of the current requirements
of the HMIS


There are two primary risks to this approach,



Any disruptions with the central provider of the software can cause problems for all users
of the system.



T
he system requires not only Internet access, but Internet access of a certain quality.


These risks are not limited to this particular piece of software but all software that exists in the
cloud.
H
aving 2 or more backup servers hosting the software in diff
erent locations

will reduce
this risk
.


HSP chose to build a PHP
-
based Web tool connected to an Oracle hosted database. PHP is a
server
-
based scripting language with the following benefits:



PHP has a large community of skilled developers



PHP is open so
urce



PHP is flexible and simple



PHP is well supported



PHP has extensive and well
-
written documentation



PHP has strong integration with MySQL databases


The costs of using PHP are:



As HMIS requirements get more complex, PHP might no longer be appropriate



T
he District Hospital Information System is built in the .Net framework. ASP.net is a
competitor to PHP that might link better with the .Net framework.


Six CHCs do not have a DSL connection. Another 13 have inconsistent DSL connections. In
both cases, 3G mobile data connections are used to provide access to the Internet. All CHCs
suffer power outages and maintenance issues that limit access to the softwa
re
3
. What is
needed to mitigate these risks is a robust
offline
mode for the software.


Version 1 of the HMIS fails to meet this threshold. HSP claims that version 2 of the HMIS has
vastly improved offline capabilities, but this was not verified. Ther
e is need to strengthen the
code to increase performance, meet functional requirements, and achieve ease of use.

Software Requirements and Specifications

The major requirements for the CHC HMIS as defined by Population Council, Thai Nguyen
Health Departm
ent, and other stakeholders are to:




3

We recommended to HSP that they conduct a randomized sample survey on the state of information
infrastructure in CHCs.

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







10

of
24




A
ssist commune staff in improving commune
-
level patient care services



I
mprove the timeliness, quality, and usability of commune
-
level HMIS patient reports



I
mprove management of the CHC system at CHC, district, and provin
cial levels



I
mprove the data available for planning



I
mprove quality of record keeping at the CHC level



P
rovide commune
-
level experience in health information system development in
Vietnam



M
ake additional data available for research and analysis of health
in Vietnam


The current version of the software as implemented
meets
these requirements.

Benefits

All CHC staff are happy with the new software and see the benefits as


1.

Reduced time to record patient data for medical examinations. A rapid assessment
show
ed that the time to examine and collect data on a patient was reduced by over 50%,
as shown in the following table


CHC

Paper
-
based HMIS

Digital HMIS

B
a

Hang CHC

10:08

4:50

Cai Dan CHC

9:35

8:54*

Dai Tu
Town
CHC

9:38

4:11


* CHC staff experienced issues entering the Health Insurance Card expiry date.


2.

Creation of a searchable patient history for continuity of care

3.

Automated management of CHC drug inventory

4.

Automated generation of summary statistics

Challenges



Lack of capa
city to solve basic IT problems at the CHC, district and provincial level



CHCs without a second computer face workflow challenges in using the software



Offline mode is cumbersome, but critical to success in a low connectivity environment.



User interface wi
ll need to be improved, but this should not be changed within the AP4
lifecycle



Outside of reports and form generation there is no mechanism for interacting with or
analysing the data



Providin
g maintenance and support to 181

CHCs will be challenging. Need to develop
formal mechanisms for handling issues.

Privacy & Security

We did not have enough time to properly assess the security of the digital HMIS. However,
there were some worrying signs.

1.

HTTP over SSL is not used.

This means that all data is transferred in the clear over the
Internet.

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







11

of
24


2.

Each CHC computer has a local copy of the patient data for that respective CHC. It is
not clear what procedures are in place to secure this data. The CHC computers are
readily acce
ssible to all manner of people.

3.

CHC staff do not use proper security procedures such as locking the desktop after use.

4.

CHC staff should have limited access to the administrative functions of the computer
software. This was not the case.


These are a few basic items that need to be addressed. A physical and information security
audit should be conducted and a threat assessment developed.

Assessment of Software Provider


HSP has succeeded in developing a web
-
based tool capable of wide sca
le use in primary health
care facilities. The software has made it easier for staff to enter patient data, search for
commune
-
level population data, and aggregate necessary data into required report formats. In
most of the Global South, the state of the
art remains a .Net based desktop application
connected to Microsoft SQL Server. HSP has been ambitious in attempting to build a web
-
based HMIS. If done properly, the rewards will far outweigh the risks.


However, an application with this level of compl
exity should be expected to be completed in 6
-

9 months including testing
4
. It has taken the software developer
1
2

months to develop the first
version of the software
, and another 15 months to develop Version 2
. It is difficult to determine
the cause of

these delays, but they could be caused by one of several reasons:



HSP does not have enough qualified developers to maintain their current workload



HSP is lacking the capacity to solve the challenges of developing a web
-
based HMIS



HSP has not prioritized t
he HMIS work in their portfolio


As with any new software, there will be bugs, new requirements, and design decisions that need
to be reconsidered. HSP has limited interaction with the end users and failed to make the
distinction between mission critical
fixes and issues that can be held to the new release. There
have been three issues in particular that should have been fixed prior to the release of version
2
5
.

1.

Mistakes in the drug inventory tallies

2.

Printing mistakes in the forms

3.

Prescriptions are being
printed in the wrong sequence


These bugs affect the day to day work of CHC staff and should be fixed immediately.


HSP also has a tendency to release code at random intervals, causing client relationship issues
with Population Council and with stakehol
ders in Thai Nguyen. A release schedule should be
developed in consultation with Population Council. These dates should be advertised to the
TNHD, District Health Director, and CHC staff. Population Council should be given access to
the development serv
er to sign off on the state of latest release. At this point, the code can be
migrated to the live servers.




4

This metric is based on international software development standards. It is not clear that it should be
applicable to Viet Nam.

5

These three problems exist in version 1 of the software. They have been corrected in version 2.
Version 2 is not widely d
eployed, so we still encountered these complaints. The main point is that certain
bugs should be fixed as soon as possible regardless of the ongoing work to create a new version.

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







12

of
24



HSP has not provided the best ongoing maintenance and technical support for the software.
Part of the problem is a lack of clear support guideli
nes between Population Council and HSP,
as the level of support and expected response times have not been agreed upon. Furthermore,
how HSP is handling support requests
i
s

opaque to the users
,

who don’t know if their request
has been received, viewed, or
solved. Using a support ticket tracking software can help the
communication between HSP and the users.


Linking Commune HMIS with District Hospital HMIS

Linking the two HMIS is doable, but not a trivial task and it is not reasonable to expect that the
full functionality of each scheme will be easily made available within the other for several
reasons. To achieve the goal of complete connectivity will requ
ire phasing out one system or
the other. However, a more focused form of connectivity could be developed around certain
transactions or segments of the patient histories.



The two HMIS use different identification schemes. The District Hospital HMIS
iden
tification scheme is not currently universally unique within Thai Nguyen province.
This can be corrected by attached a two letter hospital code to each identification
number.



Each system is written in a different programming language, uses a distinct so
ftware
framework, and runs a different database software.


What is needed to connect the two HMIS is the following:

1.

Use the CHC identifier as the basis for linking the two systems

2.

Issue identification cards with the CHC identifier at the CHC level

3.

Alter D
istrict Hospital HMIS to accept CHC identifier as a secondary identifier

4.

Develop a new module in District Hospital HMIS to accept data from the CHC HMIS

5.

Develop a new module in CHC HMIS to accept data from the District Hospital HMIS

Software Development
Process

The software was developed by HSP without much interaction with the end users or provincial
management.



CHC staff have not been included in the software development process in ways other
than as bug testers.



District staff have not been included i
n the software development or design process and
feel no ownership of the products.



Responsibilities for key aspects of the software ecosystem are unclear, with no obvious
linkages between PC, TNHD, Districts, and HSP.



There has been no user testing of

design elements. This can and should be done
before the release of the updated data entry forms.



There has been n
o

clear technical roadmap for the release of all the features. Suggest
this be done in accordance with the workload of the CHC staff.

M
oving towards mHealth

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







13

of
24


The number of fields of information that must be filled out by CHC staff preclude the effective
use of mobile phones. However, the decision to go with a web
-
based application allows the
opportunity to migrate to a tablet based approa
ch. This would require several changes to the
current web
-
based application:




HTML templates would need to be converted to be HTML5 compliant



The user interface will need to be altered to rely less on keyboard typing



The offline version will not work
unless a dedicated tablet app is created. This would be
more costly than adapting the web
-
based application to work on mobile devices (tablets)


Another, potentially beneficial application of mobile devices at the commune level would be the
development of

a mobile messaging platform to distribute periodic updates on performance.
The same system could be used to enable specific queries on patients.

Making the Software Open Source

Making software open source requires a combination of clear licensing, stro
ng documentation,
accessible code libraries, and systematic stewardship.


Open source does not mean
free
. It means that other programmers can have access to and
modify the underlying software code with restrictions. To make the software open source,

1.

T
he stakeholders must agree on the terms of the license

2.

T
he software code should be thoroughly documented and made available to the wider
public via a code repository such as GitHub.


Most importantly, some person(s) or organization must be designated as

a steward of the
codebase. The steward makes decisions on what will be included in the codebase from those
changes or enhancements provided by the programming community.


Because HMIS is a very specific application of information technology this applic
ation is not
likely to engender substantial interest from a large developer community
.

Making the code
available would allow flexibility in choosing software developers for future implementation of this
HMIS.

Information for Health Management

For informat
ion to be used for health management it has to pass through a number of steps
,



1.

Collection of patient data based on forms and logbooks

2.

Processing
:

Computerisation, Quality control, Aggregation,

3.

Analysis and interpretation: Conversion to indicators,
trend analysis, self assessment,
comparison and local use.

4.

Dissemination: Graphs, tables, reports,

5.

Use in planning and management, monitoring, evaluation, policy making etc


To date the data in the CHC HMIS data has been collected and computerised but
has not
passed through the other steps to facilitate use. There is no clarity on what information should
be used for at various levels
.

D
ata is merely sent to the level above
as

required by the
MoH
.
CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







14

of
24


There is no system for data use for facility management o
r management of the wider health
system by the district and higher levels.
It is not clear that such a system was within the scope
of AP4.

CHC data use training

The HMU has
conducted a
needs assessment

on training requirements
, developed a curriculum
for

CHC
staff,
and
provided selected training on data use
.


The background work has been done but implementation of the management component of the
HMIS has been delayed, as



The 127 national indicators have not been calculated for TNHD at any level.



T
here i
s little clarity on roles and responsibilities of CHC and district in analysing data,



N
o feedback process
es

have been established.


This is an area of action research that will need to be done as a matter of priority and will
potentially have great impo
rtance for national HMIS rollout.

Evidence
-
based unit

The establishment of an “Evidence Based Unit” to improve the use of evidence at the provincial
and district levels has been proposed, but not implemented
. EBU

staff
are

allocated at
the
provincial leve
l.


The approach to training for analysis and utilization of data at CHC has, as a result, been largely
theoretical as

1.

T
here is no local data available in an appropriate format to calculate even basic
indicators.

2.

Training of trainers has not
yet
occurred at district and provincial levels and there is no
effective analysis of data at these levels.

3.

There is still no clarity on institutional or individual roles and responsibilities for data use
at any level. This needs to be clarified before trainin
g can be effective.

Research

There are a number of interesting questions to be asked for potential research, mainly around
selecting

the most appropriate indicators for use at each level and defining roles and
responsibilities for data use at the various levels.


While it is still too early to recommend approaches for the provincial government to commission
research for planning and ma
nagement or for use of this system for pilot
-
testing m
-
health
initiatives with village health workers, some potential research questions could be

1.

How can the logbooks and forms be simplified to fit the 127 national indicators?

2.

What are the roles and respo
nsibilities of the various levels in health management?

3.

How can data quality and timeliness be improved?

4.

How can the CHC and hospital systems be functionally linked?

5.

How can
mH
ealth be used to collect individual patient data at the community level?


CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







15

of
24


Conclu
sions

The project is conceptually sound and fills an important gap in the development of a simple,
integrated national HMIS in Vietnam.


The existing software modules for medical examination and population work reasonably well
and could easily be made con
siderably better by constructive interaction with the end users at
CHC level. The web
-
based software design is appropriate and
can be easily adapted into a
standardized national approach to using HMIS at the primary health care level. This is
predicated on

the availability of a skilled software developer. The current relationship with HSP
has not been without problems.



At the moment the CHC software is not compatible with the district hospital system.

The software
will need to be better documented and t
he source code made publicly accessible
to satisfy the basic requirements

of open source software
.
The module development timetable
and release of version 2 has been seriously delayed.


Moving forward with the software’s development will require stronger t
echnical project
management of the software developer. It could make sense to divide the responsibilities of the
current software developer.

Currently, the software developer has designed, developed, and
maintained the software. This does not need to be

the case. In fact, it may be desirable to split
these roles. In either case, clearer guidelines should be developed regarding responsiveness
and expected timelines.


The approach to capacity development for analysis and utilization of data at CHC, dist
rict and
provincial levels is generally appropriate, as are teaching methods for CHC staff to use the
computerized HMIS.

District and provincial level training in data use for planning and
management has not been implemented but the needs assessment and cu
rriculum
development has been done. The formation of the evidence based unit is probably premature
as there is no information to use for evidence, but the idea is a good one, fits well with national
plans and should be supported once it is clear what roles

and responsibilities are.


Future action research should be commissioned, in collaboration with the HMIS working group
and provincial management on

1.

Simplifying the data collection, based on a streamlined set of national indicators

2.

Strengthening HMIS
organisational structures, roles and responsibilities of the province
district and CHC

3.

Linking information generated to management and evidence
-
based action.

4.

Use of mobile technologies for community level data collection and use


The major weakness of t
he project is organisational, in that

1.

T
here is little buy
-
in from provincial management

2.

T
he links between the project and MoH headquarters structures have not been made, so
that the project is not linked to the bigger picture national HMIS rollout to ens
ure its
sustainability.

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







16

of
24


Recommendations

Recommendations fall into two groups
:


1.

Immediate recommendations to be implemented before the end of AP4 project revolve
mainly around identifying a successor to PC and getting version 2 out, with all the
software

modules up and running in all CHCs

2.

Longer term recommendations for the next phase revolve around closer linkage to the
NHDSP, project management, software design, data use and capacity development.

1.
By AP4 conclusion (September 30th)

Suspend the roll
-
o
ut of version 2 of the software until the codebase is sufficiently
polished for public release.

Recommendation 1:

Atlantic Philanthropies should identify a successor to Population Council
as soon as possible (August 1
st
) and work closely with them and th
e MoH in planning the new
project, to make the project attractive to the MoH and to ensure a smooth transition to the new
project.


Recommendation 2:

HSP must finish version 2 of the HMIS software by
August 15, 2012
.
Completion is defined as finishing al
l 7 modules. Major bugs must be fixed. A bug is defined as
system behaviour that deviates from specifications.

Example: Drug inventory tallies must always be correct if not attributable to human error.


Recommendation 3:

Conduct follow
-
up software t
raining of provincial and district IT staff
(originally done in October 2011) to enable improved local support to the CHCs by
August 1,
2012
.


Recommendation 4:

Finished version 2 of the software

should be rolled out to all 181

CHCs by
Sept 15, 2012
.
Provincial and district IT teams should lead the roll
-
out of version 2 with support
from HSP.


Recommendation 5:
Implement a help desk ticketing system for tracking bugs and feedback
from all the key stakeholders from CHC staff to Population Council by
Jul
y 31, 2012
.


Recommendation 6:

Develop a set of metrics with provincial and district staff to assess when
conversion to digital system is complete and when to discontinue use of the paper system by
July 31, 2012
.


Recommendation 7:

Migrate all code and associated software components to a Population
Council or Provincial Health Department controlled server by
August 15, 2012
with periodic
updates until project end.


Recommendation 8:

HSP must upload all code and associated softwar
e components to a
Provincial or Population Council controlled GitHub account (or similar system) by
August 15,
2012
with periodic updates until project end.


Recommendation 9:

Population Council should hire a consultant to document all aspects of
the so
ftware architecture, implementation, and functionality with a view to making the system
open source. This should be completed by September 15, 2012.

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







17

of
24



A provisional GANTT chart shows the proposed timeline



July

August

September


Rec

1

2

3

4

1

2

3

4

1

2

3

4

1













2













3













4













5













6













7













8













9














2.
Next phase

The project should be continued, consolidated and sustained for a period of at least 5 years (or
to
the end of APs mandate) to have a meaningful and sustainable impact on national HMIS
development. The gains in the province of the past three years in terms of computer capacity
development at CHC level, the initiation of a web
-
based information system bas
ed on electronic
patient records and the computerisation of 8 reporting modules should be built on and included
in the national HMIS debate and activities.



CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







18

of
24


The new project should be designed for sustainability, with more provincial and district
manageria
l involvement, with national scaling in mind
. A
ll
subsequent
activities
should be
carried out in terms of the
National Health Sector Development Plan
6

(
N
HSDP)
, in collaboration
with the MoH Partners working group on HMIS. The software used should be open
source,
developed by ongoing iterative involvement with end users, scalable to be a national data
warehouse,
and
usable for both individual and aggregate data.

MoH Alignment

The next phase should be conducted as an integral part of the
N
HSDP

which is based

on the
objectives of the health sector, focusing on improving access to the health service to
disadvantaged and remote areas, improving the quality of health services and reducing hospital
overloading. The plan is designed to ensure universal health insur
ance, enhance human
resource development, reform working and financial mechanisms and improve performance of
public health facilities.


For the Health Management Information System, the HSDP vision includes
:


1.

Refining the health indicator system, log
books, records and statistical reports,
guidelines, hospital information, preventive medicine and epidemic control information,
and information relating to teaching and studying…

2.

Improving the capacity for data aggregation, analysis and processing.

3.

Develo
ping a mechanism for information sharing and feedback.

4.

Modernizing the HMIS in appropriate manner to the financial, technical capacity and use
needs at each level

5.

Improving information dissemination by diversifying formats and ensuring
appropriateness to t
he users.

6.

Increasing use of information for direct management at each unit, by each level which
supplies information.


The HMIS project activities should support National Target Programs (NTP) on health, Food
safety and hygiene, Family planning, HIV/AIDS p
revention as well as MoH priorities such as
Human Resources for Health, Medical Equipment, Health financing and Management capacity
strengthening.

Project management

The focus should be on providing a sustainable and integrated HMIS model for scaling up t
o
national rollout (See
N
HSDP) in collaboration with the MoH. This will include



T
he Health Partnership Group technical working group on HMIS,



MoH units involved in HMIS such as Departments of Planning and Finance (DPF),
Department of science and training

(DST) and Medical service administration (MSA)



National HMIS projects (e.g. EU, WHO, WB, UNFPA, GIZ).



The Administration for Technology application, when established (planned for
September 2012) and the proposed Evidence Based Unit (EBU)



There shoul
d be a continuity of project activities between the phases. A new implementing
partner should be identified well before the end of the AP4 project (August 1
st

2012). There are
many good partner options with the requisite MoH relationships and understanding of the local



6

Vietnam Government 5 year Health Sector Development Plan 2011
-
2015

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







19

of
24


context, but what is needed is a partner with strong HMIS technical project management and
design skills. This should not be limited to local
partners.

The chosen software provider should
be able to provide
a regular technical presence in the
province to ensure institutionalisation of the software and local ownership of the products by
EBU and the IT team. The partner should also have strong ma
nagerial and public health links to
the MoH, Health Partnership group HMIS TWG and other national HMIS projects.

Software

Software design should ensure scalability within MoH reporting and data use guidelines. The
software design should contribute towards
the national vision of a national database and also
ensure

1.

Continuity throughout the health information cycle (collection, processing, analysis,
interpretation, dissemination, use).

2.

Interoperability across levels of the health system, including district hospitals and the
community.

3.

Flexible dashboard for easy use by program, district and CHC managers

4.

Support for individual patient management, facility management as well as systems
m
anagement.


Separate the functions of the technology partner into design, development, and support.

An identification card based on the CHC unique identifier should be produced. This process
should happen CHC level. It also should be interoperable with

the district hospital information
system in a functional way.

Data use

The management and data use component should be intimately linked to and support national
HMIS developments in the HSDP outlined above. Focus should be on

1.

Institutionalisation of HMIS

by

a.

S
treamlining of data collection tools in line with the national indicator set

b.

D
efining roles, functions and procedures for each level

c.

D
efining products for decision making, monitoring and evaluation at all levels

2.

Capacity
d
evelopment to fill the abov
e roles and necessary skills for analysis and
interpretation, dissemination and stimulating the culture of information


This can be ensured by hiring a strong public health professional to work closely with provincial
level to develop and support evidence
-
based decision
-
making systems and to feed best
practices from the province into the MoH HMIS technical working group and other HMIS
development processes.

Capacity Building

Capacity building activities should build on current experiences and be driven by

a strong
provincial Evidence
-
Based
U
nit, supported by training and research partners at national and
provincial level. This will require increased involvement of MoH in HMIS at all levels from
province to the CHC. Focus should be first on motivating and s
upporting evidence
-
based
decision
-
making at the various levels, based on the national indicators, MDGs, and other
national and local priorities in the HSDP and linked to establishment of clear roles and
responsibilities for the HMIS at all management level
s.

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







20

of
24


Appendix A: Terms of Reference for evaluation

IMPROVING PRIMARY HEALTH CARE IN THAI NGUYEN PROVINCE:

INTERVENTIONS TO IMPROVE DATA, EVIDENCE AND CAPACITIES



Purpose:


The purpose of this evaluation is to guide the project, “Improving primary health care in Thai
Nguyen Province: Interventions to improve data, evidence and capacities” in improving its
design and implementation, and in identifying future directions after t
he project ends, especially
with reference to the Health Management Information System (HMIS).


Objectives:


1.

To determine the extent to which the goals and objectives of the project have been
accomplished.

2.

To consider and discuss the likelihood that conti
nuation along the lines currently
undertaken will result in a model of evidence
-
based decision making that will be of
national value in Vietnam.

3.

To recommend appropriate directions that might be taken in a follow
-
on project to
continue development of the p
roject’s aims.


Background


The Population Council and its partners are in late stages of implementation of a project funded
by Atlantic Philanthropies (AP) entitled, “Improving Primary Health Care in Thai Nguyen
Province: Interventions to Improve Data, E
vidence, and Capacity.” The project was intended to
last for three years beginning in March, 2009; the time period has been extended until 30 June,
2012, and a further extension to 30 September, 2012 is anticipated.


The goals and objectives of the proje
ct, as stated in the proposal of January 20, 2009, are as
follows:


Project goals

1.

To strengthen the Thai Nguyen primary health care system by improving access, equity,
quality and utilization of services, especially among the poor, ethnic minorities and
re
sidents of remote areas.

2.

To develop and strengthen the culture and practice of using evidence to guide health
policy, planning and management in Thai Nguyen province.

3.

To contribute to global knowledge and experience on the implementation and evaluation
of
evidence
-
based approaches.


Project objectives

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







21

of
24


1.

To develop and maintain an effective and sustainable HMIS model in the province,
especially at commune level.


2.

To develop a critical mass of skills, capacities and motivation:

a.

Within TNHS leadership and staff
to analyze and use HMIS and related data as
evidence for planning, policy and management.

b.

Within TNMC (now TNUMP) faculty to teach and conduct research using
Evidence
-
Based Planning, Policy and Management (EBPM).




3.

To test the initial efficacy and sustain
ability of a distinctive, integrated, province
-
led
model of EBPM.


Considerable progress has been made towards these goals and objectives. Regarding the
HMIS, a system core has been developed, first using MYSQL and then using Oracle as the
base, and the k
ey modules required in the contract are in place: population management,
patient management, drug management, materials and equipment management, human
resources, and preventive medicine. A computer has been installed in each of the 180
communes in Thai Ng
uyen, the software installed in them, and two staffs from each Commune
Health Center (CHC) trained in its use. IT staffs at provincial and district levels have been
trained to support the system. It is expected that by the middle of 2012, all reports from
“Commune Health Statistics” (i.e., the 8 BXC reports) will be entered into computers at all 180
CHCs.


Meanwhile, as part of the broader objective of “evidence
-
based program planning and
management”, other partners have carried out important health researc
h in Thai Nguyen
province, and Hanoi Medical University (HMU) has embarked on a program to build the capacity
of health workers in the province to understand and use data from both the HMIS and the field
research. A total of 21 papers have been written or
are in preparation regarding various aspects
of health in Thai Nguyen by TNUMP, PHAD and PC. An “Evidence
-
Based Unit” has been
created within THNS, and with research and HMIS data in hand, a process of training is
underway to provide capability and experie
nce in using data for management at the provincial
level. Meanwhile, curricula have been developed and tested to strengthen the ability of staff at
CHC and district levels to use data for program management and decision making at their
levels.


Evaluation
timing and team:


The evaluation will take place during a one
-
week period of field observation in June, 2012, with
additional time required for preparation and report preparation. The evaluation team will consist
of two independent consultants, one with a
strong background in HMIS software, the other with
a strong background in information for health management. Observers will join the team from
time to time from the Population Council (PCVN, the lead agency in the present project) and the
Institute for Pop
ulation, Health and Development (PHAD, the agency expected to take the lead
role in a follow
-
on project), the Thai Nguyen Health Department (TNHD) and if possible the
Ministry of Health (MOH). PCVN will provide secretariat support for the consultants.

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







22

of
24



A
ctivities:


The team will accomplish its mission through the following activities:


1) Read and review available materials on the project, as provided by the project, prior to joining
the evaluation team.

2) Meet with representatives of each of the project

partners:

a) Population Council, Vietnam (PCVN), in Hanoi

b) Thai Nguyen Health Department (TNHD), in Thai Nguyen

c)

Institute for Population, Health and Development (TNHD), in Hanoi

d) High
-
Technology Services Providing and Solutions Company (HSP), in H
anoi

e) Thai Nguyen University of Medicine and Pharmacy (TNUMP), in Thai Nguyen

f) Hanoi Medical University (HMU), in Hanoi

3) Over two or three days, visit the project activities at commune, district and provincial levels in
Thai Nguyen.

4) Communicating

in person or otherwise as appropriate, develop conclusions regarding the
objectives of the evaluation.

5) Prepare a report on the evaluation.

CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







23

of
24


Annex B: Evaluation Activities


Monday, 25 June 2012




Meeting with Dr Hau


AP
,
#31, Alley 151B, Thai

Ha St, Dong Da Dist




Meeting with Duc


HSP
,
1014 Hoang Quoc Viet, Ha Noi, Viet Nam




Meeting with Dr Hoi


HMU
,
1A Ton That Tung, Dong Da, Ha Noi





Tuesday, 26 June 2012




Visiting Ba Hang CHC, Ph
i

Yên District
,
Ba Hang town, Pho Yen Dist



Visiting C

i Đan CHC, Sông Công Town
,
Cai Dan commune, Song Cong Town




Visiting Tân Th

nh CHC, Thái Nguyên city
,
Tan Thinh CHC, Thai Nguyen city





Meeting at TNHD office (with TNHD staff + EBU members
)
,

27 Ben Tuong, Thai Nguyen
city




Meeting with TNUMP 284 L
ươ
ng Ng

c Quyen





Wednesday, 27 June 2012





Visiting Hung Son CHC, Dai Tu district
,
Hung Son commune, Dai Tu District




Visiting Dai Tu CHC, Dai Tu district
,
Dai Tu

town, Dai Tu district




Visiting Dai Tu DHC





Visiting Dai Tu District General Hospital




Visiting Binh Thuan CHC, Dai Tu district
,
Binh Thuan commune, Dai Tu dist.




Visiting Luc Ba CHC, Dai Tu district
,

Luc Ba commune, Dai Tu dist.



Thursday,
28 June 2012





Visiting Phu Binh General Hospital
,
Huong Son town, Phu Binh dist.





Visiting Phu Binh DHC




Visiting 1 CHC in Phu Binh district
,
Phu Binh dist.




Demo of DHIS2 by HISP Vietnam




Follow
-
up meeting with HSP at Hanoi Towers


Friday, 29 June 2012




Feedback to Population Council



Feedback to Dr. Phuong, Atlantic Philanthropies
-

Vietnam



Meeting with Dr. Liam, Population Health and Development Institute (PHAD)



Meeting with Dr Bao


Pathfinder International

Kim Ma T
huong street, Ha Noi


CHC HMIS Evaluation


June 2012

Arthur Heywood and Mahad Ibrahim







24

of
24


Appendix C: Questions for evaluation team


On the software:

1. Is the basic design of the software, i.e., the core
-
cum
-
modules with interface, appropriate
for the purpose, notably given the long
-
term aim of making th
is the basis of a national system?
Would it be better to develop or apply some other system?

2. Given the structure of the district hospital HMIS in Thai Nguyen, is it feasible within a
follow
-
on project to effectively link the commune
-
based HMIS to th
e district hospital system, and
to the public health system in Thai Nguyen as a whole?

3. What steps should be taken to make the project system genuinely “open
-
source”, i.e., not
only legally available to all, but effectively documented and usable?

4.

Do the consultants recommend continuing with the same software supplier and structure,
or should a change be made?

5. What recommendations can be made to make the existing software more efficient?



On utilization and management:

1. Is the system
of data entry screens and feedback reports on the right track? What changes
should be made?

2. Is the approach to training for analysis and utilization of data at CHC, district and provincial
levels appropriate?

3. Specifically, what teaching methods are
appropriate to

a. lead the CHC staff to use the computerized HMIS to improve health care?

b. for district and provincial level staff to learn to use the data for planning and management?

4. What approaches/techniques are advisable to improve data
quality?

5. What would the evaluation team recommend regarding the establishment of an “Evidence
Based Unit” to improve the use of evidence at the provincial levels?

6. What approaches do the evaluators recommend to enable provincial governments to
commiss
ion research for planning and management?

7. What advice to evaluators have on ensuring appropriate levels of privacy and confidentiality,
given that individual level data will be put on line?

8. One possibility going forward is to use this system as a
platform for pilot
-
testing m
-
health
initiatives at the sub
-
commune level, e.g., with village health workers. Can the evaluators give
us advice/perspective on this?

From Dr. Hau, Atlantic Philanthropies

1. The applicability of a number of studies conducted
within the project term. Are they relevant
to TN DoH's health management practice?

2. As a result of that switching, how satisfied do CHC staff and TN DoH feel about the software
program? How do the software programmer think about the investment into the p
rogramme
development?