Output 1: 3000000 people reached with hygiene promotion activities

flashypumpkincenterSoftware and s/w Development

Dec 14, 2013 (7 years and 7 months ago)



Type of Review: Annual Review

Project Title:Zambia Sanitation and Hygiene Programme

Date started:

22 Nov 2011

Date review undertaken:

29 Nov

Instructions to help complete this template:

Before commencing the annual review you should
have to hand:

the Business Case or earlier project documentation.

the Logframe

the detailed guidance (How to Note)

Reviewing and Scoring Projects

the most recent annual review (where appropriate) and other related monitoring reports

key data from ARIES,
including the risk rating

the separate project scoring calculation sheet (pending access to ARIES)

You should assess and rate the individual outputs using the following rating scale and
description. ARIES and the separate project scoring calculation sheet

willcalculate the overall
output score taking account of the weightings and individual outputs scores:



Outputs substantially exceeded expectation


Outputs moderately exceeded expectation


Outputs met expectation


moderately did not meet expectation


Outputs substantially did not meetexpectation


Introduction and Context

What support is the UK providing?

DFID is supporting UNICEF with a grant of £19 million from November 2011 to March 2015 to enable 3
million people in rural districts of Zambia to use improved household toilets and to practice hand
washing with soap or ash thereafter.

The Zambia
Sanitation and Hygiene Programme includes a number of mutually supporting
components, these being:


ity Led

Total Sanitation (CLTS)
The approach focuses on convincing communities to end
open defecation through generating a sense of disgust and shame, and mobilising collective action to
build and use toilets. CLTS is widely seen as a key component of effective sanitation programming at
scale sufficient to reach MDG targets. The approach adopted by UNICEF is to train and use voluntary,
based facilitators (‘Sanitation Champions’) to engage with communities, reporting to a
District Technical Team (with four Government staff from

different departments) and District Councils.

Legal Enforcement (LE)
: In towns and peri
urban areas within the project area, CLTS is
complemented with the legal enforcement of Public Health Legislation, as it applies to public premises
(for example, res
taurants, hotels, bus stands, markets, colleges and government offices) and/or the
private sector (for example, factories and farms). The legislation obliges the owner or responsible
organisation to provide appropriate toilets and hand
washing facilities.T
his activity is being undertaken
by Environmental Health Technicians, reporting to the District Health Officer.

Sanitation Marketing
: building the capacity of the private sector to respond to increasing demand for
sanitation, and help households to improv
e their latrines. This is potentially very important, as the MDG
and DFID support is based on ‘improved’ rather than ‘basic’ toilets. The former include a cleanable
slab, and a closely fitting lid over the drop hole, as well as a hand
washing facility such

as a ‘tippy tap’.

School sanitation and hygiene

designed to establish appropriate, gender sensitive and child friendly
sanitation facilities in 1,000 rural schools, complemented by a hygiene promotion component and the
based management of the inf
rastructure created.

National hygiene promotion campaign

involving both inter
personal communication and mass
media focusing on hand washing with soap or ash after visiting the toilet, before handling or consuming
food, and after cleaning an infant.

Monitoring system

to efficient

and cost effective
track project performance and
inform reporting. UNICEF’s proposal specifically details the use of mobile phone based technology to
support the monitoring system.

UNICEF is being supported by

DFID to undertake the Zambia Sanitation and Hygiene Programme
because of its previous engagement in sanitation in Zambia, with interventions supported in 20
districts. UNICEF also successfully piloted the introduction of a Zambian adap
tation of CLTS in Ch

Province. The project builds on this foundation.

What are the expected results?


of the project will be reduced diarrhoea morbidity amongst rural children under 5 from 15%
to 12%. This amounts to 14,300 fewer cases of diarrhoea in children each year. Impact will be measured
through an Impact Evaluation due to report in 2015


will be 3 million additional people consistently using improved household toilets and
practicing hand
washing with soap or ash. The result will be measured using a project specific
performance management system.

The four project

are: (i) 3,000
,000 people reached with sanitation and hygiene promotion
activities (50% weighting); (ii) 1,000 schools and health centres have appropriate sanitation and hand
washing facilities, with hygiene promotion and management systems (20% weighting); (iii) nati
provincial and district level administrations have capacity to plan, implement and monitor sanitation
promotion (20% weighting); and (iv) the private sector in 74 rural districts is enabled to supply sanitation
and hygiene services (10% weighting).


The Zambia Sanitation

and Hygiene Programme is an
important element of DFID’s global support to
household sanitation, noting that DFID has championed the cause of sanitation in recent years.
3,000,000 users amounts to 14.6% of DFID’s global result for 2014/2015 (20.5 million people).
Zambia sanitation

programme is second only to the sanitation programme in DRC, and is larger than
DFID’s sanitation programming in Nigeria, Bangladesh and India.

What is the context in which UK support is provided?

Estimates for rural sanitation coverage in Zambia vary significantly, relating to differing definitions,
sampling methodologies and how data are analysed. The Joint Monitoring Programme (JMP)
estimates rural sanitation coverage to be around 43% (the propor
tion of rural population using
improved toilets), based on an extrapolation of qualifying data points. To date, this proportion has been
increasing at around 1% a year, taking into account population growth.

JMP also indicates that 26% of the rural popu

or 2.2 mi
llion people, practices open def
Based on JMP data, to meet the MDG sanitation target, an estimated
3 million people will
need to be using improved sanitation by 2015. Like most countries in sub
saharan Africa, Zambia is

on track to achieve this target. UK support is designed to ensure that the sanitation target will be met
by 2015.

Compared to sanitation, even less is known about hand
washing practices in Zambia. Anecdotal
evidence suggests that hand
washing with s
oap or ash after using the toilet is rarely practiced. The
status of hand
washing is being investigated as part of the project’s baseline, the information collected
will be used to inform the national hygiene promotion strategy that forms part of this proj

Poor sanitation and hand
washing practices are major contributors to water borne and faecal
disease, especially diarrhoeal disease and acute respiratory infection. Children under
five are
particularly vulnerable. Ensuring access to appropriate
household sanitation also has far
consequences for the quality of life of women and girls in particular, well beyond any health impact,
given their specific needs for privacy and security.

The inclusion of institutional sanitation and hygiene

in the project (specifically sanitation and hygiene
promotion in rural schools and health centres) is also important for a number of reasons. Firstly,
schools and health centres, especially primary schools, are important parts of rural infrastructure.
ieving total sanitation coverage in a community is anachronistic if the local school lacks adequate
facilities, for example. Secondly, school children can be mobilised as important change agents in a
community led total sanitation programme

but they are
more effective if they have access to
sanitation and hygiene in their school. Lastly, although more robust evidence is needed, school
sanitation and hygiene is likely to have a significant impact on the quality of the learning experience,
especially for ad
olescent girls.

In supporting the achievement of the sanitation MDG target in rural areas, the Zambia Sanitation and
Hygiene Programme is filling a critical programmatic gap in the Government’s National Rural Water
Supply and Sanitation Programme (NRWSSP)
. Sanitation and hygiene form a component of the
NRWSSP, but to date have received less priority and attention, certainly compared to the development
of new water supplies. The programme also fills an associated gap in the funding provided by WASH

Partners (CPs). WASH CPs have tended to fund water supply activities, rather than
sanitation and hygiene.


Section A: Detailed Output Scoring

Output 1: 3,000,000 pe
ople reached with

hygiene promotion activities

Output 1 score and performance description:
Output moderately did not meet expectation

Progress againstexpected results:



Milestone Dec 2012

Results Reported

1.1: Number of people reached
with hygiene promotion
activities undertaken

communities, focusing on the
importance of using latrines and
washing with soap or ash


350,000 people

(to be corrected to
650,000 people

see note below)

323,000 people

(to be corrected to
830,000 people

see note below)

1.2: National
sanitation and
hygiene behaviour change
communication strategy and
plans established with Ministry
of Local Government and
Housing (MLGH) and
implemented, with both
community level interventions
and mass media components

Strategy, and
Annual Plan for
, agreed with
including MLGH

Draft strategy
prepared by
section, currently
being circulated for
being discussed with

1.3: Number of verified Open
Defecation Free (ODF) villages
achieved through DF
ID support

1,000 villages

1,500 villages (ODF
reported by
Champions but not


The milestone and results reported against output indicator 1.1 actually refer to number of
additional people with improved sanitation (currently an outcome level

indicator). These are
corrected to reflect people reached with hygiene promotion based on the numbers of villages
triggered (8,300) and an estimated average village population of 100.

Additional Comments

320,000 people using improved sanitation, or
92% o
f the annual target, is an estimate based on data
received from Champions in 27 out of 32 districts using an interim paper
based reporting system that is
entered into a spread
sheet at district level. Averages were used by UNICEF to estimate change in the
reporting districts.

Noting that these results have been reported in 8,300 triggered villages, it is surprising to see

that of
these, only 1,500 or 18% are reported as ODF, even taking into account the fact that in some areas,
the process has
been initiated two months ago. This sugg
ests the urgent need for follow
up by
Champions in the targeted villages.

Several fiel
d visits, undertaken by DFID associated with the Annual Review and by two consultants,
engaged by UNICEF to support the
recent mid
term review of the NRWSSP, suggest that until
recently, most district WASH staff did not have the resources to check this dat
a. Furthermore,


sanitation champions seem to lack encouragement and support from their Districts, whose ownership
of the programme seems weak. Furthermore, evidence from the these visits as well as the WASH
Bottleneck Analysis, supported by UN
CEF, suggest
s that Districts’ involvement is limited to the Local
Council, rather than involving a broader range of important stakeholders who have a related mandate
and capacity.

Theseissues need to be addressed by UNICEF to ensure that the Champions’ interest (and t
impact in targeted communities) is sustained, and does not fade. We understand from UNICEF that
these concerns are shared, and that a number of corrective actions are already being taken, including
the transfer to funds to districts for monitoring vis

Despite these concerns, it should be noted that this activity only started in April 2012, and that the
results have been achieved over an eight
month period.

Compared to sanitation and hygiene promotion by Champions, rather less has been ach
ieved in t
of the National Hygiene Communication S
trategy. This is critical, given the fact that few people seem to
practice hand
washing with soap or a
sh after using toilets, and the fact that CLTS, as an approach, is
designed around toilet use rather than hand
washing. It should be noted that UNICEF previously
agreed to prioritise this issue in Quarterly Review Meetings on June 29

and October 5

2012. It is
imperative that the strategy is developed by UNICEF with Government, and subsequently owned and
ented by Government with UNICEF support.

In terms of the numbers of ODF villages being reported, it is critical to note that none of these are in
fact verified, as required by the indicator. A robust verification system, based on a protocol and
guidance on its application, is an essential part of

any CLTS strategy. It has, to date, not been
established in Zambia, plans to outsource this to an INGO having been shelved due to a number of
issues including cost. Relying on reports from CLTS Champions, who have a clear incentive to report
success, ma
y lack credibility. Without a common approach to measuring ODF, there is also risk that
different Champions will apply different criteria. Furthermore, it makes it difficult for Government to
recognise ODF (which may or may not have been achieved). It is u
nderstood from UNICEF that work
on an ODF protocol and guideline begins in early December. However, the lack of the verification
system at this stage in the project is a critical deficiency, and a major contributing factor to the ‘B’ grade
awarded against
this output.


UNICEF should continue to build
District Capacity

to lead and support the programme, focusing
on the 32 1

and 2

wave districts, delaying further expansion until March 2013. UNICEF should
ensure that Champions receive the

support that they need to sustain and scale up CLTS

with more
emphasis on post triggering
, and that District staff are able to support the process,
monitor progress, and report resultsand challenges to

and UNICEF (see related
s associated with Output 3).

UNICEF should prioritise the development and roll out of an effective
hygiene communication
strategy and plan
, focusing on hand
washing, supporting changes in practice being achieved
through CLTS, with an appropriate mix of i
personal and mass media communication.The
strategy and related plans should be agreed with

and other stakeholders by 31 March 2013,
with implementation starting straight away. The strategy and plans can later be revised taking into
account the re
sults of formative work undertaken as part of the Impact Evaluation.

UNICEF should work with

and other stakeholders to develop, agree and establish an
effective strategy and protocol for
verifying ODF status
, and guidance on its use. The strategy,
protocol and guidelines need to be in place by 31 March 2015, given the numbers of ODF villages
being reported and the fact that the Programme is being expanded at this time to a further 30 (or
more) districts.

The logframe will be revised to reflect corre
cted milestones and achieved for the indicator on
number of people reached with hygiene promotion’. The outcome level indicator on number of
people with sustained access to improved sanitation will be moved to output 1 to ensure frequent


monitoring. An a
dditional indicator on access to handwashing facilities will also be added. This is
further reflected in a revised output of ‘3,000,000 with sustained access to an improved sanitation

Impact Weighting

Revised since last Annual Review?
This weighting has been revised from 40 to 50% to reflect

the fundamental importance of th
is o

and ensure weightings for all outputs add to 100%.

This assessment is based on the current status of this output, taking into account the

various recommendations made above. Should these recommendations not be translated into practice,
the risk rating
become High.

Revised since last Annual Review?

Output 2: 1,000 schools and health centres have appropriate sanitation facilities
including hand
washing facilities and soap available, with hygiene promotion and
management system

Output 2 score and performance description:

Outputs substantially did

not meet expectation

Progress against expected results:



Milestone Dec 2012

Results Reported

2.1: Number of schools and
health centres that have
appropriate sanitation facilities
including hand
facilities, together with an
appropriate management
system and hygiene promotion


Note: Process of
reviewing and
updating minimum
standards and
designs should be
initiated by UNICEF,
Government and
Partners by
December 2012


No progress on the
review and updating
f minimum


The review of the logframe completed in September 2012 added a number of caveats to the indicator

the original indicator in the business case simply referred to ‘the number
of schools and
health centres that have

latrines, hand
washing devices and soap availability’. The addition of the term
‘appropriate’ is important, referring to the user/toilet ratio and gender sensitive, child friendly designs.

Existing Ministry of Education (MoE) Guidelines are widely considered to be in need of change, not
least because the technical specification seems over
and expensive. In particular, there are
concerns about whether (i) a 20
year fill time for a pi
t and (ii) 1 to 20 (drop holes to students) ratio cited
by MoE are fully justified
. UNICEF and USAID report that a double ‘VIP’ toilet block of 2stands costs
about £2,000, excluding hygiene promotion. This implies that sanitation facilities for a typical

school of
500 students and 5teachers, running a doubleshift system, would cost about £28,000.

UNICEF has to date found it difficult to persuade MoE to review its designs in this context, and there
are indications that

may still go ahead and use the

existing design, resulting in schools


By comparison,
WHO advises ratios of 1:25 for Girls; 1:50 for boys, plus the provision of urinals for boys. Pit fill time depends
on many factors but is typically between 5 and 8 years, at which point they must be emptied.


receiving a fraction of the infrastructure required given its cost at the budget available. Whilst it is
recognised that many schools run a shift system, and many have existing, albeit inadequate sanitation
, the impact is likely to be negligible in terms of the learning experience and health gains.
Inadequate numbers of toilets may result in a net health risk to students, as well as establishing or
reinforcing a negative impression of sanitation and hygiene
that could be transferred to the home

Taking into account that UNICEF, DFID and USAID (which is also supporting School WASH as part of
its Education Programme) all face the same challenge,
that these three organisations account for
majority of investment in School sanitation in Zambia, DFID recommends that UNICEF continues to
negotiate wi
th MoE to revise its standards. T
he focus
should be
the development of an interim
standard with an averagecost of less than £ 12 per student (exclud
ing hygiene promotion and the cost
of establishing a management system). This is in line with the Business Case and UNICEF’s
proposal.In this respect, the CLTS approach being rolled out by UNICEF may be linked to an interim
school sanitation design that is

largely based on traditional concepts, local materials and self

The recommendation was made in the June Quarterly Review and reiterated in the October Review.

If standards cannot
revised, it is estimated that UNICEF would only be able to achieve
no more th
% of the target of 1,000 schools

In these circumstances, the component should be dropped on the
basis of its limited impact and cost

In general, it is also considered that UNICEF should focus the institutional sanitation component on
schools rather than health facilities, which are fewer in number and which require relatively simple and
affordable interventions which can be afforded by
the Ministry of Health and its CPs.


UNICEF should focus its interventions on institutional sanitation on schools rather than rural health
facilities. The logframe should be amended accordingly.

UNICEF should scale up its advocacy wit
h MoE and other stakeholders to conduct a review of
School WASH standards and guidelines, working in close consultation with USAID and DFID. The
target is to establish revised standards, with an average unit cost of £12 per student for toilets,
urinals and

washing stands, by mid 2013. Whilst this unit cost seems very low in comparison
to the current standards (around £55 per student), it should be taken into account that many
schools have sanitation structures which can be rehabilitated. In the meantim
e, school toilet
construction using the current standard (the 1:20 ratio and 20 year pit life cited by MoE) and
supported by DFID should be suspended.As part of this process and to inform advocacy, UNICEF
should demonstrate a cost effective approach to sch
ool sanitation, based on its work on community
based approaches to sanitation, by mid 2013.

The logframe milestones will be updated to reflect the above.

If standards are not revised in line with these recommendations, DFID will drop this component of
e project
and re
programme the funds

Impact Weighting (%):

Revised since last Annual Review?

. R
elates to the possibility of MoE not approving a more affordable design by the
2013, at which point this component of the project

will be dropped and the funds reprogrammed by

Revised since last Annual Review?


Output 3: National, provincial and district level administrations have capacity to plan,
implement and monitor sanitation promotion

Output 3 score and performance description:

Outputs substantially did not meet expectation

Progress against expected results:



Milestone Dec 2012

Results Reported

3.1: Number of districts
implementing Total Sanitation
and Hygiene Plans


18 plans

0 plans

3.2: Number of National
Sanitation Technical Working
Group Meetings convened by

with CPs participating

3 meetings in 2012

2 meetings held in

3.3: Sanitation and Hygiene
Performance Monitoring
System established as a
component of Government’s
WASH Information
Management System

Interim PMS
established and
used to report CLTS
results in 2012

Interim PMS
developed and
partially successful


This output is closely related to Output 1 that is focused on the roll out of CLTS. Whilst progress
against Output 1 merits a ‘B’, the re
sults against this output merit

a ‘C’. This implies that more needs to
be done to develop the capacity and systems n
eeded to scale up and sustain CLTS whilst maintaining
a minimum level of process quality

District Total Sanitation and Hygiene Plans (DTSPs), involving inputs from all major stakeholders at
District level including a number of different departments, led b
y the District or Municipal Council, are
increasingly viewed as critical components of CLTS. Responsibilities for sanitation and hygiene are
distributed amongst a range of stakeholders, not only involving the local Council. The inclusion of
these stakehol
ders in related planning can mobilise considerable technical resources as well as funds
and transportation. It also reduces the risk of contradictory approaches being implemented in the same
district, for example, related to hardware subsidies. Whilst guid
ance for DTSPs should be provided by
, to date little progress has been achieved. UNICEF is however well
positioned to pilot the
approach in a number of Districts to inform national guidelines.

UNICEF is also well
positioned to support

to conven
e Sanitation Technical Working group
(TWG) Meetings. Two such meetings have been convened in 2012. The first, established by UNICEF,
was to present approaches to sanitation marketing. The second was called by government to inform
the Mid Term Review of the

NRWSSP. Neither are in this sense ‘regular’ meetings, and the TWG has
still to be institutionalised. It should be stated that the role of UNICEF in this respect

supporting Government by for example offering to

an effective secretariat

eporting results to Districts, Provincial Government,
, CPs and amongst the latter, DFID, has
been constrained by a lack of timely, reliable data. The interim PMS is partially effective. Whilst data on
new access is reported, there is as yet no
information on the use of toilet facilities or the type of toilet


(basic or improved) that has been built. The interim PMS does include information on other aspects of
environmental sanitation such as dis
h racks and garbage pits that are

not part of a typi
cal sanitation
focused CLTS programme. Efforts to collect data using the interim PMS have been constrained by
District staff lacking funds and transport, and at least some Champions lacking bicycles. Meanwhile,
the development of a mobile phone based PMS
has been delayed by procurement issues and more
recently the reluctance of

to link this system to a future WASH IMS.

MLGH, with support from KfW has developed a water supply and sanitation financing mechanism to
promote an enabling environment that
will facilitate adequate financing to the sector, promote effective
and equitable distribution of financial resources, and provide for sound financial management. The
financing mechanism will be piloted in 3 districts from mid
2012. As set out in the bus
iness case,
channelling sanitation funds through the financing mechanism would strengthen government
leadership, alignment with the NRWSSP and the profile of sanitation within the WASH sector.


UNICEF should pilot District Total Sanitatio
n and Hygiene Plans in 18 ‘1

Wave’ districts, based on
the guidance being prepared with

in early December.

UNICEF should re
establish the Sanitation TWG with

and other interested stakeholders,
with regular meetings and a standing agenda based on shared concerns.

UNICEF should ensure that the use of toilets and their type (basic or improved) is captured by the
interim PMS, noting that this may be simplified wit
h the exclusion of other indicators that relate to
broader concepts of environmental sanitation. At the same time, UNICEF should ensure that
districts have the information and resources needed to collect this information.

Indicator 3.3 should be changed, d
eleting the phrase ‘as a component of Government’s WASH
Information Management System’. This reflects the fact that the WASH IMS is now being reviewed
by GRZ and its future seems uncertain.The PMS will in any case demonstrate the concept of
monitoring WASH

infrastructure, its quality and use, using mobile phone technology, and this
information may be used to inform the development of the future WASH IMS.

UNICEF and DFID will consider participating in the pilot of the financing mechanism with
earmarked sani
tation funds. The decision will be based on a review of the financing mechanism
guidelines and procedures, and technical support being provided by KfW.

Impact Weighting (%):

Revised since last Annual Review?

Medium. R
elates to the possibil
ity of Government not buying in to the project but seeing it as a
parallel process that they are not responsible for. If recommendations made are not enacted by
UNICEF, this rating would increase to High.

Revised since last Annual Review?

Output 4: The private sector in 74 rural districts is enabled to supply sanitation and
hygiene services in response to household demand

Output 4 score and performance description:
: Outputs moderately did not meet expectation

Progress against expected



Milestone Dec 2012

Results Reported


4.1: No of local enterprises
established and able to offer
sanitation and hygiene related
services to households in rural




Inception report for
mapping prepared


After some delay, a partnership agreement has been established by UNICEF and an INGO with a track
record in this area to assess demand and supply capacity and to develop some products for sanitation
marketing. Whilst this work will be useful, it will be ch
allenging to ensure that 2013 target of 74 local
enterprises can be met. Equally, the proposed mapping may not be able to assess the assumed
baseline of 25 local suppliers

this information is more likely to emerge from District Total Sanitation
and Hygie
ne plans (see Output 3).

During the Annual Review, it was commented that, without a baseline or effective interim PMS, we do
not know what proportion of toilets constructed through CLTS is improved. This data is now likely to be
available in March 2013,
once the baseline is completed. The proportion will determine whether this
output is a vital component in terms of meeting the project’s outcome (which refers to 3,000,000 rural
users of improved sanitation) or a simply a useful contribution to the sector,

with possibly more
relevance to peri
urban or urban areas.


UNICEF should ensure that information on the number of sanitation providers is captured to inform
the baseline, and that the proportion of people building improved toilets
through CLTS is taken into
account during the development of a sanitation marketing strategy.

Impact Weighting (%):

Revised since last Annual Review?

given concerns about meeting the 2013 target of 74 sanitation enterprises establi
and able to offer related services

Revised since last Annual Review?

Section B: Results andValue for Money.

1. Progress and results


Has the logframe been updated since last review?

Individual outputs and Indicators in the logfram
were revised in September 2012

by DFID and
UNICEF. The revision was necessary to improve the specificity and relevance of outputs and
milestones, and to improve the measurability of indicators. The opportunity was used to reconcile a
number of differences
between the logframe presented in the DFID business case, and the logframe
that formed part of UNICEF’s
Programme proposal

Further changes to the logframe are recommended in this reviewagainst each output. In addition, the
outcome level indicator on
handwashing facilities was removed as it duplicated that on use of improved


sanitation with handwashing facilities. The final logframe is attached as annex A.

1.2 Overall Output Score and Description:
B: Outputs moderately did not meet expectations


Direct feedback from beneficiaries

Feedback from beneficiaries was obtained during two field visits to Chad
za District, Eastern Province
and to Maz
buka District, Southern Province. Responses from women and men were collected
separately. This information

was supplemented with information collected during the Mid Term Review
of the National Rural Water Supply and Sanitation Programme, with two sanitation consultants visiting
rural communities inCopper Belt and North
Western Province

In general, benefici
aries (both women and men) were happy with the project and appreciated the
promotional activities undertaken by Champions, often assisted with local Councillors. There was no
indication that people were put off by the CLTS approach. Few households seemed u
nprepared to
invest their own resources in toilet construction, although in some areas, people were concerned about
the effect of termites on toilet slabs.

At this stage, relatively few ODF communities were visited, compared to numbers in which triggering

had happened. This relates to the lack of follow up from Champions. Whilst this may need more
emphasis in training, it is also a function of the degree of support and encouragement provided by
districts. Some champions still lack bicycles; given the settl
ement pattern, the result is having to walk
many hours a day just to reach communities. Based on interviews with Champions, not having a
bicycle is interpreted as not being valued by the District.

The quality of toilet construction varied considerably, w
ith both good and poor examples. More practical
advice may need to be provided in challenging areas characterised by unstable sandy soils or hard
rock, etc. Most toilets built were seen to be used, those which were not used have only recently been
d. Hand
washing tippy taps or similar seemed common, although there was less evidence
that these were being regularly used.

Whilst District stakeholders also appreciated the project, levels of understanding and buy in varied,
suggesting that more needs t
o be done to communicate about the project at this level. The Rural Water
upply Unit in Councils seems insufficient to support sanitation, hygiene and water supply interventions

typically there is no dedicated staff member for sanitation, and water supp
ly is prioritised. Lack of
funds for monitoring and lack of transport compounds this problem.

1.4Summary of overall progress

Based on logframe outputs, indicators and milestones, progress is tabulated below.



Dec 2012

Results Reported

1.1: Number of people reached
with hygiene and sanitation
promotion activities undertaken
in communities, focusing on the
importance of using latrines and
washing with soap or ash



350,000 people

323,000 people


National sanitation and
hygiene behaviour change
communication strategy and
plans established with MLGH
and implemented, with both
community level interventions
and mass media components

Strategy, and
Annual Plan for
2013, agreed with
ing MLGH

Draft strategy
prepared by
section, currently
being circulated for
internal comment

1.3: Number of verified Open
Defecation Free (ODF) villages
1,000 villages

1,500 villages (ODF
reported by


achieved through DFID support

Champions but not

2.1: Number of schools and
health centres that have
appropriate sanitation facilities
including hand
facilities, together with an
appropriate management
system and hygiene promotion




Note: Process of
reviewing and
updating minimum
standards and
designs should be
initiated by UNICEF,
Government and
Partners by
December 2012


No progress on the
review and updating
of minimum

3.1: Number of districts
implementing Total Sanitation
and Hygiene Plans



18 plans

0 plans

3.2: Number of National
Sanitation Technical Working
Group Meetings convened by
MLGH with CPs participating

3 meetings in 2012

2 meetings held in

3.3: Sanitation and Hygiene
Performance Monitoring
System established as a
component of Government’s
WASH Information
Management System

Interim PMS
established and
used to report CLTS
results in 2012

Interim PMS
developed and
partially successful

4.1: No of local enterprises
ished and able to offer
sanitation and hygiene related
services to households in rural





Inception report for
mapping prepared

Key challenges

Key challenges identified and discussed with UNICEF include the following, arrange

in priority order:

UNICEF resourcing and overall support

Both the annual review and the associated Mid Term Review of the NRWSSP have concluded that

whilst households and champions are meeting expectations in terms of the outputs achieved to date

the p
rocess as a whole needs more support

in particular from UNICEF. The support required
the WASH section to be continuously

major technical inputs from UNICEF‘s
Communication and M&E sections
; and ensuring
procurement of critical serv
is not
information needs to be communicated at the highest level to the Government and WASH CPs.
requires mobilizing UNICEF management to engage with GRZ on WASH. In general terms, e
this level of support associated with most projects of this scale may be a challenge for a medium sized
Country O
ffice, requiring additional support from the Regional Office. The decision to declare
the project to be a ‘Flagship’ is welcome,

but needs to be followed up with evidence that related
activities are being prioritized throughout the organization.

Building Government leadership and ownership at Central and District level

Securing government leadership and ownership at both Cent
ral and district level is a challenge first
raised by UNICEF in its 2011 paper on scaling up CLTS. Whilst the project details have been shared
with government from the start, institutional memory is weak, with frequent changes in staffing. At the
same time
, more clarity is needed in relation to how project systems (the PMS) and institutions
(Coaches and Champions) fit into Government’s NRWSSP structure. Project terms like CLTS or CAT

may need to be renamed to fit into Government’s sanitation lexicon.


lding district capacity for planning, supervision, monitoring and reporting

linked to
staffing, funds and transportation

As identified in the previous section, there is a clear need to ensure that districts have the capacity,
resources and incentives to
lead sanitation and hygiene interventions. At the moment, whilst responses
vary, it seems that some districts see the project as a largely external undertaking, and not a major
responsibility. The challenge relates to their understanding of sanitation and
hygiene, compared to
water supply that typically receives far more attention; their understanding of the project and its
approach; limitations in financial and transport resources, and the lack of sufficient human resources
dedicated to sanitation and hygi
ene. The
recommendation to


an inclusive District Total
anitation and
lan is designed in part to facilitate Government leadership at District level.

Verification of ODF

From a programmatic perspective, the lack of a strategy, protoc
ol and guidelines to verify ODF status
at village, Ward and ultimately district level is a major challenge at this stage in the project. Such a
system forms a core part of a CLTS programme of significant scale. In particular, verification of ODF
status is
a prerequisite of Government recognition, for example though an awards ceremony. This itself
is an important way of developing Government understanding and ownership, and communicating the
programme throughout the country. The verification system, once dev
eloped, needs to be applied to
over 1,500 self
reporting communities.

Establishing a cost effective, affordable model for School Sanitation

The challenge of establishing a cost effective approach to school sanitation has already been
highlighted under
Output 2. The estimated cost of establishing appropriate sanitation hardware in a
school may be as high as £20,000

several times more than the funds available for this activity. It
would seem that MoE is reluctan

to change its standards, which seem over

designed and
unaffordable. UNICEF, in concert with USAID, DFID and others, must now address the issue with the

Building consensus and engagement with CPs (Donors and NGOs)

Finally, based on discussions with a number of WASH CPs, some donors an
d NGOs have relatively
little understanding of the project and its fundamental relationship with the Government and NRWSSP.
In particular, there is some apprehension that CLTS may not be suitable in rural Zambia compared to
other methodologies such as PHAS
T, typically implemented with a hardware subsidy. There is also a
feeling that the project ‘belongs’ to UNICEF, and to that matter DFID, rather than being owned by the
sector as a whole. To an extent, underlying these views is a general lack of information

understanding, and according to at least one CP, a rather confrontational style of communication


1.6Annual Outcome Assessment

The project outcome

3 million people consistently using improved household toilets and practicing

with soap or ash thereafter

has much in common with the first out

statement. The
first annual outcome assessment therefore shares the concerns raised about the achievement of this
output, noting the comments relating to district capacity and the Perf
ormance Monitoring System
(Outcome 3).

Whilst the reported result of 323,000 new users is 92% of the milestone of 2012, we do not know what
proportion of the toilets constructed can be classified as improved, or the proportion of these toilets that
used. Furthermore, of 8,300 ‘triggered’ villages in which CLTS has been introduced, only 1,500

18% are reported to be ODF. Post triggering follow

has now to be prioritised by Champions and
participating Districts. UNICEF is focusing on this issue, a
nd has delayed the introduction of the
programme in a further 30 districts as a result. If the follow up strategy is properly established in the


current set of 32 districts by March 2013, results should accelerate, enabling the target to be achieved
by Mar
ch 2015.

2. Costs and timescale

2.1 Is the project on
track against financial forecasts:

The most recent financial report submitted by UNICEF indicates that by 31 December, projected

are estimated to total £
This is

less than the forecast

of £

out in the
Business C

for 2011 and 2012
, and 9% less than

the figure of £ 5,000,000 set out in the
UNICEF proposal to DFID (tr

1 and 2


Note: UNICEF financial

are not necessarily translated into actual
. For
example, based on the agreement not to support School sanitation pending the revision of standards,
there should be no expenditure against this budget line. In general, actual expenditure is like
ly to be
significantly less than the commitments being reported.

The difference between UNICEF’s commitments and the amount planned in the Business Case is
explained by (i) the decision, endorsed by DFID, to consolidate promotion activities in 32 distric
ts and
postpone expansion to a further 30 districts, and (iii) procurement delays, relating to the Impact
Evaluationand the development of a mobile phone based Performance Monitoring System.

Whilst the overall situation looks reasonable, it should be und
erstood that UNICEF commitments
include £ 1,083,681 for school sanitation. As school activities have been suspended by UNICEF with
DFID agreement, pending the review of standards, actual expenditure is less by this amount. This
makes the gap between expend
iture and the Business Case forecast much higher, increasing from
12% to 33%

2.2 Key cost drivers

Key cost drivers are set out in the following table, based on UNICEF financial proj
ections for 2012,
provided in
October 2012.

Budget Area


Commitment (2012)

Proportion of total
Commitment (2012)

Training & M&E

Funds transferred to 32
districts for training and

£ 1,744,524

38% total

School Sanitation and

Sufficient for 360
schools, currently on

£ 1,083,68

24% total

Major Contracts and

Includes Impact
Evaluation and
Sanitation Marketing

£ 812,672

18% total


Vehicles including 4x4
cars, motor bikes;
bicycles, computer
equipment, smart

£ 703,619

15% total

Based on 100% the forecast for 2011/2012, and

75% of the forecast for FY 2012/2013.


UNICEF Zambia Staff

National & International
staff: WASH and
Logistic Support Staff

£ 163,387

4% of total

Miscellaneous start up

HACT assessments,
printing of CLTS
manuals, review


2% of total


£ 4,596,103

100% total


7% Expenditure

Not Included as based
on actual expenditure

The fact that
only 4
% of reported commitments relate to staff costs may indicate that UNICEF has not
invested in sufficient staff for an undertaking of this complexit
y. In general, tec
hnical staff costs for
UNICEF programmes of this scale and type fall between 6% and 8%.

Any increase in UN Subsistence rates would increase project costs significantly.

2.3 Is the project on
track against original timescale:

Associated with this
financial situation, the project is also off track against the original timescale. In
particular, this relates to the following:

The delay in establishing a project baseline, now due by March 2013. To date, the contract has not
been signed by the contract
ee. This is related to delays in establishing a fairly complex TOR that
includes the project baseline, endline, formative research and Impact Assessment, and then finalising
the same based of DFID criteria.

The delay i

establishing a Performance Management System using mobile phone technology


associated with a complex TOR, UNICEF procurement rules and Government’s concern about how
such a system could relate to the WASH Information Management System which remains und
development and is soon to be reviewed.

The need to consolidate work in 32 districts before expending CLTS to another 30 districts (originally
scheduled for November 2012
, now postponed until March 2013
). This decision was taken to ensure
due attentio
n was given to follow up activities by participating districts and champions.

The decision to postpone work on the construction of school WASH facilities pending the establishment
of a revised guideline by MoE, supported by UNICEF, on grounds of cost effe
ctiveness and

3. Evidence and Evaluation

3.1 Assess any changes in evidence and implications for the project

Much of the global evidence that under
pinned the Business Case remains valid. Sanitation has been
shown in a number of intervention studies to demonstrate significant benefits in terms of reduced
diarrhoeal disease and for hand washing to have impacts on
both diarrhoeal disease and less well
established, on acute respiratory infections. There is a dearth of empirical data that shows the likely
impact on disease from programmes operating at scale under non
experimental conditions. The
evidence is summarised

in a soon to be published DFID Evidence Review.

There remains a debate on the importance of the standard of toilets. The Joint Monitoring Program
established a relatively high level of sanitation technology as being required to meet the MDGs,
although th
ere is little empirical data to suggest this is required. In recognition of this, more recent


assessments introduced the concept of sanitation ladders and placed greater emphasis on eliminating
open defecation. In this context the Zambia Sanitation and Hyg
iene programme has the potential to
significantly improve the global evidence base through the impact evaluation on overall impact of
programmes operating at scale and by establishing the relative benefits of basic and improved toilets.
This reinforces the

need for urgent action to improve monitoring by the programme.

In schools it is hypothesised that the provision of separate facilities for girls will have a positive impact
on school attendance and retention, specifically reducing absenteeism during menst
rual periods. A
systematic review funded by DFID and published in 2011 concluded there was no evidence to either
support or reject this hypothesis, noting this was an important area for research giving the plausibility of
the effect. A more recent review b
y Jaspar and Bartram concluded there are positive benefits for girls’
attendance from improving access to sanitation, but only four papers could be included in the review of
which two were from developed countries. Under the DFID funded SHARE programmes fu
research is being conducted in Kenya on the role of sanitation on girls’ attendance and retention at
schools. This again highlights the potential for this programme to produce significant new global

3.2 Where

an evaluation is planned what p
rogress has been made?

An impact evaluation is planned in project intervention areas (32 districts) in 2015. The evaluation uses
randomised sampling over time, rather than using a control group, and focuses

measuring health
impact of the project. ToRs h
ave been quality assured by UNICEF and DFID. Baseline data will be
collected in early 2013, with the results made available in 2015.

4. Risk

Output Risk Rating:

4.2 Assessment of the risk level

The risk rating is based on the following


Output 1
: Risk Level: Medium
. Whilst good progress has been made in terms of numbers of people
with access to toilets promoted by CLTS (92% of target); the fact that we do not know what proportion
of these are improved, and what proportion are b
eing used, is important. The effectiveness,
sustainability and scalability o
f the underlying strategy depends on effective post
triggering follow up
and the verification and recognition of ODF communities. To date, it appears that only 18% of triggered
munities are ODF (reported)

and there is no system for verifying this in place. Whilst the risk
associated with this output is medium, it becomes a high if corrective actions are not undertaken within
a relatively short time frame of 3

6 months.

ut 2: Risk level: High
his component of the project may have to be ended prematurely unless
UNICEF successfully advocates for a more cost effective design with MoE.

Output 3:
Risk Level: Medium
Whilst good progress has been made by Champions,
understanding, ownership and leadership of the

programme seems limited, and

efficient systems for
performance monitoring have yet to be established. At the same time, district participation is vital in
terms of the introduction, scale up of CLTS and foll
ow up activities to ensure that new practices are

UNICEF is aware of this issue, and is speeding up the transfer of funds to the 32 Districts
which will resource their involvement.

Output 4: Risk Level: Medium
. Noting the lack of data on

the type of toilets being built

through CLTS,
it is premature to comment on this component of the project. Nevertheless, there is a m

risk that
the intended number of sanitation entrepreneurs will not be supported to meet demand by 2015, noting
this component is already delayed by six months or so.


Whilst the Project outputs are still assessed as medium risk, this rating is based on the assumption that
UNICEF will implement the recommendations made in this Annual Review in a timely fashion. Thi
s was
effectively agreed by UNICEF’s Representative when feedback was provided on 29

Should recommendations not be acted on, however, the risk rating would increase to High.


Risk of funds not being used as intended

There is a medium risk of funds not being used as intended. Here the main concern relates to funds for
monitoring which are transferred to districts. The paper
based system currently being used to report
progress is
vulnerable to fraud
, even tho
ugh UNICEF is providing strict guidance to districts
on how the funds are to be used.

The introduction of mobile phone technology
may lessen

this risk, as
the GPS coordinates of each toilet location visited are automatically generated. Otherwise, the
ciary risk is low, reflecting the limited spend on sanitation hardware (contracts would be limited to
the construction of school toilets, based on a revised guideline).

4.4 Climate and Environment Risk

Improved sanitation and hygiene will increase

vulnerable populations resilience to climate change.
However increased frequency of severity of climate shocks, especially floods, could damage household
and institutional latrines, reducing programme sustainability. The school sanitation programme
ides an opportunity to raise youth awareness around climate change issues, but this is considered
outside of DFID’s programme scope.
DFID Zambia’s Strategic
Climate Change P
rogramme Review

undertaken in October 2012 included a review of the sanitation and h
ygiene programme and made the
following recommendations for climate proofing:

UNICEF to include in sanitation marketing consultant ToR’s need to consider climate resilient
and low carbon construction

UNICEF to commit to designing Institutional sanitation
to be designed to be low carbon

. Value for Money

5.1 Performance on VfM measures

The Value for Money (VfM) assessment considers whether the costs and benefits of the programme
are in line with the original business case and still represent a good return on investment.

The key VfM measure in this project is the unit cost per beneficia
ry. In this case the cost
benefits are
compared at household level.
In the business case, it is stated that monitoring the cost per
beneficiary/community/ward of achieving ODF and ‘total sanitation’ will be a priority. Experience of
CLTS in Zambia and else
where is that it takes a relatively long time (up to a year) to put systems in
place and identify, train and deploy facilitators, and initiate the CLTS process in communities. Progress
then accelerates rapidly. Unit costs should therefore reduce over time.

Based on the
provided by UNICEF in October 2012,
the average cost per beneficiary
in 2012 is around
£10.70 per person

(DFID investment only
, excluding expenditure on school
). This includes significant up front commitments.
In t
ime, it is expected that this initial rate will
be reduced to an average cost of
£3.00 per person

(£18 per household) or less over the whole
programme. This is lower than the unit cost anticipated in the Business Case of
£3.70 per person

The total cost o
f the intervention, including community contribution and a proportion of government
salaries and overheads, is more difficult to assess. Based on findings during the Annual Review’s field
visits, a typical family spends £8

£10 on a toilet

£2 per
. The contribution is higher if a
concrete slab is purchased, going up to around £4 per person. These toilets are expected to be in use
for around 7 years. More work is needed to establish more accurate figuresand to develop some


meaningful comparato
rs, possibly derived from similar programmes implemented in other parts of sub
saharan Africa.

The economic benefits are estimated on the basis of cost savings to households. According to a WSP
report published in 2012
, lack of sanitation costs the averag
e Zambian household
the equivalent of
£10.50 every year, taking into account access time, premature death, productivity losses and the cost
of providing health care.

Over a ten year period the total benefits are calculated to have a Net Present
Value (NPV)

of £70.56 per household, discounted at 10% a year. These significantly exceed the
expected average cost per household covered of £18.00.

Social costs are not included but are known
to be significant. Such research is however based on a number of assumptio
ns and caveats that
should be qualified.

The Impact Evaluation service provider,
Boston University has
a proposal

to do
additional data collection and analysis
on cost effectiveness

which would contribute valuable
information on value for money for this programme and the sector more broadly.

5.2 Commercial Improvement and Value for Money

Funds from DFID are routed through UNICEF, which undertakes procurement of supplies and service
and transfers funds to Government for training, supervision and reporting purposes.
unds to the
government are channel

to districts, with varying degrees of control
depending on
the results of
astandard partner risk assessment. Channelling funds to d
deemed necessary by UNICEF
avoid delays in disbursement to districts and because

its central Government partner,
, had
failed to liquidate a previous disbursement made by UNICEF within a statutory six
month period
UNICEF has provided d
istricts with elaborate guidelines on how funds for monitoring progress should
be used, with individual budget lines. This is unusual, but a welcome and necessary precaution to
reduce fiduciary risk.

In terms of Commercial Practice and V
M, UNICEF foll
ows relatively robust rules. In the case of major
contracts, technical proposals were assessed by several partners as well as UNICEF. These included
Government and a representative of DFID. Procurement of supplies follows strict procedures, with
products b
eing tested before delivery to check compliance with the specification.

Such procedures take time, however. Furthermore, the complexity of several TORs was significant,
reducing the number of proposals received and their quality. This may have contributed

to not
identifying an organisation capable of delivering the Performance Monitoring System through open
competition. As a result, the development of the system has been delayed by six months or so,
impacting on value for money.UNICEF has now opted to sing
le source a par
ticular organisation to do
. Other delays in staff contracting may have been related to the position of Head of Human
Resources being vacant for several months. It is now filled.

Meanwhile, UNICEF has agreed to defer the contracting of c
ompanies to construct school sanitation
facilities until the national guideline is revised, recognising that the costs of going ahead would be very
high to the extent that the target would not be met.

5.3 Role of project partners

In overall terms, noti
ng the points in the preceding sections, it is concluded that UNICEF needs to
improve the resourcing of this project, noting its ambitious targets and its current status. This includes
the allocation of necessary human resources from relevant programme and

operational sections, as
well as inputs from senior management related to relations with government and CPs. Whilst the
project is set to offer good value for money, and a positive rate of return within a few years, UNICEF
needs to ensure the timely deliv
ery of inputs and the completion of a wide range of interrelated
activities to achieve stated objectives. With this in mind, the declaration that this project is now a
UNICEF flagship

and regarded as such by the whole Country Office as well as the Region
al Office is

Zambia: Economic Impacts of Poor Sanitation in Africa; WSP, 2012

These were not DFID funds


very important, if it translates into strengthened resourcing and increased priority.

5.4 Does the project still represent Value for Money : Yes

5.5 If not, what action will you take?


6. Conditionality

6.1 Update on specific conditions


7. Conclusions and actions

The project has made significant progress in terms of the role out of CLTS, achieving 92% of its annual
target in this respect. However, more data is needed to establish the quality of toilets being built, and
whether or not they are used. Meanwhile, the c
onstruction of school toilets has been suspended whilst
UNICEF works with MoE and partners to establish a cost effective and affordable approach. Capacity,
especially at District level, to plan, implement and monitor the programme, needs urgent attention,
least to secure local ownership of the project which is otherwise at risk of being seen as a parallel
process largely outside of Government. Advocacy is needed to address this issue at central level and
with selected CPs. Work on sanitation marketing h
as only just started; it is too early to say much about
this and its future relevance and impact.

Project Improvement Plan

In view of the project’s ‘B’ rating and its scale and
significance to DFID and UNICEF
, a six
Project Improvement Plan
will be


and DFID senior management by mid
January 2013
. The

will be used to review progress in March and June 2013

at which point the
Head of DFID Zambia will decide whether the PIP should continue, further changes be made or the
amme (or components of it) be discontinued


will clearly indicate the actions
needed to get the
project back on track, specific
,milestones and a
time frame
.The PIP will respond to the recommendations under each
output in this annual review and the following general actions
relating to all outputs and the outcome:

UNICEF should continue to prioritise the Zambia Sanitation and Hygiene Programme and ensure it
eceives the resources and support it needs to implement the project properly, in accordance with
the agreed proposal and logframe, reflecting the status of the project as a UNICEF flagship. This
recommendation applies not only to UNICEF’s WASH Section, but

other Programme and
Operations sections, and to all levels of management.

UNICEF should re
acquaint Government stakeholders and CPs with the project, its rationale and
approaches, emphasising its association with the NRWSSP and related programme docum
and the routine sharing of information. This applies to major donors and NGOs, and the staff of
other Ministries, including the Ministry of Health, Education, Community Development and
Traditional Affairs.


UNICEF should develop and implement a stra
tegy to strengthen Government Leadership of the
programme at both Central and District level, focusing on technocrats within

and Local
Councils. The strategy should reflect the relationship between the project and the NRWSSP

will include considera
tion of participating in the pilot of the financing mechanism

8. Review Process

The Annual

Review was conducted by the DFID’s WASH Policy Team Leader with support from the
Professional Evidence and Applied Knowledge Services provider. Further inputs were made by the
Team Leader, Human and Social Development, DFID Zambia, who also coordinated th
e process, and
the Programme Officer, Human and Social Development, DFID Zambia.

The Review was held over a three
day period, 27
November 2012, immediately after the MTR of
the NRWSSP. The Annual Review
included a field visit to Maz
buka District on
November 27
briefings and meetings with UNICEF,
and CPs
(Danida, KfW and USAID)
on November 28
, and an extensive debrief with UNICEF
senior management and UNICEF WASH
on November 29

This i
nformation was supplemented by a three
day field visit
District in Eastern P

from 12
14 November,
and the preliminary
of the
of the NRWSSP
. The MTR
three weeks input from two sanitation specialis
ts and a school WASH
specialist from 5

23 November

The annual review report was presented to DFID Zambia on 12 December for discussion. Revisions
were incorporated in the document and reflected in the Logframe.