Namibia looking forward:

simpleluncheonManagement

Nov 10, 2013 (3 years and 7 months ago)

72 views

Stakeholder data
validation workshop


July 2, 2012



Windhoek

Namibia looking
f
orward:

Effectiveness of capacity development

support to local partners

Why we are here today


To share Pact Namibia’s internal
evaluation findings with stakeholders


To get stakeholder validation of the
evaluation recommendations

Agenda

Time

Activity

9:30

10:45

Presentations

10:45

11:00

Tea break

11:00

11:30

Q&A

11:30

noon

Small group work:

Discuss findings &

r
eview
recommendations

Group 1.
Capacity development impact

Group 2.
Graduation to direct USAID funding

Group 3.
Lessons learned for capacity development
organizations

Noon

12:45

Report back

12:45

1:00

Conclusions & next steps

1:00

2:00

Lunch

Background


USAID Namibia began scaling up HIV programs
in 2003.


In 2007
USAID
awarded Pact a 5
-
year project
which included grants, organizational and
technical assistance to CSOs, predominantly in
areas of:


HIV/AIDS prevention


Care and support to
o
rphans, HIV
-
vulnerable children
and home
-
based clients


Strengthening capacity of the Ministry of Gender Equality
and Child Welfare

Background


In 2010


award changed to focus on two
objectives:

1.

Strengthening
capacity of the Ministry of Gender
Equality and Child Welfare to effectively coordinate
activities relating to gender equality and orphans and
vulnerable
children

2.

Strengthening six civil
-
society
organizations to be
eligible for
U.S.
direct funding while maintaining high
standards of existing services

Tools to strengthen civil society


Organizational Development (OD)
Roadmaps

measures CSO partner organizational
systems and structures objectively


Basis
for identifying
organizational
efficiency
gaps and
prioritizing
interventions


Progression on a scale from 1 (nascent) to 5 (mature
)


Comprehensive Institutional Strengthening
Plans
to identify, schedule and monitor all capacity
-
building activities


Tools to strengthen civil society


Comprehensive Institutional Strengthening
Plans
to identify, schedule and monitor all capacity
-
building activities


Organizational Development (OD)
Roadmaps


10 capacity areas:


Purpose &
planning

HR management

Monitoring &
evaluation

Networking

Governanc
e

Organizational
sustainability

Financial
management

Grants &
compliance

Operations
management

Projects &
services

Organizational Performance Index
(OPI)

Scored on a scale of 1 to 4

on each of the following dimensions:


Effectiveness.

Achieving outcomes, meeting
s
tandards


Efficiency.

Delivering services,
i
ncreasing
r
each


Relevance.

Engaging target populations,
p
romoting
l
earning


Sustainability.

Mobilizing resources,
i
ncreasing
l
egitimacy


Why do this evaluation?


1.
Assess the sustainability of graduated partners’ capacity
development

2.
Review the quality and quantity of
services
offered to
beneficiaries of former and current CSO partners
(partially done)

3.
Assess the effectiveness of the Pact Namibia capacity
building assistance to the partners that have graduated
or plan to graduate

4.
Draw lessons for other Pact programs that are facing
similar challenges of graduating partners

Evaluation questions


To what extent was Pact Namibia’s
capacity development support effective?


To what extent were partners ready for
and able to effectively manage direct
funding?


To what extent did service quality and
coverage change over the last two years?



Process

3 phases:

1.

Designed
t
erms of reference and data
c
ollection
t
ools (virtual group)

2.

Collected data (Pact staff from 5 country
offices, for neutrality)

3.

Analyzed data, drafted and disseminated
report (current group of Pact staff)


Data collection tools

1.
Partner survey (
Mobenzi
)


17 current & former partner organizations interviewed


Quantitative & qualitative

2.
Organizational Performance Index (OPI)


12 of 17 partners

3.
Pact staff
s
urvey (
Mobenzi
)


4 core Pact Staff from each technical area

4.
USAID tool

5.
Historical OD Roadmap scores

Data analysis


Quantitative data:


Merged datasets and did selected frequencies with statistics and
cross
-
tabulations using SPSS and Excel.


Qualitative data: thematic analysis.


Partner breakdown



OD
Roadmap
support,

graduated

to
direct funding

OD
Roadmap
support,
not
graduated

No OD
Roadmap
support,
graduated

No OD
Roadmap
support,
not
graduated

Total

Number of
orgs

3

(18%)

3

(18%)

2

(12%)

9

(53%)

17

Number of
partner
respondents

11

(23%)

12

(25%)

3

(6%)

22
(
46%)

48

Study limitations

Sample size


Only 17 organizations


Limited
number of respondents in some
organizations; some interviewed were previous staff

Time


Could not conduct survey of beneficiaries

Presentation of the findings


Capacity development
i
mpact


Graduation to direct USAID Funding


Lessons learned for capacity
d
evelopment
o
rganizations


Preliminary recommendations



Capacity development impact

Key areas of analysis

1.
How did
length
of support influence
partners

OPI
scores and perceptions of usefulness of support?

2.
How do Pact and
partner
perceptions of support
compare?

3.
How
do OD Roadmap supported partners differ from
other partners?

4.
Which areas of Pact support
did partners rate
as most
useful and
of highest quality?

5.
What were the most significant changes resulting from
Pact

s
support?

1
. Partners scored longer support less
useful

3.0
3.5
4.0
4.5
5.0
1-3 years of
support
3+ years of
support
Usefulness rating


Strategic

Financial

Program thematic

Monitoring & evaluation

2
. But Pact scored longer support more
useful

Partners score longer
support
less

useful

3.0
3.5
4.0
4.5
5.0
1-3 years
of support
3+ years
of support
Partner rating

Strategic

Financial

Program

thematic

M&E

Pact scores longer support
more

useful (except financial)

3.0
3.5
4.0
4.5
5.0
1-3 years
of support
3+ years
of support
Pact rating

Strategic

Financial

Program

thematic

M&E

3. Longer support


higher OPI scores

0%
20%
40%
60%
80%
1-3 years of
support
3+ years of
support
% of partners scoring
over 3

Strategic

Financial

Program thematic

Monitoring & evaluation

4
.
OD Roadmap partners scored
higher on
OPI

67%

17%

0%
50%
100%
OD Roadmap
Non-OD Roadmap
OPI score above 3

0%
10%
20%
30%
40%
50%
60%
Technical
Support
Financial
management
support
M and E
reporting
support
Advocacy
Support
Strategic
planning
support
HR
Management
Other
% of all partner staff

Non
-
graduated partners

1a. But non
-
graduates more keen on financial
management

5. Non
-
OD Roadmap partners rated Pact support more
useful

3.50
3.75
4.00
4.25
4.50
Strategic
Support
Financial
Support
Program
Thematic
M&E
Usefulness rating

Non
-
OD Roadmap

OD Roadmap

6. Partners
rated M&E support
highest

3.50
3.75
4.00
4.25
4.50
Program
Technical
Financial
Planning
Strategic
Planning
M&E
Quality rating

Usefulness rating

7
.

Most
significant changes

M&E and Financial management
the

cornerstones
of
programs


(partner,
Pact and
USAID responses)


M&E


Timely reporting


Capturing the right and required data


Use data for decision making


Finance


Proper management of organization finance & USAID funds


Complete and accurate finance reports


Proper budget
management

7. Most significant
changes


Moved from no system to very good
system in all areas
(partner
responses)


Buy
-
in from
organizations
(
Pact
responses)


How do we sustain the change?


Experience
handover and staff retention


Resources



Summing up


More
years of support
=

lower usefulness rating, but
higher OPI scores


Pact and partners see usefulness differently over the
duration
of
support


OD Roadmap support =
lower
usefulness rating,
but
higher OPI scores


M&E support rated most useful and
of highest
quality


Most significant change in M&E and
financial
management

Graduation to direct USAID
funding

Key areas of analysis

1.
Capacity
-
building support that partners
would like more of before graduation

2.
Graduate vs. non
-
graduate performance
on OD Roadmap
and
OPI

3.
Prerequisites
for
graduation

4.
Key capacity
-
building support
for
graduated partners

0%
10%
20%
30%
40%
50%
60%
Technical
Support
Financial
management
support
M and E
reporting
support
Advocacy
Support
Strategic
planning
support
HR
Management
Other
% of all partner staff

1.
Partners want more technical support before
graduation

0%
10%
20%
30%
40%
50%
60%
Technical
Support
Financial
management
support
M and E
reporting
support
Advocacy
Support
Strategic
planning
support
HR
Management
Other
% of all partner staff

Non
-
graduated partners

G
raduated partners

1b. And graduates far more interested in technical
support

1
1.5
2
2.5
3
3.5
4
0
1
2
3
4
5
6
7
8
9
10
11
2a.
Graduates scored higher on OD
Roadmap

Graduates

Non
-
graduates

1
1.5
2
2.5
3
3.5
4
0
1
2
3
4
5
2b. But OPI not different

Graduates

Non
-
graduates

1.
Strategic
plan

2.
Governance

3.
Strong financial systems and internal
controls

4.
Administrative systems, including human
resource
management

5.
Skilled staff and technical
capacity

6.
Strong
monitoring
and e
valuation

3. Graduation prerequisites from Pact and partners

4. Key post
-
graduation capacity
development

Priority


Resource mobilization
support


Technical/
Program
s
upport

As
needed


Monitoring and
evaluation


Database development


Evaluations


Routine data quality assessments


Financial m
anagement

Summing up

1.
Graduated partners would have liked more technical
support before graduation

2.
Graduated partners averaged 3.4 on Roadmap, up from
2.5 for non
-
graduated partners (no difference on OPI)

3.
Six graduation prerequisites: strategic plan, governance,
finance, admin, technical capacity, M&E

4.
Key post
-
graduation needs: technical and resource
mobilization support

Lessons learned for capacity
development organizations

Key areas of analysis

1.
Retention of systems after capacity
building,

and impact of
who decided to engage in it

2.
OPI
scores and trends
across partner subgroups

3.
Different capacity needs
identified
by partners
and donors

4.
Challenges and advantages of
graduating to
direct funding

20%
40%
60%
80%
100%
Strategic
Planning
Financial
Management
Program
Thematic
Area
Monitoring &
Evaluation
%
of orgs still using
system

1a. OD Roadmap
-
supported partners more
likely
to

Non
-
OD Roadmap Support

OD Roadmap Support

retain systems

Who decides to
acquire system


% retaining

system

N

Partner alone

95%

60

Both partner

and Pact

83%

87

Pact alone

75%

24

1b. Retention likelier if system is partner
-
driven

20%
40%
60%
80%
100%
Partner's
Decision
Both Decision
Pact's
Decision
% retaining system

1c. Retention likelier if system is partner
-
driven

Program
thematic

Financial

Strategic

Monitoring &
evaluation

2. OPI scores


Average OPI scores show lower resource
mobilization but high social capital


Graduated partners generally score
higher


Current partners score low on resource
mobilization and service delivery













Relevance













Effectiveness

2a. Average OPI shows lower resource mobilization, high
social capital













Efficiency













Sustainability

0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Results
Standards
Delivery
Reach
Target
Learning
Resources
Social
Capital
Average OPI













Relevance













Effectiveness













Efficiency













Sustainability

2b. Graduates need to focus on
results

and external evaluation of
standards

0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Results
Standards
Delivery
Reach
Target
Learning
Resources
Social
Capital
OD Roadmap graduates

Average OPI

2b.
Current partners scored lower on













Relevance













Effectiveness













Efficiency













Sustainability

Average OPI

Non
-
graduated OD Roadmap partners

0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Results
Standards
Delivery
Reach
Target
Learning
Resources
Social
Capital
resource
mobilization and service delivery

3a. Different stakeholders, different top
-
ranked capacity
needs

OD Roadmap partners

1. Monitoring & evaluation

2. Technical

support

3. Financial management

4. Program development

Pact

1. Resource mobilization

2. Financial

management

3. Institutional support

4. Technical support

USAID

1. Resource mobilization

2. Financial

management

3. Administration

4. Monitoring & evaluation

Non
-
OD Roadmap
partners

1. Technical support

2. Financial

management

3. Advocacy

4. Monitoring & evaluation

3b. Graduated partners & USAID: contrasting
expectations

Partners expect:

1. Same relationship/treatment
with

USAID as partners had
with Pact



2.

Continued technical
assistance, but often lack
adequate budget


3. Increased USAID capacity
-
building support after
graduation


USAID expects:

1. Strong organizational
capacity and governance
systems, ability to work
independently and maintain
capacity even after staff
turnover

2. USAID

support to partners,
but also partner ability to
address own needs

3. Performance like long
-
term
international USAID partners,
responsive and accountable

4. Graduates found both challenges &
benefits

Challenges

1. Increased accountability in
line with U.S. Government rules
& regulations

2. Less

frequent capacity
-
building support

3. Would

like clearer guidance
from USAID on budgets,
funding & programs

4. Would

like more regular
feedback on financial reports

Benefits

1. Shorter reporting channels
give more time to prepare

reports

2. Improved interaction with
USAID &

understanding of
USAID programs

3. Less cumbersome financial
reporting requirements

4. International exposure &
recognition

5. Good working relations

Summing up

1.
Current partners most likely to retain M&E, financial
and technical
systems

2.
Sustainability
of capacity building depends on whose
decision it was to engage in capacity development

3.
On average partners tend to score lower in resource
mobilization on OPI scoring

4.
Partners still need to develop capacity, but are
struggling to understand how to access support under
direct funding.

Preliminary recommendations

Preliminary recommendations


Capacity development impact


Graduation to direct USAID funding


Lessons learned for capacity
d
evelopment (CD) organizations


General

Capacity
d
evelopment impact


Capacity development support should be provided for
three or more years (more research is needed).


Capacity development organizations need to keep their
support “fresh” for organizations served for longer
periods.


OD roadmap support is an important approach for
preparing effective partners and should continue to be
used.


Capacity development support should increase focus on
knowledge transfer and resource mobilization for
increased sustainability.

Graduation to direct USAID funding


The OD Roadmap should be considered as the tool to measure
preparedness for an organization’s readiness for graduation.


The prerequisites for transition to direct funding should
include
:


S
trong
strategic planning
capacity


G
overnance structures


S
trong
financial systems and internal
controls


S
trong
administration (including HR)
systems


S
killed
staff and technical
capacity


S
trong M&E


A
n
ability to
mobilize resources


A
n
ability to advocate with donors to meet (internal and external)
constituent needs
.


Lesson
l
earned for CD organizations


CD organizations should ensure ownership of
interventions for maximum impact in the area of system
retention.


CD organizations need to work with partners to diversify
their funding base.


CD service providers need to help partners to remain
relevant and vibrant by planning to incorporate internal
capacity development support on an on
-
going basis.


Increased communication on expectations will improve
the transition to direct funding for both partners and
USAID.

General


Pact should consider expanding this dataset to
include more Namibian respondents and more
international graduates (33 worldwide) and
partners in order to make the findings more
robust.


Pact should assess the impact of the capacity
development approach (and the transition to
direct funding) at the beneficiary level.

Thank you!