Introduction

In 2012, the Department for International Development (DFID) launched the £355 million Girls’ Education Challenge (GEC) Fund. The GEC set out to support up to one million of the world’s most marginalised girls to improve their lives through education. The GEC was designed to attract innovative approaches to reaching marginalised girls – girls aged six to 19 who have not been enrolled, or have dropped out of school, or are in danger of doing so.

The GEC has three funding windows – the Step Change Window (SCW), the Innovation Window (IW) and the Strategic Partnerships Window (SPW). DFID aims to promote innovation and private sector partnerships and to find the best projects capable of reaching and delivering education outcomes, at scale, for marginalised girls.

Summary

The GEC Evaluation Manager (EM) identified what parts of the delivery process have: worked well on the GEC to date; were found to be a challenge at the start but were addressed over time; and those parts that remain a challenge and could be addressed in future programs. For example:

  • The GEC targeted marginalised girls. Some of these girls are extremely marginalised. But the GEC was not set up to ‘systematically’ target, reach or benefit the ‘most’ marginalised girls.
  • Central management of the GEC through a Fund Manager (FM) was more effective than the other options considered at the start.
  • An inception phase for the Fund Manager (FM) and the EM to design and set up the GEC program was critically needed. And so is an inception phase for all projects.
  • Robust Monitoring and Evaluation (M&E) is achievable in complex, multi-country development programs, like the GEC, that include high risk project environments. However, approaches need to be tailored to the objectives of the funding window, the different types of interventions and the delivery contexts.
  • More could have been done to involve DFID country offices to use their local knowledge, networks and resources as a central part of the GEC. 
  • The GEC succeeded in including private sector partners in its portfolio by adapting approaches and tools that were better suited to working with the private sector. 
  • Payment by Results (PbR) may not be universally appropriate for all types of projects, organisations and contexts. PbR incentivises performance where it fits the project’s objectives.

GEC evaluation at a glance

This brief is based on the GEC Process Review Report produced at the end of 2015 by the EM. The purpose of the review was to learn lessons about the early stages of the GEC delivery process, from the start up until the end of the baseline phases for each of the funding windows. It assessed three specific parts of the GEC: the challenge fund design; the approach to monitoring and evaluation (M&E) requirements; and the approach to Payment by Results (PbR) – which in the GEC specifically means making payments for learning outcomes that are achieved and evidenced.

The review was based on an extensive document review and qualitative interviews with GEC stakeholders. This brief covers the key lessons learned, conclusions and recommendations that we made to help DFID and other policy-makers design and develop similar programs in the future.

Coffey International Development Ltd is leading the GEC EM consortium, which includes Research Triangle International (RTI) and Opinion Research Bureau (ORB). We are responsible for establishing, leading and managing a rigorous monitoring and evaluation framework to assess the effectiveness and impact of the GEC. We regularly share key findings and lessons learned to inform the future design of the GEC and wider DFID programming and policy-making.


Lessons learned

The GEC has not systematically targeted the ‘most’ marginalised girls, because at the outset, the evidence base explaining how and why girls were marginalised was weak.

At the start, the GEC Business Case set out to reach the ‘most’ marginalised girls. DFID decided not to prescribe the types of marginalisation factors projects should focus on. Instead these were left to projects to define. This was appropriate because there was a lack of evidence about what specifically caused girls’ marginalisation from education. It is clear that the girls targeted by GEC projects are marginalised, and in some places (e.g. Afghanistan), extremely marginalised. But they are not necessarily the ‘most’ marginalised. Projects were not required to identify, assess or define their target groups in this way. At the time, this would have been difficult given the lack of evidence that was available. We now know much more about which girls are marginalised – how, where and with what type of effects on their education.

Central management of a complex program through a Fund Manager is more effective than other options.

For a complex fund like the GEC, central management through a FM is likely to have been more effective than other options considered at the business case stage. This has ensured effective management, value for money, and consistency in applying financial and management requirements across all projects and countries – including: the grant application process; the development of guidance for projects; and decision-making about financial reporting, M&E requirements and PbR guidance. DFID did not have the capacity required, in terms of the number of staff, in-house technical expertise and level of flexibility, to manage a program the size and complexity of the GEC. By using a FM, a relatively small team within DFID has been able to have greater oversight and maintain a stronger focus on the strategic vision for the GEC.

A program inception phase to design and set up a complex program like the GEC is critical.

A complex program with rigorous requirements for M&E and PbR needs time at the start to design and setup the necessary management systems. The FM was contracted less than two weeks before the launch of the first and largest funding window, the SCW. There was no time to discuss and agree definitions and methodologies for PbR. Nor was there time to think through and develop M&E requirements or to communicate these requirements to applicants. This resulted in confusion for both applicants and projects from the outset. It had knock-on effects throughout the contracting and baseline stages. Once the SCW and IW were launched, DFID, the FM and the EM had to catch up with the fund design process, which led to delays and frustration for projects.

An inception phase for projects provides time to develop approaches and ensure compliance with program requirements.

DFID and the FM adapted the design of the IW. A two-stage approach to contracting (i.e. separate contracts for inception and implementation) was used. The introduction of an inception phase allowed time  for projects to develop their designs and M&E frameworks, ensure compliance with program requirements and develop stronger relationships with the Fund Manager and DFID. It gave DFID and the FM greater leverage over the quality of project designs compared to SCW projects. At the end of the inception phase DFID made go/no-go decisions about whether each project had achieved the standard required to progress to the implementation phase. This was an important mechanism for quality control while also allowing time to help projects achieve the standards required.

With the right type and amount of resources and management support, robust M&E is achievable in complex, multi-country development programs working in challenging environments.

It is possible to achieve rigorous M&E standards that are consistently applied in a complex, challenging multi-country, program environment. Robust M&E is expensive. It requires a significant amount of time and capacity to be effectively delivered. With substantial guidance, one-to-one remote and face-to-face support from the FM and EM, as well as an adequate budget, all projects in the GEC were able to produce M&E frameworks that met the program’s standards. The quality and comparability of the data collected at baseline will enable the GEC to assess whether projects are beginning to have an impact at midline and will provide robust evidence to inform decisions made under PbR. Projects’ capacity to meet the GEC M&E requirements has also increased. Projects have been able to use learning and tools from the M&E process to support other work. However, projects’ M&E capacity has been variable across both the SCW and IW, which at times has undermined the quality of the evidence produced. 

Involving DFID in-country staff is beneficial at all stages of the program, even for a centrally funded program.

Even on a centrally-funded multi-country program like the GEC, DFID country offices can offer useful insights and support for program design and delivery. Involving them from the start and throughout the life of the program would reduce potential overlaps in programming and improve synergies. When DFID country officers were more engaged with projects, the EM or the FM, their support added a lot of value. DFID country advisors provided contextual understanding, knowledge about saturation in specific areas, existing relationships with government, and links with other DFID (bilateral) programs.

Different tools and approaches are needed to involve private sector partners in development programs.

Organisations in the private sector have different interests and modalities to working than INGOs. They also require a more flexible approach and have different management requirements for effectively engaging with international development programs. The FM’s approach to the SPW successfully led to different types of public-private partnerships that are focused on delivering education outcomes. This is an innovative feature of the GEC. By working with different private sector organisations throughout the application stage, the FM was able to provide advice and help bring together organisations with complementary offerings. Over time, the FM adapted its tools and approach so that these were better suited to working with the private sector. 

PbR requirements lead to robust evidence of results, but may not be appropriate in all contexts.

PbR is difficult for many organisations, especially for smaller organisations and those operating in fragile contexts. PbR does not necessarily work when it is universally applied to all types of projects, organisations and contexts. Different types of PbR mechanisms need to be tailored to different types of projects, contexts and incentive structures. Applying PbR requires robust and credible data to evidence and demonstrate the results that are delivered, which triggers the right level of payment.

The level of rigour of project M&E increased for both the SCW and IW projects as a direct consequence of the PbR mechanism. In the end, projects working in fragile contexts were exempt from some M&E requirements for PbR. This was a result of projects’ concerns about the ethical, security and practical challenges of the PbR requirements, such as the use of control groups in high risk and fragile environments.

Recommendations

A universal approach to targeting girls marginalised from education was appropriate at the start of the GEC. However, DFID now has a growing evidence base that it can use to target specific groups of girls who experience particular barriers to education or are marginalised from education at different stages in their lives. 

Before the GEC, the global evidence base about what marginalised girls from education was weak. We now know more about how girls are marginalised from education, why this happens, and in which contexts. DFID has an opportunity to develop approaches that target specific types of needs. This would better address the context-specific nature of the problems that girls face at different stages in their lives and school experiences.

Central management of a complex, multi-country, multi-window challenge fund like the GEC has been effective in achieving a consistent approach across a global program. 

But global programs should ensure that linkages with DFID country offices and support through mechanisms such as Country Monitors are built into the program design from the start.

In the future similar programs should include an inception phase. This should allow enough time for DFID, the FM and the EM to design and set up an approach to program and performance management, including PbR and M&E requirements. 

An inception phase to develop program management processes and systems should be included at the start of a challenge fund. The inception phase should allow time to fully develop key performance indicators and project guidance supporting PbR and M&E processes. If an EM is contracted, the EM’s contract and inception phase should start at the same time. This helps ensure that the design of the approach to M&E complements the program design.

A rigorous approach to M&E can be achieved in complex and challenging program environments. But it needs to be appropriate to the funding objectives, project intervention designs and the delivery context.

This means that a universal (one size fits all) evaluation design should not be applied to all projects. It may not always help them deliver their objectives and may not be appropriate. Each project should be asked to carry out an evaluability assessment of their theory of change, for the purpose of working out the best approach to evaluating a project’s impact and performance. The requirements for implementing a rigorous M&E approach should be clearly communicated at an early stage of the program. This allows projects to plan and budget their M&E appropriately. It also avoids discouraging smaller organisations or organisations with particularly innovative, but riskier approaches from applying for funding.

Decisions about the use of PbR should be made early on in the process and its use should be tailored and adapted to the type of organisation, the project objectives and the context. 

Policy decisions need to be made at the early stages of the program design about whether PbR is to be applied, for what reasons, and under what program conditions. PbR approaches need to be tailored to the design and delivery context for each project rather than applied to all projects in the same way. PbR should be designed and applied to take into account: the type of organisation that is being incentivised by the PbR; the type of interventions being delivered; the nature of the target group; the type and scale of effects that need to be measured; and the type of project conditions in which PbR will be used.

Developing and implementing PbR across a program takes time. So its use should be considered from the outset and then planned appropriately. Sufficient time should be allocated at the program inception phase for the development and introduction of a PbR strategy and methodology, as well as related performance measurement, management and reporting requirements.

Read the full GEC Process Review Report.

Authors: Simon Griffiths, Ashley Craft, Maha Batran, Elizabeth Edouard