Community Contribution

Adaptive Learning: Five lessons from the Family Care First initiative in Cambodia

Apr 05, 2019

Naomi Joswiak and Christina Synowiec

Normal 0 false false false EN-US X-NONE X-NONE /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri",sans-serif; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin;}

This blog is the second in a two-part series about what Rapid Feedback Monitoring, Evaluation, Research, and Learning (RF MERL), an M&E innovation being tested within USAID, learned from our first completed engagement. The first blog, “Testing behavior change communications to reduce family-child separation in Cambodia,” captures evidence from two behavior change interventions we tested to reduce unnecessary family-child separation in Cambodia. In this blog, we will discuss what we learned about applying the RF MERL approach itself within USAID-supported activities.

In part one of this blog series, we discussed what the RF MERL Consortium learned from our partnership with Cambodia Children’s Trust (CCT) and Friends International (FI) in addressing unnecessary child-family separations in Cambodia. In the second part of this series, we reflect on what we’ve learned from applying the RF MERL process and approach — including key takeaways from our relationships with partners and adaptive learning methods in general.

RF MERL is a rigorous, flexible approach that generates timely information for programs to adapt and improve on activity design throughout implementation. RF MERL is one of several M&E innovations being piloted in USAID by the U.S. Global Development Lab, Bureau for Policy, Planning and Learning, and the Bureau for Global Health, to test new approaches within USAID to better understand the impacts of development programs. In applying the principles of adaptive learning to our own process, we wanted to share five takeaways from our first engagement, the Family Care First (FCF) initiative in Cambodia.

1. Prioritize adaptive learning activities in program areas with the greatest uncertainty.

Applying adaptive learning to ongoing implementation activities can have the largest impact when using a technique called feedback loops, where areas with the most ambiguity are repeatedly tested and refined. Jean Arkedis, senior program director for Results for Development’s Evaluation and Adaptive Learning practice, stressed this point: “Embrace and articulate your areas of uncertainty and make them the focus of early, quick and lean testing.” When we first began work on FCF, the RF MERL team was eager to design and implement Feedback Experiments, or rigorous tests comparing multiple intervention options to generate feedback and iteratively improve on already established activity design. However, we quickly realized that there were opportunities to implement smaller-scale adaptive learning activities within earlier stages of the program. These initial activities were formative in defining the activities that would influence the program longer-term, relying on approaches that involved less quantitative rigor given the level of uncertainty in activity design. Some of the preliminary questions we considered included: who should we target during a behavior change campaign?; what messages will resonate with our target audience?; and how should we disseminate the campaign? Ultimately, revising our initial plan to first include these less rigorous activities, such as formative research, rapid prototyping, and lean testing, helped to inform the design of the feedback experiments and generate more confidence in our learnings.

2. Build in continual feedback loops that evolve in their design alongside the implementer’s learning questions.

When the Consortium initially designed the RF MERL approach, we discussed how we would rely on continuous feedback loops to provide regular insights on emerging issues and progress achieved. In practice, this often involves routine monitoring or multiple rounds of data collection throughout an activity to allow for frequent readjustments during implementation. While working on FCF we also learned that carrying out multiple iterations allowed us to modify and revise future activities as our level of certainty grew and the research questions evolved. We started by developing and refining the initiative’s theory of change and identified significant uncertainty around social behavior change as a key outcome. From this, we used formative research to examine the best ways to reach the donors and community members we were targeting through the behavior change communication campaigns. We took these learnings through several rounds of lean testing to fine tune multiple messaging formats for the campaigns, before finally using that evidence to scale up the behavior change campaigns during the Feedback Experiments.

3. There is no “one size fits all” approach to adaptive learning and flexibility is required in designing program activities.

Having a certain level of flexibility allows us to propose the most relevant activity based on criteria like how far along the program activities are and the level of uncertainty in the program design. Starting with less rigorous activity options in earlier phases, such as refinement of the theory of change and formative research, allows us to use faster and less costly methods to identify and answer early and essential design questions that will inform the later and more rigorous activities like a feedback experiment. FCF helped us refine this flexibility in our approach — as previously noted, we quickly realized that jumping straight into experimental design was premature and instead should start with less rigorous but more informative design activities — which helped us hone the selection of available RF MERL activities geared toward USAID-supported programs. Understanding when and why an RF MERL activity is most useful has given us the ability to select the most relevant activities for each engagement.

4. An in-country presence helps gain implementer trust and buy-in.

Our implementing partners, funders and our own team agreed that one of the biggest drivers of success and buy-in for this engagement was embedding a team member in-country. This team member coordinated the design and implementation of RF MERL activities in partnership with the implementing partners, USAID stakeholders, and data collectors. Additionally, an in-country presence gave CCT and FI confidence that RF MERL was not just an external evaluator, but rather that RF MERL was a committed partner dedicated to capacity building and actionable learnings. Having a person embedded in-country meant dedicated access to MEL support, more comprehensive understanding of the operational constraints on-the-ground, and more in-depth assistance on implementation of the rapid feedback activities. While we aim to create lasting impact for the implementers, the in-country coordinator bridged the gap between multiple stakeholders to establish strong relationships and alignment between RF MERL, CCT, and FI, and assisted the implementers in thinking through their design and implementation plans through the RF MERL lens. For future projects, we plan to work with an in-country coordinator at key points throughout engagements to ensure fidelity to RF MERL principles and strong working relationships with implementers.

5. Adaptive learning engagements, which are co-designed and co-implemented, require high levels of engagement.

While RF MERL activities are designed to fail fast (or succeed swiftly), the collaborative nature of an engagement may be more time-consuming than is feasible for some implementing partners. While it is important for our partners to be aware of the time investment required for a typical RF MERL engagement, this relationship is also valuable in building their capacity for adaptive learning activities in the future. This is particularly true for the design stage of the engagement while working together to select activity areas and research questions that could benefit from rapid feedback support, and during ongoing monitoring and evaluation activities while carrying out data collection and analysis activities. As a result, this participation can result in improved MERL capacity for implementers involved in these activities. We recognize that smaller implementing partners might not have enough resources to take forward engagements without additional support, particularly for higher-rigor activities, so it is an important part of any scoping and design discussions to ensure that the RF MERL activities selected are appropriate from both a research and implementation lens. Ultimately, our goal during these engagements is to build strong learning partnerships with implementers who are involved throughout to ensure that the program activities are relevant and useful moving forward.

More about Family Care First Cambodia:

Working with the Royal Government of Cambodia and private donors, the United States Agency for International Development (USAID) launched the Family Care First (FCF) Cambodia initiative in 2014. Guided by the collective impact model for structured, multi-sector collaboration, one of two cooperative agreements was issued to Save the Children for the Cambodia Families are Stronger Together (FAST) project. The FAST project includes more than 25 implementing organizations, including Cambodian Children’s Trust and Friends International, and seeks to develop a comprehensive care system in Cambodia to prevent family-child separation and promote the reintegration of children from RCIs to families. The FCF partnership with RF MERL was supported by USAID’s Global Development Lab (Lab)/Center for Development Innovation (CDI) and the Bureau for Democracy, Conflict and Humanitarian Assistance (DCHA)/Center of Excellence on Democracy, Human Rights and Governance (DRG)/Empowerment and Inclusion Division (EI), and USAID/Cambodia.

More About RF MERL:
The United States Agency for International Development (USAID) signed a cooperative agreement with a consortium of four organizations including Results for Development (R4D), Abt Associates, Mathematica Policy Research, and the Notre Dame Initiative for Global Development in September 2015 to implement Rapid Feedback Monitoring, Evaluation, Research, and Learning (RF MERL). RF MERL is an innovative initiative under the Monitoring, Evaluation, Research, and Learning Innovations (MERLIN) Program through the U.S. Global Development Lab in partnership with the USAID Bureau for Policy, Planning, and Learning and the Bureau for Global Health.

Learning at USAID

Collaborating, Learning and Adapting (CLA)

Monitoring

Evaluation

Insights & Practice

Community

Adaptive Learning: Five lessons from the Family Care First initiative in Cambodia

Adaptive Learning: Five lessons from the Family Care First initiative in Cambodia

Tags