Theory of Change Review as an Adaptive Management Tool

This blog was written by Nancy Peek, M&E Fellow, USAID/KEA/FFP and Laurie Starr, Strategic Learning Advisor, Save the Children

Introduction

Adaptive management is a key principle that supports ADS 201 and is described as a structured, iterative process for making decisions in “response to changes in context and new information…that promotes intentional learning and minimizes the obstacles to modifying programs.” The USAID Office of Food for Peace (FFP) has identified a number of specific processes and approaches to employ with implementing partners to incorporate learning and information into new and ongoing activities. One such approach, as outlined in the 2016 “Office of Food for Peace Policy and Guidance on Monitoring, Evaluation, and Reporting for Development Food Security Activities”, is a requirement for partners to develop a Theory of Change (TOC) for their activities and to review it “whenever there is new evidence, or when there are changes in the context that affect assumptions or hypothesized pathways of change” and, at a minimum, annually.

While USAID partners and staff are familiar with the TOC as a design tool for activity application or start-up, the TOC review is intended to encourage stakeholders (activity staff, recipients, etc.) to engage with the TOC throughout implementation and to view it as a “living” document that is revisited and adjusted as needed to inform adaptive management. In practice, many partners and stakeholders do not have experience interacting with TOCs to translate and validate what is contained in a TOC in comparison to the reality of implementation on the ground. Based on lessons learned from TOC reviews with FFP development partners, this document offers considerations for the purpose, timing, process, and outcomes of a TOC review. We hope that organizations eager to encourage regular use of TOCs to inform adaptive management find this guidance useful. This guidance can also be adapted and used by USAID staff who may be interested in a periodic, systematic review of a project-level theory of change or conceptual framework.

Purpose and Objectives

The main purpose of a TOC review is to ensure alignment of the TOC with the goals, outcomes, or problems that you are trying to address in your activity, and reflect on and learn from evidence and experience to-date in order to inform adaptation, innovation, and improvement in activity implementation and management.

Some key objectives of a review could be to:

• Discuss additional knowledge of the local context, observed changes in conditions, considering the nature and sequence of changes, and relate the observed to the anticipated pathways of change and expected outcomes (along with inhibitors and facilitators of these changes).
• Verify if the causal pathway logic leads to expected outcomes. Consider whether existing pre-conditions are still sufficient to assure the desired change and establish whether they are still valid and supported by data/evidence available.
• Consider how the status of necessary outputs or outcomes that are controlled by other actors may influence pathway logic and progress.
• Consider how the validity of assumptions may influence pathway logic or progress
• Identify areas in which cross-cutting issues can be more clearly incorporated into programs.
• Identify interventions that have not been implemented and new interventions that were introduced since the TOC was developed.
• Assess how useful the M&E data are in measuring changes and issues of interest to the activity.
• Analyze progress toward program goals using M&E data.

The focus of a TOC review may vary by phase of activity implementation:

• Activity start-up or refinement phase: After activity award, the TOC from the proposal should be reviewed based on up-to-date reflection of the context and on-the-ground implementation reality. The TOC review can further contextualize the TOC, identify evidence gaps and areas that need further analysis, re-visit assumptions, and refine the design of the activity. This is also an opportunity for field staff to come together at the outset to develop a shared understanding of activities, outcomes, causal pathways and interactions. Finally, the first year is a critical time to assess whether the MEL plan includes sufficient means for regularly monitoring change for critical TOC outcomes.
• Early implementation years: The review may focus on firming up TOC output to outcome pathways. Using monitoring data, the stakeholders should review the practice level outcomes and early indications of systemic changes. In addition, the review should also look at the outputs to promote sustainability and indications of early outcomes.
• After midterm: Towards the midterm of the activity, the TOC can be revised based on recommendations from a midterm evaluation (if applicable) and the review should begin to verify lower level results and their contributions to expected outcomes. TOC reviews near the end of the life of the award can verify that higher level outcomes will be achieved and final targets reached.

Timing

Aside from set annual reviews, additional reviews with all or a subset of activity stakeholders may also be useful, especially for activities operating in dynamic environments.

Examples of evidence or situational factors that might prompt a special review of a theory of change are:

• An intervention or lower-level outcome failed to influence the next level outcome as expected.
• An intervention was applied in an unanticipated way or led to an unexpected outcome.
• Activity monitoring or a special study revealed that the quality or efficiency in reaching outcomes depends on additional factors not portrayed in the TOC.
• New research external to the activity supports previously unknown causal pathways or refutes previously accepted pathways portrayed in the TOC.
• Significant changes in the political or environmental conditions of the local context.

Guidance for the Review Process

A TOC review should be informed by what has been learned from a variety of available data sources such as, assessment findings, qualitative inquiries, community consultations, monitoring data, recent evaluations and assessments, sector specific research or analysis, learning events, stakeholder consultations, partner and program participants’ experiences, and secondary or other sources of information. Keep in mind that a TOC review does not have to wait until all the data are available, rather the activity staff can continually review and revise as the data become available to them. Likewise, a TOC review does not have to encompass the entire diagram, but can focus on key pathways or elements of interest (keeping in mind that any linkages will need to be re-assessed).

Other considerations for undertaking a TOC review are to think through who should lead the process and who should be invited to participate. Partners may choose how to best facilitate the review, but the process should emphasize the use of data to review the progress or non-progress along the change pathways to inform decision making. The following sections provide guidance on the process for preparing for and conducting a TOC review as well as considerations for who participates and how.

Preparing for a Review

For annual reviews, monitoring and other contextual data should be collected, analyzed, and organized early in the preparatory process so that the review itself can focus on learning from the data and using that learning to inform programmatic or operational change.

Before the review sessions, the MEL team in collaboration with the technical/sector teams should address the following:

• What information needs to be compiled/in place before the review?
• Identify the associated interventions and their status for all outputs on the TOC
• Align indicators (from LogFrame or indicator table) to TOC
• Gather actual values for relevant indicators, % of target achieved, and map to the TOC diagram
• Check for (new) interventions not considered in TOC and identify outputs to add to the TOC
• Gather information on the status of all assumptions in the TOC, i.e., whether they are true, and any changes to context
• Assess the status of necessary outputs or outcomes that are controlled by other actors
• Synthesize relevant findings from formative research and other studies
• Gather evidence on the extent to which rationales are holding
• Prepare any presentations to be used during the review (i.e., overview of pathways, assumptions)
• Capture staff observations on change that may not be formally monitored
• Create a facilitation plan and agenda for completing the review. Determine if breakout groups will review different sections of the TOC or if all participants will complete a full review of the  TOC. Groupings may depend on staff availability and the nature of intervention integration across the activity.
• What resources are needed?
• Facilitator(s) – internal or external
• Designated note taker or rapporteur(s)
• Large TOC diagrams (printed as large posters for capturing changes directly on diagrams or projected on the wall)
• Electronic versions or printouts of supporting documentation for each participant (assumptions, rationales, LogFrame)
• Flip charts, sticky notes, markers
• Appropriate venue for the sessions
• Who is invited to participate?
• Include participants with diverse perspectives, across sectors, field locations, etc.
• At minimum, include representation from: senior management (Chief of Party (CoP) Deputy Chief of Party (DCoP)), sector/team leads, field supervisors, field staff who have regular contact with participants, gender specialists, MEL team, any sub-partners, relevant USAID staff.
• External stakeholders may be included, but consider implications on the team dynamics and feelings of openness and honesty that may be precluded if external groups are present during the review discussions.
• One option is to get input from key external stakeholders beforehand to feed into the internal staff review. After the review is complete, provide feedback to them and allow them to validate changes.

Conducting the TOC Review Sessions

There is no prescribed method to conduct the review sessions, and partners may choose the best process for their context. The following section outlines a basic structure and components for a three-day review workshop that can be adapted as needed, given the purpose of the review and time and resources available.

½ Day: All participants

1. Introduce the purpose, objectives, and process for the TOC review. Discuss expectations, emphasizing the USAID intent to foster and maintain a culture of learning, starting with their observations and experience.
2. Begin with a review of the overall TOC: Purpose, elements, key guidelines (present examples of deficiencies, if any).
3. To ensure everyone understands how to tell the story of change, walk through a few sample pathways beginning from outputs of the key interventions (1 or 2 interventions per sector/purpose)
   
   
   2 Days: All or participants from designated breakout groups (plus CoP, DCoP, MEL team) 
   Proceed, intervention by intervention, sequentially until all interventions have been considered (approx. ½ day to cover sector-specific portion of TOC).
   
   
   
4. Lead open-ended discussion to identify and list observed changes in the community as a whole and among beneficiaries of different interventions (without the TOC).

• Identify perceived promoters and inhibitors of the observed changes (both within the activity and external factors).

• Identify external inhibitors and promoters of hypothesized changes.
• Examine validity or invalidity of assumptions.
• What is the status of outputs or outcomes controlled by other actors?

• Are targets appropriate and being met?
• Are results along the pathway adding up to achievement of higher level results/outcomes? Anything unexpected?
• Return to the TOC - are indicators relevant for measuring the results in the TOC? Are they capturing important changes in the causal pathways? Are there information gaps? Are data being collected which is not used/needed? Are there outcomes for which new measures are needed?
• What are the implications for the coming year’s targets?
• What are the implications to implementation?

• Significant changes to their sector in the TOC
• A review of contributions from other sectors/teams for discussion, identifies any information needed from another sector/intervention or from outside the activity that is currently not received, and invites suggestions about how to obtain the information
• Changes that will affect contributions to other sector interventions and data availability for other sectors/portions of the TOC
• A summary of recommendations to change future targets

After the Review

Once the review session is completed, finalize the adjustments the TOC diagrams and supporting documentation to:

• add, remove or replace activity outputs, based on actual and current plans for interventions;
• adjust pathways and restate assumptions to reflect current knowledge about actual outcomes and the status of external factors (assumptions, outputs/outcomes of other actors’ efforts);
• refine and update any supporting documentation, as needed, including adjustments to indicators used to measure elements in the TOC.

It is important to work closely with the Agreement or Contract Officer’s Representative (A/COR) before and throughout the TOC review process, so that USAID may anticipate any potential changes to the activity. The Office of Food for Peace considers changes to TOCs, with justification, at any time. The revised TOC is submitted for AOR approval with an explanation of the implications of the TOC changes to the interventions, implementation, and MEL Plan.

Aside from documenting changes, it is important to have the technical team, key staff, and CoP follow through the TOC implementation. Consider creating a tracker or having an After Action Review some months after the TOC review to ensure the implementation has been appropriately adapted as identified during the review. Look for strategic times during the year to “check-in” on interventions and their progress against the TOC, such as during annual work planning or reporting.

Other Tips and Considerations

• Avoid getting tripped up in the details of individual staff’s or any one team’s work. Keep the discussion focused on the strategic vision and common goal for the program by emphasizing broader learning questions: Are we on our way to reaching outcomes? What interventions are not contributing and should be removed? Where are there challenges that require more effort? Is the overall activity being implemented as designed?
• Designate note takers to help with documentation and take pictures as changes are made.
• Agree on process, facilitation plan, composition of breakout/sector teams, deliverables/outcomes of the process ahead of time.
• Make particular note of changes that have financial implications which require special consideration from the AOR/COR.
• Throughout the review, integrate discussions about the learning culture of the program. Encourage thinking about linking collaborating, learning, and adapting (CLA) and M&E. Get people to think about change that is happening beyond indicators and what isn’t necessarily in the MEL plan.
• The TOC review can provide a good opportunity to concurrently update activity stakeholder mapping as you become aware of new actors and changing roles and relationships.
• One of the biggest challenges partners have faced in undertaking TOC Reviews is that documents are typically not complete, organized nor up-to-date, leading to insufficient information to be able to conduct the review or delayed reviews as staff spend time getting the documents in good shape. Preparing for a TOC Review provides a good opportunity to shore up data collection and information systems as well as assess data quality.

For more information on how to conduct a Theory of Change Review, contact Nancy Peek, M&E Fellow, USAID/Kenya and East Africa/Office of Food for Peace at [email protected] or Laurie Starr, Strategic Learning Advisor, IDEAL Program [email protected].