The promise and pitfalls of indices to measure complex governance outcome concepts

A recurring frustration among international development implementors and evaluators is our over-reliance on reductionist indicators to measure outcome-level changes. Such indicators often do not capture the degree of change desired in a project or the complexity inherent in that outcome. Nevertheless, they are pervasive in monitoring and evaluation plans because they offer a clear, binary judgement of project success, which is attractive for reasons of accountability and box-checking.

There is an opportunity cost to such indicators. If they spend time and resources on overly simplified questions rather than on ones that are truly valuable to them, project teams will have a hard time understanding and learning from challenges and successes. Indicators that are meant to support evidence-based adaptive management and data-driven project learning then do just the opposite, masking rather than elucidating.

For example, while “Number of communities with improved natural resources management” may well be an appropriate outcome-level indicator, it is static, binary, unidirectional, and overly simplified. Specifically, it does not measure or communicate the degree of improvement in natural resource management, does not allow space for recognizing a worsening of natural resource management, and does not capture the myriad of often interrelated aspects of natural resources management that may explain change pathways necessary for learning.  

Reductionist indicators are partly a result of resource limitations and funding obligations – we do not always have the funds necessary to implement the evaluation we want. But it is also because measuring complex concepts is challenging. Unfortunately, many of the outcomes we want to measure in development work are weakly-specified intangibles: “women’s empowerment,” “civic engagement,” “social cohesion,” to name a few.

To address this, we can turn to an index – that is, a composite measure that combines multiple sub-indicators in one tool in a standardized way. An index composed of multiple sub-indicators enables us to measure more complex concepts such as “social cohesion.” Because such concepts include multiple factors, an index brings these multiple factors together in a way that both operationalizes the concept and enables its unified and disaggregated measurement.

Figure 1: Example of how the CCD team unpacked the concept of "citizen security."

Developing and using indices, however, is not without its own challenges, which is why they are rarely used. We recently presented on the use of indices to measure complex governance outcomes at the annual conference of the American Evaluation Association. The presentation aimed to demonstrate the utility of indices for complex concept measurement, and some lessons to consider. We differentiated between universal and goal-based indices. Universal indices provide a normative measure in the sense that they establish a single standard for the concept and can be used to measure and compare multiple units and over time. An example of this is Freedom House’s Freedom in the World country scores or Pact’s Organizational Performance Index. In contrast, goal-based indices are tailored to a specific goal or project and are not designed necessarily to measure units outside that goal or project.

Hayat Askar discussed various types and uses of indices, with examples from government evaluation in Jordan. She unpacked how indices are selected and developed, by focusing on the “6 W’s” : Why, Who, What, When, Where, and Which. By answering her list of questions, project teams can select the index that best meets their primary purpose, needs, timeframe, resources and role, which will help them avoid potential pitfalls, especially when using and combining existing indices. She made the important point that many existing universal indices are themselves constitutive of other universal indices, and therefore caution must be taken when using them in evaluation to avoid endogeneity.

Figure 2: The CCD Index preliminary baseline results.

Chhork Boeurg and Lina Maria Jaramillo Rojas both presented on goal-based indices developed by Pact. Lina Maria discussed the challenge of measuring the concept of “citizen security” for the Conectando Caminos por los Derechos (CCD) project, which works with Venezuelan migrants in Colombia. The CCD operating context is very dynamic, and the project goals are highly complex: in such a program, change is often not linear, and regressions may occur. For this reason, the project team wanted a way to measure more nuanced changes, rather than just a binary measure of the outcome of citizen security, to support adaptive management. Specifically, their index needed to measure the overall desired outcome, and at the same time, understand the variable change pathways of components of that outcome. The index approach has enabled the team to measure both negative and positive changes and at multiple levels, while at the same time providing an overall measure of the outcome.

In addition, the index approach enabled the project to take a more inclusive definition of citizen security by including more complex concepts. Historically, citizen security has been focused on issues of state security, such as in policing. The CCD project team wanted to broaden the concept to take a human rights perspective and to consider aspects of social trust and cohesion, acknowledging the important role communities play in citizen security. The index they developed, the “Citizen Security and Social Integration Index,” started from the project’s theory of change, and is organized into three domains: community cohesion, enabling governance, and social inclusion.

Figure 3: The WE Act Index concepts.

Pact’s Women Entrepreneurs Act (WE Act) program in Cambodia uses an index in a similar fashion to track changes in women’s economic and civic empowerment (WECEI). The index provides a program outcome-level measurement while also enabling the tracking of individual women and at various levels. Adapting from the Kabeer framework, the program looked at two component outcomes, civic empowerment and economic empowerment, among entrepreneurs and non-entrepreneurs. The index itself is organized into three domains linked to the program’s theory of change: resources (access, knowledge), agency (perceptions, attitude), and accomplishment (behavior, access). The theory states that if women are empowered, through improved resources and agency, they will become accomplished entrepreneurs and leaders and will therefore be more willing to engage in and influence civic issues.

Because of the complexity of the program’s operating environment and desired change, the project team recognized that women may advance in some areas while regressing temporarily in others. For example, as her awareness about her business and political environment grows, her perceptions about her personal agency may reduce, which is sometimes an indicator of positive change on a linear trajectory. Chhork discussed the challenge of operationalizing this and other aspects of the index in terms of data collection and the calculation of changes over time. The project team designed a survey that operationalizes each index domain and subdomain into multiple survey questions. The team had to assign a weight and scoring system for each survey question to enable aggregation of each question to form a score for each indicator. In essence, this created multiple mini-indices, one for each index subdomain indicator. The index baseline and midline data are helping the project team to design very specific interventions to improve the overall index score over time.

Key take-aways for using indices to measure complex outcomes

1. An index can provide an outcome-level assessment while showing the variable pathways of composite aspects, allowing for progress in some and backsliding in others to help adaptive management.
2.  Don’t take an index at face-value: Review the index subcomponents and its source data to ensure it is measuring what you need it to measure.
3. When developing your own index, use a theory of change as the foundation of the index to help unpack a complex concept, and to justify the inclusion of an indicator in the index and its relative weighting.
4.   Minimize the amount of data needed to measure each component of the index to reduce the data and analytic burden and the non-completion or drop-out rate of respondents if using survey research.
5.   Consider hiring a methodologist to help review the index and accompanying measurement plan, especially if the index is operationalized through survey research with multiple survey questions per indicator.
6. Consider partnering with national partners including think tanks and government offices, which will build synergy and encourage the sustainability of the index after the project ends.
7.  As always, pilot test any instruments before large-scale roll out and build in extra time for the design and testing process; don’t be shy to adjust the index at midline and compare with an endline. A validity and reliability test may also be appropriate.
8.  Consider a data simulation exercise to ensure the index will provide information useful for project adaptive management before more time and resources are invested.
9. Utilize qualitative methods as part of index data analysis to ensure you have a complete picture for adaptive management.

Have you used an index to measure a complex outcome? Share your experience!