USAID Brazil Data Quality Assessment for Standard Biodiversity Indicators

EXECUTIVE SUMMARY

• This 2017 Data Quality Analysis (DQA) of the Partnership for Conservation of Amazonian Biodiversity (PCAB) Project reports on the quality of data used to measure five standard indicators of PCAB outputs or outcomes. This DQA report also includes recommendations for USAID and the PCAB implementing partners (IPs) to implement for improving procedures for standard indicator data collection and reporting.
• The BAME team administered the 2017 DQA using the revised USAID Checklist (per ADS 201.3.5.8). The BAME team employed an assessment methodology that included: (1) an examination of USAID/Brazil mission and PCAB IP project documentation provided by the USAID mission and by IPs; and (2) structured DQA Team interviews with current principal PCAB IP representatives involved in collecting and handling data to measure the standard indicators they report to the Mission.
• This 2017 PCAB DQA found that both USAID/Brazil and its PCAB IPs have taken steps to develop plans and processes for good PCAB indicator data collection and indicator reporting. The DQA also identifies areas of IP PCAB M&E Plans – particularly the Performance Indicator Reference Sheets (PIRS) – that need clarification and further information to assure high quality data collection and reporting. 
• Overall, the PCAB standard indicator data are collected within acceptable time and resource constraints, particularly where local partner activity reports and administrative records can be consulted and used as resources and documentation.
• The PCAB IPs should be recognized for their timeliness in reporting and their integrity for safeguarding the indicator data they collect, manage and report. In particular, the 2017 DQA concludes that each of the PCAB IPs has in place practical data management systems with controls and safeguards against unauthorized access.
• Standard Indicator EG 10.2-4 (Number of people trained …) is an example where PCAB data presented validity uncertainties because data collection methods appear to result in possible under-reporting of the impact of training activities. IPs report almost uniformly report the numbers of persons who successfully complete workshop training to measure this indicator. Limiting “people trained” strictly to workshop participation, however, overlooks the skills transfer and development that IPs are also providing to their local partners and beneficiary community organizations. This happens through on-the-job skills transfer when beneficiaries are working beside IP sponsored technicians, who are guiding them on the use of innovative technologies for habitat monitoring. As a result, standard indicator EG.10.2-4 may be under reported for some IP activities.
• The DQA team is main concerned about data validity with Indicator EG.10.2-2: “# of hectares of biologically significant areas under improved natural resource management (including broad management improvements, capacity building) as a result of USG assistance”. The indicator is valid to the extent that the work of the IPs in improving management does generally improve the management of the Conservation Units, though the linkage is not strong and clear. The USAID Mission appears to be aware of EG.10.2-2 issues because it has introduced a custom indicator whose validity linkage appears much stronger, that it calls a ‘subset of Indicator EG.10.2-2 “Number of hectares of biologically significant areas receiving direct application of improved natural resource management as a result of USG assistance.”
• Greatest confidence in data reliability exists for data measuring those PCAB indicators – particularly “EG.10.2-3 Number of people with improved economic benefits ….” EG.10.2-4 number of people trained….” which can be more closely tied to direct IP project interventions … and USG support. Given the nature of the indicators covered in this DQA, the team believes that the processes employed are sufficiently reliable.
• Data precision largely applies only to PCAB Standard Indicator “EG.10.2-1 # of hectares of biologically significant areas with improved biophysical conditions….” Only one IP is explicitly employing sampling methods to measure this indicator for the entire area for which they are responsible under the PCAB program. The other IP uses official GOB statistics. PCAB reporting of this indicator may be somewhat constrained, as different methods are used, making it impossible to roll up indicator data for the IPs to produce one total PCAB program number.

DQA Recommendations for the PCAB Implementing Partners are:

• Revise and update their M&E plans with completed PIRS Annexes for each of the relevant indicators (among the 12 USAID/Brazil standard and custom indicators) that they are respectively required to report to the Mission. These updated M&E plans should document with sufficient detail how these indicators are defined and how data is collected to measure activity performance and impact. USAID’s Learning Lab can provide a template M&E plan as a model, and IPs should attempt to implement this as much as possible. 
• Consider budgeting for at least one half-day IP staff orientation. This would introduce IPs to good indicator data collection practices, and would be facilitated by an outside expert. IPs could include this in their periodic or annual staff and local partner meetings. These M&E sessions should be targeted particularly at new staff members, given that all IPs have identified staff turn-over is an on-going reality of their work.

DQA Recommendations for USAID/Brazil include:

• Move quickly to prepare complete PIRS for each of the mission’s required custom indicators. PIRS for these custom indicators involves the basic process of filling out the PIRS form for each indicator, providing basic information about how they are defined and measured, listing whom is responsible for reporting, and indicating when to report.
• Consider engaging a data systems specialist to better inform the Mission about the most cost-effective ways for implementing partners to collect standard and custom indicator data. This specialist should use this DQA as a base for assessing where existing, proven, innovative, and online-based monitoring and information communications technologies can be used for remote data entry by IPs. This would allow for real time data retrieval and visualization – both graphic and tabular – of indicator data by managers of USAID/B funded PCAB program activities.
• Consider engaging an M&E expert more directly in PCAB IP gatherings at least once annually. The expert would conduct half-day M&E workshops to build basic indicator data collection skills, foster feedback to USAID on data collection and reporting challenges, share experiences, and make suggestions on cost-effective data collection and measurement. The IP workshop could be based on findings about current standard indicator data procedures from this DQA Report.