M&E Review: LEI Impact Report on UTZ Certification in Côte d’Ivoire

My posts have become  infrequent due to grad school and now thesis writing (certification impacts on cacao producers’ net incomes in Côte d’Ivoire). UTZ Certified (UTZ) just released a report on cacao certification impacts (PDF) that’s very relevant to my thesis, and an example of high-bar certification impact assessment that warrants wider attention. UTZ also released a response to the study (PDF), identifying challenges/gaps, as well as positive outcomes, and stating commitments to addressing areas needing improvement. Their candor and public accountability are commendable. Overall, this is a great model for impact reporting that I’d love to see used by more certifiers, and others undertaking other interventions, agricultural or otherwise.

Background

Côte d’Ivoire produces about 40% of the world’s cacao (ICCO, 2014). About 43 percent of the population lives below the poverty line (Hatløy, 2012). Cacao certification has grown rapidly, in part as a way to address a looming supply deficit, producer poverty, child labor and environmental degradation (TCC, 2012). Hershey, Mars and Ferrero have committed to sourcing 100% certified cacao, driving continued demand growth. Certification adds costs for producers, companies and consumers. Given these factors, it’s critical to ensure certification has positive returns. Having read myriad studies, including peer-reviewed articles, certifier reports and third-party evaluations, and having visited farms in numerous countries, I’m aware (as are many, many others) that certification impacts vary and aren’t unilaterally positive. Most research to date has focused on coffee, making rigorous, independent studies on cacao certification essential.

UTZ Certified Study Scope

UTZ commissioned LEI Wageningen UR, a research institute, to gather baseline data on UTZ certified (725) and non-certified (55) producers from 97 co-ops (89 certified), and conduct a follow up survey in 2016. Producers were drawn from across the country, and certified producers had been certified for different lengths of time. Some UTZ-certified farmers also had Rainforest Alliance (RA) certification. The goal of the research is to evaluate farmers’ knowledge and implementation of the agricultural, social and environmental practices that are included in the UTZ Code of Conduct (GAP practices); and evaluate the effects of certification yield, income and other areas. The survey aligned with UTZ’s Theory of Change, serving to test Utz’s beliefs about how it affects change.

Main findings

Note: As stated by the authors, since the study does not include data from the time before producers attained certification, it is not certain how certified and non-certified producers differed initially, and the observed differences cannot be attributed solely to certification.

• Producers that are UTZ certified, and UTZ and RA dual certified, have higher knowledge of GAP Practices than non-certified farmers
• Producer knowledge of specific GAP practices differed, with knowledge being lowest for child labor, record keeping and numerous farm management practices. Thus, there’s room for improvement in training.
• Higher knowledge of GAP is correlated with higher productivity and higher GAP implementation overall, though not for each GAP individually. For example, post-harvest processing and quality were not necessarily higher among farmers with higher knowledge.
• The longer farmers have been certified, the higher their knowledge of GAP practices
• Certified producers reported higher productivity and incomes after certification.
• Gender inclusion could be improved, as the majority of those directly reached by UTZ training are male (as are the majority of cacao farmers). Male farmers who are reached directly train their family members, so females are reached indirectly.
• UTZ’s summary/press release is here

Methodological strengths

• A third-party study provides a more independent analysis
• The report details the sample, selection process and data; methodological limitations and the claims these allow
• Surveys took place across in three cacao regions, providing a representative sample
• In regions, certified producers and controls were from agroecologically comparable areas to control for growing environment
• In regions, surveys took place in areas with marginal, good and excellent suitability for cacao, allowing analyses of how growing environment modulates outcomes when certification status is held constant. Stratified sampling was used to make the sample for each suitability level proportional to population distribution
• The study controls for the effects of cooperative membership by including only cooperative members. Many studies fail to do so.
• The study includes co-ops selling to different buyers, controlling for buying relationships and producer support programs
• The large sample size allows for higher-level statistical tests, higher validity of group means, and more confidence in statistically-significant group differences (albeit still affected by selection bias).
• The use of controls allows for an analysis of the effects of certification versus the status quo (ditto). Controls also were randomly selected, reducing bias in the results.
• The study uses regression models to address selection bias, a higher-level technique not seen in many public reports from certifiers.
• The  follow up survey will allow for a “difference in difference” analysis that enables one to make more confident claims about program impacts, by comparing change over time across groups.

Limitations

• Not clear how certified co-ops and producers were selected
• By surveying only cooperative members, who represent about 15% of Ivorian cacao farmers (Hatløy, 2012), the study precludes conclusions about how certified producers differ from the majority of cacao farmers in the country.
• Report (p 49) notes several others, such as poor recall among farmers (few keep farm/finance records), farmers’ lack of clarity about certification status and length, and certification training; involvement of some controls in certification training, limited training for stakeholder interviews; and discrepancies in data provided by UTZ, co-ops and traders. Most of these are common issues across certification studies and development fieldwork more broadly.
• Not subject to academic peer review, yet

Citations

Hatløy, A., T. Kebede, P. Adeba and C. Elvis. 2012. Towards Côte d’Ivoire sustainable cocoa initiative. Fafo.

International Cocoa Organization. 2014b. “Quarterly Bulletin of Cocoa Statistics: Cocoa year 2013/2014, XL (3), 9/5/14.” Accessed 9/9/14 at icco.org.

Tropical Commodity Coalition. 2012. 2012 Cocoa Barometer. Author.

Advertisement