Author: Arthur Gaillard is currently a PhD Candidate at the School of Sport and Health Sciences of the University of Brighton (United Kingdom). His research focuses on the use of sport in international development settings. Specifically, he is interested in the epistemic implications of the ways in which these international development programmes are evaluated. You can follow Arthur on Twitter at @ArthurGaillar14 and on LinkedIn.
“Look at the numbers!” Who hasn’t heard or even said that phrase during a debate? I must admit, I have used multiple times the insinuating power that numbers are objective, and therefore, beyond contradiction. In our daily lives, we are constantly surrounded by indicators and other quantitative measures that claim to objectively represent reality. Does your company evaluate your work with performance indicators? That is exactly what I am referring to. We tend to take these indicators for granted, but things are not that simple.
During my PhD research from 2019-2021, I conducted fieldwork with an NGO that delivers sexual health education to children in various African locations through a methodology based on the use of football drills. My PhD focused on the way the NGO evaluated its programme. My fieldwork involved taking part in the NGO’s evaluation, thus witnessing the processes through which knowledge was produced through evaluation. At the time, the evaluation mainly consisted of giving pen-and-paper questionnaires to children before and after they participated in the programme (i.e., the football drills). The questionnaire was designed to measure how children’s knowledge of HIV and values regarding teenage pregnancy changed with the programme.
Interpreting quantitative evaluation material
“What should I do? Does that count?” For more than half an hour now, Christine (the programme manager) and I have been debating and discussing Question 10 (“cite one cause of teenage pregnancy”) while inputting the results from the Côte d’Ivoire programme questionnaire onto an Excel document. Many children have answered “going out at night,” “sexual intercourse,” “desire,” “exclusion from parental home,” or “poverty,” for which Christine is not sure whether they are correct or incorrect answers. “I see what they mean, and it is factually not wrong,” she says. I am surprised, especially by “going out at night,” which I feel blames young women for being pregnant because they went out at night. It might be technically true, but is it what we are trying to evaluate? I suggest to Christine, “Maybe it’s not so much about correct or incorrect but about the message that we want to give?”. The issue here is: what exactly is the message? It is not clear for me, or Christine, or to anyone it seems.
The following morning, we continue the conversation, this time joined by Bernard (Christine’s line manager). Bernard tells Christine she can count “going out at night” as a correct answer. Seeing us surprised, he explains: “You know, here [in Côte d’Ivoire], some young people can consider a girl who goes out at night as a ‘whore’ [Bernard mimics quotation marks]. So, she goes out, she will probably drink, surely have sex and get pregnant.” Christine and I are initially speechless, but then gently attempt to point out the slippery logic of Bernard’s statement. However, Bernard ends the debate there. These answers, including “going out at night,” will count as “correct.”
A few months later, Bernard and Christine wrote an in-progress report to the funding body of the programme. They reported an increase in children being able to identify one cause of teenage pregnancy from 27% at baseline to 38% after the activities. In the report, they highlighted how children are more informed about early pregnancy, which ultimately leads to women’s empowerment:
Being informed concerning early pregnancies as well as where to access contraception can highly contribute to their [girls] feeling of control [of their sexual and reproductive health]. Being able and confident to talk about sexuality is also another step in their confidence to access SRHR services and negotiate use of contraceptive. All the above will contribute to their empowerment and lead to a more constructive relationship between men and women and the place women have in society.
The above vignette illustrates an aspect of the many microtechniques that international development workers engage in, especially when interpreting quantitative evaluation material and turning it into reports for funders. Bernard, Christine, and I, had to categorise the questionnaire responses as “correct” or “incorrect” based on our perceptions of the children’s answers. Beyond applying arbitrary decisions based on our own cultural backgrounds, we undertook a work of translation (Latour, 1996) from the field to the language of the programme’s financial supporters. This practice is common in international development. As David Mosse puts it, programmes “have to be translated into the different logic of the intentions, goals and ambitions of the many people and institutions they bring together” (Pg. 232). In that regard, Bernard and Christine translated the answer “going out at night” as a cause of early pregnancy into “being informed concerning early pregnancies,” which contributes to girls’ “empowerment.”
Microtechniques constantly underpin the production of knowledge with quantitative indicators. When the children were given the questionnaire, their coach often explained some terms, hinted at some questions, or even inserted moral remarks encouraging abstinence and about heterosexual relationships. In turn, executive managers at the top of the organisational hierarchy decide how evaluation is conducted, planned, designed, and systematised at the organisational level. These microtechniques are not necessarily problematic per se. The issue lies within the interpretation that is given to the data produced through these microtechniques, such as Bernard’s translation practices.
Indeed, the report only shows an increase in the number of children being able to identify causes of teenage pregnancy as defined by the questionnaire’s reviewers. No explanation is given regarding the causal link between 1) knowing causes of teenage pregnancy; and 2) women’s empowerment, nor there is further explanation regarding the acceptable answers. Such a situation is allowed by the seductive potential of quantitative knowledge:
“Quantification is seductive. […] It organises and simplifies knowledge, facilitating decision making in the absence of more detailed, contextual information […] These numbers convey an aura of objective truth and scientific authority despite the extensive interpretive work that goes into their construction” (Merry 2016:1).
Multiple questionnaires
When assessing the example with Bernard and Christine, one might think that their organisation, or themselves, are at fault because they influenced the production of what should be perfectly objective data. However, anthropological work shows that some degree of interpretation is inevitable. In her book The Seductions of Quantification, Merry (2016) shows how the production and application of United Nations indicators are guided by the experiences and practices of the experts in charge of developing them. For the past few decades, Global North societies have seen the proliferation of performance measuring techniques in many professional sectors – this is what has been termed audit culture(s) (Shore and Wright, 2015; Strathern, 2000). We can discuss the benefits of such an approach that claims to hold individuals and institutions “accountable”. However, we must also remain critical of the processes that lead to the construction of the tools that are used to produce quantitative data and the ways in which they are applied for governance or, as I hinted at the beginning of this blog, for rhetorical purposes.
Quantitative indicators can be a great way to measure certain phenomena. However, this measurement is never perfect or neutral. Indicators only measure according to the criteria with which they are calculated. More importantly, there are complex processes that go from one person’s mind and senses to the overall percentages that support broad statements such as “women are more empowered”. Quantitative questionnaires are not simply out there along with their “objective components”. Rather, it is the “multiplicity” (Mol, 2002) of ways in which people interact with questionnaires that results in the construction of quantitative knowledge. The division of labour that underpins the production of quantitative knowledge, from institutions creating indicators to data collectors, results in a situation where quantitative questionnaires constitute different objects for the individuals across the quantitative supply chain, in line with their perceptions and interests.
I encourage readers, instead of urging people to “look at the numbers”, to start looking into the numbers.
[Image of a hand with pens answering a questionnaire is generated by Stable Diffusion.]
