Chris Blattman on the chaotic nature of running surveys

On trying to learn about people in the context of "doing good in the developing world". Crossposting this because I found it funny, and because gratitude is healthy. (I'm grateful that I didn't become a development economist.)

I have been running about pre-testing a new questionnaire–always a humbling and frustrating experience.

Take an apparently simple question: “How many children do you have?”

Respondent: Five.
Surveyor: Do all of these children live with you?
Respondent: Well, I have two other children who live with my brother.
Surveyor: I see. So you have seven biological children?
Respondent: No, three of my children belong to my sister who died last year.
Surveyor: So you have four biological children, plus three children you have adopted.
Respondent: Well, one of them lives with his father sometimes. I also take care of the children of my cousin, but he is away at school.

Nothing is ever simple. It turns out that five or six questions are required to figure out the number of children accurately and consistently. And in Uganda the number of children is easy. In western Kenya, out of superstition people would deliberately overstate the number of children they have.

What about something even less straightforward to measure, such as women’s empowerment?

Jeannie and I are working with an international humanitarian organization to evaluate the impact of providing small grants, business skills training, and business advice networks to the most extremely vulnerable women–those with no incomes, many children to support, low social support, little family assistance, health challenges, and so forth. How to measure this in a survey?

Luckily we have the Demographic and Health Surveys — downloadable surveys and representative data on health, demographics, domestic violence, and women’s status across dozens of countries, including Uganda. Perfect, right? Now not only do we have ready-made questions, we can have representative comparison data for our sample.

Not so fast. Turns out that looking great on paper does not necessarily pre-test well. Take one question designed to understand financial independence:

Q. Do you have any money of your own that you alone can decide how to use?

Problem: the primary answer is “no, I don’t usually have any money”. The question measures access to funds rather than decision-making power. A better option might be to first ask “when money is available…”

It turns out, however, that the answer to this question is still, “it depends”. Most of all, it depends on whether the woman earned the money herself.

We had similar problems with almost every other DHS question we adopted.

Q: Are you permitted to go to the health center to buy things on your own, only if someone accompanies you, or not at all?
A. What do you mean by permission? I usually consult my husband, especially if I have to pay money. Also, I can go for a short visit, but I need his permission to stay
overnight.

Q. Do you yourself control the money needed to buy clothes for yourself?
A. What do you mean by control? You mean I keep it myself? How expensive are the clothes? Who earned the money?

Virtually all of the DHS questions required clarification, additional specification, and narrowing. Usually the questions we ended up with in the end were still simple and straightforward, but much more locally appropriate and specific, and elicited much clearer and more consistent answers that actually captured empowerment.

Part of the challenge is that there is a great deal of local variation. But I could not help feel that the same questions and challenges would have occurred in any context. It’s not easy to tell how, where, when and by whom these questions were pre-tested in the field. Yet they are the leading surveyors internationally.

Overall, this makes me very wary of the DHS data on women’s status. I don’t mean to pick on them specifically. If you ever want a surprise, go pre-test a World Bank Living Standards Measurement Survey in the field. Warning: you may never download and use data from the Internet again…