top of page

Designing Surveys to Collect Accurate Data

If you speak more than one language, you probably had the experience of asking others or being asked to translate a word. In many cases, the translation is straightforward: tree in English is 树 (shu) in Mandarin Chinese; merci in French is gracias in Spanish.

But in way too many situations, there is no perfect translation. To say you're "happy" is not the same to say you're "高兴 (gaoxing)" or "幸福 (xingfu)" in Chinese. Depending on whether you're happy at this moment or feel content in general, you say gaoxing or xingfu, respectively.

The purpose of all that is not to give you a language lesson, but rather to share a piece of my everyday reality that many overlook or take for granted: words do not mean the same thing to different people.

The complexity of meanings and the resulting myriad opportunities of miscommunication happen even within one cultural system or society. Take the word equity. In finance, it means what is left after you take the liabilities away from assets. In social policy studies, it means equal outcomes, or so we assume. Actually, I know some use equity to refer to both equal opportunities and equal outcomes. Therefore, to say you're pro-equity can mean something very different depending on your background and audience.

That brings us to the point of survey design and operationalization. For those unfamiliar with the term, operationalization means you take an abstract concept and turn it into something you can measure. What does that mean? What does it look like in practice? What can go wrong?

Take "empowerment" as an example. These days, many mission-driven organizations work to empower a community and gather data to assess the impact of the programs or policies. The most common way is to ask, "To what extent did this experience make you feel more empowered?"

Sounds straightforward, right? Not really. The problem is that empowerment is a highly abstract term, and it means different things to different people. Exhibit one: a quick search of "empowerment meaning" on Google Scholar returns 1.65 million results. Empowerment could mean becoming more confident, feeling like they have the resources to take actions to create change or a lot of different things.

In other words, you cannot assume your survey respondents understand empowerment in the same ways you do. Also, the more people respond to your survey, the more likely significant differences emerge in how your respondents understand the word among themselves. When you aggregate your survey data and infer insights about a program's impact on empowering the community, your findings are inconclusive or even misleading. There's no "community" in your data; your community might disagree with each other, and you simply don't know.

What Can You Do?

The quickest and most effective solution to this widespread and paralyzing invalidity problem in survey designing is simply framing your survey questions in more straightforward language. For example, instead of asking, "To what extent did this experience make you feel more empowered," ask, "To what extent did this experience help you gain XYZ knowledge or identify ABC resources you can access." Asking these questions will help you minimize data invalidity and help decision-makers clearly see what components of the program/policy works.

bottom of page