Tricks of the Trade: Designing a Survey in 60 minutes (or Less)

Do you need to create a survey to evaluate your project outcomes? Are you planning on collecting survey data to assess the interests or needs of a target population?


For busy nonprofit professionals who need to put together a survey in an hour or less, you will find some easy-to-apply tricks to help you do just that in this article.


Let's cut to the chase.


Step #1: Set the goal.


What do you want to do with the data? Your answers typically fall into one of two categories. Maybe you want to determine how successful the project is based on some existing criteria, such as client satisfaction and attendance. Or, maybe you want to explore the impact of the project.


The key differences here are whether you can follow some "existing criteria" or you're "exploring." If you already have a rubric to evaluate the performance or needs with, then you have developed some existing criteria. Otherwise, the goal of the data collection will be to explore the impact or the client experience.


Knowing what you want to do with the data will shape each subsequent step of the survey design.


Step #2: Choose between structured questions and open-ended ones based on your goal.


Structured questions are those that limit your responses to a pre-selected set of answers (your already developed criteria). All of the following are structured questions:

  • On a scale from 0 to 10, how did you rate your experience? (Your answer is limited to a number between 0 and 10.)

  • How many sessions have you attended? (Your answer is limited to a list of fixed categories such as "less than five sessions.")

  • To what extent do you agree with the following statement? (Your answer is limited to a list of fixed categories such as "completely agree.")

Open-ended questions are used when you want to explore the situation of interest. These are some open-ended questions:

  • Tell me more about you.

  • How did you decide to use our services?

  • Can you describe our services from your perspective?

If your goal is to evaluate based on existing criteria, then prioritize structured questions in your surveys with a couple of open-ended questions at the end. But if your goal is to explore - identifying information that might not be on your radar yet, prioritize open-ended questions and include structured questions at the end. In this case, your survey will act as a written interview.


Now you have decided what the survey will look like when you're done, it's time to create some survey questions.


Step #3: Avoid the following mistakes when writing your survey questions.


Below you will find some common mistakes people make when writing survey questions. If you could avoid these mistakes, not only will you get a higher response rate but you will also get better data!


Mistake #1: Using abstract concepts - e.g., How much do you agree with this statement - our services have improved your sense of empowerment?


Abstract concepts such as "sense of empowerment" can mean different things to different people even if you're a native speaker. For non-native speakers, survey questions including abstract concepts could add an additional challenge for them to provide accurate responses.


What if you have to ask questions about these abstract outcomes?


Try naming some concrete examples. For example, what actions or beliefs your clients develop due to your services would make you say "That client seems to feel more empowered now"? Replace the abstract concepts with those concrete examples.


One example could be: How much do you agree with the following statement - our services have helped you feel that you have more control over your own life trajectory?


Mistake #2. Using double-barreled questions - e.g., How would you recommend our career workshop and afterschool programs?


Double-barreled questions ask about two things in one question. As a result, the respondent might be confused as to which part of the question to address (the career workshop or the afterschool program). When you review the answers, it's also difficult to make sense of the data. If the respondent says "I recommend it," does the person recommend both or just one of them?


A quick solution is to break the question into two.


Mistake #3. Asking questions that include "analysis" - e.g., How much has your knowledge of resume writing improved?


This question is three questions in one. What you're really asking is: What was your knowledge before? What is your knowledge now? What is the difference?


Questions like this almost never lead to accurate answers, meaning that the respondents might underestimate (or overestimate) the actual benefit.


The solution? Ask your respondents to report their knowledge before and after, and calculate the difference when you get the data.


Mistake #4: Not quantifying the answers - e.g., How likely will you recommend our services? vs. On a scale from 0 to 10, how do you rate our services?


The two questions above ask for the same information but result in very different data. The former will generate a list of answers that include "highly likely" and "likely." In contrast, the latter will generate a list of numbers.


Numbers are effective. Get numeric data from your respondents whenever you could, especially when the data could help you demonstrate your impact.


Step #4: Try to delete at least one question you just developed.


Long surveys have a low response rate and a low completion rate. So this exercise is designed to help you keep your surveys succinct. There is no rule that says your survey must be limited to 7 questions or less; but typically if it's over 15 questions, the incompletion rate starts to soar.


Step #5: Ask what you want to know first, and make sure you record incomplete responses.


These two tricks will make sure you get more, desired data. No matter how long, or short, your survey is, ask the most important questions first, and make sure you allow your survey platform to record incomplete responses. This is because 1) as the survey continues, the respondent will likely spend less time thinking about their answers, and 2) at least some of the respondents will leave the survey before completing it.