Firefox UX
Categories: General Methodology

Writing Surveys That Work

2 responses

Rebecca Weiss, one of our metric team members and and a PhD candidate at Stanford, gave a fantastic talk about how to write surveys.  Writing surveys seems simple because of all the free online tools that exist, but writing one that actually answers the question you set out to answer – that is harder than it looks.

Surveys are not questionnaires!  A questionnaire is the tool that is used to talk to a sample of the population.  A survey is a representative sample of that population.

These are the three most common issues with questionnaires are question wording, response methods and question ordering.

Question Wording

In this example the phrase “most impact” is too vague, which means that people will interpret that in different ways.  Therefore, you can’t trust the answer to this question.

This question is double barreled – it’s asking two questions in one:  “Did you know that Mozilla is a mission-driven organization?”  and “Did you know that Mozilla makes the Internet a better place?”  The user may have two different answers, but since it’s just one question they are forced to only answer in one way and so you aren’t getting a true answer.

This question seems fine until you think about the way someone would actually respond and what that means.  if someone answered “yes” it’s better than any other browser then it’s clear.  But if someone answered “no”, you wouldn’t know if they meant “no, it’s not better, it’s the same” or “no, it’s not better, it’s worse”.

Ranking 20 features is taxing and most participants although they want to be honest will become a bit lazy here and just click through to get to the next question.  A better way to rewrite this question would be to ask them to rank the top 5 features of the 20 or to just give them 5 features to rank in order.

This example is prone to cognitive bias.  In the US, the tendency is to err toward the positive bias.

Response Methods

When creating the responses to your questions, the goal is to make it as easy as possible for every respondent to respond.  Don’t make open-ended questions mandatory.  When asking about specific quantities, use intervals or natural frequencies such as 1/10 and not 10%.  Full label your non-enumerated scales.  As Rebecca said “one person’s 3, may not be the same as another’s”.

Question Ordering

Questions might have spillover influence on future responses.  The answer to question #4 might affect responses to questions #5, 6 and 7.  This is usually why demographic questions tend to put at the end of questionnaires.

Response option ordering might skew your distribution.  People tend to focus more on earlier or later options, and spend less time evaluating middle options.

One way to protect against ordering effects is to randomize between blocks of questions and/or within blocks.

Here are 8 best practices to writing questionnaires that work:

Download the full slide deck and refer to slide 28 the next time you are writing a questionnaire.

2 comments on “Writing Surveys That Work”

  1. Steve Portigal wrote on

    GREAT stuff! Where is the recording and/or slide deck that you refer to in the last paragraph?

  2. Vincent wrote on

    The images fail to load and since there is no alt text it is impossible to know what questions you are talking abouw :( (And of course, people using screen readers or text-based browsers can’t make out the content of the post as well…)