{"id":1592,"date":"2012-10-26T13:51:03","date_gmt":"2012-10-26T20:51:03","guid":{"rendered":"http:\/\/blog.mozilla.org\/ux\/?p=1592"},"modified":"2016-02-19T14:41:37","modified_gmt":"2016-02-19T06:41:37","slug":"writing-surveys-that-work","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/ux\/2012\/10\/writing-surveys-that-work\/","title":{"rendered":"Writing Surveys That Work"},"content":{"rendered":"

Rebecca Weiss<\/a>, one of our metric team members and and a PhD candidate at Stanford, gave a fantastic talk about how to write surveys.\u00a0 Writing surveys seems simple because of all the free online tools that exist, but writing one that actually answers the question you set out to answer – that is harder than it looks.<\/p>\n

Surveys are not questionnaires!\u00a0 A questionnaire is the tool that is used to talk to a sample of the population.\u00a0 A survey is a representative sample of that population.<\/p>\n

These are the three most common issues with questionnaires are question wording, response methods and question ordering.<\/p>\n

Question Wording<\/strong><\/p>\n

\"\"<\/a>In this example the phrase “most impact” is too vague, which means that people will interpret that in different ways.\u00a0 Therefore, you can’t trust the answer to this question.<\/p>\n

\"\"<\/a>This question is double barreled – it’s asking two questions in one:\u00a0 “Did you know that Mozilla is a mission-driven organization?”\u00a0 and “Did you know that Mozilla makes the Internet a better place?”\u00a0 The user may have two different answers, but since it’s just one question they are forced to only answer in one way and so you aren’t getting a true answer.<\/p>\n

\"\"<\/a>This question seems fine until you think about the way someone would actually respond and what that means.\u00a0 if someone answered “yes” it’s better than any other browser then it’s clear.\u00a0 But if someone answered “no”, you wouldn’t know if they meant “no, it’s not better, it’s the same” or “no, it’s not better, it’s worse”.<\/p>\n

\"\"<\/a>Ranking 20 features is taxing and most participants although they want to be honest will become a bit lazy here and just click through to get to the next question.\u00a0 A better way to rewrite this question would be to ask them to rank the top 5 features of the 20 or to just give them 5 features to rank in order.<\/p>\n

\"\"<\/a>This example is prone to cognitive bias.\u00a0 In the US, the tendency is to err toward the positive bias.<\/p>\n

Response Methods<\/strong><\/p>\n

When creating the responses to your questions, the goal is to make it as easy as possible for every respondent to respond.\u00a0 Don’t make open-ended questions mandatory.\u00a0 When asking about specific quantities, use intervals or natural frequencies such as 1\/10 and not 10%.\u00a0 Full label your non-enumerated scales.\u00a0 As Rebecca said “one person’s 3, may not be the same as another’s”.<\/p>\n

<\/strong>Question Ordering<\/strong><\/p>\n

\n

Questions might have spillover influence<\/strong> on future responses.\u00a0 The answer to question #4 might affect responses to questions #5, 6 and 7.\u00a0 This is usually why demographic questions tend to put at the end of questionnaires.<\/p>\n

Response option ordering might skew your distribution<\/strong>.\u00a0 People tend to focus more on earlier or later options, and spend less time evaluating middle options.<\/p>\n

One way to protect against ordering effects is to randomize between blocks of questions and\/or within blocks.<\/p>\n

Here are 8 best practices to writing questionnaires that work:<\/p>\n<\/div>\n

\"\"<\/a> Download the full slide deck<\/a> and refer to slide 28 the next time you are writing a questionnaire.<\/p>\n","protected":false},"excerpt":{"rendered":"

Rebecca Weiss, one of our metric team members and and a PhD candidate at Stanford, gave a fantastic talk about how to write surveys.\u00a0 Writing surveys seems simple because of … Read more<\/a><\/p>\n","protected":false},"author":308,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,9575],"tags":[27256,578],"coauthors":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/ux\/wp-json\/wp\/v2\/posts\/1592"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/ux\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/ux\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/ux\/wp-json\/wp\/v2\/users\/308"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/ux\/wp-json\/wp\/v2\/comments?post=1592"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/ux\/wp-json\/wp\/v2\/posts\/1592\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/ux\/wp-json\/wp\/v2\/media?parent=1592"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/ux\/wp-json\/wp\/v2\/categories?post=1592"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/ux\/wp-json\/wp\/v2\/tags?post=1592"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mozilla.org\/ux\/wp-json\/wp\/v2\/coauthors?post=1592"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}