WEBVTT 00:00:02.640 --> 00:00:05.601 We're going to be talking  about writing good questions. 00:00:10.320 --> 00:00:15.726 The quality of the questions, along with the quality of the respondents, is really key. 00:00:15.726 --> 00:00:20.002 So, you have to have questions which people understand, that they 00:00:20.002 --> 00:00:23.692 really can address *unambiguously*; 00:00:23.692 --> 00:00:27.312 they don't have to sit wondering what it is you meant by that, 00:00:27.312 --> 00:00:30.870 and to do it pretty quickly, too – they need to be *short*. 00:00:30.870 --> 00:00:35.225 So, we have a long list of various  points you should take into account. 00:00:35.225 --> 00:00:39.840 And it starts with the the issue of *shortness and ambiguity*. 00:00:39.840 --> 00:00:44.920 So, we want things to be as *short as possible*. We want to use short questions, short words. 00:00:44.920 --> 00:00:49.514 We want to be *very specific*, and we want to be *very unambiguous*. 00:00:49.514 --> 00:00:53.490 So, you don't ask vague questions. If you want to know about whether 00:00:53.490 --> 00:00:57.240 somebody has done something, you need to specify over what period. 00:00:57.240 --> 00:01:00.240 A lot of questionnaires these days I noticed ask, 00:01:00.240 --> 00:01:05.491 'What did you do yesterday?' or 'Did you watch this yesterday? Did you do that yesterday?' 00:01:05.491 --> 00:01:09.300 And you could of course change that period as required, but 00:01:09.300 --> 00:01:15.985 it's very unrealistic to expect people to either mind-read if you're not being at all clear about when you mean, 00:01:15.985 --> 00:01:20.763 or to remember very far back. And that changes with the topic, of course. 00:01:20.763 --> 00:01:24.176 Somebody might remember when they were married from a long time ago, 00:01:24.176 --> 00:01:28.467 but the last time they visited a coffee shop – you might not 00:01:28.467 --> 00:01:32.226 get away with more than a couple to four weeks, for example. 00:01:32.266 --> 00:01:37.800 Do use *common terms* and  provide *explanations* where you're using anything 00:01:37.840 --> 00:01:42.916 that's a little bit unusual. And common terms in  English are generally fairly short. 00:01:42.916 --> 00:01:46.966 So, use short words. Short words appear more frequently, 00:01:46.966 --> 00:01:50.504 are used more frequently. The number of syllables is actually a good indicator; 00:01:50.504 --> 00:01:54.710 the more syllables, the less frequent the word is used in English. 00:01:54.710 --> 00:01:59.282 And with a lot of interactive survey 00:01:59.282 --> 00:02:02.254 tools, you are able to provide explanations. 00:02:02.254 --> 00:02:07.366 If nothing else works, then just put something parenthetically after the question 00:02:07.366 --> 00:02:10.643 to explain what it is you mean by that particular term. 00:02:10.683 --> 00:02:15.520 Start the questionnaire or the survey with *easy, uncontroversial questions*. 00:02:15.520 --> 00:02:20.160 If you're going to go on to be asking  sensitive questions or things that people 00:02:20.240 --> 00:02:26.185 might get slightly aggravated or upset about,  then try to leave that as late as possible. 00:02:26.185 --> 00:02:31.280 You want people to get engaged; you want them to feel comfortable, and to an extent trust you. 00:02:31.280 --> 00:02:37.732 And starting with controversial questions or in some way irritating questions is not a good way to do that. 00:02:37.732 --> 00:02:42.680 Do *use words to label all points on a scale*. I've got a terrible question that I've made up about Brexit. 00:02:42.680 --> 00:02:47.840 But I've labeled every single point. Now, this used to be for me a debatable issue, 00:02:47.840 --> 00:02:53.890 that I suggested that perhaps we should just label  the two ends and number the points in between, 00:02:53.890 --> 00:03:01.280 but I've seen good evidence and firm advice that you should label all of the points in between the two. 00:03:01.360 --> 00:03:06.510 Certainly on seven – this is a seven-point response – five or seven is fairly typical. 00:03:06.510 --> 00:03:10.231 And in those cases, you would expect to see words being used there. 00:03:10.231 --> 00:03:15.000 Address only *one aspect of a question at a time*. *Avoid compounds* like: 00:03:15.000 --> 00:03:20.836 'I found the website quick and easy to use,' because there is the possibility that somebody found it easy to use 00:03:20.836 --> 00:03:28.064 but not quick, or quick but not easy. So, stop that from being a problem by asking those questions separately. 00:03:28.064 --> 00:03:32.798 Just like in navigation design, *do not use overlapping categories or ranges*. 00:03:32.798 --> 00:03:36.840 People need to be very clear about where they should be clicking; so, you want 00:03:36.880 --> 00:03:41.885 '20-49', '50-64,' '65 and over', 00:03:41.885 --> 00:03:46.931 rather than '20-50', '50-65', '65 and over' because you can see in 00:03:47.040 --> 00:03:51.331 that latter example that there are two places  where people, if they were exactly 50 00:03:51.331 --> 00:03:56.749 or exactly 65 would not be clear about where you actually want them to answer. 00:03:56.749 --> 00:04:00.793 And *do not ask leading questions* if you want honest responses, 00:04:00.793 --> 00:04:07.292 or any responses at all, for that matter. Certainly, it's during various elections that have taken place over the last couple 00:04:07.292 --> 00:04:14.821 of years, I have seen numerous alleged surveys come round where clearly it is not a questionnaire or a survey at all; 00:04:14.821 --> 00:04:20.720 it is a party political statement, and I just stop  as soon as I realize that that is the case. 00:04:20.720 --> 00:04:26.218 And I believe a lot of other people do as well.  So, this example on 'How bad an idea is Brexit?' 00:04:26.218 --> 00:04:32.470 is an example of a leading or loaded question because of course some people think that Brexit is a perfectly good idea 00:04:32.470 --> 00:04:36.655 and we should not be talking about it as a bad idea in their eyes. 00:04:36.655 --> 00:04:39.840 So, this is something that you would not do.  In fact, there are quite a few things in that 00:04:39.840 --> 00:04:43.914 particular example you should not do, so just  treat that as a bad example... 00:04:46.000 --> 00:04:50.400 *Avoid question grids*; I know that they  are extremely popular, but if you 00:04:50.400 --> 00:04:54.311 can avoid them, they are worth avoiding because  they are intimidating, 00:04:54.311 --> 00:04:59.898 certainly with the large question grids, people turn over to them online typically, 00:04:59.898 --> 00:05:05.778 and they immediately get put off. It looks very complicated. It is quite complicated. 00:05:05.778 --> 00:05:11.522 If they were on the brink of not completing your questionnaire, that probably has pushed them over 00:05:11.522 --> 00:05:17.341 the edge. And if you're unlucky and are working with a service provider for your questionnaire, 00:05:17.341 --> 00:05:22.327 your survey tool, that doesn't support converting these into individual questions, 00:05:22.327 --> 00:05:26.358 then you'll find that they don't actually work on mobile phones. 00:05:26.358 --> 00:05:30.880 And that certainly does happen from time to time,  and of course we can expect more people to be 00:05:30.880 --> 00:05:35.831 using mobile phones as their primary internet  tool; so, that would be a bad plan. 00:05:35.831 --> 00:05:41.270 Really the best way of approaching these is to ask the questions separately. 00:05:41.270 --> 00:05:46.693 And this is actually quite a bit more likely in the problem domain that we're talking about, 00:05:46.693 --> 00:05:52.800 which is user experience, rather than, say for example, market research where you might have a list of 00:05:52.800 --> 00:05:58.640 10 or 20 coffee shops down the left-hand  side and the frequency of use across the top. 00:05:58.640 --> 00:06:03.760 That's still not easy for respondents, but it's  perhaps a little bit easier than asking them 00:06:03.760 --> 00:06:07.867 deep, meaningful questions like this  particular one about Barack Obama. 00:06:10.400 --> 00:06:15.967 *Ranking questions* are those where you're asked  to re-order the responses. 00:06:15.967 --> 00:06:22.042 And the *problem* with them is twofold. One is that it's actually not very easy as a participant to do this, 00:06:22.042 --> 00:06:26.590 that you have to think about, 'Well, which of these is my preference? 00:06:26.590 --> 00:06:29.840 Which is my second preferred, third preferred?' etc. 00:06:29.840 --> 00:06:34.240 And it's off-putting to them; it's time-consuming; it's hard to do in some cases, 00:06:34.240 --> 00:06:39.986 particularly on a mobile platform, but it might be  quite technically challenging, 00:06:39.986 --> 00:06:45.880 just the dexterity required, and it's also off-putting; you're making people think really hard about something where 00:06:46.000 --> 00:06:51.360 the detailed answers are not all that important.  What does it matter to you in a list of five 00:06:51.360 --> 00:06:55.853 whether somebody lists something  fourth or fifth? It's not first; 00:06:56.000 --> 00:07:00.049 it's not second; so, do you really care that  much about it? 00:07:00.049 --> 00:07:04.180 It turns out to be very easy and almost equivalent. 00:07:04.180 --> 00:07:07.600 I have not seen a paper saying  that they're equivalent, by the way, but certainly 00:07:07.600 --> 00:07:12.520 when I've done it, I've not been disappointed. Just ask people to choose their *favorite*. 00:07:13.440 --> 00:07:18.595 Or, if you've got a long list, maybe their favorite  top 'n', where 'n' might be 2, 3, 4 ... 00:07:18.595 --> 00:07:22.960 And you're not talking about order, then; you're just talking about which is the most favorite or 00:07:22.960 --> 00:07:29.040 which are the two most favorite. And when you come to  analyze those, the analysis is very much simpler 00:07:29.040 --> 00:07:34.312 because it is just the items with the biggest  numbers are the most popular. 00:07:34.312 --> 00:07:40.640 And that was the second point with the whole ranking thing – is that you have to do a *weighted analysis* in order 00:07:40.640 --> 00:07:46.381 to get sensible results from the ranking, which means  taking into account where in the list these things appeared; 00:07:46.381 --> 00:07:50.880 whereas if you're just asking people  to choose their favorite, it is very much easier 00:07:50.880 --> 00:07:55.781 and it's just slightly more complicated for their favorite two or their favorite three. 00:07:55.781 --> 00:08:02.143 And most survey tools will let you set a *maximum and minimum number of responses* for this kind of question. 00:08:02.143 --> 00:08:09.120 So, if you really insist on having two choices from a  long list, then it could complain to participants 00:08:09.120 --> 00:08:14.320 that they've not selected two or preferably  a maximum is the better way of doing that 00:08:14.320 --> 00:08:20.000 so people cannot choose the full number if they  really don't care that much. 00:08:20.000 --> 00:08:24.949 So, that is something I would just strongly recommend you avoid altogether. 00:08:24.949 --> 00:08:29.120 It's really of no particular benefit. It looks fun 00:08:29.120 --> 00:08:32.972 when you're looking at it in the tool and  maybe on the screen yourself. 00:08:32.972 --> 00:08:39.840 But when you're talking about lots of respondents getting to  it and dealing with it, it's not fun for them. 00:08:41.280 --> 00:08:47.411 Open-ended questions do allow some flexibility,  and of course they're not in any way going to 00:08:47.520 --> 00:08:51.135 replace an interviewer who can dig a lot more  deeply 00:08:51.135 --> 00:08:54.995 and try to interpret or understand what participants are saying. 00:08:54.995 --> 00:09:00.466 They have got their use, though, open-ended questions; the set of possible answers is unknown 00:09:00.466 --> 00:09:07.647 or very large, is the main reason for doing that. So, if you've got a list of items and there are other possibilities, 00:09:07.647 --> 00:09:13.547 you will have an 'other' response, and under the 'other' response you will have a text box for people to fill in. 00:09:13.547 --> 00:09:18.826 Perhaps you want to know the underlying cause of a response – 'Why did you rate us like that?'; 00:09:18.826 --> 00:09:25.528 'What was the main thing?' And that's certainly very commonly done and a perfectly good use of open-ended responses. 00:09:25.528 --> 00:09:30.638 Or you need to allow participants to express *unanticipated concerns*. 00:09:30.638 --> 00:09:35.127 So, 'Is there anything that we could have done to have prevented this or to make you happier?' 00:09:35.127 --> 00:09:40.400 – that kind of question. And these two examples from SurveyMonkey: 00:09:40.400 --> 00:09:44.879 the top one is the final question in most SurveyMonkey templates. It's open-ended. 00:09:44.879 --> 00:09:48.709 And that's recommended for almost all surveys, that 00:09:48.709 --> 00:09:52.568 you give people a chance to comment about either your organization 00:09:52.568 --> 00:09:56.545 or the questionnaire or just general comments that they might have. 00:09:56.545 --> 00:10:00.360 'Is there anything else you'd like to add?' is the kind of question you would ask there. 00:10:00.360 --> 00:10:05.987 The bottom one is actually from their market research  template, and it has a lot of open-ended questions in it, 00:10:05.987 --> 00:10:10.840 but it's one of the very few SurveyMonkey templates that actually has more than 00:10:10.960 --> 00:10:14.856 one or two open-ended questions; so, you really can  largely 00:10:14.856 --> 00:10:19.643 – and perhaps should try largely – to stick to multiple choice questions. 00:10:19.643 --> 00:10:24.719 An alternative – and you may see this yourself as a respondent to various questionnaires – 00:10:24.719 --> 00:10:30.319 an alternative to extensive open-ended questions is to invite survey participants to take part 00:10:30.319 --> 00:10:35.599 in an *online or telephone interview*. So, you might ask them a few questions, 00:10:35.599 --> 00:10:41.515 and they might express some reservations  about certain aspects of your product or service. 00:10:41.515 --> 00:10:47.346 And if they did that, you might offer them the  chance to be interviewed at depth, 00:10:47.346 --> 00:10:52.522 either by telephone or through some online collaboration tool. 00:10:52.522 --> 00:10:56.457 And of course that becomes, then, a primarily qualitative approach, 00:10:56.457 --> 00:11:00.480 and you would only do this with a relatively small number of participants, 00:11:00.480 --> 00:11:05.200 mostly because it will be moderately time-consuming; you could expect it to take at least 00:11:05.200 --> 00:11:10.156 half an hour, and you will need a qualified  interviewer to do that. 00:11:10.156 --> 00:11:16.080 Do make sure that if you're using this approach that the interviewer does have access to the respondent's initial  00:11:16.080 --> 00:11:20.966 questionnaire so they're not repeating themselves  or are not at all aware of 00:11:20.966 --> 00:11:24.405 the participant's background or complaint. 00:11:24.405 --> 00:11:27.997 *Semi-structured interviewing* is the most appropriate technique in most cases. 00:11:27.997 --> 00:11:33.498 So, this would be a follow-up set of questions and then just the interviewer exploring 00:11:33.498 --> 00:11:38.478 some of the responses that the participant has given already.