WEBVTT

00:00:00.000 --> 00:00:30.870
We're going to be talking  about writing good questions.
The quality of the questions, along with the quality of the respondents, is really key.
So, you have to have questions which people understand, that they
really can address *unambiguously*;
they don't have to sit wondering what it is you meant by that,
and to do it pretty quickly, too – they need to be *short*.

00:00:30.870 --> 00:01:00.240
So, we have a long list of various  points you should take into account.
And it starts with the the issue of *shortness and ambiguity*.
So, we want things to be as *short as possible*. We want to use short questions, short words.
We want to be *very specific*, and we want to be *very unambiguous*.
So, you don't ask vague questions. If you want to know about whether
somebody has done something, you need to specify over what period.
A lot of questionnaires these days I noticed ask,

00:01:00.240 --> 00:01:32.226
'What did you do yesterday?' or 'Did you watch this yesterday? Did you do that yesterday?'
And you could of course change that period as required, but
it's very unrealistic to expect people to either mind-read if you're not being at all clear about when you mean,
or to remember very far back. And that changes with the topic, of course.
Somebody might remember when they were married from a long time ago,
but the last time they visited a coffee shop – you might not
get away with more than a couple to four weeks, for example.

00:01:32.226 --> 00:02:02.254
Do use *common terms* and  provide *explanations* where you're using anything
that's a little bit unusual. And common terms in  English are generally fairly short.
So, use short words. Short words appear more frequently,
are used more frequently. The number of syllables is actually a good indicator;
the more syllables, the less frequent the word is used in English.
And with a lot of interactive survey
tools, you are able to provide explanations.

00:02:02.254 --> 00:02:31.280
If nothing else works, then just put something parenthetically after the question
to explain what it is you mean by that particular term.
Start the questionnaire or the survey with *easy, uncontroversial questions*.
If you're going to go on to be asking  sensitive questions or things that people
might get slightly aggravated or upset about,  then try to leave that as late as possible.
You want people to get engaged; you want them to feel comfortable, and to an extent trust you.

00:02:31.280 --> 00:03:01.280
And starting with controversial questions or in some way irritating questions is not a good way to do that.
Do *use words to label all points on a scale*. I've got a terrible question that I've made up about Brexit.
But I've labeled every single point. Now, this used to be for me a debatable issue,
that I suggested that perhaps we should just label  the two ends and number the points in between,
but I've seen good evidence and firm advice that you should label all of the points in between the two.

00:03:01.280 --> 00:03:32.798
Certainly on seven – this is a seven-point response – five or seven is fairly typical.
And in those cases, you would expect to see words being used there.
Address only *one aspect of a question at a time*. *Avoid compounds* like:
'I found the website quick and easy to use,' because there is the possibility that somebody found it easy to use
but not quick, or quick but not easy. So, stop that from being a problem by asking those questions separately.
Just like in navigation design, *do not use overlapping categories or ranges*.

00:03:32.798 --> 00:04:00.793
People need to be very clear about where they should be clicking; so, you want
'20-49', '50-64,' '65 and over',
rather than '20-50', '50-65', '65 and over' because you can see in
that latter example that there are two places  where people, if they were exactly 50
or exactly 65 would not be clear about where you actually want them to answer.
And *do not ask leading questions* if you want honest responses,

00:04:00.793 --> 00:04:32.470
or any responses at all, for that matter. Certainly, it's during various elections that have taken place over the last couple
of years, I have seen numerous alleged surveys come round where clearly it is not a questionnaire or a survey at all;
it is a party political statement, and I just stop  as soon as I realize that that is the case.
And I believe a lot of other people do as well.  So, this example on 'How bad an idea is Brexit?'
is an example of a leading or loaded question because of course some people think that Brexit is a perfectly good idea

00:04:32.470 --> 00:05:05.778
and we should not be talking about it as a bad idea in their eyes.
So, this is something that you would not do.  In fact, there are quite a few things in that
particular example you should not do, so just  treat that as a bad example...
*Avoid question grids*; I know that they  are extremely popular, but if you
can avoid them, they are worth avoiding because  they are intimidating,
certainly with the large question grids, people turn over to them online typically,
and they immediately get put off. It looks very complicated. It is quite complicated.

00:05:05.778 --> 00:05:30.880
If they were on the brink of not completing your questionnaire, that probably has pushed them over
the edge. And if you're unlucky and are working with a service provider for your questionnaire,
your survey tool, that doesn't support converting these into individual questions,
then you'll find that they don't actually work on mobile phones.
And that certainly does happen from time to time,  and of course we can expect more people to be

00:05:30.880 --> 00:06:03.760
using mobile phones as their primary internet  tool; so, that would be a bad plan.
Really the best way of approaching these is to ask the questions separately.
And this is actually quite a bit more likely in the problem domain that we're talking about,
which is user experience, rather than, say for example, market research where you might have a list of
10 or 20 coffee shops down the left-hand  side and the frequency of use across the top.
That's still not easy for respondents, but it's  perhaps a little bit easier than asking them

00:06:03.760 --> 00:06:34.240
deep, meaningful questions like this  particular one about Barack Obama.
*Ranking questions* are those where you're asked  to re-order the responses.
And the *problem* with them is twofold. One is that it's actually not very easy as a participant to do this,
that you have to think about, 'Well, which of these is my preference?
Which is my second preferred, third preferred?' etc.
And it's off-putting to them; it's time-consuming; it's hard to do in some cases,

00:06:34.240 --> 00:07:00.049
particularly on a mobile platform, but it might be  quite technically challenging,
just the dexterity required, and it's also off-putting; you're making people think really hard about something where
the detailed answers are not all that important.  What does it matter to you in a list of five
whether somebody lists something  fourth or fifth? It's not first;
it's not second; so, do you really care that  much about it?

00:07:00.049 --> 00:07:34.312
It turns out to be very easy and almost equivalent.
I have not seen a paper saying  that they're equivalent, by the way, but certainly
when I've done it, I've not been disappointed. Just ask people to choose their *favorite*.
Or, if you've got a long list, maybe their favorite  top 'n', where 'n' might be 2, 3, 4 ...
And you're not talking about order, then; you're just talking about which is the most favorite or
which are the two most favorite. And when you come to  analyze those, the analysis is very much simpler
because it is just the items with the biggest  numbers are the most popular.

00:07:34.312 --> 00:08:02.143
And that was the second point with the whole ranking thing – is that you have to do a *weighted analysis* in order
to get sensible results from the ranking, which means  taking into account where in the list these things appeared;
whereas if you're just asking people  to choose their favorite, it is very much easier
and it's just slightly more complicated for their favorite two or their favorite three.
And most survey tools will let you set a *maximum and minimum number of responses* for this kind of question.

00:08:02.143 --> 00:08:32.972
So, if you really insist on having two choices from a  long list, then it could complain to participants
that they've not selected two or preferably  a maximum is the better way of doing that
so people cannot choose the full number if they  really don't care that much.
So, that is something I would just strongly recommend you avoid altogether.
It's really of no particular benefit. It looks fun
when you're looking at it in the tool and  maybe on the screen yourself.

00:08:32.972 --> 00:09:00.466
But when you're talking about lots of respondents getting to  it and dealing with it, it's not fun for them.
Open-ended questions do allow some flexibility,  and of course they're not in any way going to
replace an interviewer who can dig a lot more  deeply
and try to interpret or understand what participants are saying.
They have got their use, though, open-ended questions; the set of possible answers is unknown

00:09:00.466 --> 00:09:30.638
or very large, is the main reason for doing that. So, if you've got a list of items and there are other possibilities,
you will have an 'other' response, and under the 'other' response you will have a text box for people to fill in.
Perhaps you want to know the underlying cause of a response – 'Why did you rate us like that?';
'What was the main thing?' And that's certainly very commonly done and a perfectly good use of open-ended responses.
Or you need to allow participants to express *unanticipated concerns*.

00:09:30.638 --> 00:10:00.360
So, 'Is there anything that we could have done to have prevented this or to make you happier?'
– that kind of question. And these two examples from SurveyMonkey:
the top one is the final question in most SurveyMonkey templates. It's open-ended.
And that's recommended for almost all surveys, that
you give people a chance to comment about either your organization
or the questionnaire or just general comments that they might have.
'Is there anything else you'd like to add?' is the kind of question you would ask there.

00:10:00.360 --> 00:10:30.319
The bottom one is actually from their market research  template, and it has a lot of open-ended questions in it,
but it's one of the very few SurveyMonkey templates that actually has more than
one or two open-ended questions; so, you really can  largely
– and perhaps should try largely – to stick to multiple choice questions.
An alternative – and you may see this yourself as a respondent to various questionnaires –
an alternative to extensive open-ended questions is to invite survey participants to take part

00:10:30.319 --> 00:11:00.480
in an *online or telephone interview*. So, you might ask them a few questions,
and they might express some reservations  about certain aspects of your product or service.
And if they did that, you might offer them the  chance to be interviewed at depth,
either by telephone or through some online collaboration tool.
And of course that becomes, then, a primarily qualitative approach,
and you would only do this with a relatively small number of participants,

00:11:00.480 --> 00:11:33.498
mostly because it will be moderately time-consuming; you could expect it to take at least
half an hour, and you will need a qualified  interviewer to do that.
Do make sure that if you're using this approach that the interviewer does have access to the respondent's initial 
questionnaire so they're not repeating themselves  or are not at all aware of
the participant's background or complaint.
*Semi-structured interviewing* is the most appropriate technique in most cases.
So, this would be a follow-up set of questions and then just the interviewer exploring

00:11:33.498 --> 00:11:38.478
some of the responses that the participant has given already.