WEBVTT 00:00:02.720 --> 00:00:09.670 I wanted to say a bit more about this important issue of recruiting participants. 00:00:13.668 --> 00:00:19.611 The quality of the results hinges entirely on the quality of the participants. 00:00:19.611 --> 00:00:22.210 If you're asking participants to do things 00:00:22.210 --> 00:00:28.215 and they're not paying attention or they're simply skipping through as quickly as they can 00:00:28.215 --> 00:00:32.403 – which does happen – then you're going to be very disappointed with the results 00:00:32.403 --> 00:00:37.210 and possibly simply have to write off the whole thing as an expensive waste of time. 00:00:37.210 --> 00:00:42.719 So, recruiting participants is a very important topic, but it's surprisingly difficult. 00:00:42.719 --> 00:00:48.352 Or, certainly, it can be. You have the idea that these people might want to help you 00:00:48.352 --> 00:00:52.251 improve your interactive solution – whatever it is; a website, an app, 00:00:52.251 --> 00:00:56.522 what have you – and lots of people *are* very motivated to do that. 00:00:56.522 --> 00:01:01.907 And you simply pay them a simple reward and everyone goes away quite happy. 00:01:01.907 --> 00:01:05.000 But it's certainly true with *online research* 00:01:05.000 --> 00:01:09.044 that there are people who would simply take part in order to get 00:01:09.044 --> 00:01:12.200 the reward and do very little for it. 00:01:12.200 --> 00:01:18.370 And it comes as quite a shock, I'm afraid, if you're a trusting person, that this kind of thing happens. 00:01:18.494 --> 00:01:22.805 I was involved in a fairly good-sized study in the U.S. 00:01:22.805 --> 00:01:28.500 – a university, who I won't name – and we had as participants in a series of studies 00:01:28.500 --> 00:01:32.771 students, their parents and the staff of the university. 00:01:32.771 --> 00:01:37.083 And, believe it or not, the students were the best behaved 00:01:37.083 --> 00:01:42.489 of the lot in terms of actually being conscientious in 00:01:42.489 --> 00:01:47.978 answering the questions or performing the tasks as required or as requested. 00:01:48.370 --> 00:01:51.184 Staff were possibly even the worst. 00:01:51.184 --> 00:01:55.531 And I think their attitude was "Well, you're already paying me, so 00:01:55.531 --> 00:01:59.327 why won't you just give me this extra money without me having to do much for it?" 00:01:59.327 --> 00:02:05.000 I really don't understand the background to that particular issue. 00:02:05.083 --> 00:02:08.197 And the parents, I'm afraid, were not a great deal better. 00:02:08.197 --> 00:02:11.126 So, we had to throw away a fair amount of data. 00:02:11.126 --> 00:02:16.883 Now, when I say "a fair amount", throwing away 10% of your data is probably pretty extreme. 00:02:16.883 --> 00:02:20.103 Certainly, 5% you might want to plan for. 00:02:20.103 --> 00:02:25.406 But the kinds of things that these participants get up to – particularly if you're talking about online panels, 00:02:25.406 --> 00:02:32.752 and you'll often come across panels if you go to the tool provider, if you're using, say for example, a card-sorting tool 00:02:32.752 --> 00:02:38.873 or a first-click test tool and they offer you respondents for a price each, 00:02:38.873 --> 00:02:42.346 then be aware that those respondents have signed up for this purpose, 00:02:42.346 --> 00:02:46.720 for the purpose of doing studies and getting some kind of reward. 00:02:46.720 --> 00:02:51.527 And some of them are a little bit what you might call on the cynical side. 00:02:51.527 --> 00:02:57.057 They do as little as possible. We've even on card sort studies had people log in, 00:02:57.057 --> 00:02:59.524 do nothing for half an hour 00:02:59.524 --> 00:03:03.008 and then log out and claim that they had done the study. 00:03:03.008 --> 00:03:06.721 So, it can be as vexing as that, I'm afraid. 00:03:06.721 --> 00:03:10.476 So, the kinds of things that people get up to: They do the minimum necessary; 00:03:10.476 --> 00:03:12.517 that was the scenario I was just describing. 00:03:12.517 --> 00:03:16.148 They can answer questions in a survery without reading them. 00:03:16.148 --> 00:03:18.354 So, they would do what's called *straightlining*. 00:03:18.354 --> 00:03:22.873 Straightlining is where they are effectively just answering every question the same 00:03:22.873 --> 00:03:26.214 in a straight line down the page or down the screen. 00:03:26.214 --> 00:03:31.042 And they also could attempt to perform tasks without understanding them. 00:03:31.042 --> 00:03:35.994 So, if you're doing a first-click test and you ask them, "Go and find this particular piece of apparel, 00:03:35.994 --> 00:03:38.758 where would you click first?", they'd just click. 00:03:38.758 --> 00:03:41.480 They're not reading it; they didn't really read the question. 00:03:41.480 --> 00:03:44.285 They're not looking at the design mockup being offered; 00:03:44.285 --> 00:03:47.524 they're just clicking, so as to get credit for doing this. 00:03:47.524 --> 00:03:51.960 Like I say, I don't want to paint all respondents with this rather black brush, 00:03:51.960 --> 00:03:54.997 but it's *some* people do this. 00:03:54.997 --> 00:03:59.428 And we just have to work out how to keep those people from polluting our results. 00:03:59.428 --> 00:04:04.091 So, the reward is sometimes the issue, that if you are too generous in the reward 00:04:04.091 --> 00:04:07.763 that you're offering, you will attract the wrong kind of participant. 00:04:07.763 --> 00:04:12.219 Certainly I've seen that happen within organizations doing studies on intranets, 00:04:12.219 --> 00:04:17.192 where somebody decided to give away a rather expensive piece of equipment at the time: 00:04:17.192 --> 00:04:22.908 a DVD reader, which was – when this happened – quite a valuable thing to have. 00:04:22.908 --> 00:04:26.394 And the quality of the results plummetted. 00:04:26.394 --> 00:04:30.000 Happily, it was something where we could actually look at the quality of the results and 00:04:30.000 --> 00:04:36.787 simply filter out those people who really hadn't been paying much attention to what they were supposed to be doing. 00:04:36.787 --> 00:04:40.000 So, like I say, you can expect for online studies 00:04:40.000 --> 00:04:43.920 to discard been 5 and 10% of your participants' results. 00:04:43.920 --> 00:04:47.179 You also – if you're doing face-to-face research – 00:04:47.179 --> 00:04:50.795 and you're trying to do quantitative sorts of numbers, 00:04:50.795 --> 00:04:55.857 say, you'd be having 20 or 30 participants, you probably won't have a figure quite as bad as that, 00:04:55.857 --> 00:05:01.511 but I still have seen, even in face-to-face card sorts, for example, 00:05:01.511 --> 00:05:04.321 people literally didn't *understand* what they were supposed to be doing, 00:05:04.321 --> 00:05:09.813 or didn't get what they were supposed to be doing, and consequently their results were not terribly useful. 00:05:09.813 --> 00:05:15.431 So, you're not going to get away with 100% valuable participation, I'm afraid. 00:05:15.431 --> 00:05:20.671 And so, I'm going to call these people who aren't doing it, and some of them are not doing it because they don't understand, 00:05:20.671 --> 00:05:26.433 but the vast majority are not doing it because they don't want to spend the time or the effort; 00:05:26.433 --> 00:05:29.410 I'm going to call them *failing participants*. And the thing is, 00:05:29.410 --> 00:05:33.349 we actually need to be able to *find* them in the data and take them out. 00:05:35.521 --> 00:05:39.466 You have to be careful how you select participants, how you filter them 00:05:39.466 --> 00:05:43.249 and how you actually measure the quality of their output, as it were. 00:05:43.249 --> 00:05:48.075 And one of the big sources of useful information are the actual tools that you are using. 00:05:48.075 --> 00:05:53.135 In an online survey, you can see how long people have spent, you can see how many questions they have answered. 00:05:53.135 --> 00:05:58.717 And, similarly, with first-click testing, you can see how many of the tasks they completed; 00:05:58.717 --> 00:06:01.059 you can see how long they spent doing it. 00:06:01.059 --> 00:06:04.751 And with some of these, we actually can also see how successful they were. 00:06:04.751 --> 00:06:09.901 In both of the early-design testing methods – card sorting and first-click testing – 00:06:09.901 --> 00:06:12.873 we are allowed to nominate "correct" answers 00:06:12.873 --> 00:06:16.691 – which is, I keep using the term in double-quotes here because 00:06:16.691 --> 00:06:20.000 there are no actually correct answers in surveys, for example; 00:06:20.000 --> 00:06:22.000 so, I'm using "correct" in a particular way: 00:06:22.000 --> 00:06:27.006 "Correct" is what we think they should be doing when they're doing a card sort, *approximately*, 00:06:27.006 --> 00:06:30.446 or, in particular, when they're doing a *first-click test*, 00:06:30.446 --> 00:06:33.814 that we think they ought to be clicking around about here. 00:06:36.650 --> 00:06:40.774 Surveys as a group are a completely different kettle of fish, as it were. 00:06:40.774 --> 00:06:43.746 There are really no correct answers when you start. 00:06:43.746 --> 00:06:46.682 You've got your list of research questions – things that you want to *know* – 00:06:46.682 --> 00:06:50.663 but what you need to do is to incorporate questions and answers 00:06:50.663 --> 00:06:55.702 in such a way that you can check that people are indeed *paying attention* 00:06:55.710 --> 00:06:58.664 and *answering consistently*. 00:06:58.664 --> 00:07:03.580 So, you might for example change the wording of a question and reintroduce it later on 00:07:03.580 --> 00:07:06.048 to see if you get the same answer. 00:07:06.048 --> 00:07:09.740 The idea is to be able to get a score for each participant. 00:07:09.740 --> 00:07:14.206 And the score is your own score, about basically how much you trust them 00:07:14.206 --> 00:07:17.052 or maybe the *inverse* of how much you trust them. 00:07:17.052 --> 00:07:20.000 So, as the score goes up, your trust goes down. 00:07:20.000 --> 00:07:24.196 So, if these people keep doing inconsistent or confusing things, 00:07:24.196 --> 00:07:29.310 like replying to questions with answers that aren't actually real answers – you've made them up – 00:07:29.310 --> 00:07:33.650 or not answering two questions which are effectively the same the same way, etc., 00:07:33.650 --> 00:07:37.828 then you would get to a point where you'd say, "Well, I just don't trust this participant," 00:07:37.828 --> 00:07:40.724 and you would yank their data from your results. 00:07:40.724 --> 00:07:44.826 Happily, most of these tools do make it easy for you to yank individual results. 00:07:44.826 --> 00:07:49.682 So, we have to design the studies to *find* these failing participants. 00:07:49.682 --> 00:07:56.254 And, as I say, for some these tools – online tools we'll be using – that is relatively straightforward, but tedious. 00:07:56.254 --> 00:08:02.946 But with surveys, in particular, you are going to have to put quite a bit of effort into that kind of research. 00:08:06.999 --> 00:08:09.071 Steps we can take in particular: 00:08:09.071 --> 00:08:12.061 Provide consistency checks between tasks or questions. 00:08:12.061 --> 00:08:18.435 Ensure that "straightlined" results – where people are always answering in the same place on each and every question down the page – 00:08:18.435 --> 00:08:22.901 ask the same question again in slightly different wording or with the answers in a different order. 00:08:22.901 --> 00:08:26.827 Now, I wouldn't go around changing the order of answers on a regular basis. 00:08:26.827 --> 00:08:29.439 You might have one part of the questionnaire where 00:08:29.439 --> 00:08:32.735 "good" is on the right and "bad" is on the left; 00:08:32.735 --> 00:08:37.160 and you might decide to change it in a completely different part 00:08:37.160 --> 00:08:39.819 of the questionnaire and make it really obvious that you've changed it 00:08:39.820 --> 00:08:41.871 to those who are paying attention. 00:08:41.871 --> 00:08:47.322 But whatever it is that you do, what you're *trying* to do is to find people who really aren't paying much attention 00:08:47.322 --> 00:08:51.860 to the directions on the survey or whatever the research tool is, 00:08:51.860 --> 00:08:54.742 and catch them out and pull them out of your results. 00:08:54.742 --> 00:09:00.000 And of the issues you should be aware of if you're paying for participants from something 00:09:00.000 --> 00:09:04.913 like your research tool *supplier* is that you can go back to them and say, 00:09:04.913 --> 00:09:09.598 "These people did not do a very good job of completing this survey, this study." 00:09:09.598 --> 00:09:12.912 And ask them to refund you for the cost of those. 00:09:12.912 --> 00:09:15.686 You tell them that you're having to pull their data out of your results. 00:09:15.686 --> 00:09:19.720 Also, it helps to tidy up their respondent pool. 00:09:19.720 --> 00:09:23.700 Perhaps it's not your particular concern, but if you do end up using them again, 00:09:23.700 --> 00:09:28.148 it would be nice to know that some of these people who are simply gaming the system 00:09:28.148 --> 00:09:30.274 have been removed from the respondent pool. 00:09:30.274 --> 00:09:34.902 So, reporting them – getting them removed from the pool – is a sensible thing to be doing. 00:09:34.902 --> 00:09:38.918 And, finally, devising a scoring system to check the consistency 00:09:38.918 --> 00:09:45.598 and also checking for fake responses and people who are just not basically doing the research as you need them to do it.