WEBVTT 00:00:03.040 --> 00:00:06.880 Let's talk about the difference between big,  innovative changes to our product and small, 00:00:06.880 --> 00:00:11.928 incremental improvements, and the kinds of research  that you might need in order to make these changes. 00:00:16.417 --> 00:00:19.440 We'll start with the incremental  improvements because that's really 00:00:19.440 --> 00:00:23.494 the most frequent kinds of changes  that we make as designers and researchers. 00:00:26.560 --> 00:00:31.600 While we all like to talk about designing things  from scratch or making huge, sweeping changes, 00:00:31.600 --> 00:00:35.768 the vast majority of people spend a lot of their  time working on existing products and 00:00:35.768 --> 00:00:38.125 making them a little bit better every day. 00:00:38.125 --> 00:00:44.016 So, imagine you're building your new job marketplace to connect job seekers with potential employers. 00:00:44.061 --> 00:00:48.406 The product works. It's out in the real world being used by folks to find jobs every day. 00:00:48.472 --> 00:00:52.246 It's great! You made a thing that people are using, for money. 00:00:52.246 --> 00:00:56.982 Now, your product manager is looking at the metrics and they notice that a bunch of people are signing up 00:00:56.982 --> 00:01:00.274 and looking at jobs but they're not applying for anything. 00:01:00.274 --> 00:01:03.284 Your job is to figure out why. 00:01:03.284 --> 00:01:05.250 So, what do you do? 00:01:05.250 --> 00:01:08.071 You can go ahead and pause the video and think  about it for a minute if you want. 00:01:08.071 --> 00:01:11.891 There are a lot of different options you could go with here, but at the very least you're going to want to figure 00:01:11.891 --> 00:01:17.135 out the following things: Where are people stopping  in the process and why are they stopping there? 00:01:17.520 --> 00:01:23.073 You'll probably want to dig into metrics a bit and  figure out if folks do anything besides just look at jobs. 00:01:23.073 --> 00:01:28.019 Do they fill out their profile? Do they  look at job details? Do they click the Apply button? 00:01:28.160 --> 00:01:33.610 And then do they give up at that point? Or do they never actually even get to that point? 00:01:33.920 --> 00:01:38.240 Once you know where they're giving up, you'll  probably do some simple observational testing 00:01:38.240 --> 00:01:41.660 of actual users to see what's happening when  they do drop out. 00:01:41.660 --> 00:01:45.042 You'll probably also want to talk to them about why they're not applying. 00:01:45.042 --> 00:01:49.329 Maybe you'll find out that they get frustrated because they can't find jobs in their area. 00:01:49.495 --> 00:01:53.192 Well, that'd be great because that's really easy to fix; if that's the problem, 00:01:53.192 --> 00:01:57.894 maybe you can try letting them search  for jobs near them. That's an *incremental change*. 00:01:57.894 --> 00:02:02.970 Now, what do we mean by that? It doesn't necessarily mean that it doesn't have a big impact on metrics. 00:02:02.970 --> 00:02:05.816 Things like this can be hugely important for your metrics. 00:02:05.816 --> 00:02:11.022 If you manage to get lots more qualified candidates to apply to jobs, that's a huge win for the employers who 00:02:11.022 --> 00:02:15.535 are looking for great employees and it doesn't matter that it was just a simple button that you added. 00:02:16.320 --> 00:02:21.173 But it's not a wildly innovative change.  In fact, it's a pretty standard feature on most job boards, 00:02:21.173 --> 00:02:25.552 and it's a very small improvement in terms of engineering effort, or at least it should be. 00:02:25.602 --> 00:02:30.584 If it isn't, there may be something wrong with your engineering department... which is a totally different course. 00:02:30.690 --> 00:02:35.583 This change is *improving an existing flow*, rather than completely changing how something is done 00:02:35.635 --> 00:02:38.495 or adding a brand-new feature. 00:02:38.495 --> 00:02:42.703 OK, now, imagine that you're doing some observational research with your job applicants 00:02:42.703 --> 00:02:47.607 and you learn that for whatever reason they really don't have very much access to computers 00:02:47.607 --> 00:02:50.708 or they're not used to typing on a keyboard. 00:02:50.708 --> 00:02:55.478 This might lead to a very different sort of change than just searching for jobs in their area. 00:02:55.537 --> 00:02:58.946 Rather than making a small, incremental improvement to a search page, 00:02:58.960 --> 00:03:03.087 you might have to come up with an entirely  different way for candidates to apply for jobs. 00:03:03.440 --> 00:03:06.995 Maybe they need to film themselves using their  phone cameras. 00:03:06.995 --> 00:03:12.337 This is a much larger change; it's *less incremental* since you're probably going to have to change 00:03:12.337 --> 00:03:16.620 or at least add a major feature to the entire job application process. 00:03:16.620 --> 00:03:21.090 You'll probably have to change how job seekers get reviewed by potential employers as well since 00:03:21.090 --> 00:03:25.286 they'll be reviewing videos rather than text resumes – which they might not be used to. 00:03:25.286 --> 00:03:31.084 This is a big change, but it's still incremental because it's not really changing what the product does. 00:03:31.084 --> 00:03:34.832 It's just finding a new way to do the thing that it already did. 00:03:37.920 --> 00:03:39.978 OK, now, 00:03:39.978 --> 00:03:45.635 let's say that you have the option to do some really deep ethnographic research with some of your potential job applicants. 00:03:45.635 --> 00:03:50.754 You run some contextual inquiry sessions with them or maybe you run a diary study to understand 00:03:50.754 --> 00:03:55.258 all of the different jobs that they look at and learn why they are or aren't applying. 00:03:55.258 --> 00:03:59.770 Maybe in these deeper, more open-ended research sessions, you start to learn that the reason that 00:03:59.770 --> 00:04:06.090 a lot of potential job applicants drop out is because they just don't have the skills for the necessary jobs. 00:04:06.090 --> 00:04:09.528 But what could *you* do about that? Well... 00:04:09.528 --> 00:04:14.621 our only options are either to find different applicants, find more suitable jobs or create some way to train our 00:04:14.621 --> 00:04:19.191 users in the skills that they need for the kinds of jobs that are available. 00:04:19.191 --> 00:04:26.902 All of those are really pretty big, risky ventures, but they just might be what we need to do to get more applicants into jobs. 00:04:26.902 --> 00:04:30.480 These are very big, and a couple of them are fairly innovative changes. 00:04:30.480 --> 00:04:36.188 If the company pivots into, say, trainings and certifications or assessments, that definitely qualifies as 00:04:36.188 --> 00:04:43.049 innovation, at least for your product, but *how* does the research change for *finding* each of these sorts of things? 00:04:43.098 --> 00:04:47.120 Couldn't you have found out that applicants aren't qualified with the same types of research that you used 00:04:47.120 --> 00:04:51.340 to learn that they wanted to search by location? Maybe. 00:04:51.340 --> 00:04:55.280 Sometimes we find all sorts of things in  very lightweight usability-type testing, but  00:04:55.280 --> 00:04:59.960 *more often* we find bigger, more disruptive  things in deeper kinds of research 00:04:59.960 --> 00:05:05.569 – things like contextual inquiry, diary studies or longer-term relationships that we build with our customers. 00:05:05.680 --> 00:05:10.862 Also, bigger, more disruptive changes often require  us to do more in-depth research 00:05:10.862 --> 00:05:16.667 just to make sure that we're going in the right direction  because the bigger it is the more risky it is. 00:05:16.960 --> 00:05:20.818 Let's say we ran some simple usability testing on  the application process. 00:05:20.818 --> 00:05:25.469 That would mean we'd give applicants a task to perform, like find a job and apply to it. 00:05:25.469 --> 00:05:30.464 What might we learn from that? Well, that's the place where we'd learn if there were any bugs or confusing 00:05:30.464 --> 00:05:34.670 parts of the system – basically, *can* somebody apply for a job? 00:05:34.670 --> 00:05:41.234 It takes more of a real conversation with a real user or a potential user to learn why they're not applying for jobs. 00:05:41.234 --> 00:05:46.078 It's not that one kind of testing is better than the other; it's that you can learn very different things with the 00:05:46.078 --> 00:05:51.200 different types of testing. Some types of research tend to deliver more in-depth learnings that can lead 00:05:51.200 --> 00:05:55.600 to big breakthrough changes, while other types of  research tend to lead to smaller, more incremental 00:05:55.600 --> 00:05:58.705 but still quite useful and impactful changes. 00:05:58.705 --> 00:06:04.231 Both are extremely useful on agile teams, but you may find that the latter is more common just because many 00:06:04.231 --> 00:06:09.052 agile teams don't really know how to schedule those big  longer-term types of research studies, 00:06:09.052 --> 00:06:15.041 while running quick usability testing on existing  software is quite easy and can even often be automated.