WEBVTT

00:00:00.000 --> 00:00:31.600
Let's talk about the difference between big,  innovative changes to our product and small,
incremental improvements, and the kinds of research  that you might need in order to make these changes.
We'll start with the incremental  improvements because that's really
the most frequent kinds of changes  that we make as designers and researchers.
While we all like to talk about designing things  from scratch or making huge, sweeping changes,

00:00:31.600 --> 00:01:00.274
the vast majority of people spend a lot of their  time working on existing products and
making them a little bit better every day.
So, imagine you're building your new job marketplace to connect job seekers with potential employers.
The product works. It's out in the real world being used by folks to find jobs every day.
It's great! You made a thing that people are using, for money.
Now, your product manager is looking at the metrics and they notice that a bunch of people are signing up
and looking at jobs but they're not applying for anything.

00:01:00.274 --> 00:01:33.610
Your job is to figure out why.
So, what do you do?
You can go ahead and pause the video and think  about it for a minute if you want.
There are a lot of different options you could go with here, but at the very least you're going to want to figure
out the following things: Where are people stopping  in the process and why are they stopping there?
You'll probably want to dig into metrics a bit and  figure out if folks do anything besides just look at jobs.
Do they fill out their profile? Do they  look at job details? Do they click the Apply button?
And then do they give up at that point? Or do they never actually even get to that point?

00:01:33.610 --> 00:02:02.970
Once you know where they're giving up, you'll  probably do some simple observational testing
of actual users to see what's happening when  they do drop out.
You'll probably also want to talk to them about why they're not applying.
Maybe you'll find out that they get frustrated because they can't find jobs in their area.
Well, that'd be great because that's really easy to fix; if that's the problem,
maybe you can try letting them search  for jobs near them. That's an *incremental change*.
Now, what do we mean by that? It doesn't necessarily mean that it doesn't have a big impact on metrics.

00:02:02.970 --> 00:02:30.584
Things like this can be hugely important for your metrics.
If you manage to get lots more qualified candidates to apply to jobs, that's a huge win for the employers who
are looking for great employees and it doesn't matter that it was just a simple button that you added.
But it's not a wildly innovative change.  In fact, it's a pretty standard feature on most job boards,
and it's a very small improvement in terms of engineering effort, or at least it should be.
If it isn't, there may be something wrong with your engineering department... which is a totally different course.

00:02:30.584 --> 00:03:03.087
This change is *improving an existing flow*, rather than completely changing how something is done
or adding a brand-new feature.
OK, now, imagine that you're doing some observational research with your job applicants
and you learn that for whatever reason they really don't have very much access to computers
or they're not used to typing on a keyboard.
This might lead to a very different sort of change than just searching for jobs in their area.
Rather than making a small, incremental improvement to a search page,
you might have to come up with an entirely  different way for candidates to apply for jobs.

00:03:03.087 --> 00:03:31.084
Maybe they need to film themselves using their  phone cameras.
This is a much larger change; it's *less incremental* since you're probably going to have to change
or at least add a major feature to the entire job application process.
You'll probably have to change how job seekers get reviewed by potential employers as well since
they'll be reviewing videos rather than text resumes – which they might not be used to.
This is a big change, but it's still incremental because it's not really changing what the product does.

00:03:31.084 --> 00:04:06.090
It's just finding a new way to do the thing that it already did.
OK, now,
let's say that you have the option to do some really deep ethnographic research with some of your potential job applicants.
You run some contextual inquiry sessions with them or maybe you run a diary study to understand
all of the different jobs that they look at and learn why they are or aren't applying.
Maybe in these deeper, more open-ended research sessions, you start to learn that the reason that
a lot of potential job applicants drop out is because they just don't have the skills for the necessary jobs.

00:04:06.090 --> 00:04:30.480
But what could *you* do about that? Well...
our only options are either to find different applicants, find more suitable jobs or create some way to train our
users in the skills that they need for the kinds of jobs that are available.
All of those are really pretty big, risky ventures, but they just might be what we need to do to get more applicants into jobs.
These are very big, and a couple of them are fairly innovative changes.

00:04:30.480 --> 00:05:05.569
If the company pivots into, say, trainings and certifications or assessments, that definitely qualifies as
innovation, at least for your product, but *how* does the research change for *finding* each of these sorts of things?
Couldn't you have found out that applicants aren't qualified with the same types of research that you used
to learn that they wanted to search by location? Maybe.
Sometimes we find all sorts of things in  very lightweight usability-type testing, but 
*more often* we find bigger, more disruptive  things in deeper kinds of research
– things like contextual inquiry, diary studies or longer-term relationships that we build with our customers.

00:05:05.569 --> 00:05:30.464
Also, bigger, more disruptive changes often require  us to do more in-depth research
just to make sure that we're going in the right direction  because the bigger it is the more risky it is.
Let's say we ran some simple usability testing on  the application process.
That would mean we'd give applicants a task to perform, like find a job and apply to it.
What might we learn from that? Well, that's the place where we'd learn if there were any bugs or confusing

00:05:30.464 --> 00:06:04.231
parts of the system – basically, *can* somebody apply for a job?
It takes more of a real conversation with a real user or a potential user to learn why they're not applying for jobs.
It's not that one kind of testing is better than the other; it's that you can learn very different things with the
different types of testing. Some types of research tend to deliver more in-depth learnings that can lead
to big breakthrough changes, while other types of  research tend to lead to smaller, more incremental
but still quite useful and impactful changes.
Both are extremely useful on agile teams, but you may find that the latter is more common just because many

00:06:04.231 --> 00:06:15.041
agile teams don't really know how to schedule those big  longer-term types of research studies,
while running quick usability testing on existing  software is quite easy and can even often be automated.