A/B testing is all about changes in behavior. We present people with alternative designs and we look to see how much that alters their subsequent response. So in the simple A/B case, we show them design A, we show them design B, and we measure typically a completion goal, which a lot of subject areas in user experience we refer to as conversions.
So signing up to a newsletter, adding an item to a shopping basket, making a donation to a charity. These are all things that are important to their respective organizations. And typically for the interactive technology that we're working on. So websites and and apps, for example. So these are the things often that we're measuring, but they're not the only things that we can measure. We can measure really straightforward stuff like time spent on page, time spent in the site and also bounce rates.
For example, we'll be looking at some of those a bit later on. Just a reminder that because A/B testing is done very late in the day with live sites and large numbers of users, you really want to make sure that your solution is sound before you get this far. You're not going to be able to test everything that is possibly worrying you or possibly causing problems to users. It's just too long involved and potentially expensive in terms
of user loyalty and also the amount of effort you'd have to put into it. So we are looking at using A/B testing to basically polish the solution rather than to rework it. Bear that in mind and make sure that you've done adequate testing up to this point. Also, bear in mind that A/B testing tends to be focused on individual pages, so it is possible to have multi-page tests, but
it's a more complex area than we're going to be looking at in this lesson. So experiments have research questions that basically the things that you're trying to answer and because A/B testing focuses on changes in behavior, the research questions are going to be centered on defined goals. And as I've mentioned already, typically conversions. So will as an example, moving the add button above the fold improve sales conversions? I would imagine it would actually do something. I always find people
are making the mistake of getting too talkative on the first screen of the page and the actual “buy this” or “add to basket” button gets pushed further and further down until users actually don't even see it. Will a more clearly worded charitable purpose increase donations? If people have a better understanding of what your charity's about or where this money is going, would that improve conversions for those users? So both of these can be A/B tested by using goals that you almost
certainly have already defined in your analytic solution. So these are very good candidates for A/B and multivariate testing. But I'll give you some examples of bad questions too. So obviously I will repeat the words “don't ask this” when I've mentioned them because they're not meant as examples that you should be taking away. Conversely, research questions that are not directly related to improved goal completions tend not to be suitable for AB testing.
And a kind of vague question like “will better product photos reduce questions to customer service?”, don't ask this, is the sort of thing that you simply cannot effectively test in A/B testing. And the reason is that there are all kinds of channels to customer service and only some of them are through the website and only some of them can be effectively measured as goals. So it's just not a suitable scenario for A/B testing. There is a related question you could ask though,
which might be just as good, although not exactly equivalent, and that would be: “Will better product photos improve sales conversions?” Because if it reduces queries to customer service, it's almost certain that people are going to be much more confident about placing orders, adding those things to their basket. So that is a very easily measured outcome in terms of A/B testing, and that is the kind of question that A/B testing is very good at.
So simply rewording or rethinking the question in terms of defined user and business goals is one way of getting to a satisfactory conclusion, even if you have a slightly squiffy question to start with.