Many years ago, I was delivering a training course on products that would be introduced for commercial sale in a telecoms environment. The information for that course had been developed and provided by the product management team.
Taking this information on good faith, I went and delivered the course that they had requested. Unfortunately for me, within minutes of starting delivery, it became apparent that… well, the information I had been given didn’t match the reality of the business. I was training people in abject nonsense.
At the end of the session, I explained that I was sorry for the content and that in order to be able to go and revise the content before I delivered it again – I would need the participants to say that the training hadn’t met their needs. Then I handed out feedback forms for those participants. What happened? They all rated the training as excellent. Despite the fact that it wasn’t. I knew it. They knew it. So why did they give me such good feedback?
This is an important question because it also relates to user testing. We’ve all watched a user struggle with something in a test only to report that they found it incredibly easy when asked about their experience at the end.
So what is it that leads people to give positive feedback even when that feedback is not merited? And how can we avoid that to get better feedback?
People Don’t Want to Hurt Other People’s Feelings
That group I was training? They were people I knew. They didn’t want to give me bad feedback, in part, because they thought it might be hurtful to me. This remained true even when I’d told them, I wanted bad feedback.
This may be a big challenge to overcome in the classroom but it’s actually easier to do in the user testing arena. You need to tell a small fib to your participants when you begin the testing process. Explain that you are testing something for a third party and that you have no relationship with that third party.
This means that they know that their feedback has no impact on you. They can’t hurt your feelings because you’re not attached, in any meaningful way, to the product under test.
Author/Copyright holder: romana klee. Copyright terms and licence: CC BY-SA 2.0
People Often Assume That is Them Being Tested
Whether you’re saying; “Hey! I learned nothing useful in that training.” or “Hey! I found that form a total PITA to complete.” It can lead to an assumption that you’re the one at fault. That you’re the person being tested and not the training or the product.
Once again, if people think they’re under test – they’re going to give you different feedback than if they’re testing the system. You’ll often hear; “I’m really sorry about that I just blundered a little at that point.” as opposed to “It was really unclear what I was meant to do at that point.”
So start your testing by explicitly stating; “We are not here today to test you. We are here to test the system. We know this system has some issues. We need your help to identify those issues.”
This helps the tester separate themselves from the test and sets them up to give honest valuable feedback.
Author/Copyright holder: Dennis S Hurd. Copyright terms and licence: CC BY-NC-ND 2.0
People Often Focus on Completion Rather Than the Journey
In the classroom, everyone wants the certificate of completion, even though it’s really just a bit of paper that you print with no more meaning than a bit of paper that they could print themselves. In user testing there’s a certain sense of achievement that you have finished the test. (In the same way that finishing a test at school made you feel good).
The trouble with this is that when people feel good, they tend to produce nothing but positive feedback. Countering this requires a certain amount of additional effort. What you want to do is walk them back through the process and collect feedback at teach stage of the process. Refer to your observations and bring up any issues that you saw them encounter on the way through.
Push for a little more information and you’d be amazed at how quickly it’s forthcoming. Sure, it means for a slightly longer test but it also means that you’re going to get actionable, useful data rather than another meaningless pat on the back for a frustrating process.
People Can Show You More if You Let Them
You’re the expert and that’s why you’re conducting the tests. The assumption of your own expertise can lead you to assume that you’ve picked up all pain points through observation. This makes it less likely that you’ll ask your users to show you more.
Back when I started developing training, I had no real idea what I was doing. I knew that the product was “fit for purpose” but I also knew it could be better. At the end of each day’s session (it was a 2 week course), I would leave the room and give the group 15 minutes to come up with 3 highlights of the day and 3 things they thought could have been improved upon. The feedback that came out of those sessions was invaluable to finalizing the product (and in fact after 3 run-throughs of the course, new groups could no longer think of any improvements).
You have to ask; “How would you improve this product if you could?” at the end of a session too. You won’t always get valuable input but you may find that you do.
Author/Copyright holder: Derek Bridges. Copyright terms and licence: CC BY 2.0
Getting better feedback out of users is a conscious process. It doesn’t require huge amounts of effort to put processes in place to elicit better feedback.
You may also find that having a second observer or videoing the session can also extract higher levels of feedback – reviewing or having a second pair of eyes can help pick up things you missed that might have been obvious if you hadn’t have been focused on something else at the time.