Mobile app user research in the lab may not be as effective as remote research. Users of mobile apps are going to face continued distractions when using their smartphones and there’s no better simulation for these distractions than monitoring the user in their ordinary environment. There is a strong business case for remote research for mobile apps and it should help designers build better mobile user experiences.
Author/Copyright holder: Intel Free Press. Copyright terms and licence: CC BY-SA 2.0
Remote research is used in many UX projects because it allows the researcher to learn about the use of a product in the “natural environment of use”. This is becoming an even more important area of research in the mobile age where smartphone users are interacting with mobile apps “on the fly” and in amongst many distractions. The context of use for mobile apps often informs how those apps are used.
That is not to say that remote research should be the “holy grail” of mobile UX research – in order to gain the most valuable insights, UX researchers should look (wherever possible) to combine remote and other methods to get the complete picture of mobile app use and usability.
Remote Research – A Non-Technical Solution
Not all remote research requires an investment in software or hardware; in fact there is one remote method which relies on good old fashioned pen and paper. That’s the diary (or journal) study.
A diary study is a long-term study in which users make notes on when they use the product, how they used the product and any frustrations that they encounter. It is important for the researcher to choose a sample size large enough for results to be statistically significant and which allows for the inevitable “drop out” as some study participants fail to complete their diaries or are unable to continue with the study.
Author/Copyright holder: Ildar Sagdejev. Copyright terms and licence: CC BY-SA 4.0
In theory diary studies provide examples of how users work with their apps in their own environment. However, it’s worth noting that even the most dedicated diary keeper is occasionally going to forget to record an interaction or two. One big advantage of the diary study is that it allows for very honest and personal feedback with each user enabled to provide detailed feedback on any given event.
Project teams running diary studies need to take an active role in supporting the studies; this can help prevent drop out and also act as a useful reminder to participants to update their diaries.
Remote Research – Technical Solutions
Online Surveys
Online surveys can be delivered to the device on which the app is running, as part of the app itself or via e-mail. As with all surveys they need careful design to avoid leading the participant to the conclusions that the researcher favours and to ensure that the data provided is mainly qualitative so that it can be easily analysed and used to convince decision makers to support the design team’s conclusions.
Surveys are an excellent tool for collecting a lot of data in a fairly short period of time. As long as the sample sizes (of returned surveys) are statistically significant the data they provide can be highly informative for a project. Given the proliferation of cheap (or even free) survey design and distribution tools online now; they are also budget friendly.
Results do need to be treated with caution, however, there’s the chance that results do not properly distinguish between what people do and what they say they do.
Surveys may also identify areas of interest without explaining the “why” of things; this creates a case for further user research.
Behavioural Analytics
Author/Copyright holder: Photo.iep. Copyright terms and licence: CC BY 3.0
It can be a very good idea to combine behavioural analytics, which is when the user’s actions and behaviour on the device are tracked by software, with surveys to get a clear distinction between the “what users do” and “what they say they do”.
There are challenges associated with behavioural analytics; firstly they do not allow the researcher to examine the motivations for a behaviour – they only report on what is happening and not why the user chose to do something or how they made the decision.
It is also difficult to examine the context of use with behavioural analytics packages and it may be useful to carry out a small scale field study or observation to answer the question of context after a behavioural analytics research exercise.
Automatic Logging
Automatic logging is simply an automated recording of the events generated by or sensed by the mobile device. It is carried out without any action required by the participant at all (that does not mean it is carried out in secrecy, it is important that users opt in to any form of research).
This allows the researcher to build an accurate picture of the way that interaction occurs with the mobile app. Like behavioural analytics there is a short coming with regards to the ability to understand a user’s motivations and the context of use.
ESM (Experience Sampling Methods)
ESM is basically a brief, usually single question (or at most 3-4 questions) survey that is sent to the user on their device at the moment of an interaction. It allows the researcher to collect data while the experience is fresh in the user’s mind.
ESM sampling needs to be carefully designed to ensure that cognitive biases don’t prejudice the results.
The Take Away
Remote research is an excellent way of discovering the way that mobile apps are used “in the wild”. It is reasonably cost-effective and budget friendly as a rule and while some care needs to be given to the design of the research; it may be the best way to examine the user-experience of mobile products.
References
UX Mag has a great round up of mobile testing tools here - https://uxmag.com/articles/mobile-testing-toolbox-part-1
And also here - https://uxmag.com/articles/mobile-testing-toolbox-part-2
Header Image: Author/Copyright holder: highwaysengland. Copyright terms and licence: CC BY 2.0
