Here’s the short watchable version for people who don’t have good attention spans. I get you. It’s a long piece. Worth it though.
Mobile research has come of age
Smartphones have opened up the possibility of cost-effective mobile research. These projects can be conducted at the point when participants are actually making decisions, giving us true in the moment insights.
But we need to make big changes to the design of our mobile research studies if we want to make the best of these approaches.
In this post, I’ll outline 5 success factors when writing tasks and questions for mobile research projects. Rather than emulate the way we’ve always tended to ask questions online, when asking in the moment we should:
- Give thought to the context in which mobile research participants are completing tasks
- Consider how to make use of the mobile participant (ie, one that can move)
- Be considerate of the limited time people have when on the go
- Adopt a friendly, informal tone of voice; try to generate a rapport!
In the moment offers us brand new opportunities as researchers – but it does require us to do things a little differently!
1. Ask in the moment
Smartphones are now the default
The smartphone fundamentally changed our online behaviour. Before the smartphone, the internet was generally accessed from a desk, via a computer. Being comfortably sat down, and often focussed on a particular task, we tended to engage in the activity for longer.
These days the main method of getting online is via our smartphones (See Ofcom’s reports for data on the UK market), and instead of using it to find an answer, or solve a problem, we’re more likely to be going online just to fill in little bite-sized gaps whilst on-the-go.
The big change offered by smartphones is their immediacy. Because we carry them with us everywhere, we’re able to interact with the digital world as and when we choose. We may be using our phone to give us directions to a meeting, listening to a podcast on the train, or just idly browsing our social media feed while we wait at the bus stop. The point is we all now carry our computers, and our internet access, in our pockets.
The fact that research participants can immediately access our research project wherever they are, and whatever they’re doing, offers exciting new opportunities. And studies have time and time shown us that the answers that participants give when asked whilst actually making a decision (the moment of truth) can often be very different to the more considered responses they give after the fact.
Poor recall challenges traditional research methods
We often ask participants to reflect on very low conscious decisions (“which chocolate bar should I get?”) which they may have done weeks ago (How many chocolate bars have you bought in the past 30 days? Think back to the most recent – why did you choose this bar on this occasion?).
If these questions can be answered at all, any answers given are likely to be heavily post-rationalised – far more focussed on “sensible-sounding” reasons for their purchase than may have been the case. Price, promotion, the impact of brand – these all sound like good reasons why I may have bought something three weeks ago. But, had you asked me at the time I would have told you that I needed an energy boost, the product had lots of sugar, and I liked the pretty orange colour of the wrapping.
Whilst in-the-moment doesn’t completely get over the issue of post-rationalisation – we are still asking participants for their conscious, rather than unconscious, motivations after all – it does at least reduce it, and eliminates the fug of faulty memory from the equation.
Tools already exist that allow us to run a variety of mobile research projects: qualitative studies, self-ethnography, communities, even quantitative diaries – but as researchers we’ve been slow to adapt to this new paradigm.
The trouble is, we’re so used to thinking of research as something reflective – where we’re asking participants to consider their past behaviour rather than what they’re doing now – that we default to asking questions in this way even when asking participants in the moment, and we end up losing what makes this approach so unique.
To make the most of mobile research we need to change the way we’re speaking to participants – to ask them questions which make sense in their current context, rather than just carrying over the tools we’ve used online to the mobile space.
2. Consider the context
Mobile is the behaviour, not just the device
When researchers talk about mobile research they usually mean “asking participants questions via their mobile phones”. But we should really be thinking about what it means if the participant is “mobile” – ie not tethered to their computer.
Research conducted on smartphones means that the participant can move about, so they won’t always be in the same place. They may not even be in the same place when they start a survey as they are when they finish it.
It’s easy to view this as a problem. What does it mean for an advertising study if a participant could be standing beneath a billboard as they complete the questionnaire? Lots of online surveys simply exclude those who attempt to complete on mobile in order to ensure a more consistent response.
But to do this is to exclude a high proportion of participants who prefer this channel (increasingly the case amongst younger participants), or the growing number of households that have smartphones and tablets but no laptop / desktop computer.
Excluding these participants from our studies could well lead to some subtle biases in our results. For example, we can easily imagine that a test for a new app-based concept would not perform as well if we excluded phone/tablet only households from the sample!
Think about where participants might be
Rather than excluding these participants then, we just need to carefully consider the various contexts in which participants may be completing a survey, and decide whether the questions we’re asking are likely to be affected if participants complete them in different settings.
If we believe this could be the case, then providing some clear instructions about where to complete the survey at the beginning, or in the invite, should be enough to ensure a reasonably consistent response.
For in the moment projects specifically – where participants are actively recruited to take part using their phones – every task should clearly set the context in which it is to be completed. If you want participants to take photos of the clothes in their wardrobe make sure to make it clear from the get go that the task needs to be completed at home, rather than on the bus!
It’s fair to assume that all questions within the same task will be completed in the same context (or at least within easy walking distance). This means you can’t mix and match contexts within the same activity. You can’t, for example, ask people to take photos of clothes in their wardrobe and, within the same task, ask them to video a shopping trip.
There’s also the time element of context to consider. Tasks should be assumed to be completed within a single session. You might be able to get away with a task that asks questions of the beginning and end of a bus ride, but not one that asks about their breakfast and evening meal at the same time.
3. Do mobile research
Embrace mobility in your design
Consideration of participants’ context is the single most important factor when conducting in the moment research, and is truly what differentiates this approach from a typical online survey.
But we shouldn’t simply treat it as an inconvenience – the fact that research participants can move about can be used to your advantage! Just as we should be clear about the need to complete a survey at home – if our study relies on participant’s recall of advertising or brands – so too can we set tasks which rely on participants being able to get about.
For example, let’s say we’re interested in customers’ decisions about which brand of deodorant to buy. Previously we would really only have had these options:
- rely on participants’ ability to recall their decision after the fact (which we know to be affected by post-rationalisation and faulty memory)
- set up mock-up shelves in central venues or labs (which creates a set of conditions which may not exist in the real world)
- switch to qualitative shopalongs (which are great, but expensive to conduct at scale)
Today, participants have the means to answer questions when they naturally go shopping for deodorant, so we can just set them a task to complete when they get to the shelf. And – if we want to – we can even ask them to video themselves making the choice and narrating their decision-making as it happens.
Set tasks that involve going, doing or even buying
Mobile research allows us to create tasks that ask participants to go somewhere, or do something.
We might recruit regular McDonalds visitors and tell them to visit Burger King instead for 3 weeks, and keep a diary of their experience.
Or perhaps we expose participants to something completely new – a new product, service, or retail experience – and have them capture their genuine first impressions as they try it for the first time.
And whilst having participants complete advertising recall projects is problematic when you can’t guarantee that they won’t be able to see the advertising as they complete the survey, you could instead turn this into a different sort of experiment; you might have a week of asking participants to record their encounters with advertising in a category as it happens, and then a second week where you tell participants to actively seek out the client’s advertising, like a scavenger hunt – to really test its reach.
4. Split your mobile research into short tasks
Don’t disrupt natural behaviour
If the advantage of mobile research is the ability to capture how people think and feel in different contexts, then we must also be aware that we’re essentially interrupting them as they go about their day.
Unlike traditional research, where we’re assuming that people are making some time to sit down and complete the exercise we’re setting them, with mobile research we want the occasion to be the focus, rather than the research task.
Let’s imagine we’re conducting a media diary, and asking participants to record the different shows they’re watching over the course of a typical week.
A 2-3 minute task to tell us what they’re watching and how they would rate it can be completed in an ad break, but if we ask them a 20 minute survey we’re actually taking them away from the TV and missing shows they would otherwise have watched.
Long tasks like this also run the risk of quickly fatiguing the participant, meaning they start to become more selective about the occasions they record (since each time they do, they’re going to have to complete another long survey).
If participants can’t be bothered to complete the exercise, we risk generating a completely inaccurate picture of their habits – perhaps one that is focused more on the “big event” shows, or their personal favourites, rather than all the silly things they watch to fill the time between.
Small is beautiful
Keeping tasks short does not mean a loss of detail. In-the-moment studies can end up being highly complex. Within a single project you may have:
- different tasks on different days
- different tasks available to participants based on specific conditions (eg one set of questions for men, and another for women)
- a mix of diary tasks, one-off activities
- photographic tasks, video vox pops, surveys and moderated discussions
But it’s important that participants are able to complete any single task quickly. In practice this often means splitting tasks into neat bite-sized chunks – each with a very specific focus.
If you have a task that takes more than about 5 minutes to complete, or requires participants to be in different locations in order to answer different questions then you should consider breaking it up. Doing so gives participants more freedom to complete at their own pace, and fit the study around their lives.
With any repeatable diary exercises you should be absolutely ruthless about every question being asked, and it’s also important to give participants a clear structure on how to complete them so they know what information you’re looking for and can start to learn the format. If you can make it as easy as possible to complete the diary you’ll get more responses from participants.
You should also be careful never to have too many tasks open to the participant at once – as this is likely to overwhelm them. Again, by splitting tasks up, you can stage the tasks across different days, or following the completion of prior exercises. In this way tasks are quick to complete, participants feel they make progress through the study, but rich detail arises from the sum total of the project’s disparate elements.
5. Be human
Use personality in the first person
As we’ve seen, although tasks are short, mobile research studies can be complex. And because we’re interested in participants’ thoughts in different settings and at different times, they almost always take place over an extended period with the participant dipping in and out of the project as required. As a consequence maintaining participant engagement is absolutely paramount.
However, in our default mode of thinking of research as reflective, and especially in the world on online quant, we tend to adopt a passive voice when asking questions.
Rather than introduce ourselves personally as the researcher, we talk about ourselves in the third person; we’ll say “we are a research agency looking into companies and brands”, rather than “I am interested in your thoughts on companies and brands”.
Avoid stilted questionnaire jargon
Furthermore, in an effort to eliminate the possibility of participant ambiguity, we often add complex framing devices to provide context to participants.
For example:
“Consider the hot coffee drinks you have bought from Starbucks in the past 30 days. By hot coffee drinks we mean any caffeinated hot beverages made from ground coffee beans. Please do not include teas, hot chocolates (except caffeinated mochas), or iced coffees.”
The combined impact of the use of a passive voice, and these framing devices is a very formal, cold, and often complex form of writing.
Or, put another way, we just don’t come across as very friendly.
But there really is no need for these sorts of framing devices when we conduct in-the-moment research, because the moment itself provides this context.
If we’re asking people to complete a diary task every time they buy a hot coffee drink from Starbucks then every the questions can be assumed to be answered in this context. This frees us to ask shorter, and more natural questions, such as:
“How much did you pay for it?”
“How would you rate this coffee?”
“Who are you with?”
These are quicker to read, quicker to complete, and far more engaging for participants.
Engagement can be further enhanced by introducing the research team at the beginning of an in-the-moment project (with photos, a bio, or a short video). By letting the participant know who is working on the project we create a connection – and participants are far more willing to complete tasks and offer opinion, if they feel that a “real person” is on the receiving end.
By introducing ourselves, and adopting an informal, natural, approach to questioning we build rapport with participants, which is only fair, they’re carrying us about with them, so should feel comfortable bringing us along for the ride.
How to do mobile research well: a quick cheat sheet
- Ask in the moment: set tasks to gather in situ feedback at point of decision / purchase / behaviour.
- Consider the context: design mobile research tasks that take account of participants’ location, exposure to brands or distraction levels.
- Do mobile research: assume your respondents are on the move, and build this into your mobile research plan.
- Split your research into short tasks: people will give you great feedback if it is quick, easy and involves low mental effort.
- Be human: leave your ‘research-ese’ dictionary at home and engage your mobile research participants with friendly, warm and natural language.