Diary studies using WhatsApp

When you need to capture ongoing evidence of product usage with users the standard approach is to use a diary study. Users are tasked to diarise their experiences, whilst the researcher prompts and probes users to reveal insights into behaviour.

I don’t think they are used quite as often as they could be – I have only been asked to use them a handful of times. The perception is that they take a long time to set up, are difficult or time-consuming to manage, the research participants can be unreliable and they often yield insights that are not actionable.

I’ve recently used WhatsApp very successfully within the private beta phase of product development to capture those vital first, second and third-time usage experiences of an app. I didn’t experience any of the problems outlined above – in fact I came away thinking that WhatsApp is a great platform for diary studies. Here’s how I set it up:

1, I used a recruitment agency to find six users of the app – this was completely painless: Penetration of the app was around 40%. These users were already using WhatsApp and so no onboarding was required. The recruitment agency was responsible for incentives.

2, I arranged to meet the users one at a time. I carried out an interview with them to capture current attitudes, behaviour etc. and then shared the login with them via WhatsApp and observed / recorded the vital first-time usage experience.

3, In correspondence, I avoided email completely. ‘WhatsApp Web’ is the desktop platform for Windows and Mac that makes this job really straightforward.

4, It’s worthwhile noting that the client was using Crashlytics to monitor app performance during the test – my job was to capture end to end user experiences.

5, The users were given simple, bulleted instructions to feedback at each point in the journey on the days they used the app. I found that users often ignored the instructions and I had to remind them to cover all of the four points I was interested in, however, a simple nudge and emoji to keep the tonality friendly worked perfectly.

6, The app has a high usage frequency and so we only needed to run the study for a week to obtain 18 usage examples. Users fed back using photos, screen shots, written notes and video. I considered enabling users to record their screen to share with but decided it wasn’t necessary.

7, At the end of the week I ran a focus group with the 6 users. I used the screenshots, video and photos that had been sent to me as stimulus for discussion. They were naturally inquisitive to share experiences and this led to a really interesting discussion. It ran over, but the users were responsible for that, not me!

8, Afterwards I created a WhatsApp group for them to stay in touch. Unexpectedly, the users were really keen to be involved in the future iterations of the app – something I didn’t anticipate, but that obviously reflected well on the research experience!

Overall, I was able to bring together the videos of the first-time user experience, the written and visual feedback from the diaries, the Crashlytics data and the output of the focus group discussion. The research study took just under three weeks from recruitment to report delivery and was significantly less expensive to run than anticipated.

The insights confirmed initial fears regarding bugs in the app, as well as revealing some unexpected user behaviour. As a result of the research the client widened the scope for the app, affecting plans for future iterations, as well as influencing the marketing and support functions within the organisation.

If you’d like to use a similar approach and don’t have an experienced user researcher within your organisation, or if you have any other research requirements, do get in touch with me via the contact page.