Research is a systematic way to collect and synthesize information about a topic. By the end of data collection we want as much data as possible and by the end of synthesis, as little as possible. You must find the haystack first if you want to find the needle in it. The quality of the research process lies in the quality of the transition between unfiltered raw data and the final insights. In this process, researchers need to find a good trade-off between data validity and effort spent. There are also some systematic biases that chew away big chunks of data validity. A big dataset of user thoughts can only be processed one by one. As we have a tendency to avoid such tedious tasks, our attention drifts away along with data validity. In this article, we share some of our best practices at Pine that help us sweeten this tradeoff and multiply the value of qualitative data analysis.
In UX and product design context, the topic of research is usually on the human side of the product: how, why and who will use it. Research questions and hypotheses are formulated around these ‘W’-s. Then we choose a data collection methodology that fits these questions and the project budget. In an example research project for our in-house app Like Locals, we started with the question “What are future travelers interested in when they prepare for a short trip?”. We decided to sample questions asked in facebook groups like “Travel addicts” as data.
In UX research, we usually use qualitative methodologies for data collection. This is just one fancy way of saying we work with non-numerical data and don’t have the luxury of statistical modelling to make an exact synthesis of the data (UX researchers are very VERY happy if they get their hands on a relevant quantitative dataset). Traces of user needs in a facebook post like “What scams should I be aware of when I visit Barcelona?” are very hard to quantify but are certainly there. Qualitative data always makes very clear sense as long as we see only a few samples of it. Now, take 200 similar posts. Our brains can’t take in so much information. We have to filter noise and boil down correlations by making a series of subjective decisions on what to keep and what to lose.
Data synthesis and categorization
Our brains have a very similar functionality. The raw data that endlessly comes in through our ears and eyes. This raw data must be filtered and processed if we want to make meaningful observations about the world. Our brain solves this issue by abstracting and categorizing data. The seemingly independent data points of seeing “two men shaking hands” and seeing “both of them wearing suits” makes sense to me under the category of a “business deal”. Bam! Combining these two data points, my brain saved half of its brain fuel (and opened a wide door to a range of cognitive biases and stereotypes). From this point my brain pushes countless samples of raw data under one rock labelled “business deal” and doesn’t pay attention to details in the raw data anymore.
Let’s see an example. Below are 6 posts from future travelers. Try to sort them in categories!
- Q1: Ever been to Barcelona? What are your thoughts? ⭐️
- Q2: I covered Barcelona a lot lately. However how expensive is Barcelona?
- Q3: Now going to Budapest Hungary 🇭🇺 for 2 days, Any suggestions for must-see places & must-try foods??
- Q4: Michelin Starred Onyx in Budapest surprised us with great hospitality and food! 🌍🥄
- Q5: Any recommendations on where to go in London ( what to eat ) I don’t want to miss anything!!
- Q6: (…) then Mykonos and Santorini! Looking for recommendations for other islands, excursions, food, places, etc. cheap, please!
There are multiple ways to sort these comments into meaningful categories. Here are two solutions:
Affinity diagrams everywhere
Both of the categorizations make sense – we will have to see what the rest of the data say about them. One of them might fit the original research question better, but you can never know which will be more insightful for the client’s day-to-day work.
Categories like these are usually built with a technique called affinity diagraming. The insights – in this case, the Facebook posts – are written on post its and grouped on a large flat surface by the designers. Affinity diagramming is a nice analogue task, a relaxing way for designers to stop staring at their computer screen for a few hours. However, in most affinity diagramming workshops there is a large data corruption agent at work, chewing away from the quality of our data and halving the value of the outcomes: our own laziness.
There is one annoying step of affinity diagramming that people like to curtail as much as possible. Each insight somehow needs to end up on a post-it. If you want to group 200 insights, you need to write 200 post-its, right? My hand aches already.
Let’s not lose or corrupt any data!
Designers usually summarize each post-it instead of writing it down word-to-word. When I summarize the post-it for “Q1: Ever been to Barcelona? What are your thoughts? ⭐️” I will write something like this: “Thoughts on barc.?”. If Réka, our graphic designer wrote the post-it for that specific insight, she would probably summarize it with a big hand-drawn ⭐️. My interpretation is quite different from hers. We both make sense of the raw data in our own way and in the process, we arbitrarily filter out important aspects of it. Whichever interpretation ends up on the wall, we lose one or the other category of the tables above.
Another problem with writing post-its by hand is related to the effects of fatigue. Sorting is biased because we get more and more tired by writing post-its. As I get on with writing those 200 post-its the more I write the sooner I want to finish. A pressing feeling starts to grow in my stomach that throws me back to when I was six and had to finish a handwriting homework for Ildikó néni I really didn’t want to. Imagine if the same happened with a computer performing a correlation analysis. “Some of the data is promising but we are not quite sure, the computer lost its focus by the time it got to January”.
Let the automatisation begin!
Our solution? Printing post-its. You can easily stick 6 post-its to an A4 sheet and layout 6 insights for each of them in PowerPoint. Then all you need to do is hit print. Check out this tutorial on how to do these layouts and set up your very own post-it printing infrastructure!
However, laying out the PowerPoint slides is quite a heavy and again, boring task when you have to repeat it for 200 insights. To make this job easier, at Pine (in collaboration with Demola Budapest) we have developed an in-house tool to do these layouts for us. If we have a data sheet with the insights in one column we only need to lay out one slide and the algorithm will do the rest. All it takes is a few clicks and our dataset of insights is ready to print!
*Drop us an email if you are interested in this software. We haven’t released a public version…. Yet!
Affinity diagramming on steroids
When we really want to make the most out of affinity diagramming, we print the same post-its for each participant of the affinity diagramming workshop. Each participant then creates their own systems from the insights on their own. The result is several different categorization systems for the same data. Réka and I would surely find some overlapping categories – giving a sense of reinforcement to those categories and showing that they are not entirely subjective. But her brain would also find the hidden meaning in visual aspects of the data that I easily miss.
This kind of multidimensionality can only be achieved if the boring, fatigue-inducing elements of the workshop are omitted.
With this tool, we can print additional data on each post-it as well. Sometimes, for example, it’s important to see which interview participant each insight came from. We see categories that are filled with the thoughts of more than one interviewee as more generalizable. Post-its can also be displayed on the wall of the meeting room for days after the workshop. The “affinity wall” serves as a reminder of what we learnt and a war-room feeling to make us proud.
During the home-office situation during COVID-19, we experimented with Miro, which researchers use to copy-paste multiple rows from an excel sheet to generate similar post-its on a digital board. We experienced that the sheer quantity of digital post-its is a bit frightening, especially for clients. As the digital board lacks the tinkering element, participants get bored easier – but still, we found Miro a suitable alternative.
Working on the same data with multiple attendees helped us saturate the meaning of the collected data in a personalized and actionable way. Product and service design research, especially in crisis intervention scenarios, is all about producing actionable research results that help clients do their jobs better. In a research and consultation project with Brain Bar, a Budapest-based future festival, we worked side-by-side with the agenda manager, digital marketing specialist and community manager of the organizer team.
As a psychologist, I will never know what will catch the attention of a social media marketing specialist. During the workshop, he took notes of some phrases and buzzwords the target group was using during the interviews. The systems they designed from the raw data and the sparking conversations we had about them, guided us to understand what they considered actionable.
Research co-creation for us is helping our clients empathize with the end-users in a structured and deep way. Workshops often lack depth because team activities are too dominant. It might seem weird to spend the first half of the workshop in silence, however, this is necessary if we want to unlock all the value of the data. We think in very different ways, coming from our personality and educational background. This knowledge cannot be fully applied to the data during a fast-paced brainstorming situation. It also cannot be applied if 70% of the data is missing from the post-its.