Why a Single Word Can Flip a MaxDiff Result
By Conveo
- article
- AI
- Artificial Intelligence
- AI Agents
- AI Moderated Interviews
- Conversational AI
- Depth Interviews
- Qual-Quant Hybrid
- Qualitative Research
- Remote Qualitative Research
Summarise with AI
In one of the last MaxDiff studies I ran before joining Conveo, an option ranked at the top of the list. It looked like a clear winner. When the qual surfaced later, the reason became clear: respondents had assumed the option was free. It wasn’t. The “preference” I was about to recommend to a stakeholder wasn’t a preference at all. It was a misunderstanding.
Every researcher who runs MaxDiff has had a version of this. You have the ranking, but not the reasoning behind it. Without the reasoning, you can’t tell whether a top result is genuine or a misinterpretation, or whether a low-ranking idea is genuinely unloved or just badly worded.
You either trust the result, dismiss it, or carry the uncertainty into the debrief. The difference between a real preference signal and a false one can come down to a single word, and traditional MaxDiff has never given us a way to find out.
I’ve run MaxDiff studies throughout my career, at Sky, Starbucks, and Instacart, and that gap bothered me on every project. So when I joined Conveo as Insights Lead and had the chance to shape the product roadmap, this was the first thing I brought to the team.
This article explores my experience working as a researcher in a tech business, collaborating on the development of Conveo MaxDiff. You can learn more about a solution here.
Where the Idea Started
The idea came to me at a pub on Bainbridge Island, Washington. I was still working at Instacart, talking through product ideas with a colleague who works in research at Microsoft. Somewhere in the conversation, the question I’d been carrying for years finally had a possible answer: what if you could combine MaxDiff’s structured preference ranking with the kind of dynamic, human conversation that explains it?
My first version of the idea was different from what we ended up building. I thought: what if respondents could just say their MaxDiff selections out loud? The more I turned it over, the more I realised it was a misstep. The moment you ask someone to vocalise a preference, you change the nature of the exercise. So I shelved it.
What I landed on instead was probing after the selection. Let people complete the MaxDiff the way they always would, then bring in AI to have a real conversation about why. That’s a fundamentally different proposition from an open-ended survey question bolted on at the end. It’s a dynamic conversation that knows what the respondent just said and can go somewhere useful with it.
Why the Reasoning Matters
Very large MaxDiffs are often used to filter ideas. You take the top five for deep qual; the rest get dismissed without anyone understanding why they ranked where they did.
But sometimes there’s a great idea hidden in a low-ranking idea where the wording was just wrong. There’s something worth pursuing, but the description didn’t land. With a quant-only MaxDiff, you’d never know.
Probing after the MaxDiff surfaces that reasoning directly, in the same session and from the same participants. No second study, no follow-up fieldwork. You walk into the debrief with the ranking and the why behind it.
Working With Engineering
Once the idea was clear to me, the next step was making it clear to the engineering team. Engineers at a research tech company haven’t typically run MaxDiff studies, so I started by explaining why it exists, how it’s used, and why it matters to the people who commission this kind of work.
Rik, the engineer who took the lead on the project, went far deeper into the methodology than I had. He has a statistics background, and once he understood the application, he started working through the algorithm in detail. Now, he knows more about how MaxDiff works beneath the surface than I do.
One of the most important decisions in building this product was when in the process to probe. Too early and you interrupt the flow of the exercise. Too late and the connection to individual choices is lost. We went back and forth on this, testing different approaches.
Rik came up with what we landed on: probing at the top selection, the bottom selection, and somewhere in the middle. Simple in concept, but it took real iteration to get there.
One thing about how we work here shaped this product: engineers come into client calls directly. Not as observers. They’re in the conversation, because our clients are often technical too. Rik wasn’t building from a brief; he was hearing from the people the research was for.
What This Taught Me
When we launched, I heard from researchers at Harvard, Microsoft, and other organisations who recognised immediately what the product was for. As a researcher, that was pretty validating. I love research. I’ve always thought of myself as a bit of an innovator, but actually building something new and watching it land has been the most fun job I’ve ever had.
