How to Judge Whether You Can Trust a Synthetic Respondent: 5 Questions to Ask Your Research Provider
By aytm
- article
- Advertising Testing
- Agile Quantitative Research
- Brand Tracking
- Conversational AI
- DIY Surveys
- Insight Communities
- Pricing Research
- Product Development Research
- Survey Research
- Survey Software
- Synthetic Data
Summarise with AI
This is the sixth and last article in aytm’s Innovation Intelligence series, each of which explores a different stage in the innovation research funnel. These articles summarise elements of the Product Innovation curriculum in the aytm Lighthouse Academy.
This field guide to synthetic respondents covers where a synthetic model’s quality comes from, the trade-off most pitches leave unspoken, and how to tell a number you can trust from one that only looks clean. It’s part of aytm’s ongoing work on data quality in the age of AI. For more on working confidently with synthetic data, explore the learning resources at the aytm Lighthouse Academy.
Explore the full suite of innovation courses here.
Almost every research platform is adding a synthetic layer, often as a toggle you can switch on inside a study you were already running. Venture money has flowed into synthetic research tools over the past year, and the shift is arriving whether or not any one team went looking for it.
But first, let’s define the term. A synthetic respondent is a model-generated stand-in for a real person: an AI trained on human data that produces survey answers, personas, or a whole simulated sample. Some tools use it to fill a quota gap, some to pre-test a questionnaire, some to stand in for fieldwork outright. The same question sits under all of them.
But that question isn’t whether to use synthetic respondents or not. Instead, it’s how to tell a good synthetic answer from a bad one while a decision still hangs on it.
Fidelity Is The Whole Point
A synthetic respondent is worth about as much as its fidelity, meaning how faithfully the model reproduces a real population: its averages, its correlations, and its edges. Generic models tend to do this poorly. In one study presented at the 2024 Quantitative UX Conference, Paxton and Yang compared model-simulated attitudes against 500 real respondents. They found the synthetic answers captured only about 1% of the variance in real responses, a median correlation near r = 0.10. The tools that clear that bar share a method: their builders fine-tuned them on very large volumes of real respondent data.
So fidelity has a source. It rises with the depth and quality of the real data underneath the model, and the architecture matters less than most pitches imply. Variance is the part that counts, because the correlations and the surprises are usually why you ran the study at all. A model that flattens them has removed the finding you were looking for. Evaluate a synthetic tool, then, and what you’re really evaluating is the data it learned from.
Fidelity and Privacy are the Same Dial
Here’s the tradeoff most pitches skip. Fidelity and privacy sit on one dial, and turning up either turns down the other. A model earns fidelity by reproducing the structure of what it trained on, and pushed far enough, it can reconstruct the very people it learned from.
This deserves care, because responsible market research already gates personal information at the panel level, and the data that reaches a client is de-identified. The synthetic risk is a subtler one. A model tuned for high fidelity can memorize the pattern of a small, rare segment so precisely that the combination of attributes points back to real individuals, even when no name is attached.
The exposure is re-identification, and it can happen even when no raw PII ever leaves the panel. The standard protections against it include aggregation, added noise, and differential privacy, and they all work by blurring detail. They blur it most where the data was thinnest to begin with: the low-incidence, niche segments.
Every provider sits somewhere on that dial. Where they sit is their working position on quality and privacy at once, and it’s usually invisible from the outside, because the training data is rarely public. You’re asked to trust a tradeoff you can’t inspect.
A Confident Claim About a Hard-to-Reach Audience Deserves the Hardest Look
A crisp, high-resolution answer about a rare population should invite scrutiny rather than confidence. A high-fidelity read on a niche audience is close to a contradiction. Either there weren’t enough real people beneath it, and the detail is manufactured, or there were, and a small identifiable group’s privacy paid for the resolution.
Take a genuinely hard audience, say stage-four cancer patients. Reaching them at any real scale in an online quantitative study is difficult without a proxy, and much of the credible work with groups like this happens in specialized qualitative or clinical settings rather than a standing panel.
So when a synthetic tool hands back clean quantitative data on them, the question worth asking is what real sample sits beneath it. A model can only be as faithful as the data it actually collected. Researchers have long known the older form of this failure: take a finding from a thin cell and stretch it over a whole population. What’s new is that a model can do it automatically and out of view, so the overreach never announces itself.
Why a Precise Answer Can Still Mislead You
Synthetic output tends to look trustworthy, and it’s worth understanding why. Real research runs on estimates, and every honest estimate carries a confidence interval that signals how sure you should be. Synthetic data often erases that signal. It comes back smooth, with no visible sampling noise and tight intervals, so it can read as more precise than real data while being less accurate.
Precision is how tightly answers cluster. Accuracy is whether they land on the truth. Synthetic output can give you the first while quietly costing you the second, which is how a team ends up confident and wrong. For early exploration, that’s a fair trade, and synthetic earns a real place there.
For pricing, segmentation, brand tracking, or a launch call, you need grounded truth, and accuracy is exactly what the output won’t show you. Two clean estimates, one synthetic and one from verified humans, look identical on a slide. What separates them is provenance, the traceable record of how the number was made.
Two things help here. An experienced researcher can often smell a too-smooth result, the answer that fits a little too neatly to be real. And a norm is settling across the field: work that drives a direct decision stays human, while synthetic does its useful work earlier, in exploration.
The Field Doesn’t Have a Language for This Yet
Panel research spent years building a vocabulary for data quality: removal rates, attention measures, and fraud signals, the kind of evidence you can put on a report and compare across studies. For synthetic respondents, that vocabulary barely exists. There’s no shared definition of what good means, no agreed measure, and no comparable score. It’s hard to hold a provider to a standard the field hasn’t written, and that gap is part of why the tradeoff stays so easy to miss.
The timing sharpens it, because AI is raising the stakes on both sides of the same market at once. It makes synthetic tempting, and it also makes fraud cheaper and faster. Greenbook and Rep Data’s State of Survey Fraud 2025 audited 4.1 billion survey attempts, and roughly a third were fraudulent, about a quarter inattentive, and close to 70% of the bad data slipped past standard cleaning, while hyperactive respondents in blocked traffic tripled over a single summer.
When both the promise and the threat accelerate together, you can’t sort good data from bad by looking at the charts, because clean and contaminated data both render tidy. What’s left to judge by is whether a provider will show their work.
Five Questions to Put to Any Provider
You don’t need the field’s missing vocabulary to interrogate a synthetic offering today. Five questions get you most of the way there, and they work on any provider, including the platform you already use:
1. Can they trace how the number was made, study over study, on a basis you can compare?
Provenance is what lets you defend the number when it reaches the boardroom.
2. Where did the real data underneath this model come from, and how much of it is there?
Fidelity is inherited from that foundation, so this is the question behind all the others.
3. For a niche segment, how many real respondents actually stood behind this answer?
A confident claim on a rare population should come with a real base, or an honest flag that it doesn’t.
4. Where does this provider sit on the fidelity-privacy dial, and can they say so plainly?
A vendor who can name their position has thought about the tradeoff. One who can’t hasn’t.
5. Does the output carry a confidence interval, or does it only come back smooth?
Erased uncertainty is a warning sign, not a feature.
The Shift This Comes Down To
The synthetic market will keep growing, the technology will keep improving, and none of that changes the structural truth underneath it. Every layer of AI-augmented research inherits the quality of the real data that trains, validates, and grounds it. Deep real data raises the ceiling for everything built above it. But shallow real data caps it, however advanced the model.
At aytm, we hold ourselves to the same questions above, and our answer to the first one is a verified panel of 100M+ respondents across 50+ countries, maintained under ISO 20252, ISO 27001, and ISO 42001, the first responsible-AI certification of its kind in market research.
We put that on the table because the whole point is that you should be able to ask any provider, us included, to show their work. Judge the synthetic layer by the real data beneath it, ask where the resolution came from, and treat provenance as part of the deliverable. Do that, and you’ll be able to tell a number you can trust from one that only looks clean, before the decision, not after.
For the full picture, read aytm’s whitepaper, Real data is the best AI asset, on the real-data foundation every synthetic layer inherits.
Explore the Product Innovation curriculum and aytm’s behavioural-validation methods at the aytm Lighthouse Academy.
