Here is the trap almost every founder walks into. You ask a chatbot what your customers want, it gives you a tidy, confident answer, and you go build the thing. Then nobody shows up.
Angus Sewell (Angus the Nontechnical) tells a painful version of this. He built a free tool for property managers, got warm replies, and then watched it get used exactly zero times. His diagnosis is sharp: ask one agreeable model what people think and, as he puts it, “it mostly hands your own opinion back to you with better grammar.” His fix is a panel of eight AI personas deliberately cast to disagree, interviewed like a focus group so you hear objections instead of applause.
It’s a genuinely good trick, and I’d push it further and pull it back at the same time. Further: forcing personas to hold different objections is the only way synthetic research earns its keep, because it attacks your blind spots instead of confirming them. Back: the method still cannot save you from the thing that actually sank his tool.
Because look at what happened. Real humans said they loved it. Then their behaviour said no. Synthetic personas have the identical failure mode, only cheaper and at scale. Work out of UC San Diego and KU Leuven this year got AI-generated respondents to hit around 95% correlation with real survey data, which sounds like game over for the focus group. Except Kantar’s own testing found synthetic answers skew relentlessly positive and low in variance, and even the optimists say the same thing in the footnotes: validate against real people or don’t trust it.
So use the panel for what it’s good at. It’s a fast, near-free way to generate objections you’re too close to see, pressure-test your pitch, and decide what to go and ask a human. A traditional in-person focus group runs roughly $8,000 to $12,000 once you count the facility, the incentives and the moderator, so replacing the early guesswork with an AI panel is a real saving. Just don’t let it replace the one signal that counts, which is whether a live person parts with time or money.
The uncomfortable rule stands. Stated preference is marketing. Behaviour is truth. Angus’s flopped tool proves it better than any framework. Build the panel to sharpen your questions, then go and watch what people do when nobody is being polite.
Did you know: in Kantar’s tests, AI “respondents” leaned so consistently upbeat that the bias, not the insight, was the most reliable thing about them.