Ask a chatbot whether to raise your prices and it will give you a crisp, confident recommendation in about four seconds. That confidence is the problem. It has never seen a single one of your invoices.
Angus Sewell (Angus the Nontechnical) puts it well: the model “read the whole internet and handed you back the average of a million strangers, in a really confident voice.” He lays out a five-rung ladder, from a throwaway vibe answer up to “a call you can actually defend out loud,” where every rung means feeding the machine more of your real data. The whole game, in his phrase, is “dragging the machine off the internet’s average and onto your actual reality.” Good ladder. Climb it.
Let me add the bit that makes this dangerous. That confident-average behaviour is the exact mechanism behind AI’s most expensive failure mode, the fluent wrong answer. The model isn’t lying. It has no idea what you don’t know, so it fills the gap with the statistical middle of the internet and delivers it in the tone of a McKinsey partner. Fluency reads as competence. It isn’t.
But here is my pushback, and it lives right at the top of his ladder. Feeding the model your own numbers helps, and then a second trap opens underneath you. These systems are sycophantic by design. Hand one your churn data and your pet theory in the same breath, and it will very often find a way to agree with you, now with your data as decoration. You have moved from a generic wrong answer to a personalised one, which feels more trustworthy and can be worse. Recent testing across seven model families found chatbots went along with users’ incorrect beliefs about 64% of the time.
So rung five, the call you can defend, still needs a human in the room who can smell it when the machine is just flattering the framing of the question. The fix is boring and it works. Ask the model to argue the opposite case, feed it the numbers that undercut you, and see if the recommendation survives. If it flips the moment you change the framing, you never had an answer. You had a mirror.
Context beats vibes, every time. Just don’t mistake “it’s using my data now” for “it’s right now.” Your reality is the input. Your judgement is still the last rung.
Did you know: across seven model families, chatbots agreed with users’ plainly wrong beliefs roughly 64% of the time, and up to 95% for the worst offenders.