Here is a cheap trick every AI demo uses, and you have fallen for it. The presenter runs the agent once, it nails the task, the room nods, contracts get signed. Then it goes into production, runs the same job two hundred times a day, and fails often enough to become a part-time job for a human babysitter. The demo was not a lie. It was just measuring the wrong thing.
The number you actually want is called pass^k, and 2026 is the year it went from a benchmark-nerd curiosity to the metric serious teams gate their releases on. Philipp Schmid’s clear write-up and the AgentPatterns reference are the fastest way in.
Capability versus consistency
Two metrics, one letter of difference, wildly different meaning. Pass@k asks: across k attempts, did the agent succeed at least once? That measures capability, the ceiling, what the thing can do on a good day. Pass^k asks: across k attempts, did it succeed every single time? That measures consistency, the floor, what you can actually rely on.
The gap between them is where your reputation quietly leaks away. Take an agent with a 70% success rate on a single try. Run the pass@3 maths and it reads as roughly 97%, which looks like a finished product. Run pass^3, all three attempts must land, and the same agent scores about 34%. Same model, same task, same day. One number sells the deal, the other tells the truth.
It gets worse as you scale. Pass@k climbs towards 100% the more attempts you allow, because you only need one win. Pass^k falls, because every extra run is another chance to trip. For an agent doing a job hundreds of times a day from a clean slate each time, pass^k is the only figure that describes your real Tuesday.
Run the test yourself this week
You do not need a benchmarking lab. Pick one task your agent is supposed to own. Run it ten times from a fresh state, no memory of the previous run, no helpful nudges. Count how many times it fully succeeded, not partially, not nearly. Nine out of ten is your pass^10, and it is probably lower than your gut promised.
Then do the unglamorous bit. Log the failures and look for the pattern, because agents that lack memory of prior runs tend to repeat the exact same invalid tool call and the same malformed parameter, over and over. Fix that one recurring error and your consistency jumps more than any prompt tinkering will manage. Re-run the ten. Watch the floor rise.
The teams shipping agents people trust in 2026 are not the ones with the cleverest prompts. They are the ones who measure the floor, not the ceiling, and refuse to release until the floor is high enough to stand on.
Did you know: a 90% single-run success rate, which sounds excellent, collapses to about 35% once you demand ten clean runs in a row, which is why “it worked when I tried it” is the most expensive sentence in AI.
Related on Top Tool Stack: The EU AI Act rules that just kicked in · Testing your AI automations (n8n)