Pick a winner of this week’s AI benchmark race and you can have any answer you like, because every lab turned up with its own scorecard. Anthropic released Claude Fable 5.2, and on Anthropic’s own table it leads the field on FrontierSWE v2 with 69.9 percent, against 61.2 for Google’s Gemini 4 Carbon. Google, meanwhile, points at the Artificial Analysis Intelligence Index, where Gemini 4 Carbon sits top at 59. Both cannot be first, and the fact that both can claim it tells you most of what you need to know about AI benchmarks.
Everyone marks their own homework
The open secret of the model-launch cycle is that the headline numbers almost always come from the company doing the launching. A lab picks the benchmarks it wins, builds the chart around them, and leaves the losses off the slide. Fable 5.2 shows how fast the story changes under independent eyes: the testing outfit Vals AI ranks it first overall on its index at 71.31, yet places it second on SWE-bench Multimodal, behind Anthropic’s own older Opus 5.5. The ranking depends entirely on which exam you hold up to the light.
What a benchmark actually measures
A benchmark is a narrow test. It tells you how a model performs on a fixed set of questions, on a particular day, under conditions the examiner chose. It does not tell you how it copes with your messy codebase, your weird edge cases, or the request you phrase badly at 5pm on a Friday. The distance between “best on FrontierSWE v2” and “best for the thing you actually do” can be enormous, and no leaderboard closes it for you.
There is also the small matter of the numbers creeping upward every few weeks while the lived experience of using these tools improves far more slowly. Each launch promises a new frontier. Each one becomes, within a fortnight, the baseline the next launch beats. The benchmark is less a measure of capability than a marketing unit, denominated in percentage points.
What this means
Treat every vendor benchmark table as an advertisement until a neutral party repeats it, and even then treat the neutral party’s test as one data point rather than a verdict. For real work the only benchmark that counts is your own: run the two or three models you are weighing up against the tasks you genuinely need done, and skip the leaderboard theatre. On current independent numbers, Fable 5.2 and Gemini 4 Carbon are close enough that the honest answer to “who won” is “whoever wrote the press release you happened to read.”
Sources: llm-stats; Vals AI; Artificial Analysis; vendor benchmark tables, November 2026.