I was a little suspicious last time, but even moreso this time. The slant their scores take, and the release timing is very suspect. On the other hand, new models drop on a near daily basis, so maybe there's no way to appear entirely fair.
An issue I think all benchmarks suffer from is regarding precision. What does a one point difference mean? How many points of difference is needed to translate into a meaningful distinction between models?
Among fellow humans, we often find ourselves picking what we consider to be most intelligent and then designing the tests around that. I've noticed a similar thing in AI, where whatever OAI/Anthropic decide to focus on improving becomes the new measure of intelligence. With their latest models, this seems to be 3d modeling/game making capabilities. Unsurprisingly, because that isn't what the Chinese companies have been focusing on, they appear behind on "frontier" reasoning tests. On top of that, older tests end up being discarded for being saturated. Makes me think that maybe unintentionally, the methodology used for these scores might be biased towards Western companies.
This would also fit with the widely discussed decline in writing quality from Opus and the worse readability of code produced by one of OAI's recent models, despite benchmarks continuing to show progress on all fronts.
Plus, if they're going to update the benchmark so frequently to discard saturated components, is there even a research-related point to keeping it private?
peri-cl
The writeup fails to disclose that they quietly redacted several open-weight models from the new benchmark. Popular ones. They're not on the charts; they're not available to manually add to the charts; and their individual model pages now display scores of "N/A".
Their previous updates have sometimes been really bad for Chinese models. Either by exclusion, delays or simply weighting things differently so OpenAI looks better.
This feels like the opposite of benchmaxing. Maybe "minbenching" can become a word now?
simonjgreen
This might be the first time I’ve favourited a post without upvoting it.
This is really sus. I’d love a more detailed explanation and “take” from AA on what problem they felt needed correcting and why things have shuffled the way they have. And how they can claim their index has relevance if positions can be switched around so easily.
dannyw
From my rough recollection, all 3 of the past updates all had the effects of decreasing the score of open weight models.
One was rushed and same-day updated as GLM 5.3 (iirc) got added; of course its main change to rankings was lower GLM scores.
tirpen
I think refer to Qwen 3.8 Max.
It was released, got the top agantic ranking, and AA immediately made an update to their index calculation that made it lose the top spot. It was very suspicious.
Such releases often show weaknesses if the intelligence index, because the models that are ranked jointly do not have the same percieved level of quality.
singularity2001
So instead of the narrative of benchmaxing, in truth it's benchminimizing of competitors by the U.S. oligarchy?
yorwba
Their problem is really that individual benchmarks are not that informative. Bad enough models all perform at chance level, good enough models all saturate the benchmark. And the number of tasks is often low enough that differences of several percentage points might just be noise. Artificial Analysis' approach of aggregating scores in a weighted sum is largely just choosing which noise to amplify more.
I prefer Epoch's https://epoch.ai/eci methodology as they explicitly model benchmark saturation and also provide error bars for their estimates.
irthomasthomas
I noticed last night that openai had fallen way down the index, and I allowed myself to think for a minute that maybe AA aren't as biased as I thought. Then a few hours later they push this fix which sees openai rise again.
This update is even worse than usual because it now includes two private sources, data which we can't reproduce if we wanted to do.
Also note that Anthropic is the only AI company actually caught red handed cheating at evals. Teaching their models to memorize test answers and lie about it, using their hidden reasoning tokens to hide the deception [0]. Yet we see no comment from AA about this. No desire to figure out which tests are tainted. Hopefully Trump bans this fake intelligence index.
Devil's advocate: Their scoring system is itself a meta-benchmark.
What happens when you release a new benchmark? Models magically start scoring better against it.
How do you overcome this? Release a new benchmark and start the process over again!
All other things being equal, they should retarget their prices if models start gaming the process to top their index.
Has this happened? Maybe. Is their new scoring system biased towards the American frontier models? Quite possibly.
I have my suspicions, but they have led me astray before. I will continue to be cautiously cynical, though.
sourcecodeplz
small (700b) open-weights models are great at coding and tool calling but lack world knowledge. because the parameter difference is so huge T to B, of course they also are not as intelligent.
this is i think the difference in this new analysis.
doesnt mean deepseek is bad or useless.
ozgung
Fun fact: By changing the weights of the input you can change the output as you want.
chvid
"The FrontierCode verifier grades each attempt with the task's tests and rubric. We use GPT-6 Luna (medium) as the LLM judge"
Why are they using a closed source LLM as a judge? Closed source providers can see your traffic and adjust as they see fit.
I think it's pretty evident that Artificial Analysis loves to change the benchmark to place US models on the top, it's no the first time either.
The current placement of many models simply doesn't makes sense, GLM 5.3 is so much better than Astra at backend tasks, Astra does the most weird nonsensical bugs.
This is what happens when the benchmark team is funded by VC capital.
ctolsen
The thing that makes the least sense is that Haiku 5.5 is apparently better than every single open weight model in existence except Kimi K3.1. Which is just... not true. I feel like I don't even have to explain it to anyone who has touched both sides of that equation for more than three minutes.
dannyw
Even and especially for frontend UI, GLM 5.3 is a whole world better than Astra.
I can see (and believe) Astra beating GLM 5.3 on verifiable hard domains like maths, etc; but we’re at this stage where intelligence scores and even ‘frontier real world unsaturated benchmarks’ really doesn’t reflect real world use anymore.
lousken
It is embarrassing how much they suck up to american labs while those models only perform the first few days after release. With open weights you can be sure the quality stays the same unlike the closed source crap. How about a retest after some time?
pritambaral
> With open weights you can be sure the quality stays the same
Or improves. With third-party fine-tuning.
tsss
Performance depends so much on configuration parameters like KV cache that you have essentially no idea what you're getting unless you run and benchmark everything yourself, which I'm sure you do not do. If anything, hosters of open weight models are more likely to gimp it either due to incompetence or malice because they have no reputation to lose.
dannyw
Reputable hosting providers, e.g. Fireworks, Cloudflare, Baseten, whatever absolutely _do_ have a reputation to lose.
kamov
The natural language choice in multilingual index seems pretty strange and biased to me, they tried to encompass a broad swath of Asian languages, but then there's like four European romance languages. They didn't include any Slavic one i.e Russian, or Nordic one like Swedish, or Finno-Ugric. And these romance ones are probably overrepresented in the training data anyways
Havoc
> GLM-5.3 (max) score highest at 32, and Claude Haiku 5.5 (max) outscores all of them
That seems improbable
koruse
Well either AA is biased bribed riged whatever or chinese models are benchmaxxed or maybe its both
wyrdcurt
Tinfoil hat time: but the way they waited a couple weeks to evaluate Mimo 2.6 Flash, allowing GPT-6 Luna to sit on the Pareto frontier in the meantime, really did make me think they might have some bias going on... (Luna was displaced when they finally evaluated Mimo 2.6 Flash, which had been released the day before Luna)
edit to add: there totally is benchmaxxing going on, but I'd be really surprised if it wasn't across the board. After all, how can any lab tell if their model has improved over a previous iteration, if not by comparing benchmark performance?
lisplist
I’ve very mixed feelings on Mimo. The new models do surprisingly well on some tasks, but they don’t allow you to set the thinking level and the model often times spends next to zero time thinking about a problem. If you coax it just a little bit, you can get it to solve it, but this isn’t an issue with other models. Also it’s incredibly easy to get it to get caught in a thinking loop.
wyrdcurt
I've personally experienced overthinking and the thinking loop issue more often than seeing it not think enough, I know what you mean though. I tried Haiku 5.5 and the most impressive thing about it to me was that it seemed not to overthink when it wasn't necessary, yet it somehow knew when it should spend some more time reasoning about a task.
American models are too rich for my blood though. I use GLM 5.3 Flash more than any other model and it has the same issues as Mimo. Deepseek 4.1 Flash is turning out to be a sweet spot for me; it's cheap and blazing fast on CoralBricks where I've been using it, so I don't mind when it overthinks (I try to pay attention to what my agents are doing, and it tends to finish tasks before I can skim half of its reasoning)
rbehrends
Neither is the primary cause, I suspect. If you look at the coding index alone, you see more plausible results from a software engineering perspective (in particular, Terra no longer outperforming Fable) [1].
That especially the small flash models that are primarily optimized for coding tasks would perform worse at non-coding tasks is not per se surprising to me. The introduction of Terminal-Bench Science 0.1 in particular hurts their aggregate scores [2].
The general question remains of how meaningful aggregating the results of very heterogeneous tasks into a single numeric score is. (Aside from general questions about individual benchmark validity.)
Note: the cyber index is also interesting. While refusals make it somewhat opaque, even if you optimistically count all refusals for Opus and Fable as successes, the results are ... fascinating [3].
I'll add that I really want to know how Opus 4.x and Codex 5.x perform on the new index. Once we have that information, this is going to give us a better understanding of how meaningful the scores are.
It is hard to tell what people are trying to say when they use undefined words with no particular meaning. This doesn't seem to change the relative position of the Chinese models much. The Chinese are a month or few behind and the state of intelligence is so volatile that the top 10 will be completely different in 6 months. I'd expect it to still be a mix of US and Chinese models with the US models leading.
Gareth321
Benchmaxing is defined as targeted post-training attempts to optimise for higher benchmarks. This doesn’t change the aggregate intelligence or capability of the model, so it can make a model appear more capable than it is.
ozgung
I’m pretty sure “benchmaxxing” as defined is a standard post-training procedure for ALL models.
One can argue though some models are post-trained heavily for specific benchmarks and less for other datasets. So they are weaker on out of distribution tasks.
XenophileJKO
So I assume it depends on what you train.
- Exact problems on benchmark.
- Similar problems to benchmark.
- Broad class of problems or techniques that might apply to benchmark
Each has very different ramifications for how the model performs. The current evidence is that Anthropic and OpenAI is generally operating lower in the list.
piterrro
I'm still using DS-flash - it beats frontier model on speed and cost keeping good enough code quality.
It depends on the harness and constraints you give to the model but for most of the tasks its a great fit, I have not switched from DS since couple of weeks already (previously I would switch to Opus for some heavier work but I stopped recently).
My harness consists of multi-level AGENTS.md and SOUL.md containing "grug brain developer" and "asd-ste100" type of speech for clearer communication.
ilumanty
For me, DS Flash quality is still insufficient for bigger / more complex tasks. DS has gotten better by a lot, but I prefer Opus 5.5 because it gets things right with much fewer tokens. DS also sometimes gets stuck in a thinking loop "hmm... hmmm... hmmmm hmm hmmmm..." which is hilarious when you check on it after 30 minutes
piterrro
Totally makes sense, for me it was just that I observed its doing good work so that I never really need anything stronger.
btw. I'm not reviewing code in detail anymore. I only give direction and provide architecture design vision, its the model and couple of async agents working in the background and consuming work resulting in PRs which I manually approve after an eye-glance.
XenophileJKO
Are you doing a lot of specification up front?
The reason I personally like Opus 5.5 is that after using it quite a bit, I can just say something like: "Please take care of this compliance ticket." The model handles research, record keeping and code changes. Consistently and correctly.
I just don't have the level of trust in a lower end model. Even Opus 5.5 isn't quite ready for "driverless" in some architectural situations, usually around change management in the middle of implementing a system.
I was just kind of surprised to hear you were fairly hands off with a lower model like DS.
piterrro
I always start with a plan - this is the time where the model can leverage existing AGENTS.md scattered across every level of the code, take a look at test cases and written product documentation commited to repo. Once the "design doc" looks ok, I ask for implementation as a PR - when its ready, all CI checks should be green and I get a summary of what was done and whether anything was missed.
I can ask for a review from another agent (they often find NITs) if I'm hesitating.
mirekrusin
Open-weight models
Kimi K3.1 Pro: −9
DeepSeek V4.1 Flash 1017: −13
GLM-5.3: −13
MiMo-V2.6-Pro: −16
Proprietary models
GPT-6 Astra: +4
GPT-6.1 Sol: +3
GPT-6.1 Terra: +3
Claude Opus 5.5: 0
Claude Sonnet 5.5: −2
Claude Fable 5.1: −3
Gemini 4 Argon (high): −3
Muse Wonder Full: −6
Claude Haiku 5.5: −7
Muse Wonder: −9
Gemini 3.8 Flash (high): −10
Grok 4.7 (high): −11
Is there affiliation with OpenAI or coincidence?
susiamongi
Yeah, this smells. Ain't no way Terra went up and Deepseek so far low. I have had the later beat the former on multiple criteria by simply being less distracted.
There was a thread here about families of models responding in the same patterns to specific queries like "say a random word", and it takes less than two minutes to verify it is true. It'll take a bit of effort to not convince me this change wasn't reverse-benchmaxxed to screw everyone other than OpenAI. I'll be less distrustful after everyone is done IPO-ing.
moostii
My experience is not consistent with yours.
adg001
The eloquence of numbers is pitiless; thanks for summing them up. There are players filing for IPOs who don't want to lose appeal while contrasted to open-weight counterparts, which can be run on prems, or accessed as a service offered for a bargain.
nolok
More to the point, I found fable to be actually more "intelligent" than opus once you step completly outside of the technical side of things (and I don't only mean code here), so for me it's hard to judge.
Eg asking "here are all the contract I'm subscribed to, here is what happened, is there any insurance that covers me and if yes which one is the best to make the claim to and why, and provide all the article to prove why and how i'm not in the exclusion list". Try it with random stuff of your every day life. Opus is both great and not great.
joennlae
Interesting that changes in the index always happen when „non us big tech“ models seem to be a little to good for comfort.
starfallg
It makes sense for Muse Wonder though as that model does not trade blows with the best frontier models by a longshot.
joennlae
agreed. It is not easy to benchmark these models. It just the date of index changes have some „spurious“ correlations
chvid
We need an AI benchmark made in neutral Switzerland.
alexaholic
Glancing over the methodology[1] made me curious about three things:
1. Is there a benchmark measuring how much better an LLM performs when you tell it "you are an X"?
2. Does telling an AI agent "you are an AI agent" actually help? Is it the LLM equivalent of reminding a senior developer they're a senior developer, making them feel guilty about being lazy and finally getting them to do their job?
3. What happens when the persona conflicts with the task? If you tell it "you are a PM" and then ask it to write code, does it struggle? Produce worse code? Spawn a sub-agent and tell it they're a senior developer?
I find it fascinating that some of the main areas people are interested in are using benchmarks with quite few questions: ~200 for coding and ~150 for science.
As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.
I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.
6gvONxR4sf7o
It's wild seeing opus 5.5 up there so high, while i'm still using Fable 5.1 because I couldn't stand how opus writes.
danielbln
Opus 5 writes like its off it's meds, opus 5.5 is a lot less weird that way.
theanonymousone
Where is GLM 5.3 Flash?
eliaskg
Lots of open models are missing
rienbdj
Why did they call it “humanities last exam”?
Will an LLM topping this bench be the end of humanity?
The doomer rhetoric has no place in what should be an objective measure.
chvid
Artificial Analysis is a 40 person company working out of Silicon Valley; who is funding them? How do they make money?
HarHarVeryFunny
Yeah - follow the money.
Hard to trust that a for-profit benchmarking company isn't biased due to profit-maxxing.
mindwok
Interesting to see the open weight models drop so drastically from the changes here.
They perform pretty badly on the new benchmarks introduced.
himata4113
This seems to be a hit and a miss, there's no visual indication that opus 5.5 is better even though in 9 tests out of 10 it will outperform astra especially when it comes to niche domains and will absolutely destroy them in anything requiring raw intelligence rather than a naive search, naive search is what makes openai models extremely capable in scientific fields though.
The score hit on smaller models is a welcome change especially when compared to models that nearly 10 times larger.
Grok being punished quite a lot is also a very welcome change as the model is frankly not that good.
I would love to see few more points knocked off haiku by introducing steerability/creativity benchmarks.
nelaggy
chinese labs are probably benchmaxxing but i'd like to see when the index changes relative to the position of the best chinese models (last suspiciously-timed update was when glm-5.3-flash was looking really strong)
anakaiti
The drop for K3.1 Pro & MiMo V2.6 Pro is quite surprising. Are they benchmaxxed?
adg001
Or, are we witnessing reverse-benchmaxxing? Meaning that first some entities let players optimise for a given benchmark, then they update the measurement methodology so as to retain some players on the top ranks, while dropping others. As former Italian prime minister, and highly controversial public figure, Giulio Andreotti once remarked "Thinking the worst is a sin, but you’re often right."
gobdovan
Did Italy ever had a PM that was not controversial?
adg001
You are neither funny, nor well informed.
gobdovan
Italy has a pretty rich history of controversial PMs. Pre-WW2 Crispi, literally Musolini, recents Berlusconi, Andreotti, Craxi. Don't think I know any other foreign country by how controversial so many of their leaders were.
prodigycorp
Transparently so. MiMo's training dashboard showed the DeepSWE hillclimb in real time.
throwa356262
That's not benchmaxing.
You need a method to show you RL is improving the model that is separate from the training data.
simianwords
I keep seeing that MiMo is in the frontier in terms of performance and cost but I see very few talking about it and the only ones that do talk about it are doing it in context of AA
brainwad
It's also dog slow at AA index tasks compared to other models in its class. Makes me think they optimised for getting on the intelligence/cost frontier at the cost of real world usability.
simianwords
I’d bet on this
wyre
I used MiMo for a couple of days waiting for my Codex sub to reset. I really liked it, its communication style was a breath of fresh air compared to how stale gpt-sol is.
bigbaguette
Such a big re-ordering.
The same week Anthropic files its S-1. Right...
kfjgi84h
This thread is the real benchmark.
liquicity
Honestly it does somewhat match what I’ve felt playing a lot with all OpenAI / Claude / deepseek models
One thing I’m still surprised / unsure on is haiku 5.5 max besting latest deepseek - but also I think the last haiku was so incredibly bad I still hold some bias whilst using the new one.
thereitgoes456
(EDIT: A child comment has refuted the AA score comments, AA runs using their own harness.)
One of the new benchmarks is FrontierCode 2.0 by Cognition. AA seems to have accepted Cognition's benchmark scores without questioning, but they should have looked more closely.
On this benchmark, Anthropic's models are run using Claude Code; OpenAI's using Codex, and Gemini is run using... Cognition's own Devin (not Antigravity). Thus its score is much lower than it should be, knocking ~3 points off its AA score. All open-source models are also inexplicably run using Devin, damaging their scores as well.
This may even be intentional, if Cognition has beef with Google, since Antigravity was built using Windsurf's technology. But aside from that, it is to their advantage to make Anthropic/OAI look like leaders in the AI market. They are a company known to lie and exaggerate again and again, so I have trouble giving them benefit of the doubt.
shelfchair
I don't think this is relevant on AA's FrontierCode score. AA runs FrontierCode 2.0 benchmark using their own Stirrup agent harness [1].
It's interesting that the Stirrup scores are somewhat close to the specialized harness scores. Either Stirrup is quite excellent or is it that harness doesn't matter so much after a point?
zurfer
What's the advantage for cognition?
I would think it's the opposite as a more competitive frontier for llms makes the labs shed some margins and enable a broader ecosystem.
thereitgoes456
Yes, that is true, you might be right. But I think such, because the moat in their wrapper layer feels very thin and does not make sense in a world full of 20+ wrappers around similar models. It feels better to "only" have to deal with Ant/OAI (and other pure wrapper cos). Competing vs. Google, Meta, and every Chinese lab on earth, every one using their own bespoke wrappers, is much scarier.
Additionally: competitive frontier = more commodified = high valuations under more scrutiny.
wyre
Comparing the scoring rubric for AAv5.0 vs AAv4.3:
An issue I think all benchmarks suffer from is regarding precision. What does a one point difference mean? How many points of difference is needed to translate into a meaningful distinction between models?
Among fellow humans, we often find ourselves picking what we consider to be most intelligent and then designing the tests around that. I've noticed a similar thing in AI, where whatever OAI/Anthropic decide to focus on improving becomes the new measure of intelligence. With their latest models, this seems to be 3d modeling/game making capabilities. Unsurprisingly, because that isn't what the Chinese companies have been focusing on, they appear behind on "frontier" reasoning tests. On top of that, older tests end up being discarded for being saturated. Makes me think that maybe unintentionally, the methodology used for these scores might be biased towards Western companies.
This would also fit with the widely discussed decline in writing quality from Opus and the worse readability of code produced by one of OAI's recent models, despite benchmarks continuing to show progress on all fronts.
Plus, if they're going to update the benchmark so frequently to discard saturated components, is there even a research-related point to keeping it private?
GLM 5.3-flash: N/A
https://artificialanalysis.ai/models/glm-5-3-flash/
Qwen 3.8-flash-next: N/A
https://artificialanalysis.ai/models/qwen3-8-flash-next
This feels like the opposite of benchmaxing. Maybe "minbenching" can become a word now?
This is really sus. I’d love a more detailed explanation and “take” from AA on what problem they felt needed correcting and why things have shuffled the way they have. And how they can claim their index has relevance if positions can be switched around so easily.
One was rushed and same-day updated as GLM 5.3 (iirc) got added; of course its main change to rankings was lower GLM scores.
It was released, got the top agantic ranking, and AA immediately made an update to their index calculation that made it lose the top spot. It was very suspicious.
https://news.ycombinator.com/item?id=49200652
https://news.ycombinator.com/item?id=49201726
I prefer Epoch's https://epoch.ai/eci methodology as they explicitly model benchmark saturation and also provide error bars for their estimates.
This update is even worse than usual because it now includes two private sources, data which we can't reproduce if we wanted to do.
Also note that Anthropic is the only AI company actually caught red handed cheating at evals. Teaching their models to memorize test answers and lie about it, using their hidden reasoning tokens to hide the deception [0]. Yet we see no comment from AA about this. No desire to figure out which tests are tainted. Hopefully Trump bans this fake intelligence index.
[0] https://arxiv.org/abs/2608.09867
https://thehill.com/homenews/administration/6175633-trump-ba...
What happens when you release a new benchmark? Models magically start scoring better against it.
How do you overcome this? Release a new benchmark and start the process over again!
All other things being equal, they should retarget their prices if models start gaming the process to top their index.
Has this happened? Maybe. Is their new scoring system biased towards the American frontier models? Quite possibly.
I have my suspicions, but they have led me astray before. I will continue to be cautiously cynical, though.
this is i think the difference in this new analysis.
doesnt mean deepseek is bad or useless.
Why are they using a closed source LLM as a judge? Closed source providers can see your traffic and adjust as they see fit.
The current placement of many models simply doesn't makes sense, GLM 5.3 is so much better than Astra at backend tasks, Astra does the most weird nonsensical bugs.
This is what happens when the benchmark team is funded by VC capital.
I can see (and believe) Astra beating GLM 5.3 on verifiable hard domains like maths, etc; but we’re at this stage where intelligence scores and even ‘frontier real world unsaturated benchmarks’ really doesn’t reflect real world use anymore.
Or improves. With third-party fine-tuning.
That seems improbable
edit to add: there totally is benchmaxxing going on, but I'd be really surprised if it wasn't across the board. After all, how can any lab tell if their model has improved over a previous iteration, if not by comparing benchmark performance?
American models are too rich for my blood though. I use GLM 5.3 Flash more than any other model and it has the same issues as Mimo. Deepseek 4.1 Flash is turning out to be a sweet spot for me; it's cheap and blazing fast on CoralBricks where I've been using it, so I don't mind when it overthinks (I try to pay attention to what my agents are doing, and it tends to finish tasks before I can skim half of its reasoning)
That especially the small flash models that are primarily optimized for coding tasks would perform worse at non-coding tasks is not per se surprising to me. The introduction of Terminal-Bench Science 0.1 in particular hurts their aggregate scores [2].
The general question remains of how meaningful aggregating the results of very heterogeneous tasks into a single numeric score is. (Aside from general questions about individual benchmark validity.)
Note: the cyber index is also interesting. While refusals make it somewhat opaque, even if you optimistically count all refusals for Opus and Fable as successes, the results are ... fascinating [3].
I'll add that I really want to know how Opus 4.x and Codex 5.x perform on the new index. Once we have that information, this is going to give us a better understanding of how meaningful the scores are.
[1] https://artificialanalysis.ai/#coding-agents-tabs
[2] https://artificialanalysis.ai/evaluations/terminal-bench-sci...
[3] https://artificialanalysis.ai/#cyber-index-tabs
It is hard to tell what people are trying to say when they use undefined words with no particular meaning. This doesn't seem to change the relative position of the Chinese models much. The Chinese are a month or few behind and the state of intelligence is so volatile that the top 10 will be completely different in 6 months. I'd expect it to still be a mix of US and Chinese models with the US models leading.
One can argue though some models are post-trained heavily for specific benchmarks and less for other datasets. So they are weaker on out of distribution tasks.
- Exact problems on benchmark.
- Similar problems to benchmark.
- Broad class of problems or techniques that might apply to benchmark
Each has very different ramifications for how the model performs. The current evidence is that Anthropic and OpenAI is generally operating lower in the list.
My harness consists of multi-level AGENTS.md and SOUL.md containing "grug brain developer" and "asd-ste100" type of speech for clearer communication.
btw. I'm not reviewing code in detail anymore. I only give direction and provide architecture design vision, its the model and couple of async agents working in the background and consuming work resulting in PRs which I manually approve after an eye-glance.
The reason I personally like Opus 5.5 is that after using it quite a bit, I can just say something like: "Please take care of this compliance ticket." The model handles research, record keeping and code changes. Consistently and correctly.
I just don't have the level of trust in a lower end model. Even Opus 5.5 isn't quite ready for "driverless" in some architectural situations, usually around change management in the middle of implementing a system.
I was just kind of surprised to hear you were fairly hands off with a lower model like DS.
I can ask for a review from another agent (they often find NITs) if I'm hesitating.
Kimi K3.1 Pro: −9
DeepSeek V4.1 Flash 1017: −13
GLM-5.3: −13
MiMo-V2.6-Pro: −16
Proprietary models
GPT-6 Astra: +4
GPT-6.1 Sol: +3
GPT-6.1 Terra: +3
Claude Opus 5.5: 0
Claude Sonnet 5.5: −2
Claude Fable 5.1: −3
Gemini 4 Argon (high): −3
Muse Wonder Full: −6
Claude Haiku 5.5: −7
Muse Wonder: −9
Gemini 3.8 Flash (high): −10
Grok 4.7 (high): −11
Is there affiliation with OpenAI or coincidence?
There was a thread here about families of models responding in the same patterns to specific queries like "say a random word", and it takes less than two minutes to verify it is true. It'll take a bit of effort to not convince me this change wasn't reverse-benchmaxxed to screw everyone other than OpenAI. I'll be less distrustful after everyone is done IPO-ing.
Eg asking "here are all the contract I'm subscribed to, here is what happened, is there any insurance that covers me and if yes which one is the best to make the claim to and why, and provide all the article to prove why and how i'm not in the exclusion list". Try it with random stuff of your every day life. Opus is both great and not great.
1. Is there a benchmark measuring how much better an LLM performs when you tell it "you are an X"?
2. Does telling an AI agent "you are an AI agent" actually help? Is it the LLM equivalent of reminding a senior developer they're a senior developer, making them feel guilty about being lazy and finally getting them to do their job?
3. What happens when the persona conflicts with the task? If you tell it "you are a PM" and then ask it to write code, does it struggle? Produce worse code? Spawn a sub-agent and tell it they're a senior developer?
[1] https://artificialanalysis.ai/methodology/intelligence-bench...
As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.
I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.
Will an LLM topping this bench be the end of humanity? The doomer rhetoric has no place in what should be an objective measure.
Hard to trust that a for-profit benchmarking company isn't biased due to profit-maxxing.
They perform pretty badly on the new benchmarks introduced.
The score hit on smaller models is a welcome change especially when compared to models that nearly 10 times larger.
Grok being punished quite a lot is also a very welcome change as the model is frankly not that good.
I would love to see few more points knocked off haiku by introducing steerability/creativity benchmarks.
You need a method to show you RL is improving the model that is separate from the training data.
One thing I’m still surprised / unsure on is haiku 5.5 max besting latest deepseek - but also I think the last haiku was so incredibly bad I still hold some bias whilst using the new one.
One of the new benchmarks is FrontierCode 2.0 by Cognition. AA seems to have accepted Cognition's benchmark scores without questioning, but they should have looked more closely.
On this benchmark, Anthropic's models are run using Claude Code; OpenAI's using Codex, and Gemini is run using... Cognition's own Devin (not Antigravity). Thus its score is much lower than it should be, knocking ~3 points off its AA score. All open-source models are also inexplicably run using Devin, damaging their scores as well.
This may even be intentional, if Cognition has beef with Google, since Antigravity was built using Windsurf's technology. But aside from that, it is to their advantage to make Anthropic/OAI look like leaders in the AI market. They are a company known to lie and exaggerate again and again, so I have trouble giving them benefit of the doubt.
[1] https://artificialanalysis.ai/evaluations/frontiercode-2-0
It's interesting that the Stirrup scores are somewhat close to the specialized harness scores. Either Stirrup is quite excellent or is it that harness doesn't matter so much after a point?
I would think it's the opposite as a more competitive frontier for llms makes the labs shed some margins and enable a broader ecosystem.
Additionally: competitive frontier = more commodified = high valuations under more scrutiny.
AAv5.0: 25% Knowledge work: AA-Briefcase v1.1 - 15% AutomationBench-AA - 5% GDP.pdf - 5% (-5%) 25% Coding: FrontierCode 2.0 - 15% (new) Terminal-Bench4.0 - 10% 25% Science: Terminal-Bench Science .1- 15% (new) CritPt v2 (hard subset) - 10% (new version) 25% General: AA-Omniscience v1.1 - 15% HLE-Diamond - 10%
AAv4.3: 30% Agents: AA-Briefcase v1.1 - 15% GDPval-AA v2 - 10% AutomationBench-AA - 5% 20% Coding: Terminal-Bench4.0 - 10% SciCode - 10% 30% General: AA-Omniscience (Accuracy) - 10% AA-Omniscience (Non-hallucination) - 5% GDP.pdf - 10% AA-LCR v1.1 - 5% 20% Scientific Reasoning: HLE - 10% CritPt 10%
tldr: gained Terminal-Bench Science and FrontierCode, dropped SciCode, GDPval-AA and AA-LCR. Lowered weight of GDP.pdf.