Gemini 4 Carbon is the new #1 on the Artificial Analysis Intelligence Index v5
j / k
|
Down or up a comment |
h / l
|
Previous or next link |
?
|
This help |
I got it via the "Pro" plan, exclusively for personal task usage. So no agentic programming.
For starters, the limits are dreadfully low. Five messages on medium is already enough to trigger a 4 hour cooldown. Secondly, the best part is the integration with Google search and Maps. Makes it seamless for trip planning. Lastly it's the only "SOTA" model in a long time that has actively hallucinated details. I don't know if it's because the Gemini harness is not asking it to verify information, but I've had some real Deepseek V3-like lies in responses.
Either way, I find it difficult to get invested in Gemini because nobody will have access to Carbon for a few months anyways.
Honestly the fiction writing was pretty impressive!
Capabilities are also improving so fast that this fomo doesn't really keep people excited. If I can't run it, I'll sorta just click off. And given how they handled the Argon release, I've not even bothered look at this Carbon one. Fully expected it to not be usable yet.
The target is to make a celebratory announcement.
In my limited usage, it has fixed issues which has not been correctly fixed by sol-6.1 (medium) - https://github.com/auto-gnufied/markdown-modern/commit/a189a...
I am also using it via Google pro tier and did not immediately run into quota issues. But as someone else said in previous discussion of this - Google Pro tier already includes lot of stuff (including Youtube premium etc), the AI plans are just a nice addon.
Gemini models always suffered from this. They were almost always best in terms of intelligence, but one of the worst to use in practice, because they have a lot of failure modes on simple tasks that simply give factually incorrect answers, or completely break and output gibberish.
I feel like you're being overly dramatic here. Argon was 13 - 15 days.
Argon was announced September 30, made available on October 13 to Ultra everywhere, and Pro in the Gemini App. Pro plans got Argon on October 15 in AGY.
This is the first time they've slow-rolled a model. Who knows how long they'll wait on Carbon.
The Ultra-lite plans ($99/month) get some pretty good quota where you can actually use Argon some more. Maybe Google shouldn't be grouping together plans, but it should also be noted that $20/month also gets you 5TB of storage on Drive. If you paid for Dropbox, you get 3TB for $18/month (without any AI model usage).
(googler, opinions are my own)
Gemini cannot create "my maps". I like creating a map of stuff for travel and now I ask it to create an importable file (which it doesn't always get right) but it should honestly do that itself.
And I don't mean that from within the Gemini app, I'd like a shareable list of places, which you can do via a very outdated maps UI
By the time Carbon comes out, I fully expect Anthropic, ChatGPT and/or a Chinese lab to have its answer ready to ship in less time than it took Google to approve a release.
my whole family of 5 people is using in total 160 GB. personally I use about 15 GB
That's the problem for me. Google can have the world's best model; it doesn't matter to me, because I effectively can't use it.
Both Opus 5.5 and Sol 6.1 are extremely good and provide a lot of inference on their $200 plans, and then there are the Chinese models - DeepSeek 4.1 Flash, MiMo, and StepFun all offer a lot of value per cost.
At the moment, I don't see a niche where it makes sense to use Google's models, regardless of how good they are.
Yesterday I used Google AI-Search (so Gemini Flash 3.5?) for opinions on a traffic accident and it continously constructed law-interpretations and then referenced the wrong paragraphs. It talked about rules on road-crossings and cited paragraphs pertaining to alcohol intoxication as sources.
Trip planning is the most often used use-case for agents/bots I see in marketing. I'm lucky to travel once a year, but if I did have a trip, I wouldn't feel comfortable letting an LLM take the wheel. Too many variables for wrong flight or room, closed locations in itinerary, etc.
Now, critical server infrastructure? Sure, have at it, make no mistakes Claude!
Since then I tried it for experimenting on diverse fields. For example, this week there was a discussion here in HN about the non-continuation of Python funding on the JIT, so I "played" again to removing the GIL and moving to the JIT in Python, it did it. I played with many other things, even the Riemann Hypothesis (it just removed their rational parameter rounding only moves 0.875 towards 0.87495702, nothing significant though).
Many other relatively great things were done this week in parallel. I have some budget for research.
Could you expand on this? I’ve never seen a frontier model that doesn’t hallucinate details fairly often, what do you mean by “actively” here?
I used to defend Google's models months ago, but switched to Luna for less intelligent models.
But with this bs of not making it even public, pushing off release of newer models late, I lost faith in Google.
Also, their tooling still is complete shit. I'll not be forced to use Antigravity thing even if it is available and clearly better than Opus or Astra. Their subscription is useless. API can be used with anything you got but that's gonna cost.
Especially with the recent addition of graphic visualization widgets
It manages to strike the perfect balance between depth and style of prose that other Frontier models just can't quite seem to match
1. Sleep debts effect on muscle size and strength gain
https://share.gemini.google/Gvdb5Q02INrl
2. Fish oil and extra cellular vesicles
https://share.gemini.google/fem9EBCe4mfy
3. Pharmacology question about human-equivalent dose
https://share.gemini.google/RWNYJ2eiNjcU
Its the first model ever that I have been worried about a successor losing the "feel" that it has.
Without us being able to access Carbon when it is announced, there's no way to know the Carbon we get a few weeks later is the same thing as what has been benchmarked here. I think benchmarks should verify what is benchmarked is exactly the same as what's been released.
My experience is that Google is aiming to create a generalist model – but a truly generalist one! In fact, I find it particularly accurate in general terms and even in long conversations, maintaining excellent consistency and a wealth of information in response to complex questions.
I’ve tested it both as a coding agent – though, precisely because of what I mentioned above about its ‘generalism’, I don’t think it measures up to Fable 5.1 or Astra – and I’ve really appreciated its qualities when supervising my agents’ activities. It was able to keep track of them, suggest alternatives, and monitor the entire lengthy process in parallel without ever straying too far from the request or losing the thread – as often happens with Claude and OpenAI (you only need to test it in Claude Code itself, in both Ultracode and the agent team mode – albeit experimental – to realise how, once the task gets underway, both of them lose their way).
I also think Google should stop "releasing" models that aren't actually available. For their own sake, because by the time Gemini 4 Carbon becomes available, nobody will notice. There won't be a new press release or much excitement around it.
Since then none of the Google models have competed well with the others at their price points. Long context performance drop, instruction following issues, unnecessarily complex and sloppish outputs.
Argon is no different, it feels just an iteration better than 3.8 flash. I don't trust the AA ranking yet.
Also the guardrails were implemented in a very bad manner. By default the models used to raise a default guardrail error on basic queries.
- Two days ago AA announced their v5 Index which added a few new benchmarks and tweaked a few things.
- Today the newest model announced from a big lab tops the charts.
I'm not saying that AA is definitely taking money from the big labs and changing the benchmark to make the labs' new models look better, but it kind of looks that way. This pattern of behavior has happened before, where the index gets tweaked right around when a new model comes out, and the new model is on top. AA does not, AFAIK, provide a way to see how the new models would have scored under the older index versions. You could probably calculate it yourself, but I'm not going to do that much legwork just for an HN comment.
It's interesting because they publish the index methodology, so you can't really accuse them of not being transparent, but I don't really believe that the choices to update the index were completely independent of the model they happened to be evaluating at the time.
Related: The DXOMark camera scoring method also had similar behavior, where they would release new testing methodology right before flagship smartphones were released. Again, they are technically being transparent about the process, but it kind of smells like the "independent" testers may not be as independent as the average user would hope.
Check out vals.ai for an alternative. Not sure how independent they really are either, but it's at least good for a cross reference.
Example problems are:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
I have no doubt that these are very relevant task for people in those respective fields, but I suspect they have next to no relevance to some 99.9% of model users. Hence why especially smaller labs with smaller models didn't really train for such use cases and you can see it.
Personally, it's not something that matters at all to me, as I don't work in cheminformatics or bioinformatics.
This is also arguably more of a test of a model's knowledge base than reasoning ability.
But most importantly, there is also the question to which extent the aggregate score then remains a meaningful result when you mix up too many too disparate benchmarks.
The coding agents subindex, in contrast, looks at least superficially plausible [1].
[1] https://artificialanalysis.ai/agents/coding-agents#coding-ag...
So ... why bother to cheat?
I think part of it is to justify capex. "Yeah, we spent x billions on this, but look, we have the best models, we're beating OpenAI and Anthropic, we're winning, it's all worth it!"
At the previous iterations they were accused of favoring OpenAI. Now they are accused of favoring Google.
* Definitively very good at domain specific knowledge. It can read sloppy 19th century danish gothic handwriting with faded ink just fine. Not perfectly, but probably better than I could have done with a day of struggle for every page.
* Expensive.
* Bad at actually doing what I said. I had told it in the prompt to be careful not to miss text, but I hadn't explicitly told it that some images may have text in double-column layout. So on those images - over half of my 300-page document - it produced an excellent transcription of the first column, and then considered itself done. Want to try again with a more careful prompt? That'll be another $20.
My hunch is that Google will end up being best for non coding tasks. Google can concede coding to other vendors who pay Google for infra.
Until it got lobotomised. Then it lost most of it's capability.
Similar situation happened to Anthropic's Fable. It was an amazing model until Opus 5.5 came out.
So now, I'm not even looking at a "new" model until it is a month or two old. By then we will know if they will destroy ability to make it cheaper to host.
True, I use LLMs as glorified search engines to unearth material to read myself, but Gemini is much more useful if you decide to learn yourself. It also offers interesting expansions to your questions.
While I don't offload my work to LLMs (e.g.: Write this project, I'll be watching TV), I find them useful for closing my knowledge gap by collating information, and Google's Gemini is good at that.
Even if these numbers are correct, they don’t have any useful product for coding or search. Not sure what you’re referring to
I wouldn't even bet that Google is better than OpenAI and Anthropic for these categories, but they seem to be well used outside of our direct space.
Let’s see, maybe they’ll surprise this time.
I think what's happening is Chines models genuinely try to benchmax on it and hence they suffer when a new version comes out.
Is this API-only or US-only for now?
"We're taking a deliberate approach to rollout and are actively engaged in the U.S. government’s voluntary process for pre-release model testing. Starting this week, we will make Carbon available to a subset of cyber defenders and will gradually phase access to additional trusted partners through our Fairwind Program. We’re continuing to gather feedback and iterate on guardrails before making Carbon available to developers, businesses, and consumers via our product surfaces as soon as possible."
So here's a meta-benchmark: at what point does the act of choosing an LLM product for a reasoning task become less reliable (in the sense of "predictive of future success") than hiring a human being for that same task? Have we already cross that threshold? I think we might have.
And once we do... what happens then? Will the truly elite models be hidden in a sea of mediocrities simply because the goofball "managers" doing the hiring are too dumb to recognize their genius?
I take this opportunity to give feedback on Argon. The limitations are a problem. I have a Pro subscription and Argon already stops working after 3-4 prompts with long delays afterwards. It's virtually useless right now even when you pay monthly for it. It is better for some tasks than 3.1 Pro, though. Not much in my experience but it can sometimes fix errors that 3.1 has problems with. I never use Flash models since they often give abysmal results.
So I'm stuck with using Gemini Pro 3.1 for now.
AGY never even update often, and only has gemini 3.8 flash (for Google model).
Want a private benchmark and see where you'd rank? $$$.
Want to show up day 0? $$$$.
Want to show up day -0? $$$$$.
No, you can't use it. You can't even see it, and I won't provide you any information. But you must trust me that I am the genius who made the best model in the whole world.
Google's model release is pure PR puffery. It's no coincidence that it was released on the day of its earnings call.
Argon did not leave a mark or change anything
Why would anyone care about carbon as well