IBM CEO says open weight models are now good and cheap on-prem
j / k
|
Down or up a comment |
h / l
|
Previous or next link |
?
|
This help |
Over time NeoGeo realized they could sell you the hardware to play at home (“on-prem”) at arcade quality albeit at a price out of reach to most people. Alongside them, Nintendo and Atari gave you 80% of the fun at 20% of the cost. History will view games arcades as a flash in the pan compared to gaming at home.
Namco, Capcom, and Konami went into selling software instead of hardware but it feels hard to imagine buying a home copy of Fable** for $X,000 any time soon.
⸻
* That’s for video games. In fairground arcades you’d literally pay for tokens then use tokens for play. The analogy almost writes itself.
** The model, not the video game.
From a fair bit of coding with the agent, my experience with Qwen 3.8 (Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-0 on Strata, Qwen 3.8 27B on ninfer). In my particular case, I was trying to clone a recent HN post which allowed you to visualize how out of tune your singing was relative to a song, but as a web app or portable native app.
My experience is that both those models will generate an enormous amount of noise (literally: it was running audible unit tests on my machine to test the note-detection algorithms) while coming to a solution, if they do find a solution- while Claude (whatever the latest model is) or ChatGPT (ditto) will do 1/10th the amount of work to get the right solution. I suspect if I had the resources to run GLM 5.3, it would probably be significantly faster to get to a working solution than Qwen-3.8 27B, but likely still somewhat inferior to Claude or ChatGPT solutions, and also would generate a much larger energy bill on my part.
And at Q2, I think you're better of with Qwen3-coder-next on one DGX spark (or much better: on any apple silicon max, we are currently in the ridiculous situation that a second hand m1 max 64G will beat a DGX spark that's at least triple the price)
It reminds me that coin-op SF2 champion edition was like $6000 new, and you rented time with tokens. Eventually you could play a very good but more limited SNES version of SF2 at home for no tokens. And then PC hardware matured to the point where you can play the exact bits via emulator at better than original fidelity.
It will be very nice when AI gets to the emulator stage.
Do you think we'll be longing for the nostalgia hit of LLM models from 2025 in 2050?
Maybe we will collectively miss being completely right and inappropriate em—dashes.
Those early GAN image generators were so weird that they produced art that felt genuinely creative. I don't know that whatever people are using to create modern Tung-Tung-Tung-Sahur instagram reel slop is even capable of producing similar things.
(I'll run for cover now as I'm sure neogeo fans will disagree!)
(I also adore MS)
But also look at live service games and the shit modern game studios are doing to try to go back to the old world of rent seeking.
Game industry has first manipulated the economic conditions by restricting access to game content (skins, modding capabilities, always online drm, etc.) that previously was included in the price (buy game, you own that game outright).
Manipulation of public policy was a more recent thing with the laws surrounding underage gambling via 'loot-boxes'.
So I think the onus is on you to show us why these behaviors, which increase the wealth the companies are able to extract per game, are not rent seeking behavior.
As for the laws on underage gambling, countries all over the world are outlawing that, suing gaming companies, etc. The public is not in favor of underage gambling, and it's unclear to me how banning it increases the wealth of these companies? So I think the onus is on you to explain how this is an example of rent seeking.
Consider the use of microtransactions, battle passes, paid progression, and downloadable content. These systems often exploit the time, money, and emotional investment players have already made in a game. By deliberately slowing progression, withholding content, or introducing artificial scarcity, companies can encourage consumers to spend more money not because they are receiving a better product, but because they want to avoid frustration or access something they already value. The company is monetizing the player's existing commitment rather than necessarily creating new value.
This does not mean every gaming company or microtransaction is inherently rent seeking. Selling meaningful new content is a legitimate business activity. The criticism is that parts of the modern gaming industry increasingly prioritize extracting revenue from an existing player base over developing better games. When companies use their control over popular franchises and digital ecosystems to extract additional money from consumers who have few alternatives, they exhibit the very characteristics of rent seeking that economists criticize in other industries.
No, it’s not, you just made that up. Why doesn’t this match your previous definition?
The rest of your long AI-written comment has nothing to do with the principle of economic rent seeking.
Not sure why I’ve been wasting my time, but I’m out. Peace!
in the early days, 2 plays for a quarter was not unusual (but you couldn't buy one)
And the AI debt bubble price spike will probably stay there for a while.
Still (as bit over) 1/2 the ram bus width of a single HBM4 stack though! (1280b vs 2048b).
Uber took a decade but eventually, the real cost of a taxi service went up for many use cases.
Your argument is only solid if we are headed to the singularity.
Perfect encapsulation of the problem. I may steal the wording you used for future conversations.
Now I want a local LLM server :( $25k is just so much money.
Sadly, the r9700 has gone up in price, but it's still the cheapest route. I use It on an old x99 Linux desktop and it achieves 50 tps decode and 1000+ tps prefill.
My stack uses a local llamacpp ROCM build, modified by https://github.com/stew675/llama-cpp-rdna-boosts
The build uses my machine's ROCM and I just link it instead of embedded all that in an image, like the published ones do.
Then deploying it via docker compose to my local machine. The models I want to run are stored in a docker volume, so it's easy to switch the model while keeping the same build.
The biggest negative for me is that I don't think it's enough to do parallelization through llamacpp (I'd prefer to work on one thing faster than two things slowly). I haven't gotten vllm working yet. Also, once you have one, you constantly start thinking about a second card...
Some censorhip and bias towards certain ideology is baked into those models.
While it is not a problem for programming tasks, I would never use it for certain other tasks.
Self-hosting means being able to use abliterated models where the superficial instructions like not describing Taiwan as independent are burned off.
Censorship and bias are related but not synonymous. I agree that it's harder to remove political bias when the training data includes it. We know many of the frontier models are heavily trained on Reddit, for example, [which likely explains why we see such heavy left wing bias.](https://trakkr.ai/bias/findings)
More citations:
https://www.gsb.stanford.edu/faculty-research/working-papers... (article: https://news.stanford.edu/stories/2025/05/ai-models-llms-cha...)
https://www.tandfonline.com/doi/full/10.1080/29974100.2026.2...
https://ieeexplore.ieee.org/document/10817610
https://www.washingtonpost.com/technology/interactive/2026/0...
https://link.springer.com/article/10.1007/s11127-023-01097-2
https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...
I am not saying I would not use any of them at all, but I would avoid doing certain tasks with them. And with the testing I have done with abliterated models in the past, it is not 100% successful and it can still hit the same walls as original model.
In open models you can "uncensor" them: https://arxiv.org/abs/2406.11717
A lot more with Haiku equivalents (Qwen3.8 27B) or gemini. It's not frontier, but not that far away.
You may need a AMD EPYC processor and associated motherboard, don't forget a good PSU at least 1200W (300W processor + 400W by card to support spikes of consumption). On some setup (~15k€) you can use 4 CG to go up to 128G of VRAM.
I'm not using a EPYC processor for now, so I get 170tps on decode but very slow prefill, the bottleneck is the PCIe exchange between the cards, I need to setup P2P (it means what you think it means, for GCs).
My setup supports full context (2*18 bytes so 256K), I didn't setup YaRN to get 1M of context (2*20 bytes), but it's an option with 128GiB of RAM, the visual component is extracted from RAM so a little slower. I'm not using multiple agents but some people does, it's god like with 4 graphics cards, on 2 only, it's relatively quick but limited to single user and up to 3 to 4 concurrents requests only.
It's not frontier, but look AA of QFN, it's good enough. DeepSeek V4.1 can run on my setup, but it's slow. On dual B70, there is some PoCs to make it usable (>60tps). It improves every days.
If you need more information, look for local setup users on the Web. There is a lot of people sharing information about benchmarks and setups. L80 is my reference for now.
Edit: I translated the unity.
octet => byte, https://fr.wiktionary.org/wiki/Gio
Worryingly, they're taking a walled garden approach, and they don't want to expose user data to competitors. After the monumental failure of Siri over more than a decade, I don't trust Apple to execute competently on AI.
This is an alignment issue. OpenClaw with a local LLM with an email you setup for it could contact the police.
Sorry, you're claiming local LLMs are somehow deterministic and never show unexpected behavior?
Besides, I regularly see people here in HN spend a LOT more than that on hobbies and depreciating assets. A few years back, people with coastal salaries and tech jobs were tripping over themselves to spend $100k on a certain brand of trendy electric car.
This is all this stuff that dries people to pay for frontier models. Even running modest models requires a few thousand, unless you just want really lean ones. People don’t need frontier but it’s not cheap to effectively run, say, a 16gb Qwen 3.6 or 3.8 in the 30b range. Ram is like $15/gb and GPU’s with 16gb vram are hardly cheap.
The lesson it taught me is that swapping is slow and that if it happens during your software development, you're much less productive. When I finally maxxed out the RAM on my machine, I could run linux + emacs + X11 + bash + g++ all at the same time, making me about 10X more productive. It's an investment and lesson that has helped me build my career over decades.
Businesses were quite happy to spend $5k on technology for a worker if it makes things better, why wouldn't they today.
This is not a straightforward business expense conversation IMO. Otherwise yes I do agree with you. A company should invest in supporting local models for their team.
And what happens when the girl at the bar figures out that you TIMESHARE that Lambo?
Ok taking the :door:...
This sort of casual misogyny is at the root of the problem. Shame on you.
My entire career has been based on purchasing the most hardware I could afford at home and using it near its peak. I've given up other life pleasures (dinners out, nicer car/housing) several times to max out my computational capacity.
Also hardware like this, if chosen carefully, often has a viable lifetime over 5+ years
In case a future user wants to know what a 2026 model would refuse to do, I needed to rotate dozens of passwords last night both on the server and on the clients. When it was time to verify that the rotated credentials worked, I asked for a script that would rate limit the credential checks. Claude/opus5 (opus!!) refused and stopped in its tracks.
In another instance, Claude was making great headway bringing a legacy Windows app up in WINE until it’s saw “auth” somewhere and shut down. In the shutdown state, Claude asked me to reword the prompt to see if I could get it to act on less dangerous instructions. But when I reworded my request, narrowed the scope to be clearly unthreatening, or even redirected the model, it’s still rejected me. I suspect Anthropic uses the less suspicious rephrasing as training material to increase their rejection rate.
So much this. Plus everything else you mentioned. The freedom and control is real.
> I don't get to hear that it's not a doctor and I should call 911.
https://en.wikipedia.org/wiki/Darwin_Awards
Anthropic has a significant interest in reducing liability, which in this case means mostly blocking this use case.
Unfortunately, it's fairly tough for someone to determine if they are themselves a reasonable person.
A model that only ever says yes is a cognitohazard, IMO. Like being a billionaire without the money to insulate you from the consequences.
Sometimes, the requested information should be accompanied with a note like that.
Yes. I've also heard the cost (death) of people falling into pseudosciency rabbit holes.
Local models can do this and with tool use they can even call emergency services for you on your behalf.
Also, it's not like you're running any full models on local without shelling out 20k+
I've heard that there are legal software platforms like Clio that offer AI products, not sure what just eavesdropping on some lawyers at the bar.
Companies pretty much give us the full overview of what they have from a technical perspective (and often much more than that), we sign NDAs with investors that have serious teeth in them. The data they share with us would not be disclosed under any other circumstance and even our reports are shared only with the clients.
It's a nice little business. I'm not going to risk offending my customers for convenience or to save some money.
They are subsidizing all of us to the tune of $billions in losses per month. That can't go on for a decade. Probably not even for another 18 months.
I still use all the tokens Claude will give me on the max plan. But I also run deepseek v4 locally for overflow and for... contingencies.
But, there is no need to make that bet - when prices increase, I'd look into renting a previous-gen GPU on a cloud provider. If the online GPU providers raise prices, then I would look into purchasing for on-site.
Until that happens, I'm not making any bets, because if all the token providers go bust GPU rentals are going to fall in price, and if they don't go bust after raising prices, I can still get years of GPUs for self-hosting at the price of a self-owned rig.
It's not economical today to run my own hardware, but the insurance is worth it to me in the sense that if my bet is wrong, I'm still okay with paying extra for the protection.
(Of course this all depends on if their plans work out, but if not then the smaller local models will probably play a big role in the failure of their plan)
At API pricing (i.e. what all enterprise customers are paying), inference is profitable, even if they're paying usurious rates on cost of capital for capex. Where the frontier labs are getting screwed is that they're in a Red Queen's race with each other, spending tens of billions of dollars per year training the next frontier model, and not being able to recoup the investment on that model before it's obsolete.
That's why you see the CEOs calling for government intervention to "pace the frontier": it lets them reduce investment on training without risking that their competitors keep going and make them obsolete. Unfortunately for them, the Chinese labs exist, and would love nothing more than to take the frontier, so they're in a bit of a pickle.
However, that also means AI will be even more lobotomized since the cloud providers will panic over being liable
People keep repeating this, but it is only remotely close to true if you evaluate inference in a vacuum without any of the other supporting costs to facilitate that capability.
Just a few years ago people were saying the same thing about WeWork: "leasing is profitable, the unit economics of individual locations is net positive", and yet...
- they raise the prices 10x
- go bankrupt
- ban you or your country
- decide not to help you with what you want to do for opaque reasons
- you are in jail because the model halucinated something and the company promptly reported you to the police
Better avoid all off these and go local IMHO.
Just note that your local model can 100% do this itself.
The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.
But yeah, with current sub prices/budget... It's hard to go with anyone else unless you're rich.
The anthropic sub is likely still subventioned with VC money. So this may not even be available in 2 years.
But yeah, with current sub prices/budget... It'd stick with it too
It doesn’t have to be black or white…
For my toy projects, adventures, learning exercises, and throwaway MVPs I will prefer the subscription plans… they are cheap for what they offer today.
It's training and tuning new models that cost them money. And that's why they are trying to scare people into thinking AI will kill us all in the next 5 years; they want bans on the open-weight models because they know they won't be able to compete with them.
If that is true, either the American labs are not optimizing correctly or they make a ton of money with inference.
Hydro, wind, solar, nuclear, coal, they are building it all, as we speak, at least 10x more than the US. They are aiming to not just be energy independent, but energy dominant.
You can't now (and certainly won't in the future) be able to run an AI provider in the West as profitably as you can in China.
It's also very clear that providers are running variants of models, especially at the SotA levels. I wouldn't be surprised to find out at some point that 1-2 months after release OAI and Anthropic just run "good enough" quants to save themselves on inference (since they're losing money hand over fist). Another upside of local models is that I don't need to worry about model degradation at any given point.
Just think of how user-hostile other online services have gotten, and imagine how acutely painful the proprietary LLM providers could make theirs. Thankfully, open models provide some counterbalance.
Are you comparing to subscription prices or API prices? Because if it's the former it won't last forever.
You say the big rig is 25k - and then the two sparks are about 1k each?
Electricity is for "just" the big rig?
https://jacquesmattheij.com/local-inference-with-large-langu...
My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.
You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.
I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.
I have 240G VRAM total which is a good reason why that rig cost as much as it does, and if I were to build it at today's prices it would be even more expensive.
Kind of ironic, but it's the only way. I just really like this as an option. For a household LLM I have the IQ4_K_XL quant and plenty of capacity for everything you could want in a household LLM and it is trustworthy enough to not rm -rf /, etc.
And yeah that is a really nice amount! You can run and compose a lot of interesting models, vision capabilities. IDK. Plus this level of model is far more interactive in the sense that it can implement a well scoped ticket or unit of work but its not going to magically just figure some things out without guidance or a spec :)
I tried getting some 170's when I first heard about that unlock but they were already unobtanium by then. That would have been a nice way to add a lot of VRAM.
GLM 5.3 runs pretty comfortably on this rig, even with large context.
I’ve always wondered about those models. Are we just sending all of our data over to China, instead of ChatGPT or Claude? Can we trust them any more than we do ChatGPT or Claude?
At best they get a fine, and it will not make a dent in their cash reserves.
But I was not even suggesting that, just run your local LLM on Cloud infra. Do you really believe they are ready to risk their Cloud business just to spy on your prompts?
Then again, I run a business and for the business it is absolutely manageable, it is roughly the price of a single job. Privately this would be a different matter but I assume that if you would do this privately you'd simply get a single (but powerful) GPU second hand and offload to RAM. Not as fast but with llama.cpp, 'ninfer', DwarfStar and various other clones of those two you can get some pretty decent performance out of single GPUs.
I've been buying powerful hardware since the 90's, starting with SGI and then ever larger Linux boxes. I make my living with them so it's never been a huge decision to invest some of the proceeds back into the tools. But on an absolute scale, yes, it is a lot of money.
What's your use case where the open models are working well?
I’m running local models more and more and starting to move to just tailscaling into everything
I mean IBM already has a track record of building gigantic amounts of garbage and over charging large corps and govts for it coasting on their brand name and networks to get deals. So using some cheap ass model fits their MO. Having used their shit mountain called websphere, what an IBM CEO says is not something I take too seriously.
I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...
I do use Anthropic and OpenAI mostly because it costs me 1/40 and 1/50 of the api prices, which is way cheaper than Chinese subscriptions, but I’m ready to jump ship when they pull the plug. Fortunately they have nothing that forces me to keep paying them, and I don’t pay the crazy enterprise subscription costs, just the personal account prices.
I also die to be able to run the OSS setup locally to stop being spied on all the time.
For context, I averaged 10 hours a day prompting LLMs from when ChatGPT came out. At the start it was just function generation, then single file modules, then modules, and now I reason, design, implement and test with them.
An interesting thing is that with omp you can plan with a smart model and then implement with a dumber one. So for example, plan with Kimi 3.1 via OpenRouter and then implement with DSFV4 with zero data retention active, sorted by cheapest, and minimal tps accepted.
I get the people who are pissed about their cushy jobs evaporating, but if you know how to use these technologies it’s very freeing because it’s like having a company of people working for you, except they are not human so they’re not wasting their life working for you instead of using it for themselves.
You can make them do pretty much everything that is fuzzy-verifiable, you don’t have to worry about their feelings or that they won’t do what you asked them to do or stop, and you can activate them when you want at scale in parallel or not use them at all for days.
For me it’s a dream honestly. I never felt so much empowered in my life while being enabled to unlock my creativity with almost no boundaries.
I geniuinely don't see what I could make with this new tech, my brain immediately goes to "this app idea is worthless, anybody can make it now".
It's like having extra copies of yourself that you can just fork and get stuff done. E.g. the other day I was in a meeting and there were some heavy discussions about a few gnarly issues. I fired off a few prompts in parallel while the AI researches options, approaches, made a prototype etc all while I was present and paying attention still in the face to face meeting. I am so much so much more efficient as an engineer now - getting better feedback, more bonuses etc. It's never been better - I handle thinking about the requirements, the computer science and software engineering big picture etc, and the AI handles the syntax trivia for implementation.
The models are really really good now.
The only downside is that as a result of more stuff getting done, we need to have more meetings to discuss what actually needs to be worked on.
I've seen developers use Claude to make suggestions for infrastructure design that made absolutely no sense and did not consider our environment. I've seen people write English that grammatically made sense but logically did not.
I don't see enough complaints about this and it can't be just my company.
Is anyone else seeing really severe Dunning-Kruger?
I always check what AI says to me, and I use my own words. I think the average engineer is not.
They call this "slop" - I think it is well known at this point?
That it's a degradation in some environments? There's not a lot being said about this. The laments are usually developers sad that they're prompting instead of coding. My lament isn't about the art of it, it's the human reasoning that seems to have gone away in my workplace when it comes to engineers.
You are a consumer, so you naturally think like the consumer you are. But consumers want cheap stuff and only big big businesses are able to provide cheap stuff at massive scale. So stop thinking in b2c. For us little businesses the money is in b2b. Businesses pay lots of money to have problems solved. They want to focus on their core business and outsource the rest.
Consumers hate paying subscriptions. Businesses love paying subscriptions. Because then they can make their problems someone else's problem and focus on what their actual business is.
Maybe it’s 1% or 10% today, but a decade ago, having/assembling the technical chops to know how to build something was a huge part of the meat / barrier to entry.
If code becomes cheaper that may change the business fundamentals but it also means capturing customers becomes way harder.
Now that opportunity cost is much lower so experimentation is more practical than before.
I also do repairs and so I recently added a job management system into the POS, which has been really useful so far. Next step is to sort out the online store/eBay integration aspect!
I've found it very liberating that I can handle some of the tedious parts of a project to AI. In the end, I spend perhaps the same amount of time on a project, but more of it goes to important work and less gets wasted on plumbing.
I am however worried that for the kind of work that the majority of the worlds tech workers do, this is a direct replacement for.
I guess we'll have to live to see.
Sol and Opus both are quite eager to use subagents and since the subagent tasks are often well specified and then verified by the main model there's hardly any advantage to using anything better for the subagent anyway, so IMO it's a no-brainer.
For me switching the main model to Kimi K3.1 Pro would put me towards 100% open, but I don't know if it uses subagents effectively. Any other open models I noticed just don't utilize them very much.
But the lock-in is not _that_ big that a big co could not pull off a migration, but it would be like migrating from Teams to Slack, not like migrating away from Adobe.
Looking at our AI spend today vs 1 year ago we have massive changes. Anything that can bring this under control and potentially save millions a year is being explored.
We will probably end up with a combo of open weight models for most work and frontier models for key projects. For now. Who knows what the landscape looks like in 1 year.
Cheaper than the API prices you pay on openrouter. But not cheaper than this:
https://www.alibabacloud.com/help/en/model-studio/token-plan...
This is more of a statement for their consulting customers, right?
I can picture that they’d have the software side sorted out with RedHat. But I don’t know what hardware they’d push to support local on-premises models. It looks like when you buy Fusion HCI, you can choose from Dell or Lenovo as the underlying hardware supplier.
That doesn't detract from your overall point that they tend to make overpriced stuff, but I don't think they have to do that. Besides, what's overpriced? NVidia's GB300 workstation sells for 95K euros, a single RTX 6000 is now 20K. Surely even IBM can compete at that level.
I think IBM sees the Spyre as a reason to sell more Power hardware but that's just not where the industry is right now and I doubt it will be enough to create sufficient demand. Who knows, I don't have any insight in what they're doing there, but there is massive gap between A GB300 workstation and what IBM currently offers. At the same time: the RAM crunch hits all parties and that's a major obstacle as well.
That's the thing though. Software is cheap with AI, but Nvidia has Cuda. If software doesn't matter anymore (it does) because it's so cheap now, then Nvidia's moat evaporates to IBM's Spyre. In reality "just a little bit of software" has inertia and cannot be dislodged quickly, quietly, or easily. cf Microsoft.
As for CUDA: I think CUDA is a formidable moat but I also think that that is precisely the kind of thing where the likes of IBM could (if they really wanted to) make things happen. They're already in the space with Redhat and they have ample knowledge about drivers and low level code. Putting together a team that either makes a drop-in replacement or alternatively, that starts supporting their custom hardware for the big players in the inference space (VLLM, llama.cpp and Torch) would make all the difference. They'd have to keep doing it though, even if the initial results were lower than expected because that market is not going to turn on a dime.
They do? They are a spec in the ocean relative to big players offering inference and training their own models. Pretty sure one thing they are struggling with is the money to compete successfully in the ai game this time around.
Yeah they were only half a decade late to the PC revolution back in the day...
I've never seen Tandy used outside of Europe but then again, I had not traveled as much then as now so it's definitely possible they used the brand on other continents as well. I wonder how many trees were cut for those catalogs :)
If you have ever worked at IBM, it makes total sense. Completely dysfunctional middle management and on up.
It was pure coincidence that the AI-craze neatly followed the crypto-craze (many neo-clouds in the AI space were originally crypto miners).
Yes it would have costed them more to train the first few versions of ChatGPT if it wasn't for the crypto mayham, but it would still happen. The hardware was always available and still improving regardless of crypto or not.
In comparison, my highend mac book pro costs 105 euro per month (leasing cost). That's not quite enough to run the bigger models and besides running the GPU on a laptop constantly makes it a bit hostile for my lap. And the model I would need a lot more memory for the bigger models. It's nice for experimenting though.
If you get your own GPUs in some kind of server form, you could run those locally or in a data center. Or you just get hardware resources from your favorite cloud provider. Hetzner (one of the more affordable ones) GPU servers start at around 200 euro per month. I know some people that use that with open weight models. Works fine, especially if you can share the resources between multiple users.
At which point of course you are doing the same as cloud based providers are doing. But probably with a bit worse economics. It all boils down to whether you are actually saving money or paying for a lot of underutilized hardware that's running 24x7 when your people actually only show up about 8x5.
I don't think most companies should be doing that. It sounds like they'd end up with a lot of idling over capacity at a premium. The only valid reasons to do this would be securing access to GPUs or some very urgent need to keep things on premise (which a lot of companies insist is true for them until they realize how much they are going to pay).
It's also about "insurance protection" against the AI company raising the prices or changing their terms (e.g. becoming more frugal with respect to token drops).
If you are running your own models on your own hardware you take a bunch of risk as well. Is your hardware capable enough to run your models? Do you have enough of it and access to more when you need to? And is the stuff that you picked still going to be good enough in six months? And can you trust those model providers that you picked to keep up with the rest of the industry or are they going to fall behind and leave you stuck with obsolete but still very expensive technology.
Of course, this is an article about IBM who are in the business of selling consulting and services at a premium. They would probably love companies to self host their models on infrastructure that IBM supplies and consults on (for obvious reasons). The question is, should you buy what they are selling or can you do better? My simple observation here is that if companies like IBM are getting excited it's probably because they are looking to separate a lot of companies from some pretty significant cash. So, I'd be wary to buy what they are selling. But that's just me.
In a rapidly evolving market with extreme levels of uncertainty around tech and what will happen next, being reluctant to get pinned down on the wrong solutions is probably a good instinct. The biggest risk here is making the wrong bets. And you need to look at what it's going to cost you if you get it wrong and what it's going to cost you to fix it.
So it matters a lot that these open-weight models are good enough; we would likely not be rolling out this new processing if it was fully dependent on Anthropic and OpenAI.
I had a really nice time with GLM5.2 for a month or two - it was, in my experience, approx Opus 4.5ish level of capability when I think Opus 4.8 was the new hotness
Even today, I use Gemini 3.8 Flash as my daily driver for most coding. Its fast and good, even if it is not the absolute SOTA at coding benchmarks it really is very good. Yes I have tried Opus 5.5 and 4 Argon etc and yes the are better, but do I need that?
I think that we are going to be at the stage soon-enough where models are "good enough" for 99% of people's needs. The SOTA will keep on marching ever-onwards I am sure, but I would not be surprised if people stick with previous-gen/previous-previous-gen/etc models because why upgrade?
If this does play out like this, I do wonder how that works out for the frontier labs (specifically OpenAI and Anthropic, less so Meta & Google) - if open models or even just cheaper Sonnet-level models from a few generations ago are good enough for basically everyone's daily work, where is the revenue? how can you justify pumping billions and billions into new generation models and datacenters etc when there is no driving need to use them?
For work in which you are in the loop, absolutely. Do people not value their own time or something? Why in the world would you use less than the best LLM available to you? You're not going to run out of the $20 plan even if you use Opus 5.5 at xhigh and code 8 hours/day (assuming you're doing it piecewise using a Claude product) so what exactly are you conserving by using something inferior?
For large continuous volumes of programmatically submitted LLM tasks, sure.
Could your survive on say Opus 4.8? Or GLM 5.3?
I bet you could if you had to.
I think it's the same argument about cars. Why wouldn't you want a Ferrari? Do you not value your time? Why not go as fast as physically possible?! You need the absolute fastest, most-capable, best-handling car for every single journey you take! Or... Back in reality not every journey you take is trying to lap the track as quickly as possible and just any reasonable car will get you where you need to go.
This is my point. Will Anthropic and OpenAI continue to exist selling Ferraris when actually people realise they can get by with a Toyota Corolla 99% of the time? Ferrari exists in the car world so I guess there is an argument for "yes". I am not sure myself - Ferraris are after all rare and exotic cars ... I suspect that even if we are not there now, we are already seeing current SOTA models that are mostly equivalent/better than the average engineer - another generation or two they'll be waaaaay better than the average engineer. How much more car do you need, and how much are you prepared to pay for it?
At least for me, I don't like using Opus or Fable because I don't like the way it talks to me (talking down to me, refusals, moral high-grounding, overly pomposity etc). 3.8 Flash is currently my favourite model due to how it communicates and works with me and the speed is much better suited for rapid iteration. So put me in the comfortable, reliable, use-it-every-day-to-get-shit-done-with-no-fuss Toyota Corolla camp. Its good enough for me and I don't get any benefits in my day to day life by using Opus 5.5 and it's sanctimonious personality.
I think this is a very high likelihood scenario.
This is a win though. But not for the reasons people might think it is.
It is a win because it's a massive player now countering the new (temporary) behemoths. With all the leverage, budget and connections.
And contrary to the AI Labs, IBM isn't just that one SV rockstar dude, but a whole org with tons of people. Meaning that they're going to be far closer to the baseline of sane humans just from averaging alone.
IBM of today is a hallow corpse of what it was. The hardware business remains but they’ve mastered extracting rent from imprisoned customers long ago. No sane person should willing go into business with them, same as Oracle.
They are many people and none of them are in some sort of AI death cult like the EA polycule.
They're not suddenly good guys. They're just sane
If the vendor price is calibrated to a ten-month hardware payback, then at equal utilization there is no arbitrage in self-hosting: the crossover is roughly whether your sustained API spend exceeds the hardware plus that ten months. DeepSeek's own disclosed numbers give the benchmark to beat: 226.75 H800 nodes averaged over a day (~1,814 GPUs), 608B input tokens/day of which 56.3% hit the KV cache, 168B output tokens/day, ~73.7k tok/s input per node while prefilling and ~14.8k tok/s output while decoding.
That utilization is the hard part, and it is roughly what the IBM CEO is pointing at with "where are my kernel engineers". Self-hosting open weights on two or three racks is not the problem; amortizing them at a frontier lab's duty cycle is.
The frontier labs are in a tough spot. It's hard to justify their cost when you can get something that's easily as good for cheaper (and anecdotally, more reliable/accessible). I do HITL so my perspective will be different than a fully-agentic setup, but as soon as I started using open stuff regularly, I realized that any moat the US labs had built is drying up.
Though if you have the room for it, Muse Glimmer is better in practice on low reasoning strength, and while its token generation is slow, it actually gets to answers pretty quickly.
"There are dozens of us!"
The thing you need to know about Muse is that you control its reasoning/thinking depth in the system prompt (low/medium/high/xhigh):
… is pretty responsive and I've found its time to first answer to be surprisingly good.It is broadly comparable to Qwen 3.8 27B on "low" thinking, though it is weaker on very long context, one-shot stuff, but I dunno… I just like it more as a developer companion.
(I'm using Unsloth's version)
This is what will cook frontier AI and probably prevent the investment scale to happen for AGI. Humans + good enough cheap AI is just more economic and unbelievably productive. Like why we fly on jet technology from the 1960s still with incremental improvements.
It’s why Anthropic is desperate for capital and trying to get to a scale where AGI appears. But I’m afraid they won’t get there in time and the money to build the bigger model will have dried up.
Does Ferrari have any moat if it can't convince the government to ban Ford?
we technologists tend to extend what we see and think to other realms where it doesn't make sense
I really wanted to tell her -- what she needed was compute, not hardware. Her best bet was to rent GPU which is easily available from many services. If she was paid to run inference, cost is not a concern. If she did it for her own work, she wouldn't need to shell out a ton of money initially and then be left with hardware that she no longer needs when the work is done. In any case, she wouldn't need to worry about purchasing hardware or battery life. ssh to a remote machine is simply way superior than running inference on a laptop she carries around by any means except in a few rare occasions (e.g. no Internet access).
Same thing for a company. Why would any company who has a business to run get themselves burned in acquiring GPUs, setting up racks, load balancing etc when Bedrock/Azure Foundry etc is right there with tons of both open and closed models and ZDR? And then see hardware depreciate in just a few years?
I was able to get this related article
https://rcrtech.com/semiconductor-news/what-happens-to-old-a...
I am not a believer in winner-take-all mechanics for models (even for AGI -- look no further than the natural world for many examples), and a large F50 company committing money/strategy to something outside the frontier labs is a great thing for the ecosystem.
Maybe your law firm doesn't want your data going outside your business. But you obviously want high quality analysis.
In the IBM-specific sense, the most unintentionally hilarious LLM interaction I have ever had was asking Granite’s 4.2 30B reasoning variant the car wash question.
Reams and reams of corporatese about why it wasn’t wrong when it was.
Is it though? You’re likely paying more on prem per unit of electricity and the racks of NVIDIA cards also aren’t exactly cheap.
I don’t see how that leads to 10%. I can see it being a good pitch if you’re in the biz of selling on prem servers though
Happy to sell you some Oracle too.
What are the tools to look at first for usable models at that thin compute capacity and how do you go about exploring training for hobby and learning purposes?
https://karlbode.com/ceo-said-a-thing-journalism/
I imagine they are panicking a little with LLMs reducing the reliance on external companies for software support, so they are really hoping to shift it to on-prem AI, which is where the bias in his statement comes from.
The optimal inference configuration varies according to the model, the specific traffic (which changes in time), and the specific hardware it's running to. The "knowledge" for it doesn't exist yet.
Usually in similar scenarios an agent will try to experiment with various configurations and get to a good result. However, you can't have an agent test out different inference serving configurations/architectures, since every experiment might involves 10-60 minutes of spinning up a model (on very expensive GPUs)...
DeepSeek v4.1 Flash or GLM 3 are already very very capable models.
Only bottleneck at the moment is the hardware prices.
[1] - https://www.ibm.com/downloads/documents/us-en/17f55bd310d5fd...
If you absolutely need to scale something vertically, IBM Z is the final stop on the line.
X86 is great at hosting an entire farm of applications. You would never want to install salesforce or gitlab on a mainframe. That would completely defeat its purpose. If you've never encountered an appropriate workload, IBM might never make intuitive sense.
Also, the roadmap:
https://newsroom.ibm.com/2026-08-24-ibm-unveils-next-generat...
Honest question: What would such a workload look like? Why would Salesforce or GitLab not make sense on a mainframe?
Lots of work went into integration and it never went live.
For one thing, amongst many, IBM mainframes process 70% of global transactions on a value basis
Cerebras with Claude 8?