> But I also think the demand for "fast/cheap/good-enough" models is just about to take off.
There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on.
Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do.
I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.
swatcoder
Yes.
The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words.
The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of restrained but efficient model+harness-tuples that have been distilled, finetuned, and rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks with incredible efficiency and erogonomics.
theendisney
At work i only had early copilot which was hysterically bad at everything. As i wanted it to do the same task repeatedly and could spot wrong results instantly i kept evolving a prompt that attempted to correct all ways it found to do it wrong. It kept inventing new ways to get it wrong until it eventually got it right 90% of the time. My theory is that an avanced model that has no issues with a task could do the prompt enginering much better than i ever could. You could for example run x different queries that all do the same thing and compare the results y times. If there are >1 correct results and the wrong versions are all unique you should be able to drill down to a valid result with even a truly shit model running on a potato. Basically what humans do.
59nadir
Early Copilot was tab-complete in editors and was honestly the best version of LLM-assisted development I've used, because it was intentionally small in scope, trivial to verify output from at a glance and easy to opt in and opt out of. I have yet to see anything more useful in terms of code generation; very small-scale code generation (think function-level) comes close, but is a lot more tedious.
For analysis, bug hunting, overview and some porting work to popular languages and so on I think the current SotA is fantastic, but they're still very disappointing for code generation past function or small module level.
jodrellblank
> "I have yet to see anything more useful in terms of code generation"
Recently my employer hooked Claude into a bunch of SaaS services like a ticketing system and an asset inventory system, and I feel a big productivity boost is that I don't have to dig for their open browser tabs, remember their product name/URL/where I put the bookmark, find that my sessions have logged me out "for my protection" since I last looked, find my MFA code, approve sign-in on my phone, then use their mismashed/sluggish/poor-UX interfaces and slow searches to loko for things, and can instead ask Claude "search these systems for anything about <issue>" and it just does. It correlates between systems, summarizes things, and gives me references which system and where to look to check what it found. Internal search on a mess of acquired and disjoint systems.
This feeling reminds me of two classic Joel Spolsky blog posts below, and your comment here about the surprisingly useful tab-complete gives me a similar feeling. I wonder how many smaller candidates there are where LLMs are more helpful and less hyped, against the big vibe-coding solve-everything hype which are currently less helpful than their hype?
References [1]: "A lot of us thought in the 1990s that the big battle would be between procedural and object oriented programming, and we thought that object oriented programming would provide a big boost in programmer productivity. I thought that, too. Some people still think that. It turns out we were wrong. Object oriented programming is handy dandy, but it’s not really the productivity booster that was promised. The real significant productivity advance we’ve had in programming has been from languages which manage memory for you automatically. [...] Whenever you hear someone bragging about how productive their language is, they’re probably getting most of that productivity from the automated memory management, even if they misattribute it. Sidebar: Why does automatic memory management make you so much more productive? 1) Because you can write f(g(x)) without worrying about how to free the return value from g, which means you can use functions which return interesting complex data types and functions which transform interesting complex data types, in turn allowing you to work at a higher level of abstraction"
and [2]: "I have seen many language and programming fads come and go. But there’s only ONE, that’s right, ONE language feature I’ve ever seen that actually improves your productivity significantly. No, it’s not object oriented programming; no, it’s not intentional programming or assertions or programming by example or CASE or UML or XML or Java. The only thing that improves your programming productivity is using managed code – that is, using a language in which memory management is automatic."
I think this sounds fantastic, and I welcome all developments in these types of areas where you lean heavily on the hyper-attentive, very fast assistant aspect of LLMs intended to make information more easily searchable, visible and the results more accurate.
I agree that the mismatch between the expectations of OOP as a paradigm and this situation right now feel very similar. I suppose the many billions pumped into selling vibecoding and large-scale code generation via LLMs of other kinds is sort of like the massive amount of money pumped into marketing Java as a viable alternative long before it actually was.
andersonpico
we could have had that before, but in past era of tech companies, APIs were not designed with an individual needs in mind and management was all too happy to order you to update status (manually) across JIRA, Slack and whatever other tracking system they were using.
theendisney
Just like with human langages their only advantage is the community around it, the culture. (Im pretending size is a part of that Obj) No one does it but that is how you should pick it.
smartbit
Trying until you find something that doesn’t fail is the model used throughout evolution & engineering, think of bridges collapsed, airplanes falling from the sky or exploded steam locomotives.
In engineering you learn from these mistakes and try never making them again. Do we want to go through this evolution every time we solve a software issue? Just because we can with an unlimited number of cheap tokens? I think not, I’d rather use the knowledge build up that also knows about the edge cases forgotten to test. Or better, use multiple models that evaluate each other, as Entropic describes it in their recent report https://news.ycombinator.com/item?id=49316271
> We expect that agents coordinating in the wild will act in higher variance ways than we see here, because they’ll have different backgrounds and therefore different contexts. They also, presumably, won’t all be Claudes.
parasti
I've done this. Ask Claude to permutate a prompt and run Claude Code or a subagent to observe effects until success. Claude will iterate on the prompt faster than I could and likely catch more edge cases, too. It's one of those tasks where the end goal is very clear and the agent just needs to iterate on permutations. Arguably this is a perfect match, too, because prompt engineering is really just text generation, so why not have a text generation machine do it.
miki123211
Yes! As long as you have some criteria to judge the final answer, you can do a kind of "prompt-side RLVR", where you have the model generate prompt changes, try a bunch of different prompts and see which ones improve the results.
You don't necessarily need a bigger model to do this.
jimmaswell
This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always performed the best at all tasks. Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.
The Bitter Lesson is very popular right now. It seems true right now. It’s having its moment right now. That doesn’t actually mean it’s axiomatically true.
Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LLMs could eventually outperform stockfish. It’s not true today and it remains to be seen whether it will ever be true. For now, specialized models are absolutely better at specialized tasks.
Evidlo
This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess.
I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.
kmeisthax
The Bitter Lesson says that the only things that scale are search and learning.
Stockfish is the best chess search engine we've got, and you can learn some good heuristics for chess search policy that will make time-limited chess search a lot more powerful. That's perfectly in line with the Bitter Lesson.
In contrast, LLMs playing chess are relying solely on learned behavior. The inference harnesses surrounding them aren't designed to do chess things, they're designed to do autoregressive token decoding, which isn't a search process. Reasoning traces can resemble a search process, but they're far less efficient - the LLM would have to work out each legal move, test each one, calculate a score, and simulate minimax over all of that. Assuming the LLM is smart enough to even do all that.
A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.
Dylan16807
> A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.
Now let's look at the bitter lesson again. It says that general methods that leverage computation are ultimately the most effective, and by a large margin.
That's different from just saying to leverage computation (which is how I would interpret "unlocks more search/learning"). If the lesson is "more computation wins, when sufficiently channeled" you're basically looking at a truism. Of course more computation beats less when it's used right. The bitter lesson is about abandoning specialization in order to get more computation, and while there's a couple ways where that helps with chess, there's a lot more ways where it's counterproductive. It looks like it's more true for Go than it is for chess, and that it's not universally true. It probably correlates with the state space.
nickysielicki
The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.
nl
> a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games
Not really, if anything it's closer to the opposite. The Bitter Lesson essay literally has this as an example:
> These researchers wanted methods based on human input to win and were disappointed when they did not.[1]
and
> Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale[1]
The actual bitter lesson is this:
> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.[1]
Applying to the "LLMs-for-chess" example the bitter lesson approach would be to put many, many more games into the LLM.
> These researchers wanted methods based on human input to win and were disappointed when they did not.[1]
This was/is basically a strawman though. Like maybe "human input winning" was desirable for chess masters but for computer science wonks? Not the point or the disappoint. It's always neats and scruffies fighting about using some kind of recognizable method (logic) instead of magic (ML).
> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning.
More to OP's point I think: nowadays when someone wants to beat you over the head with the bitter lesson, they aren't as careful to include learning and search. They want to say learning leads to intuition (magic) whereby we can avoid work (logic/search), and maybe argue or assume from there that neats and scruffies is settled. TBF, something like reasoning in latent space does resemble intuition!
But the real lesson is confirmed every time we bother to check, and not very bitter for anyone. Search/learning/logic are ALL always necessary on any sufficiently difficult problems, and hybrids that interleave always outperform everything else. Stockfish being the example in this thread that different camps of absolutists would like to claim, but also all the MCTS examples, evolving examples, and new hybrids all the time. My favorite lately: https://arxiv.org/pdf/2511.08983
jhbadger
There's two different goals to AI research - one was to get results - a chess engine thst wins, etc. But the other goal (which seems to have been abandoned in the deep learning era) was to use AI to help understand how human minds work. A chess engine modeled after human grandmasters is much more interesting in that regard than either a min-max algorithm like beat Kasparov or modern deep learning engines.
nl
> This was/is basically a strawman though. Like maybe "human input winning" was desirable for chess masters but for computer science wonks?
Oh no!
The whole field was full of people whose entire career was built around the idea of developing smart priors.
To quote Wikipedia:
> For computer vision in particular, much progress came from manual feature engineering, such as SIFT features, SURF features, HoG features, bags of visual words, etc. It was a minority position in computer vision that features can be learned directly from data
This undersells the change though! David Lowe's reputation as the best image researcher in the world was based on his SIFT patent[1]
This approach worked until 30 September 2012.
That was a bitter day for many, many computer science researchers.
Is being wrong/ignorant about whether/how something can be automated the same as having a preference for doing it manually? Maybe so if it's your patent, your thesis I guess..
But as it relates to more/less magic, maybe the more modern lens on this is e.g. https://arxiv.org/html/2505.11581v1 . Is manual feature-engineering more like what you'd evolve, or more like what you'd get from SGD ? Feasibility and performance is always a question, there are others like what is robust, stable, adaptable, predictable, explainable. Maybe the manual-features people were interested in something besides the manual part? Maybe the story isn't so simple, and maybe it's not finished yet.
iainmerrick
I think you have it backwards.
The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data.
Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than human intelligence. The LLM is based on a massive data corpus, sure, but the amount of data specifically about chess in there pales in comparison to just playing billions of games of chess.
TwelveEyes
Also, training it on chess books is literally training it on human knowledge, and not the actual game, which is exactly what the bitter lesson says not to do.
Dylan16807
> I think you have it backwards.
> maybe if we use a blend of raw data and hand-crafted heuristics
I don't follow. They're suggesting giving raw chess data to the LLM, no heuristics involved.
iainmerrick
I was replying to this:
The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games.
If you can draw any lessons from chess commentary, I think it’s very reasonable to call it “hand-crafted heuristics.”
Dylan16807
Hand-crafted even if you're feeding in the raw commentary? That seems like a weird way to consider it. Wouldn't that make LLMs in general "hand-crafted"?
And raw games plus raw commentary is all the data you have. You can make more games but those can be fed to both stockfish and the LLM competitor. So it seems like a valid interpretation of the bitter lesson to me.
iainmerrick
Yeah, "hand-crafted" is a bit of a stretch; I mean their value is in the human insight they contain.
The key point I was trying to get at is that the human insights don't contain anything that can't be mined from vast amounts of gameplay. Every human insight can eventually be rediscovered and made rigorous by data (in chess, at least!) In the short term, those insights are useful, but in the longer term, they add nothing at all.
Note also that "raw gameplay" here can mean new games -- you can generate as much data as you need, you don't need to rely on real recorded games.
Dylan16807
> Every human insight can eventually be rediscovered and made rigorous by data (in chess, at least!) In the short term, those insights are useful, but in the longer term, they add nothing at all.
But isn't that the bulk of what we're shoving into LLMs, and it makes them much smarter? If it's useful there but not in a chess AI then that seems like a significant crack in the bitter lesson.
> Note also that "raw gameplay" here can mean new games -- you can generate as much data as you need, you don't need to rely on real recorded games.
Yeah I mentioned that, generated games are useful. But if we're being fair and letting both AIs use generated game data, does the more general LLM ever actually overtake the specialized stockfish like the bitter lesson suggests?
Another way to look at this is that giving the LLM the commentary is a way to avoid complaints of hiding data from the LLM, since it'll have strictly more info than stockfish. But if we cut that from the training data and only give it a basic description of chess and lots of raw game data then it's going to get even worse than it already is at chess. Meanwhile stockfish never had that commentary, just actual hand-crafted heuristics and training on game data, and it's very strong.
camuel
It's the exact opposite.
The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks.
Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson.
Computer chess is the canonical example of this.
klipt
But the harness still matters.
In the case of stockfish, the harness is a tree search around the neural network evaluations.
inigyou
Denser sampling only seems useful if the problem domain is in some way smooth - interpolatable. If you run it on a fractal problem domain you just learn more special cases. Chess is fractal.
antihipocrat
Maybe a future frontier LLM could approach the problem by first building its own stockfish, then applying the subsequent results
jurgenburgen
Or maybe an LLM could just tool call stockfish and doesn’t need to have more than a basic understanding of chess. The bitter lesson seems extraordinarily wasteful on the compute side.
catoc
Maybe a future LLM after that could approach the problem by first simulating a human brain, then learning from the ‘human’ gameplay.
Just kidding of course
zarzavat
Chess is a brute force search problem. Humans are not good at chess, even a small computer can beat Magnus Carlsen.
It would be better to compare models at how well they can write the code for chess engines, otherwise it's just saying that Fable is not a good CPU emulator, which is obvious.
wavemode
Both you and the parent commenter seem to be misunderstanding the point the Bitter Lesson paper makes.
The Bitter Lesson is about general-purpose algorithms vs. specialized algorithms. Historically, chess engines were programmed to look at a chess position and use positional understanding (imparted by the human programmers) to decide what the best move is. But eventually, the chess engines that actually became stronger than humans were instead programmed to just check every possible move and countermove and see which ones lead to a win. (I'm oversimplifying, but you get the point.) So even before Stockfish contained a neural network, it was considered an example of the success of the Bitter Lesson.
As it applies to AI agents, the Bitter Lesson would predict that the best possible agent would simply possess A) a way to do anything it wants, B) a way to evaluate whether what it did was correct, and C) a ton of compute. Then just turn it loose on your task. (The fact that the "brain" of the agent is an LLM is kind of irrelevant - you could also imagine the brain just being a program that generates random syntactically-correct code. What the LLM achieves is that, the random generator would take millions of years whereas the LLM is much more efficient at creating plausibly-working code. This is analogous to a chess engine's pruning heuristics.)
The hard part here is B. We've seen some great agentic successes when rewriting an existing project in a new language, since the agent can just use the project's prior test suite as its evaluator. But when developing a new project, you're still figuring out the finer details of how everything is supposed to work. As the old saying goes - writing a spec that perfectly describes how a program should work, is equivalent effort to just writing the program.
mikepurvis
But isn't that really just about giving "front end" models more access to specialized tool libraries, which include models tuned to specific tasks? Like the first model says ah, we're being asked to code something, oh and we've been provided with some example code, let me invoke a tool call to my model the recognizes many languages, that model says that we're looking at ocaml. Okay, I better pass this off to my ocaml model which will decipher the supplied code and make a plan for what we do about the user's intent. The ocaml model recognizes that there are tests in the supplied code, let's have the special testing model have a look at the testing strategy and see how that fits in with what we just implemented, etc etc.
And perhaps at the end it all gets a single pass by a god-tier model for overall sanity and congruence, but the actual work, planning, coordination, and even user interaction was done by cheaper and faster agents of much more limited capability.
BobbyTables2
It’s kinda funny that your last paragraph is basically describing why sparse files, sparse matrices, etc. are used in other contexts.
It really is absurd to ask programming questions to a model also trained about the lifecycle of a fruit fly.
Instead of building small models from scratch, we train an enormous model and use ridiculous amounts of GPU memory. In the end, the whole thing is shoved into RAM because we don’t know where the useful parts are…
We certainly would know where they were if they were just in smaller models in the first place!
squidbeak
You're making a conceptual mistake here, comparing a chess tool to its operator. Deterministic tools produce superior results compared to models in many areas, so we allow models to use tooling.
The correct analogy here is Fable as a second tier player assisting a SuperGM in running stockfish, then assessing its output to identify promising variations.
There might be a limit somewhere that prevents the bitter lesson being axiomatic - for instance where simulations for anything can be exhaustive - so that judgement isn't needed any more as an arbiter. But while there are problems sufficiently complex or large to require a breadth models don't currently have, greater scale and compute will continue to convert to better decision making, and the bitter lesson will remain true (true enough).
edot
Yes, but an LLM will just call stockfish if it needs to play chess … sure if you arbitrarily constrain an LLM to use no tools it’ll suck at chess. But no one is using LLMs in isolation. Even consumer-grade, bone-stock ChatGPT has tools.
fragmede
ChatGPT does not have stockfish as a tool it can call.
Yeah but it can just install it. It writes arbitrary code. It can do whatever you want it to do.
Animats
Good point.
Dumb AIs are needed for customer service. Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server.
Someone
> Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server.
Needs? Customers want something that immediately answers their question/solves their problem, but that’s far away, even ignoring the “run locally on a 1U server” and that that may not be in the company’s interest.
For many companies, that support line is a cost center, not a PR mechanism.
Also “Press 1 for sales, 2 for billing...” has the big advantage that it handles all accents, speech impediments, etc.
Long term I think a solution where a user’s agent trained on their voice, running on their phone communicates with the support agent of a company is where we will end up, and support phone lines will cease to exist.
cjkaminski
Yes, and the technology to improve the interface you described is already available to run hundreds of concurrent instances on a 1U server. The barrier to entry is getting the people who manage those systems to care enough to implement something better.
MrDrMcCoy
Fact. My company's largest partner is CoreWeave, and convincing leadership that we could run it ourselves on partner discounted hardware for a lot less money has gone nowhere.
inigyou
Maybe you need to walk into the office with a $1000 server running a hundred instances of whatever your code does.
MrDrMcCoy
We got rid of all our offices, whole company is remote now.
PEe9bB7D
Maybe depends on how you ask it? Directly, or let it write a chess program? I think the latter can yield way better results.
srcreigh
No. The bitter lesson is about capabilities. GP is talking about efficiency.
GP isn’t suggesting that focused narrow model(s) will be more capable than large model, but that many small focused models can have sufficient capability while being more optimal.
Also, the bitter lesson is just wrong. The bitter lesson is about hand tuned AI vs computational general methods. However in truth today’s AI uses both. We have general compute heavy models which require narrow expert instructions (eg tools internet docs).
LLMs would not be as good without expertly written context, and expert context without LLMs aren’t as good either.
anon373839
> The bitter lesson is about hand tuned AI vs computational general methods. However in truth today’s AI uses both. We have general compute heavy models which require narrow expert instructions
The models are not even really trained bitter lesson-style anymore. That concept peaked during the era of pre-train scaling, back when it was thought that making a bigger and bigger GPT-3 would automatically solve all problems through prompting. In 2026, the most important part of training is post-training, which uses vast quantities of niche, hand-curated data to fit the models for specific tasks in domains like tax law.
ZainRiz
I'd respectfully push back on the framing here.
If you look at value as purely the LLM output, then there's a valid argument that the best frontier models will always be better than fine tuned specialists. (I'm not convinced personally, but it's a defensible claim)
But that misses two dimensions:
1. The cost of acquiring that output
2. What is actually "good enough" for that specialist domain
Not every output needs to be the best to produce value.
And as specialist models increase in cost, their cost/value proposition goes down.
At some point, there's a threshold where cheaper, fine tuned models are "good enough" at the task and also substantially cheaper than the expert models.
That's where fine tuning helps.
Personally, I became a believer in fine tuning after fine tuning a 1B Qwen model as a second pass over my local voice transcription app, achieving excellent accuracy at ~zero token cost and waaaay lower latency than if I'd invoked my Claude subscription under the hood.
DrewADesign
At some point, the idea of cost/benefit analysis in the software business turned into the benefit analysis. The amount of money going into the frontier LLM model game is fantastically ridiculous. Being much better than the free resources doesn’t even touch how much better they will have to be to justify the expense of creating them, let alone continually maintaining these services.
applfanboysbgon
This idea has not failed to pan out at all. I work for a startup that is exactly what GP described, and am set for life because of how wildly successful it is. Notably, we are successful, in a genuine sense of the word: we bootstrapped from running tiny models to larger and larger models on our own slowly improving fleet of GPUs, and now have millions in revenue without a single dime of outside investment. Conversely, you cannot call taking on ~1 trillion in debt and purchase commitments to scale "success". OpenAI and Anthropic are underwater financially. To be precise, they're in the Mariana Trench.
shimman
Are you willing to say the same of the startup or are you still stealth?
wild_egg
Wait, you actually found a viable counter to The Bitter Lesson? Please say more
applfanboysbgon
This is a misunderstanding of either the bitter lesson or what was being claimed, on multiple accounts. Firstly, the bitter lesson is merely about human expertise-tuned algorithms vs. throwing raw compute at a domain. But, notably, it is still domain-specific. No matter how much compute you throw at training an LLM, it is never going to beat a Chess engine at Chess. If you give a Chess engine 1,000,000 compute units and a general-purpose LLM 1,000,000 compute units, the Chess engine is obviously superior at Chess; ergo, there is value in throwing compute units into training models for specific tasks. This is true for within several orders of magnitude of compute, in fact. It's also true that if you give the Chess engine 1000 compute units it'll still beat the all-purpose model with 1,000,000 units, so actually there's a lot of value in training for specific tasks.
Secondly, the bitter lesson is predicated on compute being cheap. There was a period where a hand-tuned algorithm informed by human expertise would outperform a raw alpha-beta search at Chess. Then compute got cheaper, and DeepBlue ascended to the top. Compute is now expensive again relative to the tasks being performed. We are absolutely still in a period where human expertise in training LLMs will outperform a naive approach with more raw compute.
CamperBob2
I don't know much about chess engines; do they still use hand-tuned algorithms, or are they more like AlphaZero, where they learn through self-play to beat any/all possible human contenders? I don't believe DeepBlue was automated to that extent, but it may have been.
In the latter case, the chess example would tend to support the Bitter Lesson, rather than refute it.
I would also be VERY slow to claim that general-purpose models will never be competitive at chess. It wasn't so long ago that transformers couldn't add two-digit numbers reliably without resorting to tool use. They are now as good at "mental arithmetic" as any human savant. It wouldn't surprise me at all to see someone come up with a model that just happens to be really, really good at leveraging the portions of its general training data having to do with chess.
In fact you could argue that AGI demands such a model, if we are to assume that LLMs are a guidepost in that direction.
applfanboysbgon
DeepBlue beat Kasparov with essentially raw compute thrown at alpha-beta search. That does support the premise of the bitter lesson in general. But that does not mean the bitter lesson is correctly being applied here. The point is that even if throwing raw compute at a task is better than careful human-crafted algorithms, it's still task-dependent. The current trend with the people blowing hundreds of billions of dollars is developing an all-purpose model that is everything to everyone, but you don't need hundreds of billions of dollars to create a task-specific model that outperforms their model at a given task.
> I would also be VERY slow to claim that general-purpose models will never be competitive at chess.
This is not the claim. The claim is that for the same amount of compute, a general-purpose language model will never beat a Chess model. I'm dubious, but allow for the possibility that a language model could eventually compete at a top level against humans with enough compute. However, it will never compete with a dedicated Chess model with similar resources. Training a model for a specific task with the same amount of compute will outperform training a general-purpose model with the same amount of compute. This should be common sense, right? The bitter lesson was only about compute over human algorithms, not at throwing compute at a generalised domain over throwing compute at a specific domain.
You made arguments against two claims that I did not make (that I was trying to refute the bitter lesson or that I claimed that LLMs could never be competitive against humans at Chess), so I'd like to ask you read my statements a little more carefully this time.
jmalicki
> Training a model for a specific task with the same amount of compute will outperform training a general-purpose model with the same amount of compute. This should be common sense, right?
I see arguments like this all of the time in ML. There is a large branch of statistical theory that can cast some doubt on this.
Stein's paradox: when three or more parameters are estimated simultaneously, there exist combined estimators more accurate on average (that is, having lower expected mean squared error) than any method that handles the parameters separately.
This goes deeply against "common sense", but is mathematically proven, empirically proven, etc.
If you are just taking a chess model vs. an LLM, you are right.
But if you want a chess model, a medical diagnosis model, and customer service chat model, at a high level the intuition of Stein's paradox is that one combined model will get you better performance than three task-specific models.
Most practitioners don't get this because what amounts to graduate-level stats theory is often brushed aside, and we keep relearning it the hard way.
AFAIK Stockfish still remains a traditional (some-variant-of-) minimax-search engine, but now uses a microscopic neural network to evaluate the positions at the maximum search depth rather than heuristics. I expect any pure ML engine can learn to exploit any fixed heuristics, but now Stockfish is trying to balance both strategies, which still heavily ends up on the side of raw search depth.
dmoy
I don't know anything about the last 8 years of chess engines, but yea maybe 8-10 years ago AlphaZero shit all over e.g. stockfish.
brucehoult
I guess you missed Leela then.
And in 2020 Stockfish 12 adding some NN evaluation. And then in 2023 Stockfish 16 entirely removing the classical position evaluation code.
Bitter lesson #1: don't waste time optimizing code when a faster processor is around the corner.
What countered it: Moore's law stopped working.
Bitter lesson #2 similarly relies on scaling laws that might have diminishing returns wrt model runtime vs intelligence. Runtime matters for turnaround on the problem you're solving.
wild_egg
Moore's Law has nothing to do with processors getting faster. Dennard scaling stopped working but Moore just slowed somewhat, not stopped.
bsder
You are technically correct. The best kind of correct.
However, what most people think of as Moore's Law--CPU speed doubles every 18 months--broke somewhere between 90nm and 22nm.
And even the actual Moore's Law--2x the transistors every 18 months--doesn't hold for all types of chips anymore. Memory only gained 2x density over 10 years.
joe_the_user
I think the gp found a viable counter to the ggp's version of the bitter lesson, which seems so extreme as to certainly exceptions. IE, they seem to say nothing but the latest frontier model is ultimately viable as an AI business.
The actual argument of the Bitter Lesson essay is pretty limited but people's interpretation of it has gradually drifted until it's seen as prediction that current LLM will reach AGI at a large enough scale.
HDThoreaun
The issue is that GP is misusing the bitter lesson. Yes, search + learn tends to be more effective than human rules based strategies, but that's not what's being considered here. The original claim is effectively that AGI isn't needed for most tasks and more value can be created by using search + learn to solve specific problems instead of applying general models to every problem. Then GP commented a non sequitur
z3t4
Do you have a website?
lqstuart
It’s not hard to fine tune a 4B parameter model to outperform GPT-5.x or Claude on a domain-specific task. You don’t even need GRPO or whatever the flavor of the month RL technique is either. I suspect what you’re saying is that massive frontier models are the best for doing big slow tasks on human time horizons—and I emphatically agree—but keep in mind LLMs are useful for quite a bit more than just multi-turn chatbots; and in those instances, throwing infinite compute and latency budgets at a problem is generally not a great way to find the “best” solution :)
miki123211
I think large-scale systems are optimal when the number of tasks is relatively bounded and costs don't matter.
When you're doing something "at scale", you want smaller models, weaker heuristics and rules written in code, as they're much cheaper, much faster, much more predictable and much easier to reason about.
If your task is "tell me which pages on the internet are most relevant to this search query", running Claude Fable to evaluate every single page would be ridiculous. You may run something like it at the end, to synthesize information from a small result set for example, as that's a "human-sized" task.
OtherShrezzing
I think if you start taking cost into account for tasks which models have saturated a long time ago, the bitter lesson stops being applicable.
If your task is "write a function to sort a list of 1mn entries alphabetically", a small coding model will exactly match the performance of a frontier model (it'll 1-shot it, with 100% accuracy). So the only differentiator is cost, and the frontier model will cost orders of magnitude more than the specialised model.
For most intelligence based tasks, you don't (and never have) needed the tool which "performs best at all tasks". You need the cheapest one which performs adequately for your immediate task.
This doesn't mean the bitter lesson is incorrect. At the frontier, it's still correct. It means that it's not applicable at all to lots of tasks.
schrodinger
If anyone else finds that BitterLesson link difficult to read because it wraps full-width, you can type "javascript:" in your URL bar and paste:
(Chrome strips the "javascript:" for security if I were to include it myself.)
xnx
Or use reader mode or resize the browser window.
CamperBob2
Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.
VibeThinker 3B constitutes extraordinary evidence, IMO. The first such evidence I've seen myself. Very small model, very low literacy, almost no world knowledge, but it is as good at math and logical reasoning as models a hundred times larger.
The Bitter Lesson is a valid and trenchant observation about how about we got here, but I think it's a mistake to assume it tells us very much about where we're going. Too much has changed recently and is still doing so.
spockz
So theoretically, if you give that model the means to find information, ascertain the quality of said information, it could still reason its way to an proper answer?
Is this whole thing than maybe a read vs write optimisation again? Spent more time and effort training more knowledge into the model upfront and get it out in a single question instead of training a small model and needing more steps to answer the same question?
algo_trader
> VibeThinker 3B constitutes extraordinary evidence.. math and logical reasoning
Any similar model aimed at coding?
A >10B model for mass spawning/swarming and reporting back to a larger model
I wouldn't use it for anything important without heavy supervision, as it's very weak outside its specialty. Not ideal for instruction-following tasks.
joefourier
> It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always performed the best at all tasks. Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.
Absolutely false. At least when it comes to multimodal inputs, even a simple classifier will outperform the largest LLMs who still hallucinate details or don’t describe audio and images accurately.
And there’s also the issue of cost/inference speed. Running a trillion parameter model for all tasks will be incredibly costly, require a cloud API, while a tiny CNN can be run locally or at a cost multiple orders of magnitude lower.
HoldOnAMinute
Someone will eventually figure out how to package it all into a single, cheap chip
bmitc
That you can then write text to program and make applications with.
noduerme
Honestly, I've been perfectly fine using OpenClaw and Claude 4.6 for the last six months, and am used to its flaws. In general it doesn't screw up anything major, but I also wouldn't ask it to reason about anything more than a few code files at the same time. Since I'm not interested in it replacing higher level reasoning or architectural design, it's pretty capable of just doing the modifications and features I suggest.
One thing notable is that it makes a difference when I specifically refer to line numbers and variable names when I'm asking it to make changes. Meaning, I also read what it writes and understand what it's doing, and have specific queries where I already know what I'd like to change I add.
MASNeo
Sounds like mainframes. But then, these never quite disappeared to the date.
eduramirezh
it feels like if we had invented hammers, and we're still on the "make them bigger, stronger" phase, but we haven't even invented nails yet.
panagathon
For large language models, isn't everything nails?
apatheticonion
Agreed. I've been "guide coding" in my editor (Zed / VSCode) for a while now and it's really enjoyable.
DeepSeek v4 flash has been dirt cheap and so fast that my development loop is;
- small prompt
- review
- small prompt
- review
I build software with the same quality I normally would but it's way faster to produce and I think more about architecture and flows than I do about implementation details. The small diffs let me accept / modify / veto diffs and if the model struggles, I just write it by hand. It prevents compounding defects from leading the model astray (like you see in vibe coding).
In some cases vibe coding is useful, like when the complete specification is available (e.g. creating a JavaScript engine that implements the standard) - but anything that requires iterative development sees vibe coding break down pretty quickly (you could argue that is the case for a JavaScript engine).
I feel energised by AI assisted coding rather than drained, as it's a force multiplier for my skills and it lets me build more than I could by myself.
That said, most of my team vibe codes and reviewing their work is like pulling teeth.
ianmarcinkowski
Basically how I feel, but I use the low-powered API models like Sonnet. I feel more energized and focused on the bigger picture than getting stuck on stupid implementation of micro-problems.
I think the muscle memory of doing those tiny problems is good for our minds, but solving larger-scale issues is also challenging.
I'm on vacation right now and getting claude to build a mostly-throwaway e2e testing harness (admittedly not small-prompt-review-repeat) for a backend API to speed up our existing e2e test suites which do click-ops to set up tests 8-10 years ago, we had a team who spent 3-4 months every year maintaining our E2E suite and people would do rotations on there to spread the knowledge.
I basically want an industry standard practice implemented on my team of 4 devs who are too busy doing other things.
jermaustin1
To me, most local models work just fine for anything you can be patient for. If I want something quicker, I will go to a SOTA model via API, but with multiple 3090s, I have never really needed a hosted model for a lot of my experiments.
For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playing my games.
But nothing one-shots a prototype better than Fable 5. I can have a prototype built in 30 minutes, hooked up to my local LLMs and Claude Code is very good at testing the interactions and even tuning the prompts of the NPCs for better experiences.
__float
"with multiple 3090s" is quite a bit of burying the lede for "most local models work just fine", don't you think?
jermaustin1
Having multiple 6 year old cards doesn't seem like it's that big of burden for local LLMs.
I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060.
My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that price tag seem a lot better than it was, but it was an investment in all things GPU that were happening in 2020 (crypto, blender, image gen), then LLMs exploded.
thayne
A single, used 3090 costs more than I have ever spent on a computer.
9cb14c1ec0
Yes, the tunnel vision around local models on this site is crazy. The percentage of people in the world who can afford the hardware is extremely low.
layer8
It seems roughly similar to the pricing level of personal computers in the early eighties (i.e. IBM PC and Apple Macintosh). I’d expect prices to come down significantly over the next few years. Not so much in the next year or two, but after that.
oblio
Just like for warships, the complexity and cost of building cutting edge hardware has grown exponentially up to a point where a significant chunk of the world's computing is dependent on 2 companies: ASML, TSMC. We shouldn't extrapolate linearly from examples from the 80s.
layer8
No, but I wouldn’t expect it to stagnate like with Intel in the 2010s either. Maybe the biggest caveat is that most people will be fine with using cloud providers, so the market for non-server hardware won’t be subject to as much competition.
inigyou
That's in nominal dollars. However, inflation since then has been about a factor of ten to fifty and it hasn't trickled down at all.
regularfry
There's a non-small contingent who lucked into the periodic games machine upgrade at the right time to snag a {3,4,5}090 rig just before everything exploded. It's a small contingent now but it was less so then. And now those people can add a second card for roughly what that whole system would have cost new originally.
hparadiz
The current supply chain problems will eventually pass.
TeMPOraL
So will the stock of 3090s, as well as their ability to run contemporary local models. And there will be no supply of newer equivalents of those GPUs, because NVIDIA has since wisened up, and is using the very capabilities you need for local LLMs as market segment differentiator.
jermaustin1
I don't think there is tunnel vision. I'm just saying that I have a couple 3090s I invested in a handful of years ago, and they are still going strong today as multiple GPU-needing technologies emerged.
I'm not saying everyone has to run local LLMs, because the APIs are in a race to the bottom, and my $10 of OpenRouter credits I bought months ago is down to $8.94 because most models give you MILLIONS of tokens for a US Quarter.
9cb14c1ec0
> I'm just saying that I have a couple 3090s
This is tunnel vision. The percentage of people who could afford the hardware you could at the time you back it so vanishingly small. I do not know a single non-tech person who has multiple graphics cards in a single computer.
rmunn
And right now the demand for GPU is far outpacing the supply, even with factories at full production, which is keeping prices high and out of reach of most people. But unless something happens to shut down the factories (not impossible, but hasn't happened yet), eventually production will catch up to demand and prices will return to sane-ish levels. Won't happen this year, almost certainly not next year... but I would be shocked if the current high prices were to persist for a decade. Eventually the percentage of people who can afford that hardware will grow to be a decent chunk of the computer-owning population. And they'll be following READMEs written by the early adopters, for installing open-source harnesses to work with open-weight models.
My personal expectation is closer to 5 years than 10, which is why I wouldn't touch Anthropic or OpenAI stock with a ten-foot pole, personally, no matter how high their theoretical valuation is. Because their business model is doomed in the long run.
TeMPOraL
Newer cards aimed at consumer market are not capable of being used for local models the way 3090s are. That's on purpose: this capability is now used to price-differentiate between "normies playing games" and "companies in data center business".
rmunn
For now. That won't last forever. Yes, it'll take quite some time to work through the current production backlog, which is why I'm predicting five years, not one or two. But the trajectory has always been "new video card comes out, game devs push the limits of what it can do, gamers buy new card so the hot new game can run faster, rinse and repeat". And that includes wanting more VRAM so the game can load more of the scene at once, load higher-res textures, etc.
Which means it's inevitable that eventually, even the consumer game market will be buying GPUs with 32 or 64 GB of RAM. And there are decent models that will run at that size. Even the "normies playing games" market, as you call it, will end up with the capacity to run local models. It'll take a few more years than it would have if the data-center companies weren't trying to buy up all the GPUs, but it's not like gamers are going to stop wanting to play games. So in the long run, Anthropic et al are still going to have to figure out how to deal with competition from local models that run on your gaming video card. Which won't ever be at parity with the models that take terabytes of VRAM to run, but are very rapidly approaching "good enough for what most people want to do".
BoxOfRain
It's a decreasing pool as well I'd say, the dev machine I built last summer would make less financial sense to me now for example.
zahlman
I mean, I'm not rushing out to buy that kind of hardware myself, but it is a matter of perspective. People commonly spend an order of magnitude more on a car, and that's just the sticker price.
Karrot_Kream
I keep coming back to this: why do I need to run a local model on my own GPU? Open models can run in dedicated clouds and while, yeah, they may be more expensive per token than my own GPU, when accounting for depreciation, energy usage, and opportunity cost (money not spent on my GPU will instead sit in my portfolio appreciating with its particular blend of returns), I'm pretty sure I break even or even net lose money with a GPU.
Don't get me wrong, there are advantages to a fully local model in that, I can have agents looping 24/7 even when my internet is not working. But this is niche enough that if I had to price the advantages they don't seem worth it.
If I'm willing to pay the Openrouter tax, I can fire up Openrouter today and just get access to whatever model I want, and still pay a fraction for tokens as what I'm paying with the big guys.
wafflemaker
My single 3080 runs so hot I don't need to warm my room in winter, and have to play games in my underwear in summer.
throwaway219450
Unless you value privacy, pay for openrouter. You still get the benefits of cheap tokens and programmatic usage.
3090 pricing is something of a wild card. Since the only big-mem consume cards are the xx90s, and a 5090 is pushing $5000, resale value has gone way up. The bottom hit ~$700 last year. It's still a very good GPU, if power hungry.
zamadatix
I got a great deal on ~72 TB of NVMe right before storage prices shot up, doesn't make it any less ridiculous that I have it or any more relevant to people talking about building a NAS now. 99% of people, even in tech, do not have the stupid amounts of hardware people like us hobby on.
oceanplexian
Most people in the US have a car, and the average new car is $40,000. Hell where I live a middle class consumer will spend double that on a Boat or an RV and think nothing of it. These aren’t elite tech workers.
It’s not unfathomable that if a personal, generally intelligent local AI provides enough utility and doesn’t require you to tweak CLI flags millions of Americans would want one.
shimman
There are many payday loan operators and those willing to sell predatory loans to those workers you mention buying boats or RVs. I've yet to see a payday loan open up in SF to help tech workers buy hardware.
ninglor
Most people in the US don't drive a new car, and used cars can be had for far less than $40k. An $80k purchase would be just shy of the median annual household income -- anyone who thinks nothing of that has financial resources far above typical. You are in a bubble.
vel0city
An $80k purchase is far more affordable when you're looking at an 84 month loan. You trade in your current $20k truck with $30k in debt on it for your $80,000 car, get a couple grand in incentives and a $10k down payment, and boom you're only looking at a bit under $1,200/mo in payments. The median household is bringing home ~$84k before taxes, hypothetical person lives in a no income tax state, they take home ~$5k/mo. Easy peasy, its not like you were planning on taking any vacations anyway since you're always working.
What matters is you've got the Duramax HD King Ranch TRD Big-Boy machine. Doesn't matter the cost. You can tow anything, drive anywhere, do anything, and do it all in comfort. Other than parking in a normal parking spot comfortably. Or even park it in your own garage at home.
I've seen this exact scenario many times personally.
duskdozer
Most people in the US can't feasibly hold a job, get groceries, or go to the doctor without having a car.
spockz
Spending that kind of moment on a product that gives you personal happiness for years up to decades and then will still have residual worth, which people save up for ages for, is an entirely different proposition than buying a product that may make you faster professionally, but which in the short time can also be achieved by a few dollars worth of subscriptions to a hosted model for even greater effect.
kevin_thibedeau
> people save up for ages for
Americans by and large don't do that. Much of the population engages in discretionary spending with debt instruments. Combined with mass innumeracy, they're all oblivious to the true cost of their purchases because they only think of the monthly payment.
sroussey
where? i would love that.
zamadatix
"Where'd I buy it" or "where is it now" ;)?
It was a 96 core gen 4 epyc+supermicro board build with consumer NVMe drives on 1x16->4x4 "dumb" bifurcation cards. I had to get a few MCIO-> PCIe adapters as well to get the full lane coverage. Mounted in a standard EATX compatible consumer case with a consumer PSU and a lot of Noctua fans - surprisingly cool and quiet for what it is.
Motherboard+CPU I got from Ebay. Rest from the best MicroCenter/Amazon/Walmart deal of that day. Bought juuuuust before the AI pricing apocalypse, largely by pure chance.
xnx
> 2x 3090s
You could sell those and have enough money to pay for hosted inference for years.
jermaustin1
They cost more to run than hosted anyway. But that isn't the point of having them. They are a playground, a backup when the internet is down, or claude is down. They can render Blender scenes pretty well. They play any game I want.
You can do each of those at various hosts and own nothing. Or own a couple "over priced" cards and do it all at home on battery power for a few hours while the power is out.
robotresearcher
For me it’s more that you can show them your financial and medical data without BigCo looking over your shoulder.
lostmsu
BigCos you are referring to are earning their money doing state of the art research, not so much from your financial and medical data. That was the age of Internet ads, which was over since AdBlock was created for anyone concerned.
Gecko4072
But after all those years you’d still have 2 3090s, which are now about 6 years old and still holding value.
irishcoffee
I keep seeing this comment. This is _hacker news_ where, back in the day, people just hacked on things, because it was a hobby. They weren't "moneymaxxing" or desperately trying to be as insanely efficient as possible. They hacked on stuff with a can of surge at 3am because it was fun.
Your comment is like a meta comment of "LLMs are generating everything, after a while the ouroboros will eat itself. (Which I agree with)" If people aren't hacking on this shit just because, you have completely conceded control of software to a handful of sociopaths, and open source software is dead.
deaux
Back in the day the business backing this platform wasn't incubating companies like Flock (YC S17). I think the increased focus on "moneymaxxing" in the community reflects a similar change by its owners.
bitexploder
Not really. 2 years ago that was a pretty normal amount of GPU hardware for a hacker or gamer. It's all relative. They are not accessible to most people yet, but for someone that cares and is a technologist? Likely accessible.
sroussey
I have trouble getting simple extraction to work sometimes. I have a block of text describing people and their roles at a company and their ages, and i asked for structured results of an array of these things with the text span that it appears in and all i can say is: nope.
jermaustin1
I've done pretty decent local prose->json extraction using Qwen and Phi and Gemma.
I'm sure most of it comes down to prompts, and all of them run over 100tps on a 3090. Smaller cards will likely be slower, but Qwen3.5 9B is small enough to fit on most consumer cards.
keeda
Yep, I've been having excellent experiences with the models even from the 2023 era. They required a lot of "holding it right" (mostly: being very precise in what went into the context) but their raw coding capabilities were astonishingly good even then.
However, back then I was getting the AI to write individual functions or classes or a test suite. I was decomposing the larger task into smaller tasks, delegating some of them to the AI, reviewing the results and composing the codebase from those. I was also essentially the harness.
Today the models can write and test and deploy an entire project. In terms of the code quality, I actually don't think today's frontier models would have written it much better than the 2023 models did. So in terms of raw coding capabilities i.e. converting a high-level specification into working code, I think we hit the peak way back in 2024 itself.
What has changed is the AI has learned how to do the task I was doing (besides being the "harness"!), which was the mid-to-higher level "engineering" aspects like decomposing a task, specifying it to a reasonable level, reviewing the outputs, and course correcting as needed.
I'm not sure if that is something the AI labs explicitly focused on during training (which may be why Meta is having its highly paid engineers do annotation work), or an emergent property of "better reasoning" (which I believe Dario implied in a podcast), or some mix of both.
But the fact remains that even the weaker models are more capable than we realize, and many being open weights, are here to stay.
mw888
It does have to be said that if LLMs keep becoming better coders at some point the bottleneck on quality is prompting. Good ideas have many hidden assumptions you think are procedural but often are pivotal to your broader vision.
I find that when I give an LLM my full handcrafted codebase, it does very well. It follows my conventions, sees the intent and can coherently build within its scope. It writes much better code than a 'vibe' prompt.
It is always tempting and I myself will continue pushing the boundaries, but when you keep an LLM in reasonable scope (that may be one line, function, file at a time, depending on your idea of reasonable), you, by definition, can get sound utility out of them.
LoveMistral
Same. Mistral 7b has been more than I ever needed for text for years now.
Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid.
Mistral 7b can do anything, and it’s basically instant even on an M3
frigidwalnut
Sounds interesting. Can you give more details on your workflow and what tasks you use it for?
LoveMistral
Code, creative writing, email summaries, automated email replies, and I prefill my invoice notes and daily updates for work.
Actually built a full invoicing product for that, using it too.
I use Mistral 7b and LlamaIndexTS on Node, I run it on a MacBook M3 and on a Linux server with only 8GB VRAM (old gaming PC).
Basically flawless, runs very fast and I don’t even know what paying for “tokens” is :)
Almondsetat
What kind of work are you doing? For example, if I have some code in the hot path and I want to do all the usual tricks to help the compiler vectorize it, such a small model is not able to do much.
LoveMistral
RAG is your friend (or any vector db). No model can vectorize an entire codebase in context.
Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details.
What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response.
If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use.
usef-
Out of interest, have you tried the newer models? You are not describing my experience recently.
LoveMistral
Yes - you are experiencing a mix of context caching and db retrieval from these mainstream model experiences.
Even the best models available lose a ton of detail over time if you were to paste in tens of thousands of lines of code.
The only way to hold huge amounts of context with a high degree of accuracy is to store it using various mechanisms (one of which is RAG).
On “effectiveness”, I mean end use case effectiveness in the tasks at hand, not whatever benchmark the model developer or vendor themselves come up with - which may or may not be useful to the work I’m doing.
casper14
What are some limitations you have found with using a smaller model like that?
LoveMistral
As a developer, none related to the model. It’s faster, lighter, makes less assumptions.
For a casual consumer, it is no doubt a worse chat buddy. Knows less historical facts à la Wikipedia.
But I’m not really using LLMs for that kind of entertainment and I don’t rely on them for fact-checking anyway. It’s almost worse to rely on a smoother talker for something it can’t possibly know.
For feeding in thousands of daily updates and getting it to predict the next one based on a crude list of tasks, it’s great. For completing code files in my style, also great. It can also handle most small customer service issues and refunds related to my apps on App Store, the back and forth to collect data from users.
For a booking engine I run with a partner, it is great at confirming bookings and following up. It handles cancellations which are about 40% of confirmed bookings due to the nature of the business. I used to feel like I was wasting my time with them - there is no way we can charge for it. Having basically a smart bot do it makes everyone happier (even the customer who knows they aren’t hurting anyone’s feelings).
For invoices, I prefill notes and things based on previous ones. It’s fine. I don’t need a massive model that takes 30s to reply etc. it’s way overkill. Maybe worse than overkill - off in a direction I don’t need.
ksec
While they are improving rapidly, or as you say even if they don't. The next stage is for hardware companies ( cough Apple cough ) to ship these Local Model ready hardware in their products.
It will be interesting to track the improvements of these 7B model over time.
There will be a turning point in the next few years where it attract enough consumer attention to create yet another Smartphone and PC super cycle.
QuercusMax
Just being able to instantly generate a complicated query expression to pull specific bits out of a JSON blob sold me. It's awesome that I can ask Claude to build a whole feature and it will often one-shot it for me, but generating utility bash / python scripts or little throwaway utility webapps is what really excites me.
riazrizvi
I think there's something subtle about language and ambiguity that means they aren't designed to become superintelligent autonomous machines. They're value is as information repositories that actual intelligent autonomous machines (us) mine and string together.
dgellow
Yes LLMs are a beautiful way to compact knowledge. It would be such a cool technology to develop and worked with if it wasn’t linked to such a toxic industry
riazrizvi
I think you're just observing ppl in one of these rare instances where enough of them come together because they are motivated. 'Toxic' is the clamoring sound of a crowded room where what gets through to your ears are just the most annoying snippets of incomplete conversations. I dare you to hang out with any actual people here, understand their viewpoint and listen to what they actually have to say in person, within the context of watching them do it.
pjc50
The "toxicity" was:
- ingesting all the current knowledge without regard for intellectual property or the work of people that went into it; then
- claiming that AI would make all those people who put in the work redundant
It's not really surprising that when the sales pitch is "this will eliminate human creative work in all writing and illustration centric industries", people got angry.
riazrizvi
These are great examples of converting the snippets into imaginary ppl. Thanks.
dgellow
I know those people. Lots of them are fantastic humans. That doesn’t change the fact the AI industry is extremely toxic
shakna
By annoying snippets of incomplete conversations, do you mean Musk's clamoring for legalisation of child porn, or Altman's obsession with doomsday in talking about AI, or Zuckerberg's "Aut Zuck Aut Nihil"?
The people involved at the leadership, in this wave, or incredibly toxic individuals. When the leaders behave a certain way, it tends to be reflected by those following after - intentional or not.
eqmvii
I see it in a slightly opposite way: even the good models are relatively cheap, and so I worry what we might miss by spending too much time playing with the Sonnets of the world when the Opuses are still objectively a bargain for the power they bring.
zahlman
> when the Opuses are still objectively a bargain for the power they bring.
The cost isn't just what you're billed. There are security, privacy etc. concerns.
Foobar8568
I know companies that are using github, even using public repo, and request their teams to not use SOTA models, but are ok with local models.
Just stupid policy.
zahlman
If you're writing open-source code then there's obviously nothing wrong with publishing it in a public repo. "Using Github" doesn't require you to use their CI, but even then, a human managing secrets for GitHub CI is worlds apart from trying to make sure an internet-connected agent doesn't leak secrets. And if you have sensitive data that you can't send to a remote model but you would benefit from the technology, then processing it with a local model without network access is the obvious way to address that.
avadodin
If Orang mane bans GitHub they've got their local clones and can whip out a local server and a CI solution.
If Orang mane bans Claude, they've got their local models.
The latter has already happened too so I'd say their risk modeling is spot on.
nowittyusername
There's A LOT low hanging fruit still out there for sure. And with antigenic systems being able to do the boring repetitive work of looking for that low hanging fruit I think we will see interesting things indeed. Also I think heuristics is where its at for such things. Once you describe some good heutistical structures for the research models to always follow related to "creativity" and such things, thats where we will see biggest difference. The agentic systems know the scientific method well and can follow it they just need the ability to be "creative" so their sampling becomes less rigid.
dominotw
ppl keep talking about the supposed unexplored and untapped "model overhang" but very few things in the world are where you can write elaborate test criteria to before using ai.
A sales person sending a prospect email doesnt have a way to write a test harness for it. Yet these tasks dominate what humans do compared to writing a crud app . otherwise anthropic wouldnt have trillions dollar valuation
cyanydeez
I've amassed access to 4 different GPU rigs with 128GB to 72GB; I didn't this before I event touched an agentic engineering harness. It was sometime in February/March when I set them to first tackle small problems, and now with deer-flow, they're scaffolding full project/scope implementation and I'm finishing off the fine details around the problematic edges.
gozzoo
> We have, right now, access to things that 10-20 years ago would be considered magic
These things would be considered magic even 4 years ago!
Der_Einzige
BTW structured/constrained generation has so many places to trivially enable jailbreaking/alignment/safety problems that closed source models heavily limit the full expresivity of grammars and capabilities, particular of on-the-fly dynamic grammar construction/reconstruction.
viscousviolin
If someone has an old GPU laying around, say a GTX 1080 with 8 GB of memory, would that be enough to get a (small?) local model running?
tobiasu
Sure. The fastest small coding model is probably Mellum2 12B-A2.5 by Jetbrains. It matches or beats all Qwen models in this class.
Can even run on a notebook CPU and comes in Base (best for FIM), Instruct and Thinking variants. mradermacher has imatrix quants for people who can't run it at Q8.
IQ4 should fit, but even if it doesn't, llama.cpp has options to partially offload models to system memory.
trucks-refinish
I'm able to run qwen 3.6 35b a3b with my rtx 2060 super which also has 8gb of vram. It's a very capable model, not as smart as a frontier model but it is still very useful.
I use an 6bit quant and get around 20 tokens per second.
bityard
A small model, yes! But not necessarily a good model.
With the additional caveat that I don't know whether that specific card is supported by modern drivers.
You'd be looking at one in the 6B or 7B parameters range at FP8. Or smaller. It's been quite some time since a recognizable company in the AI space released a model that small. You can try larger model that has been quantized down to that size, but they don't always fare well with that.
Modern text-to-speech and speech-to-text models also fit well into modest amounts of VRAM.
avadodin
You're arguing for a very specific range of weights but many slightly smaller and slightly larger models have been released including QAT and MoE versions.
An old nVidia brand card with 8GB is more than enough to see those models running at usable speeds and accuracy.
bityard
I wasn't arguing for or against anything, just trying to help answer the person's question.
andsoitis
> One thing a few investors I've talked with have mentioned: "It's weird we're not seeing more consumer AI companies. Why is that?"
What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start.
Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do the hard part of understanding specific consumer needs and wants and pursuing that.
nonethewiser
There is a thin line. Lots of AI powered things are just workflows you could implement with claude code and some skills. Or even just a prompt on the web chat. But I think these big technological swings sometimes take a long time to shake out. Society is still adapting to the internet.
I think there is more opportunity for consumer application of AI. Im surprised we dont see more models in game. Small models that do more purpose driven things not "this npc is just a chatbot with a personality in its system prompt." I mean we literally have systems in games called "AI" and we dont think these much more powerful AIs have a place?
abustamam
I don't know much about game development but given how intensive some AI models can be i feel like adding LLMs to games can have performance issues, whereas in-game AI for computer controlled characters works in-enginer.
chillfox
The average game dev cycle is about 7 years these days, so expect to wait some more years before you see llms used much in games.
marcelo-earth
And almost all consumer software has AI now.
> But what if you want to add AI to your product? Well, now you have some real inference costs on every request!
Eventually, these companies just lower their costs by using more efficient models. There were consumer companies built on GPT-3.
embedding-shape
> There were consumer companies built on GPT-3.
I don't think anyone is doubting this, but were they any good? Remembering back my first interactions with GPT-3, it was fun, but hardly reliable nor "ready for production", besides making cool demos.
pjc50
> almost all consumer software has AI now
.. whether you want it or not!
We're at the point where it's becoming a product sticker like "fair trade" or "does not contain nuts" to say a product wasn't made with AI.
shostack
Yeah, I think if we reframed it as "why is there no consumer AWS?" It would make more sense.
AI is a utility that can abstract code to such a high level it is indiscernible from natural language.
dearilos
> What would consumer AI company even be?
Something that helps homeowners build anything on their property, for example https://myhomecat.com
maniflames
What purpose does AI serve here apart from lowering costs by generating a plan?
Especially if I need to ‘consult an expert’ isn’t this just a customer company using AI?
dearilos
You can plan and design your projects without spending weeks/months on research and understanding how construction and code compliance works. It'll also help you save time/money from having to make extra runs to the hardware store or if you end up making the wrong cut/connection if you DIY the first time.
You can say what you want, and the tool has appropriate guardrails to make sure what you're going to build is safe.
NickNaraghi
> Across his various startups, Peter has seen two kinds of work:
> 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of.
> 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts.
I'm curious about not only which of these roles models will fill, but also how they will empower us to be in the mode we prefer.
kridsdale1
I conceive of this as Protoss vs Zerg. I’ve had a lot of success in my career following a Zerg strategy.
theshrike79
That's been my goal too, know a bit of everything - enough to know it exists and maybe Know A Guy I can ask if I need to know something more specific.
Now in many cases the "Guy" is an LLM. I know enough to know when the LLM is bullshitting me and when I need to consult an actual human.
drxzcl
We call it Ravenclaw vs Hufflepuff. I guess it's a generation thing.
leoh
Zerg rush
charv
> July 2009
> Thanks to Sam Altman, … for reading drafts of this.
Oh how times have changed!
satvikpendem
Have they? Some say he's still reading those drafts, along with everything else on the Internet.
swiftcoder
I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back
jbjbjbjb
I’ve been playing around with Luna, Terra and Sol and for the type of work I’ve been doing lately I actually think Sol is just a likely to trip up as Luna. Examples were Sol over assuming, persisting in the wrong direction, over engineering a little script to do some exploration of api. They can all be fixed but it’s a waste of tokens, I rather have Luna do it because course correction on small pieces of work is cheaper.
scoring1774
I've found the distinction to be in how much I care about how the final product looks. If I want high-quality code I typically find a smaller model with a well-designed spec to do better, if I want it to just run and produce something close to my vague description typically Sol does better. For most actual business use-cases I think the first is likely better but the experimentation speed up with the frontier is very nice.
mchusma
I have an agentic workflow and Luna just always gets stuck, SOL and grok 4.6 don’t. I like Luna in theory I just find not much practical work for it yet in coding type work.
Now I think Luna is plenty good for many applications inside a very good harness/scaffold. And I think there are a lot of those usecases. So I think these small models are really good for application developers.
But for entrepreneurial knowledge work all of my work still benefits a lot from more intelligence.
ZeWaka
I've found Luna to be quite capable.
SomeonesAccount
Exactly! Composer 2/2.5 were amazing, cheap, and fast. Everyone else was Gaga about GPT 5.5 and such, while we were over here doing the work with less cost and more speed
sickcodebruh
Composer 2.5 is phenomenal for so many tasks!
edg5000
Sol is 100 EUR/m and I prefer it over Fable, and you get practically unlimited usage unless you go really overboard with it. So I don't get the cost argument. Would be better to own hardware, but with the shortages that's not very attractive atm.
samspot
I received access to AI in May of this year and used smaller models a lot in May, June, and July. Haiku 4.5 is terrible for coding. This month Luna was added and it's incredibly capable. I have to reach for a better model 1/50 times instead of 1/2 or 1/3 with Haiku. Luna is better and cheaper than the Haiku and Sonnet models I've been spending most of my time with.
It's safe to say I didn't come to this conclusion because I was "addicted to chasing frontier models"
jlkuester7
Exactly. Even 32b parameter models you can run locally on consumer hardware are "good enough" at this point for some workflows!
dominotw
no they are not good enough for "most" tasks
kccqzy
> for most tasks
The word “most” is doing a lot of work here. On a percentage basis perhaps most tasks a typical SWE needs to do when they aren’t in meetings or writing docs are just glorified autocomplete. But that’s boring and that’s why people don’t usually talk about it.
People are addicted to chasing frontier models because they all have memories of spending a week on a deeply challenging algorithm problem or even have crazy complicated algorithms they cannot implement themselves and want to have the models achieve this technical breakthrough. It’s the kind of productivity boost from spending one week on a problem to spending one hour. In contrast the productivity boost from spending ten minutes to spending one minute just doesn’t occupy people’s mind.
swiftcoder
> crazy complicated algorithms they cannot implement themselves
I'm not sure I know very many engineers who would fall in this bucket. Or do you mean the business types who suddenly think AI can replace all the engineers?
kccqzy
It probably depends on the background and the company. For example if one works at a startup that happens to use technology, it’s unlikely to happen because SWEs just translate business rules to code. But if one works at the place where the technology itself is the focus, then yes most people will fall in that bucket.
In fact I noticed that this is the one place where people discussing AI on HN tend to talk past each other. On the one hand people are talking about supreme intelligence like designing new algorithms (on the same vein as finding counter examples for the Jacobian conjecture) and on the other hand people are just satisfied using AI to automate a few quotidian tasks that hitherto couldn’t be automated.
michael0church
It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model.
There are many applications where world knowledge is unnecessary or even a negative, and in which only a small amount of language skill is necessary, and there we can expect small models more intelligently used to beat large ones naively used.
giraffe_lady
Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks, even ones far outside the domain it was optimized for. IE claude fable is better at writing both code and prose than smaller code- and prose-specific models.
The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge for local tasks" type dreams.
And to be clear I'm not saying that smaller models don't or can't work well, or that we shouldn't be heading in this direction. And it's not quite the case that broad knowledge is strictly necessary. But it never seems to be negative! And so far it is the best way we've found to do... everything. Small models are good to the extent they are like big models, not to the extent that they are small.
janalsncm
On narrow domains, it is very common for small models to match or outperform larger ones at a fraction of the parameter count.
For example in language, this is called the “curse of multilinguality”. Small models that handle a single translation direction can easily outperform big ones that try to handle them all.
In any case, for most tasks the question is not “how many tasks can this model kind of do well” but “given time/cost constraints, what is the maximum level of quality we can achieve”. And for that, small models are usually very competitive.
hgoel
This argument seems somewhat strange to me, and at odds with the recent model releases.
It feels like claiming that because a current generation laptop CPU is slower than a current generation HEDT CPU, we should ignore that the laptop CPU is faster and more efficient than the previous generation HEDT CPU.
Of course a huge system could do more, but these systems are constantly evolving and just in this month we've seen comparatively smaller models approach capabilities that were frontier AI less than a year ago. What you're saying would be true when we eventually find the limits of how much models can compress knowledge. Until then, all I think you're saying is that companies that are willing to burn more money can afford to ignore efficiency.
wredcoll
I think the context here is that small models run locally, not rented from a cloud.
giraffe_lady
Yes small models are and will be useful for lots of stuff for several reasons.
But the idea they’d be better than a bigger model is cope, you’re pretty much always better off running the biggest one you can bring to bear within your constraints.
deaux
> claude fable is better at writing both code and prose than smaller code- and prose-specific models.
There's a million ways to define "better at writing prose", and it's only better at some of them. It's worse at many others.
EagnaIonat
> Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks,
Try using a LLM model for RAG embeddings and get back to us on that.
giraffe_lady
Why do you think I haven't done this?
EagnaIonat
If you have, then it explains so much more.
michael0church
Fable 5 is actually a lousy writer. Opus 4.6 is the best for writing and prose assessment. Gemini 3 is smarter at reading comprehension but tends to be more unstable in judgement.
LPisGood
Small amounts of world knowledge seems like it would inherently be tied to more hallucinations.
TJTorola
Perhaps we'll get to a point where believing any un-sourced information from an LLM will feel crazy. I don't want my model to know more than it needs to perform logic and use tools. Once it is capable of using tools I would much rather it looked up information or sourced it from existing context rather than just divine it from it's weights.
ImaCake
The problem is it needs world knowledge to know what to lookup. This puts a floor on how little it can know while being able to look up what it doesn't know. Maybe its better if it knows a lot but has a good instinct for verifying that.
pjc50
How do you actually get to the thing you're looking up? The scrapeable internet is shrinking in response to scrapers.
"Source: rare book ingested and shredded by Anthropic. No, you can't look it up and we can't show you the scan. The remaining open market copy is $5000. Trust me."
pinkmuffinere
I empathize, and I have the same preference, but I wonder how this interacts with other people (many of them being our coworkers) using LLMs. There is no authoritative source for the models to pull info from, so either people will have to exercise good judgement and double check important claims, or they will trust too blindly and fall close to the level of whatever LLM they use. In that case, I prefer my coworkers to use an LLM that does have world-knowledge -- I will still hear them spout ridiculous claims, but at least it should be less frequent. It strikes me there's a sort of prisoners dilemma here, where if nobody trusts others to critically evaluate info, it's in our interest to make the tooling do it instead, to whatever degree that is possible. Maybe I'm too cynical about working with others though.
DennisP
Only if we require the knowledge to be built into the weights. Give it access to a search engine and a big library of ebooks, and it might do better.
Zambyte
Probably. You can solve it with either some grounding context, or spending hundreds or thousands a month extra on a model that has more knowledge baked in. With modern harnesses, the choices is obvious.
jbstack
Doesn't matter if you aren't asking the type of questions where hallucinations are relevant e.g. you're seeking pure reasoning rather than factual information.
cpill
yeah, I think they will get smaller so they can be run everywhere, and really just be an interface to various non AI systems.
throwaway63467
I’m kind of cautiously excited for the next five to ten years, with these AI chips becoming incredibly fast and RAM capacities ramping up its in the cards that we’ll have chips like today’s ATMEL microprocessors that fit on a single board computer and can run small models locally, then all our gizmos can have local AI and I can have a truly intelligent home. Of course there will be a huge push to put all of it in the cloud but maybe we have a chance to take this technology home for good as it’s hard to imagine people will submit to this kind of surveillance required for AI home automation
24/7 (then again I might be wrong). Exciting times.
dTal
Is a "truly intelligent home" something I should want? None of the current generation of "smart" addons are true value adds - their entire purpose is data collection. The pretext is always absurdly thin. I just thought "I bet there's a wifi enabled microwave", googled it, and indeed, Samsung have released such a thing - you can control it with your voice! Wow! Never mind that you can't un/load food with your voice, or that you're never more than 10 feet from it in the kitchen anyway. Oh but there's an app. Of course there's an app. It "suggests recipes". Right.
I think hoping for a locally hosted "smart" home is backwards. They have no other function than to invade your space. "Smart" objects are agents, and they don't work for you.
tuesdaynight
Same. I was in the doom AI crowd ~2 years ago, then I realized how many problems could be mitigated or solved by having these local models, and I became part of the cautiously excited crowd
mathgenius
Can we stick one of these in something that looks and sounds like HAL 9000 ?
throwaway63467
Hard to decide between Star Trek computer voice, Glados or Jarvis, HAL would also be cool.
verdverm
with wireless tech, you can embody Ai in just about anything, waiting for that hacker post about turning their toaster into a thinking machine, I have a Anki vector I've been meaning to do this with (has camera, speakers, microphone, and screen built in)
teiferer
A friend of mine told me earlier today that they had a discussion at work (a coding shop) about "downgrading" to luna from sol for cost reasons and that many were quite unhappy about this because they didn't want inferior tech to be forced upon them. Do they have a point? Is sol actually worth the extra cost? Especially if you ramp up the effort level?
bunderbunder
I don’t love the “forced upon them” framing; if that’s really how people are thinking about it then maybe they should pause and reflect for a moment yhat it isn’t their money being spent. Amd the default isn’t always having the latest and greatest, it’s not paying for anything at all.
Now, if the debate is really about which option is more cost effective, then we could easily run an A/B test to find out. Though TBH my instinct is that that experiment is likely to cost more than the potential cost savings.
What I will say is that my own sense from experimenting around in a non-rigorous way is that the answer depends on how you use the tool. For actual vibecoding you should always go for the SOTA model because it will need less oversight. It’s also less likely to get stuck in a vicious loop that fruitlessly wastes tokens. But for a more hands-on approach where you move in small, carefully planned increments that you review and test in human-comprehensible chunks, smaller models may be preferable. SOTA ones don’t do that much better when working that way, and the slower inference adds a detrimental amount of friction to the work cycle.
teiferer
> we could easily run an A/B test
I'd object to "easily". It's already hard to measure whether AI is generally worth the cost. Let alone compare models in such detailled ways. It's mostly handwaving and gut feeling Doesn't mean the conclusions are wrong, but biases are strong.
abustamam
I recently just had fable run itself into a loop. Im sure it would have stopped itself at some point (perhaps when i ran our of tokens?) But I still stopped it early when I noticed it trying to get something to work when the solution was in a file in a sibling repo.
pseudosavant
It is a good question. Luna is definitely a very capable model. Much more capable than the top SOTA models from 12 months ago. It definitely isn't at the same level as Sol, but you get 20x the tokens for the cost, and it has a much faster tokens/second rate.
If this is a cost conscious company where I'm going to get a fairly limited amount of Sol, or a nearly unlimited amount of Luna, I'm probably choosing Luna.
deaux
> Much more capable than the top SOTA models from 12 months
Confidently wrong here. It's absolutely not across the board "more capable" than GPT 5 or Opus 4.1 or even Gemini 2.5 Pro. It's potentially better at certain specific tasks, mostly agentic coding implementation work. I.e. tool calling and usage of bash. It's worse at a large range of other tasks.
ImaCake
Luna on xhigh unlocks gh copilot for me. Sol, even discounted, is too expensive to use and is much slower. Luna on the other hand seems to be so cheap you don't have to think about cost.
azuanrb
I’m building an internal tool for our company, basically an agent to help with on-call and alerts via Slack. I have evals running across a few scenarios, and my favorite models so far are Sol medium and Luna xhigh.
Sol medium has been a nice balance between intelligence and response time. Luna xhigh can achieve similar scores on the evals, but it takes noticeably longer. My impression is that the higher reasoning effort helps compensate for the lower base intelligence.
Cost is definitely a big factor, but latency and intelligence matter too. If I had the budget, I’d take Sol medium over Luna xhigh.
From using both on real scenarios, Sol is noticeably better at navigating around issues, exploring alternatives, and being creative when the obvious approach doesn’t work. That matters quite a bit when you’re investigating live alerts, where the path to the root cause isn’t always straightforward.
usef-
Luna as a doer, with a smarter model planning, can be a good compromise. Using sol for everything can be expensive without much gain, as a lot of steps don't need that sort of intelligence.
praveer13
Luna is great at doing targeted smaller work, I use sol max for creating a plan and targeted /goal prompts after I finalize the design. Or Claude with ultracode for design and planning and adversarial review by sol max and then delegate to Luna for smaller goal prompts
shepherdjerred
I choose to use Luna for most tasks because it is cost efficient, even though I get a pretty generous budget from my company.
Sometimes I will use Fable or Sol for large features/projects, or research/exploration.
I would not be at all happy if I were forced to use Luna, though. I’d probably start looking to leave. I don’t want to work somewhere where I don’t have choice over my tools.
noodletheworld
Is sol better?
Yes. Categorically. Anyone who tells you otherwise and that luna is “just as good” does not know what they are talking about.
Going from sol to luna is a downgrade.
It is not a question, it is a fact.
> Is sol actually worth the extra cost?
Is a question only you can answer, because it has no generic answer.
Right now, for me, being able to use sol is worth the cost, but using it all the time is not.
I’m sure going from using it to using luna feels rubbish; but there are realities about costs you have to face sooner or later.
Maybe like… give your team credits and make them pick the right tool for the job; and if they burn their credits on sol in 20 minutes, well, tough luck buddy, looks like you're coding by hand for the rest of the month.
Team will quickly shift. People hate losing access to ai.
vatsachak
Idk... Luna is great if you generate specs before implementation.
Sure a Lexus is better than a used Prius, until you include price
noodletheworld
Anyone who can’t tell the difference between driving those two cars isn't actually driving.
You cant just go “oh hey, I guess they're both cars so I’m taking your lexus away, catch a cab its cheaper” and expect people to just hug you be be like “yay, thanks! I still have a job I guess! :party:”
:P
defrost
Straw argument.
Obvious to the meanest intellect they can tell the difference between the cars.
noodletheworld
So can anyone using sol and luna.
Don't complain to me if someone responds using a stupid metaphor that proves the opposite of the point they were trying to make.
DarmokTanagra
They should try "downgrading" to people who don't need a random number generator to do their job for them.
brikym
It's silly to discuss it. Just do the evals.
dude250711
Yes. If they don't like the cost then they should fire the "leader" who introduced the AI there to begin with.
kingstnap
Smaller models + more effort has strong diminishing returns, especially if your goal is to save money.
Sol already lacks judgement. It will absolutely add idiotic tests and comments. Luna is that but worse so if you account for things like going down wrong paths, producing bad results, overthinking then it could easily cost you more to get less.
glimshe
> There's obviously a lot we can optimize here, but if you're charging what the WSJ or The Economist charges, you'd better be delivering similar value.
Gosh, watching paint dry has been a better value than reading The Economist in the last 5 years or so.
That aside, I had good results with Luna. I'd be interested in hearing about a comparison that takes into consideration response time (not TPS), cost and performance of the popular models at different settings. That chart has some of that. For instance, is Luna Max a better value than Terra Medium?
[deleted]
Comment removed.
my-next-account
Are ya kidding me
Myrmornis
Is there a resource somewhere that tells me, "Given that you have X Gb RAM on your laptop, these are the current local models you should consider trying and here's how to configure them to leave enough memory for other applications on your machine."?
Are you familiar with Ollama [1]? It is a particularly easy to use tool to download and run local models. They sort models recent popularity and specify size for the various quantization levels.
I would try using ~1/2 your available ram and iterate from there.
If you have 32GB of RAM, I would give Qwen 3.8 a try. All you would have to do is run "ollama pull qwen3.8:27b" then "ollama run qwen3.8:27b". If you have 16GB of RAM, I would try Gemma 4.
[1] ollama.com
blueferret
Several exist actually. Try whichllm.app or fitmyllm.com.
- Linux, general use case, balance
- 16 GB RAM
- 10 GB VRAM
Recommendation: Kimi-K3
This checks out.
rurban
whichllm sucks. It only recommends gguf models, and llama.cpp is not supported.
fitmyllm is much better
sandos
It says Kimi-K3 for very set of parameters I put in!
64 or 128GB RAM, 6 or 64GB of VRAM....
Not sus at all.
tencentshill
Roleplay, windows, 16gb/4gb
Kimi-K3
I think the slop site is broken or compromised...
crossroadsguy
It gives astronomically optimistic results for my 16GB M1 Pro :)
By the way own exploration sort of led me to qwen3.5:9b for the best case scenario balanced model considering almost 10-11GB of RAM is almost always gone anyway. Even with aggressive app quitting.
sandos
I just asked my current LLM for that advice. Funny that they dont block it, I guess they are not very threatened.
For my anemic 6GB built-on 14GB Qwen seems to be the best bet, not great reviews but from my limited testing its pretty impressive.
jillesvangurp
We've been testing and developing a few openai based applications. Relatively simple stuff. I topped up my api credit with 10$ a few months ago. I just checked, we burned something like 61 cents over this period. About 126 requests over the last month. That's just us testing but it shows me that this stuff is relatively cheap. The reason we don't have a lot of requests is that most of this stuff pretty much works on the first try. This is not hard to build.
I think AI is an underused capability in applications. People come up with all these big plans to build agents. But even a simple button to "calculate a color palette that takes into account these symbolic names the user entered" is a simple prompt that comes back quickly and doesn't require a lot of reasoning. We use that with Luna and we give it some more context.
Image based stuff is more expensive but still quite reasonable. We do some things with Sol that work on images. It's not fast but the result is worth waiting for. These are features that save the user time and feel a bit magical. They just click a button, wait 20-30 seconds and then magic happens. Beats them clicking around for half an hour.
Local model hosting would force me into higher cost and overhead for less intelligence. That's only a good tradeoff if you are really going to use lots of tokens or you need this for other reasons (e.g. data security). But otherwise you just end up paying for underutilized GPU capacity.
magimas
I think in general these smaller applications of AI are seriously overlooked even though they will actually add lots of economic value for companies and lots of quality-of-life features for users.
2001zhaozhao
A dream of mine is to be able to host a LLM-powered video game that I can host on a home server running a decent mid-range GPU like the RTX 5060, and the LLM is fast and intelligent enough to make for a fun game experience for a few dozen concurrent players. People can ask for features and they just get made and added to the game on the fly for the lobby to enjoy. The hosting costs would be manageable enough that I don't have to charge anything for the game.
I think with one more year or so of small model progress, that might just be possible to accomplish.
U4E4
I’m doing this now. Homebrew DnD engine and mobile app chat interface. Rich 3d sim and haptics for the dice rolls. But otherwise no fancy graphics rendering. The juice is in the imaginative generate group story telling. Local real time STT on self hosted video chat, to convert the banter and side talk into next player action dice rolls. Qwen on an m3 ultra 96gb handles NPC, dm, bot party members, arc, consistency, history, memory, adventure beats. There’s even a mechanic for bards to compose an end-of-adventure ballad that compresses the shenanigans into verse.
Bootstrapped it with fable, opus or sol 5.6 when I had extra usage to burn over the last month. Playtesting it solo as both DM and PC was fun too. Character consistency and originality is remarkably compelling.
Upshot is that for the gaming hours that my friends have available, the social interpersonal gaming experience competes well for their AAA gaming time. There’s even some good async passive campaign continuation patterns that run in our signal thread to keep things going between sync sessions.
And players have requested features directly in the session transcript that have been direct implemented.
0xdeadfa11
I'm very interested in this. There already are a few commercial and open source solutions but none are of my liking.
Are you going to publish yours somewhere? I'd love to play Cyberpunk 2020 with AI
dgunay
Luna max is suitable for like 90% of the kinds of code changes I want to make. I only find myself actually reaching for a Sol or Fable tier model if the problem is very complex. If you're willing to build the guardrails and do some extra planning, Luna is very capable.
bartleeanderson
Nobody is questioning what is meant by small? When you started mentioning frontier models that don't run locally, I just go TB;DR "Too big, didn't read"
daquisu
I read the full post. It is more about cheap models with good enough performance rather than small models. "Local" was not mentioned at all
weinzierl
Small is relative. I'm looking for models that I can with run around 100 MiB mark (RAM just for the weights) to demo what you can do with this little memory.
I know of SmolLM 2 which in Q4 is borderline regarding the size and rather dated. There is also TinyStories, which is also old and also focussed on children's stories.
Is there anything newer in this category? Or should I try to distill something down to this size?
a13n
Regarding the Pareto frontier and related benchmarks, I have a hard time taking anything seriously that claims that Opus is anywhere near the intelligence of Fable. Are there any benchmarks that haven't just been benchmaxxed that more accurately represent actual usage?
kakugawa
FrontierCode is prob the closest. [1] It's closed source (so no direct benchmaxxing), and it was calibrated by 20+ open source maintainers. It shows Opus 5 (medium), beating out the other reasoning levels by a large margin. i.e. Opus 5 w/ higher reasoning levels actually reduces performance. [2]
However, you'll have to gauge for yourself how closely their tasks resemble your tasks.
I love how "Small Models" apparently is "Model of unknown size but probably smaller than another model that we also don't know the size of".
delis-thumbs-7e
I was wondering the same, trying to find if OpenAI has started to release the ampumt of model parametres. Basically they seem to mean ”cheaper”.
low_tech_punk
The tokens per second speed measurement is highly inflated nowadays because most of the tokens went into thinking. I wonder if there is a more realistic measurement for "effective speed", which accounts for thinking efficiency.
ak_t
Many benchmarks now measure the total cost or energy usage per completed task.
yipinwong
"Small models" nowadays work like someone who has IQ 100+ while SOTA ones are like 150, "relatively".
Given sheer number of turns I can make with small models, I can do a lotta stufff
- cheaper, and faster
Harness makes differences: There have been many HN posts about how one made tiny models work better at certain tasks using harnesses.
These "small" models with right context, and guidance, they work wonders.
---
I've been saying Luna has been my go-to AI in previous comments and why Luna is still more compelling than GLM-5.3-flash.
I think technically sound its just that the frontier models are just sooo in the news that al the money is around intelligence and big breaks in intelligence.
Once the economics change and we see less bit model deployments there will be much more work optimizing models at the pareto front.
Also yes people need to be able to sustainably run these things to start running a business. Building a business on the model pricing today is as smart as building a retirement plan on a gambling schedule
caust1c
IMO big models are not a product in and of themselves. Inference is just a new type of compute. I'm confident that in two or three years, every product will have inference capabilities integrated into the experience, and models will become less and less distinctive from one another.
What most products need from a model is a pretty short list: the ability to make tool calls well, accurate recall, and the ability to follow directions without wavering (whether or not those directions are baked into the weights or provided in a system prompt). That covers 95% of inference utility in products.
We're nearly there, and I believe these capabilities will fit on small models.
Because of this though, I predict hardware demand will stay high despite demand for "hosted" inference dropping. Unless there's some regulatory shenanigans that step in to say otherwise.
abustamam
As someone who uses a lot of opus, I'm curious what changes to my workflow I need to make to make small models work. My workflow currently is for a large feature I'd explain the feature and how I envision it technically and run the grill me skill where it'd ask me questions that I may not have thought of, then it'll write a spec and execute from there. It spawns an orchestrator so supposedly its supposed to use haiku for mechanical things but im not confident that works.
How much of this can be replaced with small models, and what changes to my workflow do i need to make?
pornel
Opus is good at deciding what to do beyond literal interpretation of the prompt (like checking assumptions before coding). Smaller models need to be told not only what to do, but how to behave.
oh-my-pi's Advisor feature can fill a lot of this gap (a smaller model told to reflect on its own output catches when it's doing a rush job and skips instructions).
Computer0
I’d imagine it’s more focused on individual methods and how they connect
butterisgood
I don’t care about the cost if the results are weak. I did a bunch of work with Sol only to find Opus 5 spot a bunch of bugs.
And it was correct. The results weren’t as good as they should have been for Sol. How am I going to trust Luna?
highfrequency
> the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts... ~95% of the work he does falls into bucket 2. It's hopping on calls. Nudging people. Blocking and tackling.
This is a good insight broadly!
anuptalwalkar
I kind of agree with your assessment. Running models not just on local, but cheap and lightweight frameworks will drive the next phase.
100% agreed. Small, cheap, and hosted models. Luna (and open weight models and others) is ridiculously cheap @ $0.2/$1.2, easily accessible, and more than good enough for basic use cases (e.g. summarization, simple tool calling, etc.).
marius_
I wouldn't call $0.2/$1.2 "ridiculously cheap"
scottgk
Luna is definitely "good enough" for so, so many use cases. And that price point...
Goes to show that open source = cheap, closed source = expensive is wrong-headed. Closed source is going to beat open source on price point due to scale (try buying a few GPUs to run your own model on...) and labs innovating across the full inference stack.
That leaves the value prop of open source as privacy / customizability.
zatkin
Maybe I'm being super reductive here, but operating small models at the core of your business kind of moves the needle from making external API calls (against frontier models) to running internal API calls (against your locally-run models). It seems like if we want local models to take off, it will need to become easier to run local models for cheap. I'm thinking like reducing the barrier of entry for running "local models" in the cloud providers like DigitalOcean, AWS, etc.
malfist
You should be glad to know digital ocean already offers this
regularfry
In theory so does AWS, but the Bedrock model selection is badly in need of a refresh.
spl757
I only run local models and I don't give them access to much externally. I don't do anything serious with it, but it comes in handy and I know that they can do so much more. I'm on a meager RTX 3060 12GB and a GTX 1660 Ti with 6GB for some extra vram space. When I first started playing with local models, I was really impressed with what I was able to achieve locally.
That's great, but the thing that worries me is that many companies have billions invested in the AI bubble. It's around 1.5 trillion last time I looked. It's all circular spending between the companies building out the infrastructure, and the models. None of it is profitable. They will want to recoup that 1.5 trillion from consumers, which means using online-only pay-as-you-go cloud models. They will inevitably see that people using capable local AI are "lost customers" and they will try to kill the ability to locally host AI or somehow enshitify it enough to make paying a subscription more palatable.
I'm not saying I believe that will happen, I'm just worried that it will. Is anyone else worried about that as well?
mlnj
I am very excited that more makers will come up with fast memory for consumers rather than enterprise. Companies can only pre order so much RAM.
At some point there will be a surplus of fast memory and even in a crash the current generation of SLMs are bounced to be plenty to build a lot of intelligence at home.
mumbo_rmj
My money would be on ‘for security purposes we just can’t let these get into the wrong hands’ and then the lobbyist go to work.
nullbio
The future is self-hosted models trained on your own reasoning traces autonomously, as you sleep, using QLoRA and whatever else.
oybng
An absolutely nothing post at #2 on the frontpage
mattmaroon
The demand for fast, cheap, good enough models has always been borderline infinite, it’s the supply that’s going to take off.
jmtulloss
I forked my Big Serious Harness™ that models construction projects into a harness for building a vibe coded family assistant. I couldn't figure out how to make the toy operate at toy prices until Luna. Now you can vibe code all the little apps you might want for your fam for like $5 and operate it day to day for a few cents.
maestroquirk
We've been running a small finetuned VLM for OCR and yeah... cost is like 1/3rd of the usual vision APIs. Small models are kinda the whole product for us
MaxikCZ
Which vlm are you using? Ive found a reddit thread listing like 20 of them, but its hard to find any concrete info on what is worth even tryin.
possibilistic
> Peter runs multiple companies. Beyond Segment, he's raised $100m+ for Charm Industrial, and just recently closed a Series A for Revoy. He's incredibly organized and efficient with his time.
You can do this before an exit? Build and fundraise for multiple (3?) companies at the same time?
zachthewf
Segment had a $3B+ exit to Twilio back in 2020.
MASNeo
We mostly use specialized small models because larger are to expensive/slow and are prone to hallucinating quite a bit. Not sure how this is a surprise, seems more like a best practice.
whiplash451
Is there evidence that small models hallucinate less than big ones?
gpiechnik2
I hope that one day it will be possible to use models as libraries on the backend (without having to host them somewhere/use a provider)
n4pw01f
I’m getting surprisingly good results on GEMMA4 2 bit, running on a Nvidia nano and raspberry pi
1saadcodes
The cost difference is pretty dang nice. Going from roughly a dollar to $0.10 for the same kind of task makes it so that products that didn't make financial sense before become possible
tosh
I think we'll see more of this soon
replit is already leading the way with free luna usage
ittsel
Watch reasoning tokens though. We tried a small reasoning model that burned ~2800 thinking tokens per call, 3x the cost of a cheaper non-reasoning one despite a better price sheet.
zmmmmm
The "good enough" concept is interesting because of how systematically people over estimate it. So often, things that are lower quality but thought to be "good enough" turn out to be either not good enough or not worth it compared to just using the higher quality "thing".
I will believe that smaller models have hit that bar empirically when I see them in production. At the moment, even frontier models are stuck in most of the scenarios I am seeing for high value tasks at the "not good enough" gate - so small models are not even close to being on the scene there yet.
Kim_Bruning
It's kinda fun that people get to experience what the 8-bit era was like!
fitsumbelay
using small local models - with a little bit of extra work - for the first time over the past few days was _really_ illuminating and inspired similar thoughts about how far you can practically get with so little. column of zap emojis, mane ...
verdverm
I only use open models now, I really think the era of open models is upon us, big or small, but I also agree small models reached the point where you don't have hand hold them with qwen 3.8 27B
dzonga
small models + a good application layer - are more than enough, good for routine business tasks.
the application Layer i.e having a good graph RAG & connecting it up together is the missing piece for most.
sroerick
Can you elaborate on this?
lantry
The model doesn't have to be smart if all it's doing is pushing a few different buttons.
I don't have to be an automotive engineer to start my car and put it in drive.
sroerick
I mean - how specifically are you using knowledge graphs?
dzonga
there's this [0] - Zep.
e.g for a lot of small business - they generate invoices - feed that into a RAG system. Then have an agent etc do follow ups. for agents then the smaller models are enough.
That's why the market has our solutions but economics.
Zigurd
I recently had some relevant experience: for a couple of months now I've been experimenting with on device models to summarize feeds in a Bluesky client I am developing. The feature extracts topic areas, categorizes posts, and creates a summary under each topic.
At first the results were hot garbage, and progress was slow. I hooked up the settings to download models from Hugging Face conveniently, so I could run experiments faster, and I massaged the prompts a bit. Last week this feature made a qualitative jump from science experiment to something I'd actually use.
The fact that all runs on the device means I've got no variable costs associated with adding this to what will be, at best, a pretty low revenue product. I've tested it on trailing edge devices like an M1 Mac and a Pixel 8, and performance is very tolerable.
The key is I'm not asking for open ended answers to open ended problems. When it proves to be useful it's not going to get less useful or more expensive.
There are vast domains of uses for LLM models with similar characteristics and likely similar results.
hartator
I have trouble seeing the points of using less capable models.
I just want the smartest, best, and most capable models. It feels smaller models for speed and cost are just transitions towards better hardware allowing the very best model.
krisoft
And that is why i always carry my groceries with an Antonov An-225 Mriya. Is it really needed? No, but i refuse to compromise on what is(was/will be) the best.
imp0cat
That plane was destroyed by Russia, wasn't it? I believe it was partially disassembled when Russia invaded Ukraine and so it wasn't possible to save it. :(
rurban
2 Ukrainian Antonovs made it to Germany, Leipzig, and are doing well flying weapons in
arjie
My experience has been that responsiveness is value. For tasks where you need steering, responsiveness allows for better steering. For tasks which you want unattended, better models are just better.
There are still tasks that even Fable is bad at doing. And many are just mundane things. Because of the fact that you have to steer it on those tasks, you might as well steer an 80% model that is 5x faster. And those do exist.
Naturally there’s a bit of a gap because the faster models need steering on tasks the slower models don’t so there’s no smooth transition but I find it worth it. Especially if you want to stay in flow.
Ironically this sometimes means starting a plan with a great model, planning with a worse model, iterating, then submitting it to a better model for review, and then having the better model do the implementation.
trvz
First, smaller models are fun for hackers: you can run them locally, or run them faster.
Second, when cloud models become unavailable or otherwise deteriorate, these will be all you have. May as well prepare.
breezybottom
If you're hacking a US-based entity, using a high-performance Chinese model through a VPN is probably safe enough. I doubt a local model is going to be sufficiently smart to hack any major company.
trvz
You misunderstood what I was referring to by “hacker” there.
shafyy
Some reasons:
- Smaller models will always be cheaper
- Smaller models will always use less energy, therefore better for the environment
It's a bit like saying you always want the fastest and best car; Sure, you can have it if you keep paying for it. But a small car will also get you from A to B, will use less gas and will be much cheaper.
ebiester
It depends on what you're trying to do. For non-coding tasks luna is quite often enough. Flash models are more than enough for summarizing a text, for example, or whipping up a small script to save me fifteen minutes. If you're on a 200/month plan, I see your point. If you're on a dollar limit - or worse, paying per token out of your pocket - you look to be more efficient.
0xbadcafebee
There's a difference between want and need. I want a 650hp V8 supercar. I need a 150hp I4 toyota corolla. Why choose a less capable car? Because I don't want to spend 10x as much money to get groceries.
polotics
Can you define your use of the word 'smartest' here just in case some of us don't quite know what you mean?
tartuffe78
Cost is the point
ewwefwef
Where i can download this luna model?
;D
agcat
I like the analogy on ways to make small model useful.
ib33
I have been using Sol Medium and it works great. I switched to 5.6 Luna Medium after reading this and I am back to feeling angry and gaslighted.
One time, it said "You were right; I corrected the regressions and committed them."
It was still not fixed.
I do admit not hitting the five hour limit in two prompts is nice. However, I wonder if becoming an angry person because of cheap models is worth it.
hnub7akvk6
Didn't know that
retinaros
somehow I cant understand how luna is a step up. to me it feel dumber than 4.1. slower too
incrudible
Where are these 180 IQ models? Even Fable needs guidance so it actually produces output I consider on par with what I could have written myself in a domain I am familiar with, and I am certainly not at the frontier in those domains.
svantana
The "IQ 180" label seems to be referring to the (human) CTO of their startup.
dev_awesome
how effective are the small model?
paganel
> Most of the "human tokens" at companies today are spent this way
It sucks that we got to this point when it comes to our civilization. At least guys like this one here can be happy with their AI-sourced money.
nocodeexportcom
Nice!!
hnrprtlpdb
Well said
senectus1
not sure how this is really about model size.
I'm keenly interested in seeing super small (like 10's or 100's of mb) special purpose models starting to come into their own.
that might actually be transformational globally not just in rich countries.
davidguetta
nice
now do the same for robotics, because itnernet trained vlm + action head was really a dumb take
robinduckett
100 t/s of crap is still crap.
[deleted]
Comment removed.
[deleted]
Comment removed.
Barbing
This is how your comment displays on my screen, perhaps a typo:
There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on.
Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do.
I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.
The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words.
The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of restrained but efficient model+harness-tuples that have been distilled, finetuned, and rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks with incredible efficiency and erogonomics.
For analysis, bug hunting, overview and some porting work to popular languages and so on I think the current SotA is fantastic, but they're still very disappointing for code generation past function or small module level.
Recently my employer hooked Claude into a bunch of SaaS services like a ticketing system and an asset inventory system, and I feel a big productivity boost is that I don't have to dig for their open browser tabs, remember their product name/URL/where I put the bookmark, find that my sessions have logged me out "for my protection" since I last looked, find my MFA code, approve sign-in on my phone, then use their mismashed/sluggish/poor-UX interfaces and slow searches to loko for things, and can instead ask Claude "search these systems for anything about <issue>" and it just does. It correlates between systems, summarizes things, and gives me references which system and where to look to check what it found. Internal search on a mess of acquired and disjoint systems.
This feeling reminds me of two classic Joel Spolsky blog posts below, and your comment here about the surprisingly useful tab-complete gives me a similar feeling. I wonder how many smaller candidates there are where LLMs are more helpful and less hyped, against the big vibe-coding solve-everything hype which are currently less helpful than their hype?
References [1]: "A lot of us thought in the 1990s that the big battle would be between procedural and object oriented programming, and we thought that object oriented programming would provide a big boost in programmer productivity. I thought that, too. Some people still think that. It turns out we were wrong. Object oriented programming is handy dandy, but it’s not really the productivity booster that was promised. The real significant productivity advance we’ve had in programming has been from languages which manage memory for you automatically. [...] Whenever you hear someone bragging about how productive their language is, they’re probably getting most of that productivity from the automated memory management, even if they misattribute it. Sidebar: Why does automatic memory management make you so much more productive? 1) Because you can write f(g(x)) without worrying about how to free the return value from g, which means you can use functions which return interesting complex data types and functions which transform interesting complex data types, in turn allowing you to work at a higher level of abstraction"
and [2]: "I have seen many language and programming fads come and go. But there’s only ONE, that’s right, ONE language feature I’ve ever seen that actually improves your productivity significantly. No, it’s not object oriented programming; no, it’s not intentional programming or assertions or programming by example or CASE or UML or XML or Java. The only thing that improves your programming productivity is using managed code – that is, using a language in which memory management is automatic."
[1] https://www.joelonsoftware.com/2004/06/13/how-microsoft-lost...
[2] https://www.joelonsoftware.com/2001/10/17/working-on-citydes...
I agree that the mismatch between the expectations of OOP as a paradigm and this situation right now feel very similar. I suppose the many billions pumped into selling vibecoding and large-scale code generation via LLMs of other kinds is sort of like the massive amount of money pumped into marketing Java as a viable alternative long before it actually was.
In engineering you learn from these mistakes and try never making them again. Do we want to go through this evolution every time we solve a software issue? Just because we can with an unlimited number of cheap tokens? I think not, I’d rather use the knowledge build up that also knows about the edge cases forgotten to test. Or better, use multiple models that evaluate each other, as Entropic describes it in their recent report https://news.ycombinator.com/item?id=49316271
> We expect that agents coordinating in the wild will act in higher variance ways than we see here, because they’ll have different backgrounds and therefore different contexts. They also, presumably, won’t all be Claudes.
You don't necessarily need a bigger model to do this.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Recent comment touching on this in relation to LLM's in more depth: https://news.ycombinator.com/item?id=49322695#49323341
Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LLMs could eventually outperform stockfish. It’s not true today and it remains to be seen whether it will ever be true. For now, specialized models are absolutely better at specialized tasks.
I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.
Stockfish is the best chess search engine we've got, and you can learn some good heuristics for chess search policy that will make time-limited chess search a lot more powerful. That's perfectly in line with the Bitter Lesson.
In contrast, LLMs playing chess are relying solely on learned behavior. The inference harnesses surrounding them aren't designed to do chess things, they're designed to do autoregressive token decoding, which isn't a search process. Reasoning traces can resemble a search process, but they're far less efficient - the LLM would have to work out each legal move, test each one, calculate a score, and simulate minimax over all of that. Assuming the LLM is smart enough to even do all that.
A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.
Now let's look at the bitter lesson again. It says that general methods that leverage computation are ultimately the most effective, and by a large margin.
That's different from just saying to leverage computation (which is how I would interpret "unlocks more search/learning"). If the lesson is "more computation wins, when sufficiently channeled" you're basically looking at a truism. Of course more computation beats less when it's used right. The bitter lesson is about abandoning specialization in order to get more computation, and while there's a couple ways where that helps with chess, there's a lot more ways where it's counterproductive. It looks like it's more true for Go than it is for chess, and that it's not universally true. It probably correlates with the state space.
Not really, if anything it's closer to the opposite. The Bitter Lesson essay literally has this as an example:
> These researchers wanted methods based on human input to win and were disappointed when they did not.[1]
and
> Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale[1]
The actual bitter lesson is this:
> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.[1]
Applying to the "LLMs-for-chess" example the bitter lesson approach would be to put many, many more games into the LLM.
Does this work? People have trained fairly small LLMs that are competitive Stockfish at the ELO 1500-2000 level, eg: https://github.com/kinggongzilla/chess-bot-3000
This seems to be evidence that large LLMs probably don't have as much chess training data as Stockfish does.
[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html
This was/is basically a strawman though. Like maybe "human input winning" was desirable for chess masters but for computer science wonks? Not the point or the disappoint. It's always neats and scruffies fighting about using some kind of recognizable method (logic) instead of magic (ML).
> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning.
More to OP's point I think: nowadays when someone wants to beat you over the head with the bitter lesson, they aren't as careful to include learning and search. They want to say learning leads to intuition (magic) whereby we can avoid work (logic/search), and maybe argue or assume from there that neats and scruffies is settled. TBF, something like reasoning in latent space does resemble intuition!
But the real lesson is confirmed every time we bother to check, and not very bitter for anyone. Search/learning/logic are ALL always necessary on any sufficiently difficult problems, and hybrids that interleave always outperform everything else. Stockfish being the example in this thread that different camps of absolutists would like to claim, but also all the MCTS examples, evolving examples, and new hybrids all the time. My favorite lately: https://arxiv.org/pdf/2511.08983
Oh no!
The whole field was full of people whose entire career was built around the idea of developing smart priors.
To quote Wikipedia:
> For computer vision in particular, much progress came from manual feature engineering, such as SIFT features, SURF features, HoG features, bags of visual words, etc. It was a minority position in computer vision that features can be learned directly from data
This undersells the change though! David Lowe's reputation as the best image researcher in the world was based on his SIFT patent[1]
This approach worked until 30 September 2012.
That was a bitter day for many, many computer science researchers.
[1] https://en.wikipedia.org/wiki/Scale-invariant_feature_transf...
But as it relates to more/less magic, maybe the more modern lens on this is e.g. https://arxiv.org/html/2505.11581v1 . Is manual feature-engineering more like what you'd evolve, or more like what you'd get from SGD ? Feasibility and performance is always a question, there are others like what is robust, stable, adaptable, predictable, explainable. Maybe the manual-features people were interested in something besides the manual part? Maybe the story isn't so simple, and maybe it's not finished yet.
The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data.
Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than human intelligence. The LLM is based on a massive data corpus, sure, but the amount of data specifically about chess in there pales in comparison to just playing billions of games of chess.
> maybe if we use a blend of raw data and hand-crafted heuristics
I don't follow. They're suggesting giving raw chess data to the LLM, no heuristics involved.
The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games.
If you can draw any lessons from chess commentary, I think it’s very reasonable to call it “hand-crafted heuristics.”
And raw games plus raw commentary is all the data you have. You can make more games but those can be fed to both stockfish and the LLM competitor. So it seems like a valid interpretation of the bitter lesson to me.
The key point I was trying to get at is that the human insights don't contain anything that can't be mined from vast amounts of gameplay. Every human insight can eventually be rediscovered and made rigorous by data (in chess, at least!) In the short term, those insights are useful, but in the longer term, they add nothing at all.
Note also that "raw gameplay" here can mean new games -- you can generate as much data as you need, you don't need to rely on real recorded games.
But isn't that the bulk of what we're shoving into LLMs, and it makes them much smarter? If it's useful there but not in a chess AI then that seems like a significant crack in the bitter lesson.
> Note also that "raw gameplay" here can mean new games -- you can generate as much data as you need, you don't need to rely on real recorded games.
Yeah I mentioned that, generated games are useful. But if we're being fair and letting both AIs use generated game data, does the more general LLM ever actually overtake the specialized stockfish like the bitter lesson suggests?
Another way to look at this is that giving the LLM the commentary is a way to avoid complaints of hiding data from the LLM, since it'll have strictly more info than stockfish. But if we cut that from the training data and only give it a basic description of chess and lots of raw game data then it's going to get even worse than it already is at chess. Meanwhile stockfish never had that commentary, just actual hand-crafted heuristics and training on game data, and it's very strong.
The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks.
Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson.
Computer chess is the canonical example of this.
In the case of stockfish, the harness is a tree search around the neural network evaluations.
Just kidding of course
It would be better to compare models at how well they can write the code for chess engines, otherwise it's just saying that Fable is not a good CPU emulator, which is obvious.
The Bitter Lesson is about general-purpose algorithms vs. specialized algorithms. Historically, chess engines were programmed to look at a chess position and use positional understanding (imparted by the human programmers) to decide what the best move is. But eventually, the chess engines that actually became stronger than humans were instead programmed to just check every possible move and countermove and see which ones lead to a win. (I'm oversimplifying, but you get the point.) So even before Stockfish contained a neural network, it was considered an example of the success of the Bitter Lesson.
As it applies to AI agents, the Bitter Lesson would predict that the best possible agent would simply possess A) a way to do anything it wants, B) a way to evaluate whether what it did was correct, and C) a ton of compute. Then just turn it loose on your task. (The fact that the "brain" of the agent is an LLM is kind of irrelevant - you could also imagine the brain just being a program that generates random syntactically-correct code. What the LLM achieves is that, the random generator would take millions of years whereas the LLM is much more efficient at creating plausibly-working code. This is analogous to a chess engine's pruning heuristics.)
The hard part here is B. We've seen some great agentic successes when rewriting an existing project in a new language, since the agent can just use the project's prior test suite as its evaluator. But when developing a new project, you're still figuring out the finer details of how everything is supposed to work. As the old saying goes - writing a spec that perfectly describes how a program should work, is equivalent effort to just writing the program.
And perhaps at the end it all gets a single pass by a god-tier model for overall sanity and congruence, but the actual work, planning, coordination, and even user interaction was done by cheaper and faster agents of much more limited capability.
It really is absurd to ask programming questions to a model also trained about the lifecycle of a fruit fly.
Instead of building small models from scratch, we train an enormous model and use ridiculous amounts of GPU memory. In the end, the whole thing is shoved into RAM because we don’t know where the useful parts are…
We certainly would know where they were if they were just in smaller models in the first place!
The correct analogy here is Fable as a second tier player assisting a SuperGM in running stockfish, then assessing its output to identify promising variations.
There might be a limit somewhere that prevents the bitter lesson being axiomatic - for instance where simulations for anything can be exhaustive - so that judgement isn't needed any more as an arbiter. But while there are problems sufficiently complex or large to require a breadth models don't currently have, greater scale and compute will continue to convert to better decision making, and the bitter lesson will remain true (true enough).
https://mcpmarket.com/server/stockfish
https://til.simonwillison.net/llms/mcp-in-claude-and-chatgpt...
Dumb AIs are needed for customer service. Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server.
Needs? Customers want something that immediately answers their question/solves their problem, but that’s far away, even ignoring the “run locally on a 1U server” and that that may not be in the company’s interest.
For many companies, that support line is a cost center, not a PR mechanism.
Also “Press 1 for sales, 2 for billing...” has the big advantage that it handles all accents, speech impediments, etc.
Long term I think a solution where a user’s agent trained on their voice, running on their phone communicates with the support agent of a company is where we will end up, and support phone lines will cease to exist.
GP isn’t suggesting that focused narrow model(s) will be more capable than large model, but that many small focused models can have sufficient capability while being more optimal.
Also, the bitter lesson is just wrong. The bitter lesson is about hand tuned AI vs computational general methods. However in truth today’s AI uses both. We have general compute heavy models which require narrow expert instructions (eg tools internet docs).
LLMs would not be as good without expertly written context, and expert context without LLMs aren’t as good either.
The models are not even really trained bitter lesson-style anymore. That concept peaked during the era of pre-train scaling, back when it was thought that making a bigger and bigger GPT-3 would automatically solve all problems through prompting. In 2026, the most important part of training is post-training, which uses vast quantities of niche, hand-curated data to fit the models for specific tasks in domains like tax law.
If you look at value as purely the LLM output, then there's a valid argument that the best frontier models will always be better than fine tuned specialists. (I'm not convinced personally, but it's a defensible claim)
But that misses two dimensions: 1. The cost of acquiring that output 2. What is actually "good enough" for that specialist domain
Not every output needs to be the best to produce value.
And as specialist models increase in cost, their cost/value proposition goes down.
At some point, there's a threshold where cheaper, fine tuned models are "good enough" at the task and also substantially cheaper than the expert models.
That's where fine tuning helps.
Personally, I became a believer in fine tuning after fine tuning a 1B Qwen model as a second pass over my local voice transcription app, achieving excellent accuracy at ~zero token cost and waaaay lower latency than if I'd invoked my Claude subscription under the hood.
Secondly, the bitter lesson is predicated on compute being cheap. There was a period where a hand-tuned algorithm informed by human expertise would outperform a raw alpha-beta search at Chess. Then compute got cheaper, and DeepBlue ascended to the top. Compute is now expensive again relative to the tasks being performed. We are absolutely still in a period where human expertise in training LLMs will outperform a naive approach with more raw compute.
In the latter case, the chess example would tend to support the Bitter Lesson, rather than refute it.
I would also be VERY slow to claim that general-purpose models will never be competitive at chess. It wasn't so long ago that transformers couldn't add two-digit numbers reliably without resorting to tool use. They are now as good at "mental arithmetic" as any human savant. It wouldn't surprise me at all to see someone come up with a model that just happens to be really, really good at leveraging the portions of its general training data having to do with chess.
In fact you could argue that AGI demands such a model, if we are to assume that LLMs are a guidepost in that direction.
> I would also be VERY slow to claim that general-purpose models will never be competitive at chess.
This is not the claim. The claim is that for the same amount of compute, a general-purpose language model will never beat a Chess model. I'm dubious, but allow for the possibility that a language model could eventually compete at a top level against humans with enough compute. However, it will never compete with a dedicated Chess model with similar resources. Training a model for a specific task with the same amount of compute will outperform training a general-purpose model with the same amount of compute. This should be common sense, right? The bitter lesson was only about compute over human algorithms, not at throwing compute at a generalised domain over throwing compute at a specific domain.
You made arguments against two claims that I did not make (that I was trying to refute the bitter lesson or that I claimed that LLMs could never be competitive against humans at Chess), so I'd like to ask you read my statements a little more carefully this time.
I see arguments like this all of the time in ML. There is a large branch of statistical theory that can cast some doubt on this.
Stein's paradox: when three or more parameters are estimated simultaneously, there exist combined estimators more accurate on average (that is, having lower expected mean squared error) than any method that handles the parameters separately.
This goes deeply against "common sense", but is mathematically proven, empirically proven, etc.
If you are just taking a chess model vs. an LLM, you are right.
But if you want a chess model, a medical diagnosis model, and customer service chat model, at a high level the intuition of Stein's paradox is that one combined model will get you better performance than three task-specific models.
Most practitioners don't get this because what amounts to graduate-level stats theory is often brushed aside, and we keep relearning it the hard way.
https://en.wikipedia.org/wiki/Stein%27s_example
And in 2020 Stockfish 12 adding some NN evaluation. And then in 2023 Stockfish 16 entirely removing the classical position evaluation code.
https://stockfishchess.org/blog/2023/stockfish-16/
Bitter lesson #1: don't waste time optimizing code when a faster processor is around the corner.
What countered it: Moore's law stopped working.
Bitter lesson #2 similarly relies on scaling laws that might have diminishing returns wrt model runtime vs intelligence. Runtime matters for turnaround on the problem you're solving.
However, what most people think of as Moore's Law--CPU speed doubles every 18 months--broke somewhere between 90nm and 22nm.
And even the actual Moore's Law--2x the transistors every 18 months--doesn't hold for all types of chips anymore. Memory only gained 2x density over 10 years.
The actual argument of the Bitter Lesson essay is pretty limited but people's interpretation of it has gradually drifted until it's seen as prediction that current LLM will reach AGI at a large enough scale.
When you're doing something "at scale", you want smaller models, weaker heuristics and rules written in code, as they're much cheaper, much faster, much more predictable and much easier to reason about.
If your task is "tell me which pages on the internet are most relevant to this search query", running Claude Fable to evaluate every single page would be ridiculous. You may run something like it at the end, to synthesize information from a small result set for example, as that's a "human-sized" task.
If your task is "write a function to sort a list of 1mn entries alphabetically", a small coding model will exactly match the performance of a frontier model (it'll 1-shot it, with 100% accuracy). So the only differentiator is cost, and the frontier model will cost orders of magnitude more than the specialised model.
For most intelligence based tasks, you don't (and never have) needed the tool which "performs best at all tasks". You need the cheapest one which performs adequately for your immediate task.
This doesn't mean the bitter lesson is incorrect. At the frontier, it's still correct. It means that it's not applicable at all to lots of tasks.
VibeThinker 3B constitutes extraordinary evidence, IMO. The first such evidence I've seen myself. Very small model, very low literacy, almost no world knowledge, but it is as good at math and logical reasoning as models a hundred times larger.
The Bitter Lesson is a valid and trenchant observation about how about we got here, but I think it's a mistake to assume it tells us very much about where we're going. Too much has changed recently and is still doing so.
Is this whole thing than maybe a read vs write optimisation again? Spent more time and effort training more knowledge into the model upfront and get it out in a single question instead of training a small model and needing more steps to answer the same question?
Any similar model aimed at coding?
A >10B model for mass spawning/swarming and reporting back to a larger model
I wouldn't use it for anything important without heavy supervision, as it's very weak outside its specialty. Not ideal for instruction-following tasks.
Absolutely false. At least when it comes to multimodal inputs, even a simple classifier will outperform the largest LLMs who still hallucinate details or don’t describe audio and images accurately.
And there’s also the issue of cost/inference speed. Running a trillion parameter model for all tasks will be incredibly costly, require a cloud API, while a tiny CNN can be run locally or at a cost multiple orders of magnitude lower.
One thing notable is that it makes a difference when I specifically refer to line numbers and variable names when I'm asking it to make changes. Meaning, I also read what it writes and understand what it's doing, and have specific queries where I already know what I'd like to change I add.
DeepSeek v4 flash has been dirt cheap and so fast that my development loop is;
- small prompt
- review
- small prompt
- review
I build software with the same quality I normally would but it's way faster to produce and I think more about architecture and flows than I do about implementation details. The small diffs let me accept / modify / veto diffs and if the model struggles, I just write it by hand. It prevents compounding defects from leading the model astray (like you see in vibe coding).
In some cases vibe coding is useful, like when the complete specification is available (e.g. creating a JavaScript engine that implements the standard) - but anything that requires iterative development sees vibe coding break down pretty quickly (you could argue that is the case for a JavaScript engine).
I feel energised by AI assisted coding rather than drained, as it's a force multiplier for my skills and it lets me build more than I could by myself.
That said, most of my team vibe codes and reviewing their work is like pulling teeth.
I think the muscle memory of doing those tiny problems is good for our minds, but solving larger-scale issues is also challenging.
I'm on vacation right now and getting claude to build a mostly-throwaway e2e testing harness (admittedly not small-prompt-review-repeat) for a backend API to speed up our existing e2e test suites which do click-ops to set up tests 8-10 years ago, we had a team who spent 3-4 months every year maintaining our E2E suite and people would do rotations on there to spread the knowledge.
I basically want an industry standard practice implemented on my team of 4 devs who are too busy doing other things.
For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playing my games.
But nothing one-shots a prototype better than Fable 5. I can have a prototype built in 30 minutes, hooked up to my local LLMs and Claude Code is very good at testing the interactions and even tuning the prompts of the NPCs for better experiences.
I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060.
My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that price tag seem a lot better than it was, but it was an investment in all things GPU that were happening in 2020 (crypto, blender, image gen), then LLMs exploded.
I'm not saying everyone has to run local LLMs, because the APIs are in a race to the bottom, and my $10 of OpenRouter credits I bought months ago is down to $8.94 because most models give you MILLIONS of tokens for a US Quarter.
This is tunnel vision. The percentage of people who could afford the hardware you could at the time you back it so vanishingly small. I do not know a single non-tech person who has multiple graphics cards in a single computer.
My personal expectation is closer to 5 years than 10, which is why I wouldn't touch Anthropic or OpenAI stock with a ten-foot pole, personally, no matter how high their theoretical valuation is. Because their business model is doomed in the long run.
Which means it's inevitable that eventually, even the consumer game market will be buying GPUs with 32 or 64 GB of RAM. And there are decent models that will run at that size. Even the "normies playing games" market, as you call it, will end up with the capacity to run local models. It'll take a few more years than it would have if the data-center companies weren't trying to buy up all the GPUs, but it's not like gamers are going to stop wanting to play games. So in the long run, Anthropic et al are still going to have to figure out how to deal with competition from local models that run on your gaming video card. Which won't ever be at parity with the models that take terabytes of VRAM to run, but are very rapidly approaching "good enough for what most people want to do".
Don't get me wrong, there are advantages to a fully local model in that, I can have agents looping 24/7 even when my internet is not working. But this is niche enough that if I had to price the advantages they don't seem worth it.
If I'm willing to pay the Openrouter tax, I can fire up Openrouter today and just get access to whatever model I want, and still pay a fraction for tokens as what I'm paying with the big guys.
3090 pricing is something of a wild card. Since the only big-mem consume cards are the xx90s, and a 5090 is pushing $5000, resale value has gone way up. The bottom hit ~$700 last year. It's still a very good GPU, if power hungry.
It’s not unfathomable that if a personal, generally intelligent local AI provides enough utility and doesn’t require you to tweak CLI flags millions of Americans would want one.
What matters is you've got the Duramax HD King Ranch TRD Big-Boy machine. Doesn't matter the cost. You can tow anything, drive anywhere, do anything, and do it all in comfort. Other than parking in a normal parking spot comfortably. Or even park it in your own garage at home.
I've seen this exact scenario many times personally.
Americans by and large don't do that. Much of the population engages in discretionary spending with debt instruments. Combined with mass innumeracy, they're all oblivious to the true cost of their purchases because they only think of the monthly payment.
It was a 96 core gen 4 epyc+supermicro board build with consumer NVMe drives on 1x16->4x4 "dumb" bifurcation cards. I had to get a few MCIO-> PCIe adapters as well to get the full lane coverage. Mounted in a standard EATX compatible consumer case with a consumer PSU and a lot of Noctua fans - surprisingly cool and quiet for what it is.
Motherboard+CPU I got from Ebay. Rest from the best MicroCenter/Amazon/Walmart deal of that day. Bought juuuuust before the AI pricing apocalypse, largely by pure chance.
You could sell those and have enough money to pay for hosted inference for years.
You can do each of those at various hosts and own nothing. Or own a couple "over priced" cards and do it all at home on battery power for a few hours while the power is out.
Your comment is like a meta comment of "LLMs are generating everything, after a while the ouroboros will eat itself. (Which I agree with)" If people aren't hacking on this shit just because, you have completely conceded control of software to a handful of sociopaths, and open source software is dead.
I'm sure most of it comes down to prompts, and all of them run over 100tps on a 3090. Smaller cards will likely be slower, but Qwen3.5 9B is small enough to fit on most consumer cards.
However, back then I was getting the AI to write individual functions or classes or a test suite. I was decomposing the larger task into smaller tasks, delegating some of them to the AI, reviewing the results and composing the codebase from those. I was also essentially the harness.
Today the models can write and test and deploy an entire project. In terms of the code quality, I actually don't think today's frontier models would have written it much better than the 2023 models did. So in terms of raw coding capabilities i.e. converting a high-level specification into working code, I think we hit the peak way back in 2024 itself.
What has changed is the AI has learned how to do the task I was doing (besides being the "harness"!), which was the mid-to-higher level "engineering" aspects like decomposing a task, specifying it to a reasonable level, reviewing the outputs, and course correcting as needed.
I'm not sure if that is something the AI labs explicitly focused on during training (which may be why Meta is having its highly paid engineers do annotation work), or an emergent property of "better reasoning" (which I believe Dario implied in a podcast), or some mix of both.
But the fact remains that even the weaker models are more capable than we realize, and many being open weights, are here to stay.
I find that when I give an LLM my full handcrafted codebase, it does very well. It follows my conventions, sees the intent and can coherently build within its scope. It writes much better code than a 'vibe' prompt.
It is always tempting and I myself will continue pushing the boundaries, but when you keep an LLM in reasonable scope (that may be one line, function, file at a time, depending on your idea of reasonable), you, by definition, can get sound utility out of them.
Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid.
Mistral 7b can do anything, and it’s basically instant even on an M3
Actually built a full invoicing product for that, using it too.
I use Mistral 7b and LlamaIndexTS on Node, I run it on a MacBook M3 and on a Linux server with only 8GB VRAM (old gaming PC).
Basically flawless, runs very fast and I don’t even know what paying for “tokens” is :)
Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details.
What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response.
If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use.
Even the best models available lose a ton of detail over time if you were to paste in tens of thousands of lines of code.
The only way to hold huge amounts of context with a high degree of accuracy is to store it using various mechanisms (one of which is RAG).
On “effectiveness”, I mean end use case effectiveness in the tasks at hand, not whatever benchmark the model developer or vendor themselves come up with - which may or may not be useful to the work I’m doing.
For a casual consumer, it is no doubt a worse chat buddy. Knows less historical facts à la Wikipedia.
But I’m not really using LLMs for that kind of entertainment and I don’t rely on them for fact-checking anyway. It’s almost worse to rely on a smoother talker for something it can’t possibly know.
For feeding in thousands of daily updates and getting it to predict the next one based on a crude list of tasks, it’s great. For completing code files in my style, also great. It can also handle most small customer service issues and refunds related to my apps on App Store, the back and forth to collect data from users.
For a booking engine I run with a partner, it is great at confirming bookings and following up. It handles cancellations which are about 40% of confirmed bookings due to the nature of the business. I used to feel like I was wasting my time with them - there is no way we can charge for it. Having basically a smart bot do it makes everyone happier (even the customer who knows they aren’t hurting anyone’s feelings).
For invoices, I prefill notes and things based on previous ones. It’s fine. I don’t need a massive model that takes 30s to reply etc. it’s way overkill. Maybe worse than overkill - off in a direction I don’t need.
It will be interesting to track the improvements of these 7B model over time.
There will be a turning point in the next few years where it attract enough consumer attention to create yet another Smartphone and PC super cycle.
- ingesting all the current knowledge without regard for intellectual property or the work of people that went into it; then
- claiming that AI would make all those people who put in the work redundant
It's not really surprising that when the sales pitch is "this will eliminate human creative work in all writing and illustration centric industries", people got angry.
The people involved at the leadership, in this wave, or incredibly toxic individuals. When the leaders behave a certain way, it tends to be reflected by those following after - intentional or not.
The cost isn't just what you're billed. There are security, privacy etc. concerns.
If Orang mane bans Claude, they've got their local models.
The latter has already happened too so I'd say their risk modeling is spot on.
A sales person sending a prospect email doesnt have a way to write a test harness for it. Yet these tasks dominate what humans do compared to writing a crud app . otherwise anthropic wouldnt have trillions dollar valuation
These things would be considered magic even 4 years ago!
Can even run on a notebook CPU and comes in Base (best for FIM), Instruct and Thinking variants. mradermacher has imatrix quants for people who can't run it at Q8.
IQ4 should fit, but even if it doesn't, llama.cpp has options to partially offload models to system memory.
I use an 6bit quant and get around 20 tokens per second.
With the additional caveat that I don't know whether that specific card is supported by modern drivers.
You'd be looking at one in the 6B or 7B parameters range at FP8. Or smaller. It's been quite some time since a recognizable company in the AI space released a model that small. You can try larger model that has been quantized down to that size, but they don't always fare well with that.
Modern text-to-speech and speech-to-text models also fit well into modest amounts of VRAM.
An old nVidia brand card with 8GB is more than enough to see those models running at usable speeds and accuracy.
What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start.
Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do the hard part of understanding specific consumer needs and wants and pursuing that.
I think there is more opportunity for consumer application of AI. Im surprised we dont see more models in game. Small models that do more purpose driven things not "this npc is just a chatbot with a personality in its system prompt." I mean we literally have systems in games called "AI" and we dont think these much more powerful AIs have a place?
> But what if you want to add AI to your product? Well, now you have some real inference costs on every request!
Eventually, these companies just lower their costs by using more efficient models. There were consumer companies built on GPT-3.
I don't think anyone is doubting this, but were they any good? Remembering back my first interactions with GPT-3, it was fun, but hardly reliable nor "ready for production", besides making cool demos.
.. whether you want it or not!
We're at the point where it's becoming a product sticker like "fair trade" or "does not contain nuts" to say a product wasn't made with AI.
AI is a utility that can abstract code to such a high level it is indiscernible from natural language.
Something that helps homeowners build anything on their property, for example https://myhomecat.com
Especially if I need to ‘consult an expert’ isn’t this just a customer company using AI?
You can say what you want, and the tool has appropriate guardrails to make sure what you're going to build is safe.
> 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of.
> 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts.
Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html
I'm curious about not only which of these roles models will fill, but also how they will empower us to be in the mode we prefer.
Now in many cases the "Guy" is an LLM. I know enough to know when the LLM is bullshitting me and when I need to consult an actual human.
> Thanks to Sam Altman, … for reading drafts of this.
Oh how times have changed!
Now I think Luna is plenty good for many applications inside a very good harness/scaffold. And I think there are a lot of those usecases. So I think these small models are really good for application developers.
But for entrepreneurial knowledge work all of my work still benefits a lot from more intelligence.
It's safe to say I didn't come to this conclusion because I was "addicted to chasing frontier models"
The word “most” is doing a lot of work here. On a percentage basis perhaps most tasks a typical SWE needs to do when they aren’t in meetings or writing docs are just glorified autocomplete. But that’s boring and that’s why people don’t usually talk about it.
People are addicted to chasing frontier models because they all have memories of spending a week on a deeply challenging algorithm problem or even have crazy complicated algorithms they cannot implement themselves and want to have the models achieve this technical breakthrough. It’s the kind of productivity boost from spending one week on a problem to spending one hour. In contrast the productivity boost from spending ten minutes to spending one minute just doesn’t occupy people’s mind.
I'm not sure I know very many engineers who would fall in this bucket. Or do you mean the business types who suddenly think AI can replace all the engineers?
In fact I noticed that this is the one place where people discussing AI on HN tend to talk past each other. On the one hand people are talking about supreme intelligence like designing new algorithms (on the same vein as finding counter examples for the Jacobian conjecture) and on the other hand people are just satisfied using AI to automate a few quotidian tasks that hitherto couldn’t be automated.
There are many applications where world knowledge is unnecessary or even a negative, and in which only a small amount of language skill is necessary, and there we can expect small models more intelligently used to beat large ones naively used.
The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge for local tasks" type dreams.
And to be clear I'm not saying that smaller models don't or can't work well, or that we shouldn't be heading in this direction. And it's not quite the case that broad knowledge is strictly necessary. But it never seems to be negative! And so far it is the best way we've found to do... everything. Small models are good to the extent they are like big models, not to the extent that they are small.
For example in language, this is called the “curse of multilinguality”. Small models that handle a single translation direction can easily outperform big ones that try to handle them all.
https://arxiv.org/pdf/2311.09205
In any case, for most tasks the question is not “how many tasks can this model kind of do well” but “given time/cost constraints, what is the maximum level of quality we can achieve”. And for that, small models are usually very competitive.
It feels like claiming that because a current generation laptop CPU is slower than a current generation HEDT CPU, we should ignore that the laptop CPU is faster and more efficient than the previous generation HEDT CPU.
Of course a huge system could do more, but these systems are constantly evolving and just in this month we've seen comparatively smaller models approach capabilities that were frontier AI less than a year ago. What you're saying would be true when we eventually find the limits of how much models can compress knowledge. Until then, all I think you're saying is that companies that are willing to burn more money can afford to ignore efficiency.
But the idea they’d be better than a bigger model is cope, you’re pretty much always better off running the biggest one you can bring to bear within your constraints.
There's a million ways to define "better at writing prose", and it's only better at some of them. It's worse at many others.
Try using a LLM model for RAG embeddings and get back to us on that.
"Source: rare book ingested and shredded by Anthropic. No, you can't look it up and we can't show you the scan. The remaining open market copy is $5000. Trust me."
I think hoping for a locally hosted "smart" home is backwards. They have no other function than to invade your space. "Smart" objects are agents, and they don't work for you.
Now, if the debate is really about which option is more cost effective, then we could easily run an A/B test to find out. Though TBH my instinct is that that experiment is likely to cost more than the potential cost savings.
What I will say is that my own sense from experimenting around in a non-rigorous way is that the answer depends on how you use the tool. For actual vibecoding you should always go for the SOTA model because it will need less oversight. It’s also less likely to get stuck in a vicious loop that fruitlessly wastes tokens. But for a more hands-on approach where you move in small, carefully planned increments that you review and test in human-comprehensible chunks, smaller models may be preferable. SOTA ones don’t do that much better when working that way, and the slower inference adds a detrimental amount of friction to the work cycle.
I'd object to "easily". It's already hard to measure whether AI is generally worth the cost. Let alone compare models in such detailled ways. It's mostly handwaving and gut feeling Doesn't mean the conclusions are wrong, but biases are strong.
If this is a cost conscious company where I'm going to get a fairly limited amount of Sol, or a nearly unlimited amount of Luna, I'm probably choosing Luna.
Confidently wrong here. It's absolutely not across the board "more capable" than GPT 5 or Opus 4.1 or even Gemini 2.5 Pro. It's potentially better at certain specific tasks, mostly agentic coding implementation work. I.e. tool calling and usage of bash. It's worse at a large range of other tasks.
Sol medium has been a nice balance between intelligence and response time. Luna xhigh can achieve similar scores on the evals, but it takes noticeably longer. My impression is that the higher reasoning effort helps compensate for the lower base intelligence.
Cost is definitely a big factor, but latency and intelligence matter too. If I had the budget, I’d take Sol medium over Luna xhigh.
From using both on real scenarios, Sol is noticeably better at navigating around issues, exploring alternatives, and being creative when the obvious approach doesn’t work. That matters quite a bit when you’re investigating live alerts, where the path to the root cause isn’t always straightforward.
Sometimes I will use Fable or Sol for large features/projects, or research/exploration.
I would not be at all happy if I were forced to use Luna, though. I’d probably start looking to leave. I don’t want to work somewhere where I don’t have choice over my tools.
Yes. Categorically. Anyone who tells you otherwise and that luna is “just as good” does not know what they are talking about.
Going from sol to luna is a downgrade.
It is not a question, it is a fact.
> Is sol actually worth the extra cost?
Is a question only you can answer, because it has no generic answer.
Right now, for me, being able to use sol is worth the cost, but using it all the time is not.
I’m sure going from using it to using luna feels rubbish; but there are realities about costs you have to face sooner or later.
Maybe like… give your team credits and make them pick the right tool for the job; and if they burn their credits on sol in 20 minutes, well, tough luck buddy, looks like you're coding by hand for the rest of the month.
Team will quickly shift. People hate losing access to ai.
Sure a Lexus is better than a used Prius, until you include price
You cant just go “oh hey, I guess they're both cars so I’m taking your lexus away, catch a cab its cheaper” and expect people to just hug you be be like “yay, thanks! I still have a job I guess! :party:”
:P
Obvious to the meanest intellect they can tell the difference between the cars.
Don't complain to me if someone responds using a stupid metaphor that proves the opposite of the point they were trying to make.
Sol already lacks judgement. It will absolutely add idiotic tests and comments. Luna is that but worse so if you account for things like going down wrong paths, producing bad results, overthinking then it could easily cost you more to get less.
Gosh, watching paint dry has been a better value than reading The Economist in the last 5 years or so.
That aside, I had good results with Luna. I'd be interested in hearing about a comparison that takes into consideration response time (not TPS), cost and performance of the popular models at different settings. That chart has some of that. For instance, is Luna Max a better value than Terra Medium?
I question this choice. Coding models have improved significantly in the past 2 years.
[1] https://qwen.ai/blog?id=qwen2.5-coder
I would try using ~1/2 your available ram and iterate from there.
If you have 32GB of RAM, I would give Qwen 3.8 a try. All you would have to do is run "ollama pull qwen3.8:27b" then "ollama run qwen3.8:27b". If you have 16GB of RAM, I would try Gemma 4.
[1] ollama.com
- Linux, general use case, balance - 16 GB RAM - 10 GB VRAM
Recommendation: Kimi-K3
This checks out.
fitmyllm is much better
64 or 128GB RAM, 6 or 64GB of VRAM....
Not sus at all.
Kimi-K3
I think the slop site is broken or compromised...
By the way own exploration sort of led me to qwen3.5:9b for the best case scenario balanced model considering almost 10-11GB of RAM is almost always gone anyway. Even with aggressive app quitting.
For my anemic 6GB built-on 14GB Qwen seems to be the best bet, not great reviews but from my limited testing its pretty impressive.
I think AI is an underused capability in applications. People come up with all these big plans to build agents. But even a simple button to "calculate a color palette that takes into account these symbolic names the user entered" is a simple prompt that comes back quickly and doesn't require a lot of reasoning. We use that with Luna and we give it some more context.
Image based stuff is more expensive but still quite reasonable. We do some things with Sol that work on images. It's not fast but the result is worth waiting for. These are features that save the user time and feel a bit magical. They just click a button, wait 20-30 seconds and then magic happens. Beats them clicking around for half an hour.
Local model hosting would force me into higher cost and overhead for less intelligence. That's only a good tradeoff if you are really going to use lots of tokens or you need this for other reasons (e.g. data security). But otherwise you just end up paying for underutilized GPU capacity.
I think with one more year or so of small model progress, that might just be possible to accomplish.
Bootstrapped it with fable, opus or sol 5.6 when I had extra usage to burn over the last month. Playtesting it solo as both DM and PC was fun too. Character consistency and originality is remarkably compelling.
Upshot is that for the gaming hours that my friends have available, the social interpersonal gaming experience competes well for their AAA gaming time. There’s even some good async passive campaign continuation patterns that run in our signal thread to keep things going between sync sessions.
And players have requested features directly in the session transcript that have been direct implemented.
Are you going to publish yours somewhere? I'd love to play Cyberpunk 2020 with AI
I know of SmolLM 2 which in Q4 is borderline regarding the size and rather dated. There is also TinyStories, which is also old and also focussed on children's stories.
Is there anything newer in this category? Or should I try to distill something down to this size?
However, you'll have to gauge for yourself how closely their tasks resemble your tasks.
1/ https://cognition.com/blog/frontier-code
2/ https://cognition.com/frontiercode
Given sheer number of turns I can make with small models, I can do a lotta stufff
- cheaper, and faster
Harness makes differences: There have been many HN posts about how one made tiny models work better at certain tasks using harnesses.
These "small" models with right context, and guidance, they work wonders.
---
I've been saying Luna has been my go-to AI in previous comments and why Luna is still more compelling than GLM-5.3-flash.
- https://news.ycombinator.com/item?id=49450353#49452248
Once the economics change and we see less bit model deployments there will be much more work optimizing models at the pareto front.
Also yes people need to be able to sustainably run these things to start running a business. Building a business on the model pricing today is as smart as building a retirement plan on a gambling schedule
What most products need from a model is a pretty short list: the ability to make tool calls well, accurate recall, and the ability to follow directions without wavering (whether or not those directions are baked into the weights or provided in a system prompt). That covers 95% of inference utility in products.
We're nearly there, and I believe these capabilities will fit on small models.
Because of this though, I predict hardware demand will stay high despite demand for "hosted" inference dropping. Unless there's some regulatory shenanigans that step in to say otherwise.
How much of this can be replaced with small models, and what changes to my workflow do i need to make?
oh-my-pi's Advisor feature can fill a lot of this gap (a smaller model told to reflect on its own output catches when it's doing a rush job and skips instructions).
And it was correct. The results weren’t as good as they should have been for Sol. How am I going to trust Luna?
This is a good insight broadly!
Not trying to plug, but I do't know any other way. I wrote a piece couple of days ago on small models and memory usage on the edge devices- https://polign.com/blog-edge-agent-memory and https://news.ycombinator.com/item?id=49450816 closing on the same problem.
Goes to show that open source = cheap, closed source = expensive is wrong-headed. Closed source is going to beat open source on price point due to scale (try buying a few GPUs to run your own model on...) and labs innovating across the full inference stack.
That leaves the value prop of open source as privacy / customizability.
That's great, but the thing that worries me is that many companies have billions invested in the AI bubble. It's around 1.5 trillion last time I looked. It's all circular spending between the companies building out the infrastructure, and the models. None of it is profitable. They will want to recoup that 1.5 trillion from consumers, which means using online-only pay-as-you-go cloud models. They will inevitably see that people using capable local AI are "lost customers" and they will try to kill the ability to locally host AI or somehow enshitify it enough to make paying a subscription more palatable.
I'm not saying I believe that will happen, I'm just worried that it will. Is anyone else worried about that as well?
At some point there will be a surplus of fast memory and even in a crash the current generation of SLMs are bounced to be plenty to build a lot of intelligence at home.
You can do this before an exit? Build and fundraise for multiple (3?) companies at the same time?
replit is already leading the way with free luna usage
I will believe that smaller models have hit that bar empirically when I see them in production. At the moment, even frontier models are stuck in most of the scenarios I am seeing for high value tasks at the "not good enough" gate - so small models are not even close to being on the scene there yet.
the application Layer i.e having a good graph RAG & connecting it up together is the missing piece for most.
I don't have to be an automotive engineer to start my car and put it in drive.
e.g for a lot of small business - they generate invoices - feed that into a RAG system. Then have an agent etc do follow ups. for agents then the smaller models are enough.
[0]: https://help.getzep.com/graphiti/getting-started/overview
At first the results were hot garbage, and progress was slow. I hooked up the settings to download models from Hugging Face conveniently, so I could run experiments faster, and I massaged the prompts a bit. Last week this feature made a qualitative jump from science experiment to something I'd actually use.
The fact that all runs on the device means I've got no variable costs associated with adding this to what will be, at best, a pretty low revenue product. I've tested it on trailing edge devices like an M1 Mac and a Pixel 8, and performance is very tolerable.
The key is I'm not asking for open ended answers to open ended problems. When it proves to be useful it's not going to get less useful or more expensive.
There are vast domains of uses for LLM models with similar characteristics and likely similar results.
I just want the smartest, best, and most capable models. It feels smaller models for speed and cost are just transitions towards better hardware allowing the very best model.
There are still tasks that even Fable is bad at doing. And many are just mundane things. Because of the fact that you have to steer it on those tasks, you might as well steer an 80% model that is 5x faster. And those do exist.
Naturally there’s a bit of a gap because the faster models need steering on tasks the slower models don’t so there’s no smooth transition but I find it worth it. Especially if you want to stay in flow.
Ironically this sometimes means starting a plan with a great model, planning with a worse model, iterating, then submitting it to a better model for review, and then having the better model do the implementation.
Second, when cloud models become unavailable or otherwise deteriorate, these will be all you have. May as well prepare.
It's a bit like saying you always want the fastest and best car; Sure, you can have it if you keep paying for it. But a small car will also get you from A to B, will use less gas and will be much cheaper.
;D
One time, it said "You were right; I corrected the regressions and committed them."
It was still not fixed.
I do admit not hitting the five hour limit in two prompts is nice. However, I wonder if becoming an angry person because of cheap models is worth it.
It sucks that we got to this point when it comes to our civilization. At least guys like this one here can be happy with their AI-sourced money.
I'm keenly interested in seeing super small (like 10's or 100's of mb) special purpose models starting to come into their own.
that might actually be transformational globally not just in rich countries.
now do the same for robotics, because itnernet trained vlm + action head was really a dumb take
>”@dang I really need an IP &/or account ban”