A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis
j / k
|
Down or up a comment |
h / l
|
Previous or next link |
?
|
This help |
Just use the LLM to make a thing that's actually good at what you need it to do.
I think it reveals how many people were only familiar with LLMs as the generic AI tool for everything. It only took one ambitious marketing campaign for someone to come along and present a simpler technology as a new breakthrough to audiences that hadn't considered anything else.
The largest LLMs are really amazing for how much they can accomplish on unseen tasks. That is undeniably remarkable. The mistake that gets made everywhere is thinking that they're the best tool for every task, because LLMs are the only AI tool in most developer's AI toolbelt right now.
That’s exactly how the biggest vendors have been marketing them, it’s no wonder non-experts think of them that way. I’m not an expert myself but have a passing familiarity with theory as a SWE and grad student who did some research for machine translation and currently am in a computing-adjacent field. I use LLMs heavily for work and have never come across any coder-oriented tutorial, guide, etc. encouraging use of LLMs as decision models specifically, even if they are being used that way “beneath the hood.” Jev’s announcement seemed sort of new functionality to me but even in the first post about it on HN (where I heard about it), iirc people were pointing out that LLMs do the same thing, at least to some extent.
I don't know of any devs in real life who would loudly and publicly get excited about anything unless it was their own completed project to show off.
The tools are a footnote. It's precisely because it's so hard to get a project that far along that they would be familiar with many more tools than they would ever speak about. I suppose some people have a low bar. I wouldn't really call them "devs", especially now in this "AI era".
I think you're coming at this from a very narrow and specific experience with a few people you know.
Jev really went from the announcement to being discussed all over overnight. We're discussing how popular things are in general, not among your narrow and specific group of people you work with.
I think it's clear we're at the tail end of this cycle. I'm very glad to see it.
Like that old adage "that's a feature not a product" - Jev is just a system prompt, not a product.
Doesn't matter it's wrong significant fraction of the time and you have no clue what went wrong because the output is yes/no.
Classifier models actually make it think about how the input should be formatted and pre-processed and that kinda breaks the dream.
Unfortunately LLMs are magic to them.
When you lack a first principles understanding of how something really works, you will bolt on hacks and duct tape on the wrong approach.
It's also nuts because "traditional ml" generally requires a fraction of the resources of an LLM. If you need to auto label, tag, categorize, it's never been easier to setup that sort of system in any project.
This is the end state of most imagined use cases for “agentic” software IMO. Use the AI to build software, with a classifier or LLM as the expansion joint in awkward spots.
I had a team proudly show me how they were feeding their dashboard data into an LLM to predict growth needs, instead of doing basic linear regression or any number of pre-existing basic statistical analysis. The part that depressed me the most on that was it was a bunch of engineers that did it, the people you'd most hope would understand what an LLM is/does.
You can go “here’s a spreadsheet of some data, do a thing to it” and it’ll go through, have it plan out the type of calculations you want to do, and it’ll write you a program to parse your sheet, calculate what you want, then output an result sheet.
The key to good results with minimal effort seems to be the planning modes to me, combined with the /goal command. A little bit of validation before implementation goes a long way.
I'm mostly starting to have some success telling TPMs and engineers to ask chatgpt to give them appropriate code that generates more accurate answers.
Today the problem you're mentioning is easily avoided if you are aware by ensuring you direct the agents to build tools for anything deterministic.
Every week "we" (myself and Claude) start by analysing revenue from the previous week. Claude will do some manual proposals but they will be based on analysis done by a deterministic script. It will then run workflows that are increasingly calling deterministic scripts - every week another few decision points the agent used to do (and that I had to then manually vet) gets replaced by deterministic scripts.
It is what allows what is a very low margin opportunity to work - it is scaling with the total amount of manual labour and total token use remaining near constant. It's an opportunity that I won't go into much detail about because the margins are thin and the moat narrow and it's basically only viable because of this approach of setting a tiny budget where ~1/3 of it or so is dedicated to systematically eliminating manual and LLM work from the process.
Even so, I expect the opportunity to actually extract profit from it is going to be short-lived - the moat will eventually evaporate if LLMs keep getting cheaper and better in favour of people "just" asking an LLM to do the whole thing end to end.
So on one hand I agree with you - there are lots of things that'd be better automated with scripts, and where there might well always continue to be efficiencies you can extract by turning things into deterministic scripts instead. On the other hand, LLMs are already so cheap for a lot of things that it increasingly doesn't matter, and good enough that where it does matter they will increasingly also just write tools themselves if you ask them and sometimes - and likely increasingly so over time - without even being asked.
That being said, the correct way would be to just use the LLM to produce the script, validate it themselves (often times there's something very minor wrong), and then use that script next time instead of an LLM to rewrite the same script over and over every time you need to produce a report.
However sometimes I think that AI use shows a revealed preference around things. In this case, the actual data just ... maybe never mattered to them at all? People wanted a shiny chart and they got a shiny chart.
For example, a client want a classifier to decide to make a go/no go decision on roofing jobs based on weather. The end result doesn’t use LLMs at all, but does use a lot of other techniques including human feedback when it makes the wrong call.
(Now the work crew gets the decision at 5 AM instead of 5:30 and the guy making the decision doesn’t have to wake up at all.)
As an example, I've been using winnow running locally as a decision model in openchamber to select between llms for doing various tasks. I don't have an eval set of different coding and other tasks that I could use to train a proper classifier. This I just download off huggingface and it works out of the box and has cut my token spend to a fraction of what it was. So it's good enough to be beneficial.
You have to know the right words to ask the LLM to make and train a classifier. You and I already know the words. But they are not actually that trivial to learn for an average joe, and people are lazy.
«We came full circle on the problem that also exists with Google Search, and why so many people don't get good results out of it.» [0] It is not that access to all information was hidden to anyone before, it was harder to find it, sure. Now it can be found easily and aggregated, however, it is not as simple as saying "Do X" and believe that X is correct. Recall the infamous Swing & Product Management meme? [1] The problem is the absence of information you provide, the LLM will have to make a guess of what you mean, much like the human. Likelihoods of this guess being correct depends on the domain and background. So yes, it'll likely build whatever you ask it (I do think we're there already), but if that matches what you intended to actually get is vastly different. You'll be settling with "Good enough" though and declare it working.
Others already pointed out that 6 months seem to be the constant of distance, irrespective of the date.
[0] - https://news.ycombinator.com/item?id=50063427#50063744
[1] - https://www.productftw.com/productftw-2-the-tire-swing-carto...
https://waymo.com/blog/2025/10/creating-an-all-weather-drive...
The corpus of code available to train LLMs on is rich in both volume and quality. The same can't be said for LLM prompts. Prompts only came into existence alongside LLMs themselves, so most of the prompts out there are actually LLM generated, and training on them causes every model to overfit to the same patterns.
LLMs can give clear and concise directives to other LLMs, but only for basic tasks. Beyond that, they frequently overstuff their prompts with too much prose and too much direction, and worse, they often fill them with subtle contradictions.
Of course, there's a spectrum here. Some models turn out to be better at this than others, and I suspect this is what differentiates Fable-type models from the rest, and why vendors have started keeping their models' reasoning confidential.
Ten years ago, in France, two doctors (one in a professional setting and the other during a private consultation) asked me about a wound I had on my shoulder; obviously I couldn't identify what it was, and they offered no suggestions, so I supposed it was not something important.
It took a visit to a cardiologist—for an entirely different reason—for him to suggest that I rapidly consult a dermatologist.
It was a carcinoma, and surgery removed it. If ChatGPT had existed back then, this carcinoma would have been removed several years earlier.
Current diagnostic setup relies on humans having vast expert knowledge, when in fact it's almost always impossible.
A well trained neural net/LLM will always be better at visual detection because of sheer number of examples it sees during training, compared to a dermatologist which, if lucky will observe no more than 20 patients/day x 270 days a year x 40 years of practice i.e. around 220k samples throughout his lifetime.
MD with other specialties will see significantly less and we are talking about top of the top specialists.
An average MD knows more amount medical diagnostics than than the average Joe, but in principle much less than a well trained AI.
Do we have those kinds of models already? Maybe not yet but, ceteris paribus, we will definitely have them in the future.
IMHO we do have those models already but they aren't available to the public, though their availability would certainly lower cost for healthcare providers.
Automation doesn't make up for carelessness.
Reasonable treatments for small, non-aggressive skin cancers (or suspected cancers) are extremely low risk. And reasonable practitioners will often suggest that you merely keep an eye on an abnormal spot - because it doesn't look worth the bother of any sort of treatment for now.
But if we think a little further and generalize then there is great risk (and by extension, harm) to delegate diagnosis and treatment (of health issues in general, including potentially or supposedly fatal ones) to ChatGPT and similar tech. That we'd end up with a net positive is far from obvious.
Conversely, ChatGPT can help avoid false negatives.
Bonus: after using them for a while, you get the beginnings of a training set if you change your mind.
You can do that with LLMs too - it's just a cost and latency question.
E.g. I use an LLM to label my e-mail. I do it because it's low risk enough that I don't care about it being incredibly precise - it does not automatically act on anything, it just labels - and it means I don't have to curate a dataset of what e.g. should get the "receipts" label.
An LLM zero-shots it well enough and cheaply enough to remove my incentive to train a model for my individual e-mail.
If I wanted to scale it up, Jev or similar would probably be better.
A dedicated classifier would almost certainly provide even better precision if I invested the time to curate a dataset.
But the threshold where that is worth it for me given the precision I get from a cheap LLM is pretty high, and it's also less flexible.
E.g. one of the nice things about using a generic model for this is that whenever I add a new tag, it starts using it immediately and odds are good that at most I need to add a line to a file with one-line descriptions of what the folder name means to get decent results.
So there's a space for both LLMs/Jev-likes and custom classifiers.
Use Jev-like models (or LLMs) when the effort to curate a dataset to train a dedicated classifier is too high, and switch to a dedicated classifier when/if you need to scale something to a point where the economics justifies the effort or when you already have the dataset anyway (EDIT: and if you start with an LLM or Jev, collect the data - it will provide a good starting point for your training set if/when you decide to train a dedicated classifier)
Cost/benefit tips in favor of supervised machine learning when training data is cheap and/or plentiful, or when you expect to be able to amortize model development costs across a whole lot of inference.
People vastly underrate the patience of people to learn and setup a classifier. Jev is one API call. Training a good classifier is hard.
Same for annoying training jobs, with sufficient data you can probably let an LLM loose on it in autotrain mode and it will come back with a decent classifier.
Heh this used to be an interview question I gave. You’d be surprised how far people would get in 30 minutes.
We have in the works a classifier for if a prospective customer will be long term profitable or not. This changes everything.
I'm curious, why do you use a classifier for long-term profitability? Wouldn't predicting expected lifetime profit with a forecasting model be more meaningful than just predicting whether a customer will be profitable or not?
Eventually we hope to identify ones who are going to be lifetime duds.
With enough marketing you can sell anything. Also a decent way to make a product is to do the hard part for someone else.
I've been training classifiers for ten years, it used to be hard. Now it's hard to do really well but it's trivial to do well enough.
100% this is a very good insight
Universal classifier just seems nonsensical. It seems clear that it can't have accurate probability distributions on any possible question you might ask it. Moreover, giving only "maybe" answers to everything adds a layer of equivocation that makes it harder to catch it in a clear mistake.
That said, I've encountered plenty of fuzzy judgment questions now (ie is this case more like example X or example Y?) that reasoning models struggle with. I could see how, in lieu of a dedicated classifier, a model like jev, ostensibly trained to guesstimate the relative likelihood of natural language hypotheses, may do well.
But it still feels wrong to call it a classifier in the traditional sense, let alone a universal classifier.
The list of categories are provided by the government and can change several times a year, and the product descriptions are almost random (think the various titles of AliExpress listings of essentially the same product).
What kind of dedicated classifier would be better? Genuinely curious.
Agreed, and that’s the premise behind Jeffy, open source, bundles pretrained task-specific models that run (and train) locally on CPU: https://jeffyclassify.com/
At the same time, I’ve been working with some people implementing it, and there’s definitely value in having a more general model like Jev, or an LLM, for initial exploration and data labeling, to then use that data and train the task-specific model. It’s a whole workflow
Also, the market/pain insight of Typesafe is super interesting. They realized that a big chunk of the market was using LLMs as classifiers, and they built a solution around that (very much to your point, a task-specific model, rather than a more general LLM), and it worked! (Also they have the ability to generate a lot of attention, given the founder being ex-OpenAI and having raised quite a bit of money too), so it’s not just a technical/engineering thing, it’s a business/market/need/discoverability issue as well
I agree that a good model will outperform an LLM, but the LLM is so much faster/easier to build. That’s the usual tradeoff.
Got the same accuracy and f1 from the Opus-label dataset than the original one (I realize that Opus might have banking77 in its training data, so might need to account for that somehow)
But the point is: it's way cheaper to run the LLM model over your data to label once, and then use a cheap classifier, than using the LLM directly for classification
An off-the-shelf API has way less friction, if the results are good enough why bother?
"at scale" or the above use case of "specialized domain"
Jev being cheaper than an a full LLM call creates places where other tools might work better later - when the volume or the available data warrants it.
System optimal, code optimal and cost optimal aren't always aligned. Pick the right tool for the job - and dont get married to it. On occasion worse is better, and moreso when your testing or it is a one off/disposable.
Anomalies are very hard to detect on certain skin tones and creating data sets from image material is quite a lot of work, even if you have access to large clinics specialised on dermatology with diagnosed and undiagnosed material.
Not seeing LLM to perform very well here and hallucination would be a major issue (not at all a solved problem, even frontier models do hallucinate/match the most probable path. Perhaps a problem that cannot be solved without arbitrary rules.)
And I guess some SVM still outperform CNN, just that CNN provide a pretty good learning environment that can improve upon ingesting more data. Although there is a risk associated with that as well.
Still, if we had well documented data material, many algorithms and AIs would perform well. The work is getting the data in a dependable state.
If you really evaluate the best options, it’s very likely that a Jev-style decision model won’t give the best results.
This kind of approach works for one-shot "let's just see if it works" type stuff, but anything that I would stand behind needs at least some level of labeled data and curated dataset.
More generally, continue to build systems. More complex, more robust, systems.
An LLM is just one of many tools available to integrate into a system, but it’s not going to replace every system: trivially, any system that requires physical manipulation of reality can’t be replaced by an LLM. I’m not going to eat my morning cereal using an LLM.
There are tradeoffs (cost, accuracy, all sorts of metrics) that play into which tool you use for which part of the system. There will be choices that are always more cost effective:
Take the problem of determining if two people in a genealogy database represent the same person. It's a decision problem. There will be some data in structured formats you can count on, but which may be missing, or may even be wrong. There will be lots of fuzzy data (born in Newtown yes, but which Newtown?) and unstructured data (notes, etc.)
Now, do you want to try to assemble a dataset to train your own classifier on that, or do you want to throw a general purpose LLM-rooted decision model on it?
Eventually, yes, it makes sense to move to a pure classifier model as it's probably a lot cheaper. But then you need to learn how to serve your own classifier model correctly. Compute costs are being driven up by demand to serve models and unless you're careful it's easy to use your GPUs less effectively than a company specializing in serving a model.
I don’t have a deep understanding of this topic, but wouldn’t you also need a good amount of relevant data to train on?
The last place I worked and my current job have a tremendously bureaucratic process for standing up new internally trained models. You could easily spend ten times as much mental energy and effort doing the paperwork and revisiting it annually as you do in the care and feeding of the model itself.
If a canned product meets the requirement, there's a lot to be said for using it. Even if it costs a bit more per decision, the all-in costs are WAY lower.
I know which I’d expect to perform better.
Have you considered putting an LLM on the task of completing the bureaucratic gauntlet
While no doubt, you can get better performance using a specialized hardware … there becomes a breaking point when the commodity / general purpose is good enough while allowing you lots more flexibility as well as typically being cheaper that is become hard to justify the specialized solution.
Thinking about this every time someone shares their setup and there's LLMs doing simple periodical things directly instead of having the LLM write a deterministic Python script that will never try to hack your sandbox lol
I think its a combination of being new to classification models and not using in anger.
LLM Decision models are still relatively new and suffer from a number of issues. Order of choices impacts probability, it starts to break down over 20 classifications and even how you word things impacts probability.
Is this a typo? are you saying that this is so new people haven't struggled with Jev enough yet? very interesting
> it starts to break down over 20 classifications
Yes, I recently ran a benchmark on exactly that: https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
At around 4 classes already LLM-based classifiers (Jev/Laya/GLiNER et al) start falling behind
My understanding of the statement is trying to use it in production for the first time. Once you start to actually use decision models beyond something simple you start to see the limitations.
There’s lots of applications where it’s not being deployed at scale. Also lots of cases where someone who knows they need something classified does not know what a classifier is.
and because they start off generalized, you can train them against multiple of your workloads (they'll start with some generalized performance/world knowledge about them)
Which allows you to embed proprietary processes/data (world knowledge only you may have) across multiple domains into a model cheaply
subnanosecond response times
model fits on CPU
Yes, this.
I don’t want to pay someone to maintain a first rate classifier when I can buy a pretty good classifier as a service. I don’t want to pay someone to maintain and build server infrastructure when I can buy that as a service.
I feel like I’m being gaslit on HN when a solution seems obviously valuable to me because it eliminates or modifies constraints, and people dismiss that as if managing constraints wasn’t most of high level engineering work.
I tuned out of HN for a few years, came back in the middle of the AI renaissance, and I am constantly raising my eyebrows and hearing this Upton Sinclair quote in my head:
“It is difficult to get a man to understand something when his salary depends on his not understanding it.”
I'm surprised he's surprised.
> A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs.
Yes of course. A 30m params Katago network with one playout is superhuman in playing go, while frontier LLMs have over 10,000 times more parameters and aren't any good at playing go. They're good at other things (such as helping you create a network that's good at playing go).
The bitter lesson tells us that general _approaches_ outperform domain-specific understanding.
That doesn't mean a general model will outperform a domain specific one.
https://en.wikipedia.org/wiki/Bitter_lesson
For smaller training data sizes, transformers don't have a clear advantage over CNNs. CNNs are still the way to go if you have problems that are not too complex, and especially constraints regarding inference costs. Think of a small edge device / sensor that counts apples in your apple picking machine.
What we sometimes see is that a backbone is trained using transformers, then a smaller CNN is destilled from that (DinoV3 for example).
This is really what’s happening right now… LLMs are excellent computer users. Which makes one of the frontiers “amazing things computers always could have done, but we lacked the programmers to do”.
If basically anyone can train a high quality CNN for just about any purpose now, what new opportunities appear that computers can help with?
>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.
This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.
You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.
I believe the normal term is "gaming a benchmark test".
Nothing in what you quoted is related to that. It's a normal way to train and test a classifier.
The only way to determine if it's brittle is to test on a different dataset and look at the sensitivity and specificity.
Even so, there is nothing wrong with a model that is trained to a narrow domain when it doesn't need to deal with "wild" images.
Jesus Christ the techbros are stumbling upon the fundamentals of fields they’re unfamiliar with and treating them as completely novel discoveries. No wonder they’re so high off their own farts in delusions of grandeur.
That's just good developer practice, not "benchmaxxing".
Benchmaxxing is when someone presents a model as being generally smart, but actually quietly overtrained it on some benchmarks that were supposed to be proxies for general information.
If you train a small model on a specific task that's not "benchmaxxing" because there's no deception involved. It's just training a model for a task.
So it is not solving a task, it is solving the same test set. You can easily see these in simple OCR datasets like mnist. Draw a number yourself. It would be much less accurate than the 99% that good architecture claims.
That is not overfitting, that is training for a specific task. The measure for overfitting is how well it works on previously unseen data, hence the 200 test images removed from training and validation. That set serves as a second order of validation for the researchers to check themselves they have not overfit the model to match only the training data.
The 200 test images here are 'wild'. Fine-tuning in-domain is the whole point. This is what model training is. A skin cancer diagnosis tool doesn't also need to generalise to dog breeds to be valid, and will almost always perform worse if it does.
I developed many models from scratch deployed in various fields.
When you want to automatically detect a pathology from a give scan, you want it to be model that you own, train, use, and update, and not a generic vision capable LLM.
This is NOT what benchmaxxing is. You want the model to be accurate in terms of your cared metric- depending on the problem.
Slide rules displaced tables, calculators displaced slide rules and calculators themselves were displaced for the most part (they are still in use, in education but not nearly as much as they used to be outside of that) by the PC.
LLMs do that sort of thing for just about every information processing aid.
The harness in my experience makes a huge different too. Out of curiosity I tried having Gemini chat (with Argon) try to solve a level of this new puzzle game I'm linking (Order of the Sinking Star). It just got in an infinite loop and crashed. Then I try Antigravity with Flash 3.8 and it solved it easily. I saw the harness looked at other sokoban puzzles, coded some stuff, tried to replicate the level, etc. All things that a pure Gemini Chat doesn't really do.
LLMs are great for the high level harness, but if you want to count letters for example, it will still rely on "tools" like a piece of code to do it more reliably. Similarly in the above example, you wouldn't have the LLM directly try to analyze the images, but instead fetch and use the best tool for the job. But that can only be as good as the harness/tools it has access to.
Doesn't matter the domain or the task, the answer from some researchers is "LLM" always.
We're messaging each other right now inside a heavily sandboxed virtual environment that interprets XML-like markup and runs a text-based JITted scripting language from the 90s. It doesn't really matter that it's extremely inefficient, since it's still effectively instant.
Effectively instant means nothing, since everything is fast for small n. I love to give this example from a codebase I have worked on during my Ph.D. The problem which took half an hour under Matlab was done in ~40 seconds with mine, however, a thousand-step version of the same problem would take 11 hours. Saving a single second from that 40 second loop would shave my work by ~20 minutes, and this will compound for bigger problems still.
Just because we see the results instantly, doesn't mean it's fast or efficient by any means. It means we might forego optimizing it for sufficiently small n, if we don't care about that part of the code. That's all.
The app is less efficient only in an abstract, impractical way.
It depends on the calculator and task at hand. Consider graphing, integration or any programmable needs, and they might converge. ALU of a processor is pretty efficient at what it does.
> The app is less efficient only in an abstract, impractical way.
Again, depends, as I said above. You tend to brush efficiency away because you only mentally measure time to completion, only, which is understandable.
If it's not, then it's not.
I remember being blown away by CNNs and GANs when they first gained popularity. I do think, with skin cancer diagnosis, a transformer architecture is going to be the better bet.
However, I think it kinda misses the potential of llms for diagnosis. What happens if we also include journal notes? What if we let the model generate questions to ask the patient and include those? What if the images aren't perfect examples and happen to contain out of distribution content like a tattoo?
There's infinite front end code to train on. Virtually infinite python and shell scripts too. As I've worked my way away from such domains, the LLM outputs have gotten worse. It just makes sense.
What is maybe more interesting is that, despite the availability of open weight medical image-language embedding models like Google's MedSigLIP AND the opportunity to adapt them, they don't seem to break into the Top-10. My impression is that the joint representation is valuable down-stream of the diagnostic classification task for interpretation, communication, and management of the risk, but the language corpus for dermatology images are full of 'human engineered' descriptive features (think Bitter Lesson), and we should be looking past clinical semantics for more informative visual features if we want to push the limits of performance and generalizability. Ultimately, this seems necessary: people are immediately and persistently aware of the presence and extent of their skin conditions, the skin is the most readily accessible to imaging by patients themselves, smartphones have remarkable macroscopic photo capability, and there is huge demand for dermatology care.
Clinically, I'd imagine that choosing the right final diagnosis with relatively low certainty is often less useful than reporting the level of confidence (high or low) of each candidate diagnosis. A set of plausibles is surely better than a single wrong ID.
Clinical tests, especially when used by nonspecialists, are useful not only as diagnostic endpoints. Often they're essential to separate plausible diagnoses from implausible. If a CNN can rule out all outcomes but one with high certainty, that's nothing to sneeze at.
And if it could also _explain_ why the certainty of each candidate was high (or low) -- as non-LLMs often can do better than LLMs. That should have value too, especially in establishing an audit trail to defend the chosen diagnosis.
I'm not a dermatologist, but I imagine they don't make their decisions on one image alone (and the decision probably isn't "is this cancer" but "should we take a biopsy"). The LLM can take into account additional information such as the patient's age, occupational exposure and medical history, the CNN can not.
But then again, maybe it's useful to have a tool which you know does not use any information besides the image to make the call.
This is like saying I compared this against photos for sure stored in a home swimming pool (CNN), compared to photos presumably stored in an ocean (generalized LLM).
I use LLMs where I can, but I'm no automatic fan of LLMs, a healthier comparison here seems to be what if a general purpose LLM was provided those CNN images, and perhaps what does a fine tuned local model with the CNN images perform like?
It still might not be better, but I would suspect it's closer, and a more realistic apples to apples comparison.
General assistant models compared to task specific models seem to be a well known spot where things don't always fit, but might be improving.
Llms have their applications. Other models still do.
That being said, I'm sure a frontier LLM could make some improvements to your CNN.
This strategy is how LLM-like models have achieved leading performance on tabular prediction tasks (TabPFN).
Which LLM? GPT-6 Astra and Claude Fable 5.1, not specified exactly which version
Obviously this is a bit like comparing an 100 iq radiologist, 20 years experience, with a just-tested-at-iq-130 6 year old and observing the radiologist beat the crap out of the kid. Was anybody really expecting a different outcome? Let's see after fine tuning.
Btw: the actual lesson from that network when it came out, which is really important is that Inception v6, the Google trained version (ie. can recognize cats, halloween masks and airplanes) beat, with a wide margin, the version that was trained from scratch on clinical images (and still didn't do so well, 62%. Not even really good enough for a first triage).
So yes, this is an important finding that LLMs are not better than experience.
Anyone wanna take a bet that the general purpose model will beat these specific models in a year or two?
First, the CNN is not 10 years old. The base is but it was fine tuned for this project on the same dataset the test images were derived from:
> The test: The lesions come from MILK10k, a public dataset released through the International Skin Imaging Collaboration [ISIC] in 2025
> The CNN is deliberately not state of the art. I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU
The test images themselves were filtered out at least, but that's not really enough. The CNN has a home field advantage because the test images are going to be in distribution with the training images.
Second, Looking at the code, for frontier models probably you should be giving them image tools and using them in an agent loop and not trying to one shot results (e.g. https://platform.claude.com/docs/en/build-with-claude/prompt...). It is also not meaningful to ask an LLM for a numeric confidence score.
Third, all 200 of the the test images were biopsied by the clinician originally. So when you ask the model "You are assisting a dermatologist" and "Decide whether this lesion can be safely discharged, meaning it needs no treatment, no biopsy and no follow-up" why would you penalize the model for making the same call the physician did even if it was benign in retrospect?
Are you evaluating the model on if it can replace the need for a biopsy itself (in which case the ground truth is whether it in retrospect was harmful or not) or in if it can match physician judgement in determining when a biopsy is needed (in which case the ground truth is if they physician ordered a biopsy)? You're asking the model to do the latter and grading it on the former.
Overall yeah probably specialized models make a ton of sense in a bunch of areas still but idk this does not seem like a good demonstration of that
I know some people want to believe that LLMs are good at everything, but they're not. They're good at language.
LLMs are very general and flexible, by all means they can do well on non-text based specialised problems, but it also does well because their harnesses know how to defer to the right tools a lot of the time, and produces good answers from inspecting the output. Like if you ask Codex/Claude Code to transcribe a videos, it's going to create a script that uses whisper to extract the transcript not manually process the binary. If you ask it for demographic information it'll pull census data and write a script which produces the summary stats and report it back to you (I imagine much of the recent math stuff relied on this).
Sometimes it'll pull a number from its training data but the strength of an LLM is its ability to autonomously seek out the right answer even when it doesn't know the answer initially.
You could definitely use train an LLM to utilise these efficient specialised models, by it may be more cost effective to ask the model to build a special purpose application with a new UI over the top of the efficient specialised model.
It's comparing a fine-tuned CNN to general purpose models on a tiny dataset.
I don't think we should need to be reminded that frontier models aren't particularly good at most tasks. Just like we shouldn't need to be reminded that a hammer is not particularly good for driving a screw, or writing an essay.
It's an admission of magical thinking about a tool that's made out of the same material of every other digital tool since the invention of computing.
It's at the forebrain of every executive in America that LLMs imbued with RLHF are on the cusp of replacing every job they can imagine, from robot chauffeurs to carpenters to surgeons. But as this paper suggests, the integration of generally capable vision systems with basic tool handling that will be needed to replace the specific skills essential to most human jobs -- especially using only today's frontier models -- still takes us unforeseeably and undeniably far into the realm of sheer fantasy.
I will not flag it because the post is fine, it's the voters who I believe are acting poorly.