Thanks! I'm putting "doing cool stuff with really basic ideas" in my bio... :)
hn_throwaway_99
> In conclusion, I’m pretty negative about AI in general, but this experience made me cautiously optimistic about the role the technology could play in protecting consumers from counterfeit products.
Whaaaattt? How could the author go through such detail and discover that Gemini completely failed in the assigned task (calling all the products fakes when some were and some weren't means that it provided zero correct assistance in this discrimination task) and then conclude "AI could play a helpful role"??? Does the author also conclude that stopped clocks could play a helpful role telling time because the are correct twice a day?
Apparently AI isn't the only thing that hallucinates.
wgrover
I think my "cautious optimism" mostly came from the fact that Gemini found subtle problems with the counterfeit packages that I (someone who looks at fakes as part of my job) didn't notice - like mismatched info between the tube and box, and a malformed Irish postal code. If I didn't notice those, then I seriously doubt that a consumer glancing at a package would notice them. In that case, Gemini's insights would be valuable info for a consumer.
Gemini wrongly called an authentic product a fake, but that was mainly because there really were typographical errors in the authentic product's ingredient list. That's more of an indictment of the manufacturer than it is Gemini.
Finally, a lot of the things that Gemini got wrong seemed to me like they could reasonably be attributed to things like optical artifacts in the photos (glare, shadows, stuff like that). Better/more photos might improve that.
All that being said, this is obviously a tiny study of a single AI tool with a single brand of cosmetic product, so it's probably premature for me to optimistic, even cautiously. I've edited the last section of the writeup accordingly.
hn_throwaway_99
Thanks for the response, and I was overly harsh, I apologize. I did appreciate the detail and thoroughness you went to explain your experiences.
Yet I still think your study was a microcosm of the greatest dangers I think about AI. That is, it was astoundingly good at doing small scale feature detection, but astoundingly bad at synthesizing an overall conclusion, and worse, it did so with characteristic "AI certainty". Also, in the real world just like you found, pictures have glare, and real manufacturers have mistakes. The horrifyingly scary thing is that even if you think Gemini did a fairly good job at feature detection, in the real world people have and will just follow the AI conclusions blindly because they're "mostly" correct, even when they lead to completely wrong outcomes.
Heck, a Tesla already killed its passenger when it rammed into the side of a truck a few years ago because it mistook glare on the truck for the sun. Military planners blew up a school of young children based on old data, yet the Pentagon tried to blacklist Anthropic because Anthropic didn't want to provide autonomous kill capabilities.
I don't mean to sound over dramatic, but again, to me your study highlights everything I think is wrong with AI and the extreme dangers it will cause if society relies on it too much, which it has already begun to do.
Terr_
Perhaps a softer critique would be that the author seems to assume that (A) feature-detection will map to (B) accurate and useful conclusions, and that isn't necessarily true.
muppetman
That was my thought too. It failed amazingly at the task yet they were still thinking it was helpful. I had to go back and reread bits because I thought I’d missed a bit where Gemini hadn’t cocked up.
Very odd conclusion to an otherwise interesting experiment.
fwipsy
Replacement laptop batteries and power adapters have a huge problem with counterfeiting as well. In most cases it's a lot easier to spot because e.g. they rearrange the logos.
Even if these techniques did not produce false positives on genuine products, I suspect that they would not scale well. If counterfeiters realized they were losing a lot of sales through AI detection, they will just use AI to catch the errors themselves.
wgrover
Regarding scaling - I think you're right if the AI mostly detects typos, like it did in this work.
I'm hopeful that future AI models (perhaps trained for this purpose, not a general-purpose model like I used here) might be able to identify more subtle variations in e.g. injection molding patterns, the surface finish of a pill, things that would be hopefully a lot harder for counterfeiters to fix than a typo.
harvey9
Why do they rearrange the logos anyway? Surely it isn't hard to scan a real one.
fwipsy
I think it's because their printing isn't quite as good as the genuine one. They need to scale up the logos slightly for them to remain legible, which requires rearranging them to fit.
cmrx64
The candycodes mentioned at the end of the article are absolutely brilliant as well. These products have extremely intimate relations with bodies and protecting that chain can avoid a lot of harm.
qgin
It's baffling that the counterfeiters don't use the exact same design for the box. At least using the same text. Almost seems harder to make text thats sort of close but not just a straight copy.
Gigachad
Because they don’t have the source images. If they just used scans, the text would be fuzzy when reprinted. So they are converting it back to text in editing so it can print perfectly. Aside from some mistakes in the OCR.
As well as the fact they have already done well enough to fool almost all buyers.
trollbridge
Any competent graphic designer can do this. I think it’s a lack of effort because… they don’t need to spend the extra $100.
tonyoconnell
Last week, my wife who grew up in a hilltribe away from the modern world was using Gemini for this exactly. She found out that some face cream she bought was fake. I've been working with AI for the last 8 years. I didn't want to pollute her mind with AI and tech because I love the way she walks around thinking about fruit and trees and nature but she learned how to use AI by herself. She uses it for learning how to do things in "the city" like booking airplanes and hotels and asks Gemini questions all the time.
Walf
I hope you also tell her it's a liar. Sure it's useful for some things, but when it doesn't know, it'll just make shit up. It also peppers its answers with references now, but checking those refs will frequently show they offer no support at all to the AI's point.
trollbridge
This is genius - I’ve had good look using AI to identify fakes (or just reproductions) in other areas. It’s surprisingly good, and works fast enough to be usable in a fast paced live auction environment.
wgrover
That's fascinating! If any of your work using AI to identify reproductions is publicly sharable, I'd love to learn more about it (amateur antique collector here). But if not I understand.
jan_Sate
It got me into thinking, since the genuine product also failed the AI test, what if someone made a fake product with a packaging that passed the AI test? It shouldn't be that difficult. All you need to do is to feed the photo of your fake product into the AI and keep iterating on the packaging.
aashu_dwivedi
Counterfeit makers are also going to start using AI to find problems and improve their offering.
hankbond
This was a fun experiment but dang the formatting of this post was so hard to read.
drabbiticus
Seems likely to be another example of LLMs replicating training corpus. My expectation is that there will be more writing showing why something is counterfeit than why something is genuine, in part because disproving that something is counterfeit from visual/physical clues alone is essentially impossible. Lack of any definitive counterfeit proof is not proof of authenticity, hence why provenance and trust as such integral market factors. For example, even if the packaging were to be identical (e.g. say it's stolen from the factory), this is still not proof that the contained product is the one advertised.
And yes, packaging mistakes with chemical names and typography are quite common for anyone who occasionally pays attention to these things.
Transformanshen
This was not quite the way I expected to use AI
cute_boi
Thanks, this is how we should use AI, not creating useless slop in a hope to get karma and stars.
- https://groverlab.org/research/2022-05-06-candycodes.html
- https://groverlab.org/research/2026-03-19-disintegration-fin...
(edit: Formatting)
Whaaaattt? How could the author go through such detail and discover that Gemini completely failed in the assigned task (calling all the products fakes when some were and some weren't means that it provided zero correct assistance in this discrimination task) and then conclude "AI could play a helpful role"??? Does the author also conclude that stopped clocks could play a helpful role telling time because the are correct twice a day?
Apparently AI isn't the only thing that hallucinates.
Gemini wrongly called an authentic product a fake, but that was mainly because there really were typographical errors in the authentic product's ingredient list. That's more of an indictment of the manufacturer than it is Gemini.
Finally, a lot of the things that Gemini got wrong seemed to me like they could reasonably be attributed to things like optical artifacts in the photos (glare, shadows, stuff like that). Better/more photos might improve that.
All that being said, this is obviously a tiny study of a single AI tool with a single brand of cosmetic product, so it's probably premature for me to optimistic, even cautiously. I've edited the last section of the writeup accordingly.
Yet I still think your study was a microcosm of the greatest dangers I think about AI. That is, it was astoundingly good at doing small scale feature detection, but astoundingly bad at synthesizing an overall conclusion, and worse, it did so with characteristic "AI certainty". Also, in the real world just like you found, pictures have glare, and real manufacturers have mistakes. The horrifyingly scary thing is that even if you think Gemini did a fairly good job at feature detection, in the real world people have and will just follow the AI conclusions blindly because they're "mostly" correct, even when they lead to completely wrong outcomes.
Heck, a Tesla already killed its passenger when it rammed into the side of a truck a few years ago because it mistook glare on the truck for the sun. Military planners blew up a school of young children based on old data, yet the Pentagon tried to blacklist Anthropic because Anthropic didn't want to provide autonomous kill capabilities.
I don't mean to sound over dramatic, but again, to me your study highlights everything I think is wrong with AI and the extreme dangers it will cause if society relies on it too much, which it has already begun to do.
Very odd conclusion to an otherwise interesting experiment.
Even if these techniques did not produce false positives on genuine products, I suspect that they would not scale well. If counterfeiters realized they were losing a lot of sales through AI detection, they will just use AI to catch the errors themselves.
I'm hopeful that future AI models (perhaps trained for this purpose, not a general-purpose model like I used here) might be able to identify more subtle variations in e.g. injection molding patterns, the surface finish of a pill, things that would be hopefully a lot harder for counterfeiters to fix than a typo.
As well as the fact they have already done well enough to fool almost all buyers.
And yes, packaging mistakes with chemical names and typography are quite common for anyone who occasionally pays attention to these things.