One trap when you name AI bots specifically is that is that robots.txt groups don’t merge. Crawlers will only follow the most specific group so if GPT bot has its own group named nothing under User-agent: * applies to it anymore so you need to repeat every disallowed line under each group.
Hissigh
Are llms.txt actually read by bots now? I found one experiment that showed that no bot has accessed the file in 3 months.
I get the odd one try on my personal site, but usually its just robots.txt... /shrug
literallyroy
I once had oh-my-pi read a website’s llms.txt while doing research many months ago.
CqtGLRGcukpy
Most of what the author is doing won't actually help.
> LLMs ignore Markdown files — log analysis showed no evidence that GPTBot, ClaudeBot, or similar crawlers request or prioritise .md content, even when listed in llms.txt.
why? You're helping AI companies to take your writing for free
mmcallister
Whilst I agree with most criticism of AI/LLMs but most of the time at the other end of someone using Claude/ChatGPT there is still a human there, a human that is trying to solve a problem or learn something. Maybe I'm naive and it's just all bots scraping sites to build their corpus of data, but at least the way I use AI is to solve problems and build things I couldn't do before (either because of time, or it was out of my reach)
1shooner
Sooo, are they reading the md?
lofaszvanitt
Maybe read how webfetch and websearch works inside the harnesses.
https://www.rebootonline.com/geo/llms-txt-experiment/
> LLMs ignore Markdown files — log analysis showed no evidence that GPTBot, ClaudeBot, or similar crawlers request or prioritise .md content, even when listed in llms.txt.
https://www.longato.ch/llm-md-files/
> Across all 50 sites, across 200+ days, across every major AI user-agent logged - the `/llms.txt` endpoint was requested zero times.
https://mishamanko.com/research/what-ai-bots-actually-read