In short
llms.txt is a proposal from 2024. The idea: a file in the root directory that tells a language model, in Markdown, what the site is about and where the important content sits. Conceived as a counterpart to robots.txt, for answer engines instead of search engines.
It is not a standard. No provider has committed to reading it.
What 30 days of logs show
Measured against the WAF logs of scmc.ch, 30 days to 11 September 2026.
| Fetch | Count |
|---|---|
robots.txt, total |
1,911 |
robots.txt, of those by AI crawlers |
417 |
llms.txt and llms-full.txt, total |
47 |
llms.txt, of those by AI crawlers |
0 |
The major AI crawlers were plentiful in that window, over 2,000 requests: ClaudeBot 810, GPTBot 395, ChatGPT-User around 400, OAI-SearchBot around 250, PerplexityBot 123, Google-Extended 99. They fetch robots.txt, hundreds of times. Not one of them asked for llms.txt even once in 30 days.
Who does fetch it
The 47 fetches came from browsers, from curl, from the Common Crawl bot and from small tools in the SEO and AI space: Shap-User, ShapBot, SEOJuice, PipericBot, Dataprovider.com, mcpsee-observatory. So from people looking, and from services inventorying sites for third parties.
That is not nothing. It is just not what the file is recommended for.
The most common mistake
“We have an llms.txt, so we are ready for AI.”
The file changes nothing about visibility as long as nobody reads it. What the models actually read is the HTML your pages deliver, and that is decided somewhere else entirely, see JavaScript shell.
What we do anyway
We keep maintaining llms.txt on our own sites. The effort is small, the harm is zero, and if a provider starts reading it one day, it is there. We simply do not sell it as a measure, and it appears in no audit as a point that brings visibility.
Anyone who creates one should know they are placing a bet, not closing a gap.
Where the boundaries run
robots.txt is an enforced standard. It governs who may fetch what, and the AI crawlers demonstrably respect it, they fetch it hundreds of times.
Structured data per schema.org genuinely is evaluated by search engines and answer engines. Effort spent on machine readability is better spent there.
Verified crawl answers whether those fetches were genuine at all, see Verified crawl.
How to check it yourself
Search your access logs for llms.txt and look at the user agents beside it. Then compare that count with the robots.txt fetches by those same user agents. The ratio tells you whether the file is being read on your site or merely exists.
A limit on this measurement
It comes from one site, over 30 days. The user agents here are not checked against the providers’ published address ranges. That does not weaken the finding, it strengthens it: unverified user agents count too many AI crawlers rather than too few, and even then the llms.txt figure stays at zero.
Sources
- Own measurement, WAF logs of scmc.ch, 30 days to 11 September 2026, CloudWatch Logs Insights
- The proposal itself: https://llmstxt.org