Verified crawl
A user agent string is free to set. At scmc.ch, 95 percent of requests carrying a crawler user agent were forged. How to tell the real ones apart, and why it only works with server logs.
Read the term →Glossary
AI visibility, term by term
What the terms around AI visibility actually mean, each one backed by a measurement from our own estate. No entry without evidence, no evidence without a date.
AI crawler
Programs run by AI providers to fetch pages so a model can answer from them. Across eleven measured hosts they came 17,290 times last month, the classic search bots only 7,976.
Read the term →JavaScript shell
If the text only arrives after JavaScript runs, an AI crawler gets an empty frame. Vercel measured this across its own network in December 2024, and for us it affects 17,290 requests a month.
Read the term →llms.txt
The proposal promises a machine-readable summary of your site for language models. In 30 days on scmc.ch: 417 fetches of robots.txt by AI crawlers, zero fetches of llms.txt.
Read the term →Scanner
35 percent of all requests to our eleven hosts come from programs hunting for weaknesses. On small sites they outnumber humans twenty to one.
Read the term →