In short

An AI crawler is a program run by an AI provider that fetches web pages so a language model can later use the content in an answer. OpenAI, Anthropic, Perplexity and Google each run their own, carrying user agents like GPTBot, ClaudeBot or PerplexityBot.

The difference from a search bot is the purpose. A search bot builds an index that later produces a list of results, and you get a place in it. An AI crawler supplies material for a written answer, in which your name either appears or does not.

What we measure

Across eleven hosts we run or look after, in the window from 11 August to 10 September 2026, 751,386 requests in total.

Class Requests
Humans 269,524
Scanners looking for weaknesses 263,067
AI crawlers, verified 17,290
Search bots 7,976
Forgeries carrying a crawler user agent 8,560

Two of those numbers are worth keeping.

AI crawlers now come more than twice as often as search bots, a factor of 2.17. Measuring your findability by Googlebot alone measures the smaller half.

The ratio depends heavily on the site. On our own company site there is one AI crawler request per 3.7 human visits. On a small, tightly focused client site it is nearly one to one, 1,678 requests against 1,851 visits. A small, tightly focused site can be more interesting to models than to people.

As a share of all traffic, AI crawlers are 2.3 percent. Small, and still the channel through which recommendations now form.

The most common mistake

“We will see AI crawlers in analytics once they matter.”

They never appear there. A browser counter only runs once JavaScript executes, and AI crawlers execute none. What analytics shows you is the human part, and the rest stays invisible until someone puts the server logs beside it. See JavaScript shell.

Where the boundaries run

Search bot. Googlebot and Bingbot build an index for result lists. Google-Extended is the special case: it belongs to the AI side but inherits Googlebot’s infrastructure and is the only one that renders JavaScript.

Scanner. Looks for weaknesses, not content. On our hosts the largest group of all, at 35 percent.

Forgery. Carries the user agent of an AI crawler but does not come from the provider’s published network. Without an address check you count them in, see Verified crawl.

AI referral. A person arriving through a link in an AI answer. That is the effect, the crawl is the precondition.

How to check it yourself

You need access logs carrying the user agent and the source address. Filter for the known user agents, check the address against each provider’s published ranges, and count separately: verified, forged, search bot. Only that separation makes the number hold up.

A browser counter is no use for this, and neither is a look at the user agent alone.

Sources