In short

A verified crawl is a request whose source address proves it really came from the provider it names. The user agent string, “GPTBot” or “Googlebot”, is not enough. It is text, and any sender can set it freely.

How verification works

The large providers publish the networks their crawlers come from. OpenAI keeps a separate list for GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot. Perplexity publishes one for PerplexityBot and one for Perplexity-User. Google offers two routes: match the address against the published ranges, or run a reverse DNS lookup that has to resolve to googlebot.com, google.com or googleusercontent.com, then a forward lookup back to the same address.

If the address sits inside the published range, the request is verified. If it sits outside, someone is claiming to be a crawler and is not one.

This only works with server logs, because that is the one place where the user agent and the address sit together. A JavaScript counter never sees crawlers at all, since they run no script.

An example from our own measurement

We analysed the WAF logs of scmc.ch across 30 days, as of 21 August 2026.

Finding Count
Requests carrying a crawler user agent 7,118
of those, from published ranges 344
of those, forged 6,774, or 95 percent
forged and let through anyway 4,811

Verification ran against the published ranges of OpenAI, Perplexity and Google. The 4,811 forgeries that passed carried no attack signature, so the firewall had no reason to stop them.

Across eleven hosts we run or look after, the picture is milder and still not harmless. In the window from 11 August to 10 September 2026, 17,290 verified AI crawler requests came with 8,560 forgeries alongside them. That is 33 percent, counted against verified crawlers plus fakes. How high the share runs depends on the estate and the window. That is exactly why a number like this belongs with its window and its denominator attached.

The most common mistake

“We see GPTBot in the logs, so ChatGPT is reading our site.”

Counting requests by user agent counts the forgeries too. At scmc.ch, the unverified number would have overstated real crawl traffic twentyfold. Every claim about AI visibility built on numbers like that inherits the error.

Where the boundaries run

AI crawler is the umbrella term for programs run by AI providers to fetch pages. Without an address check, any count of AI crawlers is a count of claims.

Web application firewall. The common rule sets look for attack patterns. Whether a bot is genuine is not something they check without a dedicated rule or bot management. The example above shows it.

Prompt monitoring measures what an AI system answers to a question. Whether the system ever fetched your page is invisible to it. A verified crawl answers exactly that question.

AI referral is a person arriving on your site through a link in an AI answer. ChatGPT appends utm_source=chatgpt.com for this. That is visitor traffic, not a crawl.

How to check it yourself

You need access logs carrying the source address and the user agent, from a web server, from CloudFront or from a WAF. Load the providers’ published ranges fresh, because they change. Then match every request carrying a crawler user agent against the range of its provider. Whatever falls outside, count separately.

We do this with our own tool for WAF logs. For CloudFront logs the check is built into Munot, where forgeries appear as their own class. We do not reconstruct visitor sessions while doing it. Only requests claiming to be a bot get checked.

Sources

All provider pages retrieved on 10 September 2026.