The user agent of an AI crawler is a claim, not an identity. In a measurement over 31 days and 156,917 requests on k-i-soft.ch, only 657 of 2,430 requests claiming to be OpenAI or Perplexity were covered by the providers’ published IP ranges, so 27 percent. For the on-demand agent ChatGPT-User, which is the actual citation signal, it was 88 of 907, so 9.7 percent. A dashboard that counts AI visibility by user-agent strings therefore overstates it several times over. Verification works by checking the client IP against openai.com/gptbot.json, perplexity.ai/perplexitybot.json and Google’s published list.
If you want to know whether AI systems read your site, you look in the server logs. Google Analytics does not help: it only counts after the browser has run JavaScript, and AI crawlers run none. The logs, by contrast, record every fetch, before any banner and before any script.
I did that for my own site: 31 days, 156,917 requests across 6,728 log files. I wanted to know which AI systems come by. What came out was something else.
First, the uncomfortable clean-up
Before any percentage means anything, you need to know what is actually in the log:
| Requests | Share | |
|---|---|---|
No user agent (-) |
47,311 | 30.2 % |
curl/8.7.1 |
42,047 | 26.8 % |
| Everything else | 67,559 | 43.1 % |
More than half the traffic on my site is not an audience. The curl requests come at about 56 per hour, evenly spread across the day. That is a machine, not a person. Just under a third sends no identifier at all.
Anyone who does not subtract those two blocks has lost before starting. Every statement about “visitors” is then more than twice too high.
What the user agents claim
In the remaining traffic it looks encouraging at first. 7,841 requests present themselves as an AI system. Four of them are there on all 31 days:
| Bot | Requests | Days seen |
|---|---|---|
| Amazonbot | 1,152 | 22 |
| meta-externalagent | 1,144 | 30 |
| ClaudeBot | 1,134 | 31 |
| ChatGPT-User | 907 | 29 |
| cohere-ai | 870 | 7 |
| OAI-SearchBot | 592 | 31 |
| GPTBot | 547 | 31 |
| Bytespider | 453 | 31 |
| PerplexityBot | 337 | 16 |
| Google-Extended | 231 | 6 |
All 48 URLs from my sitemap were fetched at least once in the period. This is where I would once have stopped, ticked the box and felt good about it.
The user agent is a claim
A user agent is free text in the request. Anyone can write GPTBot into it, it costs nothing and nobody stops them.
The large providers know this and therefore publish the IP ranges their crawlers actually come from:
- OpenAI:
openai.com/gptbot.json,searchbot.json,chatgpt-user.json - Perplexity:
perplexity.ai/perplexitybot.json - Google:
developers.google.com/search/apis/ipranges/googlebot.json
That makes every request checkable: the identifier in the user agent, the origin in the IP. If both match, it was the bot. If not, it was somebody else.
The result
| Bot | verified | not covered | share real |
|---|---|---|---|
| Googlebot | 802 | 189 | 80.9 % |
| OAI-SearchBot | 304 | 288 | 51.4 % |
| GPTBot | 239 | 308 | 43.7 % |
| ChatGPT-User | 88 | 819 | 9.7 % |
| PerplexityBot | 26 | 311 | 7.7 % |
| Perplexity-User | 0 | 47 | 0.0 % |
Of 2,430 requests claiming to be OpenAI or Perplexity, 657 were real. 27 percent.
The hardest hit lands on exactly the number that matters. ChatGPT-User is not an index crawler that drops by eventually. It is the fetch during a running answer: somebody asked ChatGPT something, and ChatGPT is pulling this page because of it. That is the signal closest to being cited.
Counted raw: 907 fetches on 29 of 31 days. Verified: 88. Counting user agents reports ten times the truth here.
Googlebot sits in the table as a contrast. The old, well-understood crawler is reported real 81 percent of the time. The AI identifiers, 27. So this is not the same game as classic SEO, even if the analysis looks identical.
Who is doing the impersonating
Most of it traces back to five IP addresses in a single autonomous system, AS396982, which is Google Cloud. Together 3,473 requests, cycling through identifiers one after another:
1376 Mozilla/5.0 AppleWebKit/537.36 (no identifier)
461 Amazonbot/0.1
457 Amzn-SearchBot/1.0
228 OAI-SearchBot/1.3
227 PerplexityBot/1.0
226 Google-Extended/1.0
53 CCBot/2.0
38 cohere-ai/1.0
36 meta-externalagent/1.1
30 Bytespider
29 anthropic-aiThe same five addresses, eleven different identities. From AS396982 as a whole came 8,505 requests from 550 IP addresses.
And they are not crawling. Under those faked AI identifiers, the following were among the paths fetched:
/.env /.git/config
/.aws/credentials /.git/HEAD
/.aws/config /.gitlab-ci.yml
/.github/workflows/deploy.yml /fetchThose are not content, those are credentials. Everything was answered with 403 or 404; on a statically served site none of it exists in the bucket. But that is not the point:
Measuring AI visibility by user-agent strings means booking attack traffic as attention.
What cannot be verified
In fairness: verification only works where the provider publishes ranges. For ClaudeBot I found no machine-readable list at the time of measurement, and none for Amazonbot, Bytespider, meta-externalagent or cohere-ai either.
Their numbers in the first table therefore remain claims, not measurements. I do not know whether those 1,134 ClaudeBot requests were real. Given what the check produced for the providers that can be checked, I would not bet on it.
That is not an accusation aimed at Anthropic, it is an observation about the state of things: with AI crawlers, verifiability is still the exception rather than the rule.
What this means in practice
robots.txt is not protection. It is a request. Anyone faking the identifier ignores the file too. That is no reason to leave it out, but it is a reason not to mistake it for a control.
A dashboard without verification is worse than none. It shows a line going up, and the movement comes from scanners. You then make decisions on numbers that measure the opposite of what the label says.
Verify by IP, not by name. For Google and Bing, additionally by reverse DNS with a forward lookup, which is the documented method there. For OpenAI and Perplexity the published prefixes are enough.
Subtract monitoring and requests without an identifier before forming any ratio. On my site that was 57 percent of total traffic.
What I changed as a result
I rebuilt the analysis script so that it loads the published ranges itself and reports verified against not covered, instead of counting user agents. Whatever cannot be verified is reported as unchecked, not as real.
That is uncomfortable, because the numbers get smaller. 88 reads worse than 907. It is, however, the only number you can work with.
Frequently asked questions
How do I detect AI crawlers in my server logs?
The user agent gives you the candidates: GPTBot, ClaudeBot, PerplexityBot, ChatGPT-User, OAI-SearchBot, Google-Extended, CCBot, Amazonbot, Bytespider and meta-externalagent. But the user agent is only a claim, anyone can set it. You verify by checking the IP address against the provider’s published ranges, otherwise you are counting somebody else’s traffic.
Can the GPTBot user agent be faked?
Yes, trivially, it is free text in the request. In my own measurement over 31 days, 308 of 547 requests carrying the GPTBot identifier came from IP addresses outside the ranges OpenAI publishes. For ChatGPT-User it was 819 of 907.
Where do I find the IP ranges of AI crawlers?
OpenAI publishes them at openai.com/gptbot.json, searchbot.json and chatgpt-user.json, Perplexity at perplexity.ai/perplexitybot.json, Google at developers.google.com/search/apis/ipranges/googlebot.json. For ClaudeBot I found no machine-readable list at the time of measurement, so those hits stay unverifiable.
Why does Google Analytics not show AI crawlers?
Because Google Analytics only starts counting after the browser has run JavaScript. AI crawlers run none, an analysis of more than 500 million fetches found zero execution. They therefore appear in no analytics interface, only in the server logs.
Method, to check for yourself
CloudFront access logs in Parquet format, 13 July to 12 August 2026, one distribution, 6,728 files, 156,917 rows. Classification by user agent, then a match of the client IP against the published prefixes of OpenAI, Perplexity and Google. No sampling, every row analysed.
One note for rebuilding it: fetching the files one by one with aws s3 cp takes needlessly long at this volume, because process startup dominates. An aws s3 sync with a date filter does the same job orders of magnitude faster.