The brand is infrastructure.
Search still sends people. But your pages now feed indexes, AI answers and the next models, often through the same crawlers. Built once, read everywhere.
Week to 20 Nov 2026 · 30,245 bot hits · 167 crawlers · 2,708 spoofs
People are no longer the only readers.
- Index22.8%
- Social23.4%
- SEO12.7%
- AI search0%
- Train23.1%
- Answer0%
- Spoof18.1%
Index
The index both search and AI answers draw on.
Googlebot, Bingbot, PetalBot: nearly a quarter of the top-10 hits. Google's AI Overviews read through the same crawler. That work has not been replaced. It has been joined.
AI search
A page fetched to be shown and cited in AI search.
OAI-SearchBot, Claude-SearchBot, PerplexityBot. They fetch pages so AI search results can show them and link to the source.
Answer
A page fetched live because someone asked.
ChatGPT-User is the clearest case. The page was read. Whether you were named is another event.
Train
Pages that may enter the next model.
GPTBot, ClaudeBot, Bytespider. They cite nothing today. What they collect shapes what the next model knows about you.
One week of requests, crawler by crawler.
From the log
- 23:59:02facebookexternalhit · Socialdeclared
- 23:56:22facebookexternalhit · Socialdeclared
- 23:55:17facebookexternalhit · Socialdeclared
- 23:54:46Amazonbot · Traindeclared
- 23:51:35Amazonbot · Traindeclared
- 23:51:16facebookexternalhit · Socialdeclared
- 23:50:17Bingbot · Indexverified
- 23:50:17Bingbot · Indexverified
- 23:50:16facebookexternalhit · Socialdeclared
- 23:50:16Bingbot · Indexverified
- 23:50:01Googlebot · Indexverified
- 23:48:58MJ12bot · SEOdeclared
- 23:48:58MJ12bot · SEOdeclared
- 23:45:47facebookexternalhit · Socialdeclared
- 23:43:50facebookexternalhit · Socialdeclared
- 23:40:29facebookexternalhit · Socialdeclared
- 23:38:12facebookexternalhit · Socialdeclared
- 23:37:33Googlebot · Indexverified
- 23:37:26Bytespider · Traindeclared
- 23:36:39facebookexternalhit · Socialdeclared
facebookexternalhit
Social · declared
2,360
"Googlebot"
Spoof · spoofed
1,821
Bytespider
Train · declared
1,086
Bingbot
Index · verified
1,055
AhrefsSiteAudit
SEO · verified
797
Googlebot
Index · verified
737
Amazonbot
Train · declared
681
Meta-ExternalAgent
Train · declared
562
PetalBot
Index · verified
506
AhrefsBot
SEO · verified
482
| Crawler | Job | Identity | Hits |
|---|---|---|---|
| facebookexternalhit | Social | declared | 2,360 |
| "Googlebot" | Spoof | spoofed | 1,821 |
| Bytespider | Train | declared | 1,086 |
| Bingbot | Index | verified | 1,055 |
| AhrefsSiteAudit | SEO | verified | 797 |
| Googlebot | Index | verified | 737 |
| Amazonbot | Train | declared | 681 |
| Meta-ExternalAgent | Train | declared | 562 |
| PetalBot | Index | verified | 506 |
| AhrefsBot | SEO | verified | 482 |
2,708 spoofs · 1,821 as "Googlebot" · show the borrowed names
- "Googlebot" · 1,821
The week total is 2,708. The list is only the borrowed names in the top 10.
Verified: reverse DNS or published IP ranges confirm the name. Declared: the bot names itself; I could not confirm it. Spoofed: the name is borrowed. A user agent is a claim, not an identity. A sample, not the internet.
A fetch, a citation and a mention are three events.
Fetch
A crawler reads the page.
Seen in server logs, declared bots only. CrawlGate.
Citation
The answer links to the page.
Seen in sampled answers. Ahrefs Brand Radar.
Mention
The answer names the brand.
Seen in sampled answers. Ahrefs Brand Radar.
Logs show the fetch, never the citation or the mention. So I read both: CrawlGate for who reads your pages, Ahrefs Brand Radar for what the answers do with them.
Start the conversation →