Skip to content

The brand is infrastructure.

Search still sends people. But your pages now feed indexes, AI answers and the next models, often through the same crawlers. Built once, read everywhere.

Week to 20 Nov 2026 · 30,245 bot hits · 167 crawlers · 2,708 spoofs

People are no longer the only readers.

  • Index22.8%
  • Social23.4%
  • SEO12.7%
  • AI search0%
  • Train23.1%
  • Answer0%
  • Spoof18.1%
Fig. 1. Share of hits by job, top 10 crawlers (10,087 of 30,245 hits), week to 20 Nov 2026. Social previews and SEO tools make up the middle. Source: CrawlGate, Glorics network.

Index

The index both search and AI answers draw on.

Googlebot, Bingbot, PetalBot: nearly a quarter of the top-10 hits. Google's AI Overviews read through the same crawler. That work has not been replaced. It has been joined.

AI search

A page fetched to be shown and cited in AI search.

OAI-SearchBot, Claude-SearchBot, PerplexityBot. They fetch pages so AI search results can show them and link to the source.

Answer

A page fetched live because someone asked.

ChatGPT-User is the clearest case. The page was read. Whether you were named is another event.

Train

Pages that may enter the next model.

GPTBot, ClaudeBot, Bytespider. They cite nothing today. What they collect shapes what the next model knows about you.

One week of requests, crawler by crawler.

From the log

  1. 23:59:02facebookexternalhit · Socialdeclared
  2. 23:56:22facebookexternalhit · Socialdeclared
  3. 23:55:17facebookexternalhit · Socialdeclared
  4. 23:54:46Amazonbot · Traindeclared
  5. 23:51:35Amazonbot · Traindeclared
  6. 23:51:16facebookexternalhit · Socialdeclared
  7. 23:50:17Bingbot · Indexverified
  8. 23:50:17Bingbot · Indexverified
  9. 23:50:16facebookexternalhit · Socialdeclared
  10. 23:50:16Bingbot · Indexverified
  11. 23:50:01Googlebot · Indexverified
  12. 23:48:58MJ12bot · SEOdeclared
  13. 23:48:58MJ12bot · SEOdeclared
  14. 23:45:47facebookexternalhit · Socialdeclared
  15. 23:43:50facebookexternalhit · Socialdeclared
  16. 23:40:29facebookexternalhit · Socialdeclared
  17. 23:38:12facebookexternalhit · Socialdeclared
  18. 23:37:33Googlebot · Indexverified
  19. 23:37:26Bytespider · Traindeclared
  20. 23:36:39facebookexternalhit · Socialdeclared
  • facebookexternalhit

    Social · declared

    2,360

  • "Googlebot"

    Spoof · spoofed

    1,821

  • Bytespider

    Train · declared

    1,086

  • Bingbot

    Index · verified

    1,055

  • AhrefsSiteAudit

    SEO · verified

    797

  • Googlebot

    Index · verified

    737

  • Amazonbot

    Train · declared

    681

  • Meta-ExternalAgent

    Train · declared

    562

  • PetalBot

    Index · verified

    506

  • AhrefsBot

    SEO · verified

    482

2,708 spoofs · 1,821 as "Googlebot" · show the borrowed names
  • "Googlebot" · 1,821

The week total is 2,708. The list is only the borrowed names in the top 10.

Verified: reverse DNS or published IP ranges confirm the name. Declared: the bot names itself; I could not confirm it. Spoofed: the name is borrowed. A user agent is a claim, not an identity. A sample, not the internet.

A fetch, a citation and a mention are three events.

Fetch

A crawler reads the page.

Seen in server logs, declared bots only. CrawlGate.

Citation

The answer links to the page.

Seen in sampled answers. Ahrefs Brand Radar.

Mention

The answer names the brand.

Seen in sampled answers. Ahrefs Brand Radar.

Logs show the fetch, never the citation or the mention. So I read both: CrawlGate for who reads your pages, Ahrefs Brand Radar for what the answers do with them.

Start the conversation →