AI crawler
Bots that AI companies use to fetch web content: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and others. Major AI crawlers do not execute JavaScript.
Notes
One distinction matters most: training crawlers (e.g. GPTBot, collecting model training data) and retrieval bots (OAI-SearchBot, ChatGPT-User — fetching in real time when generating answers) are separate and can be controlled independently in robots.txt. If you want citations, retrieval bots must be allowed.
Second key fact: per Vercel’s analysis, major AI crawlers do not execute JavaScript. And CDNs/WAFs sometimes block AI bots by default — check CDN settings, not just robots.txt.
Sources
Updated: Jun 10, 2026