Which AI Crawlers to Allow in robots.txt to Get Your Site into AI Answers in 2026
OpenAI
Anthropic
Perplexity AI
Google/DeepMind
Яндекс
Сбер
Apple
Common Crawl
The article examines the problem of website invisibility to AI assistants: often robots.txt blocks access for training and real-time bots, even though regular indexing works fine. It provides a list of relevant crawlers (GPTBot, PerplexityBot, Google-Extended, etc.), along with tips for configuring access and checking logs.
The author describes a typical situation: a website is well indexed by Google and Yandex, but does not appear in ChatGPT responses because robots.txt has Disallow for GPTBot, ClaudeBot, and PerplexityBot. There are two types of AI crawlers: those for model training (GPTBot, ClaudeBot, CCBot) and real-time ones (OAI-SearchBot, PerplexityBot, ChatGPT-User). To appear in AI answers, you need to allow access for bots from the working list as of the end of July 2026, including GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, YandexBot/YandexAI, GigaChat, Applebot-Extended, CCBot. It is recommended to check robots.txt quarterly, as new user agents appear (for example, Google renamed NotebookLM to Google-GeminiNotebook). Robots.txt is only the first layer; issues can arise from Content-Security-Policy, lack of SSR (bots cannot see content loaded by JS), and infinite scrolling. To check access, the author suggests grep on logs and curl with User-Agent spoofing. Three common mistakes: outdated robots.txt, CDN protection blocking legitimate bots, and false confidence that indexing guarantees AI crawler access. The fix in the described case took 15 minutes and quickly led to an increase in citations in ChatGPT and Perplexity.
Source: Habr — хаб ИИ —
original