Which AI Crawlers to Allow in robots.txt to Get Your Site into AI Answers in 2026
The article examines the problem of website invisibility to AI assistants: often robots.txt blocks access for training and real-time bots, even though regular indexing works fine. It provides a list of relevant crawlers (GPTBot, PerplexityBot, Google-Extended, etc.), along with tips for configuring access and checking logs.
OpenAI
Anthropic
Perplexity AI
Google/DeepMind
Яндекс
Сбер
Apple
Common Crawl
Habr — хаб ИИ27.07 · 00:01