Free SEO Tool
AI Bot Manager
Control which AI crawlers can access your site. Configure per-bot allow, disallow and crawl-delay rules for all major AI bots – then copy the generated robots.txt block straight into your file.
No sign-up required. Your data never leaves the browser.
Quick presets
5
Allowed
0
Rate Limited
10
Blocked
AI Search & Retrieval
OAI-SearchBot
OAI-SearchBot(OpenAI)Powers ChatGPT search results and web browsing.
ChatGPT-User
ChatGPT-User(OpenAI)Fetches pages when ChatGPT users share links in conversation.
PerplexityBot
PerplexityBot(Perplexity AI)Powers Perplexity AI search results and citations.
Google-Extended
Google-Extended(Google)Controls Gemini and AI Overviews training. Does not affect regular Google Search.
Applebot-Extended
Applebot-Extended(Apple)Controls Apple Intelligence features, Siri, and Safari suggestions.
AI Training
GPTBot
GPTBot(OpenAI)Crawls content for OpenAI model training. Separate from ChatGPT search.
ClaudeBot
ClaudeBot(Anthropic)Crawls content for Claude model training and improvement.
cohere-ai
cohere-ai(Cohere)Crawls content for Cohere model training.
AI2Bot
AI2Bot(Allen Institute for AI)Crawls content for AI2 research and open-source model training.
Amazonbot
Amazonbot(Amazon)Crawls content for Alexa AI answers and Amazon recommendations.
Data Collection & Scrapers
Bytespider
Bytespider(ByteDance)May ignore robots.txtAggressive crawler for TikTok and ByteDance AI training. Known for very high request volumes.
CCBot
CCBot(Common Crawl)Builds the Common Crawl open dataset, widely used to train many LLMs.
FacebookBot
FacebookBot(Meta)Crawls content for Meta AI training and Facebook/Instagram features.
Diffbot
Diffbot(Diffbot)Web scraping for knowledge graphs. Used by various AI companies.
ImagesiftBot
ImagesiftBot(Hive)Crawls images for AI image recognition and content moderation training.
Generated robots.txt rules
# AI Crawler Rules # Generated by Dynamically AI Bot Manager # https://dynamically.co.uk/tools/ai-bot-manager User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: AI2Bot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: FacebookBot Disallow: / User-agent: Diffbot Disallow: / User-agent: ImagesiftBot Disallow: /
How to use: Append these rules to your existing robots.txt file. They control AI bot access only – your existing rules for Googlebot and other search engines remain unchanged.
Note: Some bots (marked above) may not fully respect robots.txt directives. For these, consider server-level blocking via your CDN or firewall.
Understanding AI crawlers
Training vs search: why it matters
Not all AI crawlers serve the same purpose. Understanding the difference is critical for protecting your content while maintaining visibility in AI-powered search.
AI search crawlers
These crawlers power real-time AI search results. When someone asks ChatGPT, Perplexity, or Google AI Overviews a question, these bots fetch your content to generate answers.
- Allow these to maintain visibility in AI search answers
- Essential for Generative Engine Optimisation (GEO)
- Includes OAI-SearchBot, PerplexityBot, Google-Extended
AI training crawlers
These crawlers collect content to train AI models. Your content becomes part of the model's knowledge but blocking them does not remove you from AI search results.
- Safe to block if you want to protect content from model training
- Blocking does not affect your AI search visibility
- Includes GPTBot, ClaudeBot, CCBot, cohere-ai
Want to appear in AI search results?
Managing crawler access is just the first step. Our Generative Engine Optimisation service ensures your brand is cited when AI answers your customers' questions.