Which AI bots should access your website?
This directory covers identifiable AI crawlers and user-triggered agents, robots.txt control tokens, and major AI platforms with web access but without a publicly documented bot identity — what each does, and our GEO recommendation where one applies.
No entries match the current search and filters.
Fetches a page in direct response to a specific ChatGPT user action or request.
Crawls content for ChatGPT Search — OpenAI directly ties its access to being found, shown, and cited.
Collects public content that may be used to train future OpenAI models.
Visits landing pages submitted as ChatGPT ads to check policy compliance and relevance — not related to search visibility or model training.
Fetches pages when a Claude user request needs current web content.
Indexes the web to improve the quality and relevance of Claude's search results.
Collects web content potentially used to train future Claude models.
May fetch a page directly for a specific user question and cite it in the answer.
Builds Perplexity's search index; not officially used to train foundation models.
Google's core search index, which also powers Google's AI features — do not block on a commercial site.
A robots control token (not a crawler) governing whether already-crawled content trains and grounds Gemini.
Fetches pages in real time for AI-assisted answers with sources; not used to train models.
Fetches specific web content on demand to power Meta AI features.
Collects and indexes content for Meta's AI systems.
User-triggered fetch of web pages for up-to-date AI answers.
Automated crawler indexing web content for Mistral AI search functionality.
Automated crawler collecting web content for Mistral's generative AI training datasets.
Powers Siri, Spotlight and Safari; now also used as live context for Apple's AI answers.
A robots.txt control token for opting out of Apple foundation model training; it does not crawl webpages.
Crawls web content to improve Amazon products and services; may also train Amazon AI models.
Indexes web content for Amazon search experiences, including Alexa; not used for generative AI training.
Fetches current web information in response to user actions, including Alexa queries.
Builds the Common Crawl dataset widely used across the ML/LLM ecosystem.
Crawls and indexes public web pages for Exa search and AI retrieval applications.
Fetches web pages on demand in response to user requests made through Exa.
Crawls public web pages to build Linkup's search index for AI search and grounding applications.
A web crawler widely attributed to ByteDance; its AI-training use is not officially confirmed.
Bot access beyond robots.txt
Allowing AI crawlers in robots.txt does not guarantee access. CDN, WAF and bot protection rules may independently block or challenge requests. Cloudflare also provides AI Crawl Control and Pay Per Crawl options.
AI platforms without a public bot identity
These platforms power widely-used AI assistants, but haven't published a dedicated crawler user-agent or IP range — so their traffic can't be verified or selectively controlled via robots.txt today.
Alibaba's Qwen ecosystem supports web search, web extraction and URL fetching through its tools. No dedicated publisher-facing Qwen crawler identity has been confirmed.
DeepSeek developer tools support web search, HTTP fetching and browser-based access. Traffic identity depends on the selected backend; no single dedicated DeepSeek crawler identity has been confirmed.
Grok supports web search, X search and agentic browser interaction. No dedicated publisher-facing crawler identity has been confirmed.