Meta-ExternalFetcher
Operated by Meta · Supports user-initiated web retrieval in Meta products
Meta-ExternalFetcher retrieves a specific web page when a person asks Meta AI a question that needs fresh information, similar to ChatGPT-User. Meta's documentation notes this fetcher may bypass robots.txt for these user-initiated requests.
What it does
When someone asks Meta AI a question that requires looking at a specific website, Meta-ExternalFetcher makes the request on their behalf, sometimes with source attribution back to the page.
According to Meta's official documentation, this fetcher may bypass robots.txt rules because the request was initiated by a user — this differs from Meta-ExternalAgent, which strictly follows robots.txt for its autonomous training crawl.
Why we recommend keeping it open
- Responds to a specific person's question, not a bulk crawl
- Not used to train Meta's models
- Meta states this agent may bypass robots.txt for user-initiated requests, so robots.txt control here is limited regardless
Identification & robots.txt
Meta-ExternalFetcher
Additional verification information is available in Meta's official documentation.
User-agent: Meta-ExternalFetcher
Allow: /