Googlebot
Operated by Google · Powers Google Search & AI Overviews
Googlebot is Google's core web crawler for Search. Google's AI features inside Search, such as AI Overviews, draw on this same crawl and index — there is no separate way to appear in Google Search while opting out of AI Overviews specifically.
What it does
Googlebot crawls and indexes public web pages to power Google Search results. Google's Search-integrated AI features, including AI Overviews, are built on top of this same search index rather than a separate crawl.
Whether Google may use your crawled content to train its Gemini and Vertex AI generative models is a separate question, governed by the standalone Google-Extended robots.txt token — not by Googlebot itself.
Why we recommend keeping it open
- Required for any presence in Google Search results
- There is currently no way to appear in Search while excluding AI Overviews specifically
- Gemini/Vertex AI training use is controlled separately via Google-Extended, so blocking Googlebot is not required just to opt out of that
Identification & robots.txt
Googlebot
Additional verification information is available in Google's official documentation.
User-agent: Googlebot
Allow: /