Which AI crawlers a hospital website should let in

A hospital’s robots.txt file tells crawlers which parts of its website they may read. The companies behind the main AI assistants split their crawlers by purpose: one for search, one for model training, and one that fetches a page when a user asks about it. Block them all and a hospital drops out of AI search. Block only the one you object to and the others keep working.

OpenAI

• OAI-SearchBot crawls for ChatGPT’s search features. OpenAI says sites that opt out of it won’t be shown in ChatGPT search answers, though they can still appear as navigational links, and that changes take about 24 hours to apply.
• GPTBot crawls content that may be used to train OpenAI’s models. OpenAI says this setting and the search setting are independent.
• ChatGPT-User fetches a page when a user’s request calls for it. Because a person started the action, OpenAI says robots.txt rules may not apply.

Google

• Googlebot crawls for Google Search, including all Search features. Google’s guidance on AI Overviews and AI Mode points to Googlebot’s robots.txt rules, and to the nosnippet, data-nosnippet, max-snippet and noindex controls for limiting what appears.
• Google-Extended is a separate token that controls whether content is used to train Gemini models and to ground them. Google says it does not affect a site’s inclusion in Google Search, and is not used as a ranking signal.

Perplexity

• PerplexityBot surfaces and links websites in Perplexity’s search results, and Perplexity says it isn’t used to crawl content for AI foundation models. Perplexity recommends allowing it for sites that want to appear in its results.
• Perplexity-User fetches pages when a user asks a question, and generally ignores robots.txt because a user requested the fetch.

Anthropic

• Claude-SearchBot crawls to improve Claude’s search results. Blocking it may reduce a site’s visibility in them.
• Claude-User retrieves pages when people ask Claude questions. Blocking it prevents that retrieval.
• ClaudeBot collects content that may be used for training.

Microsoft

Bingbot crawls for Bing. Bing Webmaster Tools now reports how often Microsoft Copilot and Bing’s AI summaries cite a site’s pages, in its AI Performance report, in public preview since February 2026.

What a hospital should allow

• To appear in AI answers, allow the search crawlers: OAI-SearchBot, Googlebot, PerplexityBot, Claude-SearchBot and Bingbot.
• Treat training as a separate decision. OpenAI and Google say their training opt-outs, GPTBot and Google-Extended, don’t affect their search results. Anthropic lists ClaudeBot as its training crawler, separate from Claude-SearchBot.
• Expect user-triggered fetchers to read public pages anyway. ChatGPT-User and Perplexity-User may not follow robots.txt, so any public page can be fetched when a user asks about it.

A hospital that wants to stay visible in AI search without contributing to model training could use:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

robots.txt is not a lock

robots.txt is a public file of requests to well-behaved crawlers. Google says it is not a mechanism for keeping a web page out of Google, and recommends noindex or password protection instead. Patient portals, test results and appointment records belong behind a login, not on a public page with a Disallow line.

These lists change, so check each company’s documentation for current crawler names when you review the file.