ShopReady / Guides

Which AI crawlers to allow in robots.txt so ChatGPT, Perplexity, Google and Copilot can find your store

Most stores that are invisible to AI shopping tools are not missing anything fancy. They are blocking the wrong crawler, often by accident through a plugin, a platform setting or a firewall. Here is which bots to allow and why.

1. Know the difference between search bots and training bots

Each AI company runs more than one crawler, and they do different jobs. Search and shopping crawlers decide whether your products can show up in answers. Training crawlers only decide whether your pages may be used to train AI models. You can allow the first group and block the second. OpenAI says its settings are independent, so a site can allow OAI-SearchBot for search while disallowing GPTBot for training.

Sources: [1]

2. OpenAI (ChatGPT)

OAI-SearchBot is the one that matters for shopping. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and it can take about 24 hours for a robots.txt change to take effect. ChatGPT-User visits a page when a person asks ChatGPT about it; OpenAI notes robots.txt rules may not apply to it because the visit is user initiated. GPTBot collects content that may be used for training. Blocking GPTBot does not remove you from ChatGPT search.

Sources: [1]

3. Perplexity

PerplexityBot surfaces and links websites in Perplexity search results, and Perplexity says it is not used to crawl content for AI foundation models. Perplexity-User fetches a page when a user asks a question, and Perplexity says it generally ignores robots.txt because the user requested it. Changes can take up to 24 hours to show.

Sources: [2]

4. Google (Search, AI Mode and Shopping)

Googlebot covers Google Search and its features. Storebot-Google controls all surfaces of Google Shopping, including the Shopping tab. Google-Extended is not a separate crawler; it is a robots.txt token that controls whether your content may be used for training Gemini models and for grounding in Gemini Apps and Vertex AI. Google states Google-Extended does not affect inclusion in Google Search and is not a ranking signal. Never block Googlebot or Storebot-Google on a store.

Sources: [3]

5. Microsoft (Bing and Copilot)

Bingbot is Bing's standard crawler and handles most of its crawling. Bing says you control it with robots.txt and can set crawl rates in Bing Webmaster Tools. Keep Bingbot allowed.

Sources: [4]

6. A safe robots.txt example

This example allows the search and shopping crawlers above and, optionally, opts out of training only. Replace the sitemap address with your own, and keep any rules your platform already adds for cart, checkout and account pages.

# Let AI shopping and search crawlers read your store
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Storebot-Google
Allow: /

User-agent: Bingbot
Allow: /

# Optional: opt out of AI model training only
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://www.example-store.com/sitemap.xml

If you are happy for your content to be used in training, simply leave out the GPTBot and Google-Extended blocks.

Sources: [1] [2] [3] [4]

7. Check your firewall and CDN, not just robots.txt

A clean robots.txt is not enough if a firewall challenges the bots. Perplexity publishes IP ranges for its crawlers and recommends allowing requests from them; it also warns that Cloudflare Bot Fight Mode cannot be skipped with WAF custom rules and can still challenge its bots. OpenAI publishes IP ranges for OAI-SearchBot, GPTBot and ChatGPT-User as well. If you use a security plugin, host firewall or CDN bot protection, test a real product page.

Sources: [2] [1]

Sources

1. OpenAI: Overview of OpenAI crawlers (OAI-SearchBot, GPTBot)
2. Perplexity: Perplexity crawlers (PerplexityBot)
3. Google: List of Google's common crawlers (Google-Extended, Storebot-Google)
4. Bing Webmaster Tools: Which crawlers does Bing use?

Platform guides

Last reviewed November 1, 2026. Platforms change their settings often, so check the linked sources.