Fix the catalog
What a robots.txt file is, and why it decides if AI can see your store
By Ankit Minocha, founding team at Atomz. Updated January 21, 2026.
Around 40% of Shopify stores quietly block the AI crawlers that now drive discovery. Here is what robots.txt does, which bots to allow, and the exact lines to add.
A robots.txt file is the short instruction sheet at the root of your domain that tells automated visitors which parts of your site they may read. Search engines have used it for decades, and most store owners never think about it. That stopped being safe when the bots that matter changed, because the assistants now sending shoppers your way each crawl with their own user agent, and a file written for Google can quietly shut them out.
In the stores we have looked at, close to forty percent are unintentionally invisible to ChatGPT or Claude for exactly this reason. The products are fine and the site is live, but the crawler that would have read it was never allowed in.
The crawlers that matter now
Each assistant uses one or more named agents, and your robots.txt has to recognize them by name. Blocking one is the difference between being readable and being absent on that surface.
| Agent | Who it belongs to |
|---|---|
| GPTBot, OAI-SearchBot, ChatGPT-User | OpenAI and ChatGPT |
| ClaudeBot, Claude-User | Anthropic and Claude |
| PerplexityBot | Perplexity |
| Google-Extended | Google's AI products |
| CCBot | Common Crawl, which feeds many models |
What to put in the file
The goal is simple: allow the AI crawlers, keep them out of your cart and account pages, and point them at your sitemap. On Shopify you override the default by editing robots.txt.liquid in your theme, and the lines themselves are short.
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://yourstore.com/sitemap.xml
Common mistake
The most frequent error is a leftover Disallow: / under User-agent: * that the store owner forgot was there, or a theme that never allowed the AI agents in the first place. One line decides whether an assistant can read your entire catalog, so it is worth checking before anything else.
What changes when you fix it
This is not a growth hack, it is a gate you either open or leave shut. Across stores that opened it after being blocked, the pattern is a modest, real lift in direct and organic sessions as the assistants begin citing them, on the order of ten to twenty percent for the stores we tracked, because traffic that was simply unreachable became reachable.
The honest framing is that allowing the crawlers is necessary but not sufficient. Once an agent can read your store, what it finds still has to be structured enough to match a query, which is the catalog work the rest of these guides cover. Start by making sure you are not locked, then make what is behind the lock worth reading. The audit checks both in one pass.
One important update since the Shopify Spring 2026 Edition: robots.txt now governs only open-web crawlers. It does not control whether Shopify Catalog syndicates your products to the AI channels you have activated, which runs on its own settings. Shopify also auto-serves a set of agent-discovery files on every store now, so robots.txt is one layer of a larger picture.
Subscribe to The Agentic Operator