Why ChatGPT and Perplexity Cannot See Your Website: Four Places the Block Hides

The site opens, leads arrive, and yet an AI assistant says it cannot read your pages or simply never mentions them. Nobody on your team decided that. The block usually came with a hosting move, a CDN setting or a platform checkbox, and a browser will never show it. Here is where it hides, how to find it, and which crawlers deserve an open door.

IN SHORT

Why AI assistants cannot see your site

If ChatGPT, Claude or Perplexity cannot see your website, the cause usually sits in one of four layers: robots.txt, a firewall or user agent rule at your host, a setting at your CDN, or content that exists only after JavaScript runs. The owner often does not know, because a person in a browser passes all four.

That is why these blocks can last. Every check a business normally runs, from opening the home page to sending a test lead, looks fine. The crawler gets a different answer from the same address, and nobody reads that answer unless they go looking for it.

We have met this from both sides. On one recent project, after a move to a new hosting account, GPTBot and Meta's crawlers received a 403 error, and the host had added them to robots.txt on its own. Nothing on the site looked wrong.

Google lists this among its basic requirements for appearing in AI Overviews and AI Mode: crawling has to be allowed in robots.txt and by any CDN or hosting infrastructure in front of the site. The same logic applies to every other assistant, with one difference. Google and Bing report problems to you through their webmaster tools. For most AI crawlers, your own logs are the only report you will get.

A person in a browser passes every check the crawler fails.

THREE JOBS

Training, search and user fetches

Before opening or closing anything, separate three kinds of AI traffic. They come from different bots, and most advice online treats them as one.

• Training crawlers collect pages that may be used to train future models.
• Search crawlers build the index an assistant answers from.
• User fetchers open a page at the moment someone asks about it, for example when a person pastes your link into a chat.

Here is how the vendors divide them, according to their own documentation as of November 2026:

The practical consequence is large. OpenAI states that its settings are independent: you can disallow GPTBot to keep content out of training and still allow OAI-SearchBot to appear in ChatGPT's search answers. A site that closes OAI-SearchBot will not be shown in those answers, apart from navigational links. Anthropic says the same in softer terms: blocking Claude-SearchBot or Claude-User may reduce your visibility in its search results.

Google works differently. Google-Extended is a robots.txt token without its own crawler, and Google says it has no effect on inclusion in Search and is not a ranking signal. AI Overviews and AI Mode draw on the ordinary Search index, so Googlebot is the bot that matters there, and closing Googlebot costs you both the classic results and the AI ones.

There is also a trap in the most common quick test. Pasting a URL into ChatGPT exercises ChatGPT-User only. It tells you nothing about OAI-SearchBot, and a polite answer from the assistant may come from an older index, not from your live page.

LAYER 1

robots.txt, written by someone else

robots.txt is the first place to look and the place where people stop too early. Three things catch owners out.

The file may not be the one you wrote. The host in our story above added AI crawlers to the file by itself. Cloudflare's managed option places its own directives before your content, and it creates a file when none exists. Squarespace has a checkbox, "Block known artificial intelligence crawlers", that updates robots.txt with a list including GPTBot, ClaudeBot and Meta-ExternalAgent. So read the live address in a browser, https://your-site.com/robots.txt, and never trust the copy in your repository or your developer's memory of it.

Each host has its own file. The standard, RFC 9309, puts robots.txt at the top level of each service, and Anthropic reminds owners to repeat its rules for every subdomain. A shop on shop.your-site.com or a blog on a separate host can have rules the main domain does not.

The status code of the file is itself a signal. Under RFC 9309, a robots.txt that answers with a 4xx code lets a crawler treat the site as unrestricted, while one that answers with a 5xx code means the crawler must assume everything is closed. Google follows a similar path: on a 5xx it stops crawling for the first 12 hours, then works from the last good copy for up to 30 days. A misconfigured server can therefore close a site without a single Disallow line in it.

When you read the file, look for a group naming the bot, such as User-agent: GPTBot followed by Disallow: /, and for a catch-all User-agent: * with Disallow: /, which often survives from a test environment. After you fix anything, allow for caches: OpenAI, Perplexity and Meta all mention about 24 hours before a robots.txt change takes effect, and the standard itself says a cached copy should not be used for longer than that.

One more limit. The RFC is explicit that these rules are not access authorization. A crawler that respects them stays out; one that does not walks in, and the user fetchers in the table above may not read the file at all.

LAYER 2

The host's firewall and user agent rules

This layer is the hardest for a business to see. The web server or a security module in front of your application looks at the User-Agent header or the IP address and refuses the request before your code runs. Your application logs stay clean, your CMS shows nothing, and the robots.txt may be perfectly open.

Here is what it looks like on our own site. Since about October 22, 2026, our host has answered 403 to the user agent HeadlessChrome and to Bytespider, ByteDance's crawler, while Googlebot, Bingbot and the AI crawlers we care about get 200. We keep Bytespider out on purpose. The refusal is an nginx error page, so it comes from the server layer and never reaches our Laravel application.

The HeadlessChrome half of that rule taught us how far such a block travels. PageSpeed began to fail on every page of the site with the message "Browser tab has unexpectedly crashed". We found the cause by publishing static copies of an article and removing parts one at a time until the crash disappeared. The site's service worker was registering on page load, and its script was requested with the test browser's own user agent, which the host refused. Nothing in that chain mentioned AI or crawlers. The full story is in our three.js experiment. Any agent or testing tool that announces itself the same way gets the same 403.

A rule written for one bot can break a tool nobody thought about.

The price of a wrong rule goes well beyond AI. If Googlebot is caught too, Google treats a 403 like any other client error: it behaves as if the content does not exist and removes a previously indexed URL from its index. A 429, by contrast, is read as overload, so crawling slows down and the URL stays in the index for a while. If you need to slow a bot down, 429 is the honest signal, and Google warns against using 401 or 403 for that purpose.

When a host is the suspect, ask support in writing: which user agent and IP rules apply to this account, which of them were added by default, and whether they change without notice. A hosting move is the classic moment for this to go wrong, and it belongs on the same list as DNS and mail records, as in our website migration checklist. Bot protection was also what stopped our own agent in our test of 15 US online stores, where robots.txt turned out to be the smaller factor.

LAYER 3

CDN defaults that keep changing

A CDN or application firewall sits in front of the host, so its decisions never reach your server logs. Cloudflare is the example most owners ask about, and its defaults have moved enough that any statement from memory is risky. Here is what its own pages say, in order:

The last two do not agree, and from the outside we cannot tell which one describes your account. Either way, the setting belongs to your zone and you can read it: in the Cloudflare dashboard it is under Security, then Settings, filtered by Bot traffic, as AI Crawler and Agent access policies. Two details deserve attention there. Agent is where chat fetch bots live, so a block on that row hits ChatGPT-style page visits. And a Training action also reaches mixed-purpose crawlers used for both Search and Training, so a decision that sounds like copyright protection can affect search visibility.

Separately, Bot Fight Mode on the free plan challenges traffic that matches bot patterns, and Cloudflare states that it cannot be skipped with WAF custom rules. Perplexity's documentation warns about exactly this: its advice for allowing its bots is a rule matching both user agent and IP range, placed above your blocking rules, and it notes that such a rule will not get past Bot Fight Mode.

Other platforms have their own switches. Vercel's AI Bots ruleset is inactive by default and lets you deny AI Crawlers and AI Assistants as separate categories. Whatever sits in front of your site, the question stays the same: who turned this on, and which of the three jobs does it stop? On larger setups that review is part of DevOps and cloud work, at $50 to $70 an hour.

LAYER 4

Content only JavaScript can draw

The fourth layer returns 200 to everyone, and the crawler still leaves with nothing. If prices, services or the article body appear only after scripts run in a browser, the first HTML response is an empty shell.

Vercel's study from December 2024 found that the major AI crawlers it tracked, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Meta-ExternalAgent and Bytespider, did not render JavaScript. ChatGPT's and Claude's crawlers fetched script files without executing them. The exceptions were Gemini, which uses Googlebot's infrastructure, and AppleBot. That study is almost two years old and vendors change their crawlers, so test your own pages. Designing as if the raw HTML is all a crawler gets remains the safe rule.

Our home page has a 3D first screen, and it is still a fair test of this. Fetched without a browser on November 2, 2026, it returns about 182 KB of HTML with the H1 and its words in place. The scene is decoration over real text, a choice our three.js article explains.

A related failure is content hidden behind a layer: popups and embedded frames can leave very little readable for software, a finding our shopping agent test documents.

CHECK

How to check all four layers

With access to the logs, a first pass takes about an hour by our estimate. The order below goes from the cheapest test to the most reliable one.

1. Read the live robots.txt on the main domain and on every subdomain that matters.
2. Request your own site with a declared user agent and compare the status code with an ordinary browser's (first command below). Do this on your own site only: sending crawler names to someone else's server misrepresents who you are.
3. Read the server logs and count the answers each AI crawler received (second command below).
4. Check the IP addresses. A user agent is only a claim. OpenAI, Anthropic and Perplexity publish JSON lists of their IP ranges, Bing publishes one for Bingbot, and Google documents a reverse DNS lookup. A request saying GPTBot from an address outside the list is an impersonator.
5. Open the CDN dashboard. Cloudflare's AI Crawl Control shows activity by crawler and status code; on free plans the Metrics tab covers the last 24 hours.
6. Compare raw HTML with the page. Fetch a page with curl, or use View Source, and search for a sentence and a price you expect to find.
7. Ask the search engines. Google's URL Inspection runs a live test that shows the raw HTML and whether robots.txt allows the crawl. Bing's URL Inspection has a Live Check for the same question, and its AI Performance report, in public preview since February 2026, shows citations in Copilot and Bing's AI summaries.

The two commands, for a typical server. In the common combined log format the status code is the ninth field.

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://your-site.com/

grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|meta-" access.log | awk '{print $9}' | sort | uniq -c

The curl test has a real limit. It exercises rules based on the user agent, from your own IP address. A firewall that judges by IP range or reputation may treat the true crawler differently, in either direction, so only the logs from real crawler addresses settle the question.

Here is our own result from November 2, 2026, requesting our home page with curl:

Then read your findings against this table:

The last row is the honest one. An open door does not earn a citation. What does is covered in SEO, AEO and GEO in 2026 and What Is AEO?, and it is ongoing SEO and website promotion work, from $1,500 a month with us.

OPEN OR CLOSE

What to open and what to keep closed

The right rules depend on how the business earns. A store wants its products read and its carts left alone. A company that sells its research may want search and fetchers without training. That decision belongs to the owner, and it should be a written decision and not a side effect of a host's defaults.

Some things stay closed for every bot: admin panels, accounts, carts and checkouts, staging copies, and anything holding personal data. robots.txt cannot do this job, because it is a request and because listing a secret path in a public file advertises it. These areas need a login or a rule at the server. We close temporary copies of a site at the server level and fill them with anonymized data, so a crawler that wanders in finds nothing real. Customer data carries legal duties as well: California's CCPA, for one, gives consumers rights over personal information held by businesses above its thresholds.

If the worry is load and not rights, reach for softer tools first. Anthropic supports the non-standard Crawl-delay directive. It also advises against blocking its bots by IP, since a blocked bot cannot read robots.txt, where the opt-out lives.

As for llms.txt, we generate one nightly for our site. It remains a proposal, published in September 2024, and Google says no new AI text files are needed for its AI features. Treat it as a summary for agents, never as a replacement for the checks above.

KEEP IT OPEN

Make the check part of the routine

A block appears after an event, so tie the check to events: a hosting move, a new CDN or firewall plan, a security plugin, a framework upgrade, a redesign that moves content into scripts. Then repeat it on a schedule, because defaults change without any action of yours.

Writing the decision down helps more than any tool. Our own project file, AGENTS.md, has a rule that AI crawlers are never blocked, names them, and adds that blocking Bytespider is intentional. Every developer or coding agent who opens the project is meant to read that before touching anything. At handover, the client gets an AGENTS.md along with the code, deployment commands and rollback procedure, under the contract, so such rules stay with the business and do not depart with whoever set them.

Under website maintenance and support, from $1,000 a month, the onboarding audit covers crawler access, and every upgrade ends with a real test lead and three checks that have nothing to do with the code: mail authentication, robots.txt and which crawlers can reach the site. Our prices for all of this are on the pricing page.

We treat crawler access the way we treat the rest of a system: as a part shaped by what this particular business needs from it, and not as a default somebody inherited.

Defaults change without you. The decision should stay yours, in writing.

FAQ

Sources

• OpenAI: Overview of OpenAI crawlers
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity: Perplexity crawlers
• Google: AI features and your website
• Cloudflare blog: Content Independence Day, July 1, 2025