Is Your Site Blocking the AI Bots That Cite You?

No amount of citation-ready writing matters if an AI assistant cannot crawl your site in the first place. A meaningful share of marketing sites block the exact bots that power ChatGPT, Perplexity, and Claude citations, often without anyone noticing, because the block sits inside a robots.txt file nobody on the marketing team ever audits.

How much traffic are AI assistants actually sending to websites right now?

The honest answer is still a small slice of total traffic, but it is growing fast and concentrated in exactly the research moments that matter most. SE Ranking's analysis of 101,574 websites found that AI platforms accounted for 0.32 percent of all website traffic in the period it measured, up from 0.24 percent in 2025 and 0.02 percent in 2024, tracking Google Analytics data from January 2025 through April 2026. That looks tiny until you remember it was essentially zero two years ago.

The traffic that does arrive is concentrated on one assistant for now. The same study found ChatGPT drove 74.78 percent of AI referral traffic in 2026, followed by Gemini at 11.56 percent, Perplexity at 7.23 percent, Copilot at 3.51 percent, and Claude at 2.62 percent. A prospect evaluating MSPs is not spreading that research evenly across five assistants. Right now, one assistant is doing most of the deciding about which providers even get considered.

Does blocking an AI crawler actually cost you citations?

Yes, and the gap between sites that allow and block the same bot is not subtle. A cloro Research study of 1,058 highly cited and highly visible domains found that sites blocking GPTBot had a median ChatGPT citation propensity of 0.003, compared with 0.417 for sites that allowed it, a measure of how often a domain gets cited relative to how often it appears in organic Google results. That gap, 0.003 against 0.417 on the same citation propensity measure, is one of the starkest findings in the cloro Research dataset: sites that block GPTBot are not merely cited somewhat less. They are functionally invisible to ChatGPT citations by comparison.

The same pattern holds for other crawlers and other assistants. The same study found that blocking PerplexityBot dropped median Perplexity citation propensity from 1.167 to zero, and blocking OAI-SearchBot dropped median ChatGPT propensity from 0.394 to zero. A site that blocks the crawler behind a given assistant does not get cited less often by that assistant. In this dataset, it typically does not get cited at all.

Those block-rate figures come from the same cloro Research study, based on robots.txt files successfully fetched for 88 percent of the domains in its sample. A separate, continuously updated tracker paints an even higher picture for PerplexityBot specifically. Eighteen percent of the top 1,000 websites it monitors currently have a robots.txt rule disallowing PerplexityBot, as of October 2026. Whichever figure you trust more, the direction is the same: a meaningful share of high-visibility sites are blocking the crawlers that would otherwise cite them.

Why do MSP marketing sites block these bots without realizing it?

Nobody sets out to disappear from ChatGPT on purpose. Robots.txt rules accumulate from CDN defaults, security plugins, and a web developer's best guess at what counts as an unwanted bot a year or two ago, long before most marketing teams had a reason to care which specific crawler belonged to which AI assistant. A rule written to block a scraper nobody could name at the time can end up blocking the exact traffic source that matters most today, and nobody revisits that file until something prompts them to look.

That blind spot is compounded by what standard analytics actually show. AI referral traffic is small enough in raw volume, per the SE Ranking figures above, that it rarely shows up as its own clearly labeled line in a dashboard built around organic search and paid channels. A site that lost its AI visibility months ago because of a robots.txt rule nobody remembers adding would not necessarily see a dip anyone would notice, because the volume was never large enough to miss in the first place. The cost shows up as prospects who never considered you, not as a number on a report.

A marketing team can go an entire quarter reviewing traffic reports, conversion rates, and keyword rankings without ever surfacing this particular failure mode, because none of those reports are built to flag it. Organic search traffic can hold steady or even grow while AI citation visibility quietly drops to zero for an assistant that a growing share of prospects now consult to shortlist vendors before they ever run a traditional search. The two metrics simply do not live in the same report, and a team that is not specifically looking for the gap has no reason to go find it.

What's changing with Cloudflare's new default rules?

The rules governing this are also shifting under site owners who are not actively watching. Cloudflare's own developer changelog describes new default settings taking effect for new domains on September 15, 2026, splitting AI bot traffic into Search, Training, and Agent categories, with Training and Agent bots blocked by default on pages that carry advertising while Search stays allowed. Any new site that onboards to Cloudflare after that date inherits a default posture toward AI crawlers that did not exist a year ago, without anyone on the marketing team making an explicit choice about it.

The practical lesson is not about Cloudflare specifically. It is that the default behavior of your hosting, CDN, and security tooling toward AI crawlers changes on its own schedule, set by vendors solving their own infrastructure problems, not yours. A setting that was fine last year can become a block this year, and the only way to know is to check the actual robots.txt file and bot management settings rather than assume last year's audit still holds.

This also means a one-time fix is not really a fix. A site that confirms today it allows every major AI crawler has no guarantee it will still allow them after the next platform update, the next security plugin release, or the next change a hosting provider rolls out to its own defaults. Treating crawler access as a setting you check once and forget is the same mistake as treating a backup as done because it ran successfully last year. It needs a recurring check, not a one-time audit, built into whatever cadence the marketing team already uses for reviewing site performance.

How do you check whether your own site is blocking the bots that matter?

Start with the file itself. Pull up yourdomain.com/robots.txt directly in a browser and read every disallow rule against the current list of named AI crawlers, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, ChatGPT-User, and Google-Extended among them, rather than assuming a rule written for one era of bots still makes sense for this one. If your site sits behind a CDN or a security plugin, check its bot management settings separately, since a block can exist there even when robots.txt looks clean.

This is exactly the kind of structural, easy-to-miss gap Actiforge's ActiScore was built to surface, checking a site against the access and structural signals that determine whether an AI assistant can even reach your pages, not just whether your writing is citable once it does. Catching a blocked crawler is a five-minute fix once you know to look. Finding out you need to look is the harder part, and it is usually the one that gets skipped.

If you are not sure whether this kind of gap exists elsewhere in your marketing and service stack, a structured review beats waiting to notice a slow, unlabeled decline in the dashboard. Actiforge's stack builder walks through exactly that kind of audit across your current setup.

Being well written and being reachable are two different requirements now, and a site that nails the first while failing the second never gets the chance to prove it. Browse Actiforge's full product catalog to see the tools built for MSPs who want to compete on both fronts.

See the full stack