On this page
- 01Sixteen bots, three jobs
- 02Cloudflare wrote 341 of the 741 blocks
- 03Own Cloudflare account: one site in five blocks
- 04The EU rule Cloudflare points to
- 05Search bots get in almost everywhere
- 06Firms that want to be found block about as often
- 07By sector: healthcare is higher because of two shared files
- 08By province: about equal
- 09The platform decides who writes the file
- 10Why the figure is higher than in October
- 11What robots.txt cannot show
- 12What this means for your business
In November 2026, 741 of 23,676 working Belgian small business websites (3.1%) told at least one named AI crawler to stay away in robots.txt. GPTBot, OpenAI's training crawler, was shut out on 569. On 341 of the 741, every rule came from the managed block Cloudflare writes at the top of the file. Only 0.7% kept out an AI search or user bot.
So the fear that Belgian firms are vanishing from ChatGPT does not match the files. The opposite worry holds up better: if you want to keep your text out of model training, 96.9% of small businesses have not set anything up. This study counts both sides, by bot, by network, by platform, by sector and by province, with open data you can reuse.
Training bots get blocked, AI assistant search bots hardly ever
AmazonbotAmazon, training
2.5%GPTBotOpenAI, training
2.4%CCBotCommon Crawl, open web archive
2.2%Meta-ExternalAgentMeta, training
2.2%ClaudeBotAnthropic, training
1.9%Google-ExtendedGoogle, training
1.8%Applebot-ExtendedApple, training
1.7%Amzn-SearchBotAmazon, search in an AI assistant
0.3%PerplexityBotPerplexity, search in an AI assistant
0.3%ChatGPT-UserOpenAI, fetches for a user
0.2%OAI-SearchBotOpenAI, search in an AI assistant
0.1%Meta-ExternalFetcherMeta, fetches for a user
0.1%Claude-UserAnthropic, fetches for a user
0.1%Perplexity-UserPerplexity, fetches for a user
0.1%Claude-SearchBotAnthropic, search in an AI assistant
0.0%Amzn-UserAmazon, fetches for a user
0.0%
Percent of working sites; blocked means Disallow: / for that bot in its own group. Dark blue: bots that search or fetch for a user. Measured on 5 November 2026.
Sixteen bots, three jobs
AI vendors no longer run one crawler each. OpenAI alone has three: GPTBot gathers pages that can help train its models, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User fetches a page when a person asks for it in a chat. Anthropic, Perplexity, Meta and Amazon split their bots the same way. Google-Extended and Applebot-Extended are a special case. They fetch nothing; they are tokens that tell Google and Apple whether pages their normal crawlers already collect may also feed AI models.
Training and archive bots account for nearly every block we found: 714 of the 741 sites close the door to at least one of them. Amazonbot tops the list on 601 Belgian SME sites (2.5% of 23,676), ahead of GPTBot on 569 (2.4%) and Common Crawl's CCBot on 532. Search bots trail far behind. OAI-SearchBot is refused on 31 sites and Claude-SearchBot on just 8.
| Bot | Vendor | Purpose stated by the vendor | Sites blocking it | Share of 23,676 |
|---|---|---|---|---|
| Amazonbot | Amazon | Improves products, may train AI models | 601 | 2.5% |
| GPTBot | OpenAI | Model training | 569 | 2.4% |
| CCBot | Common Crawl | Open web archive | 532 | 2.2% |
| meta-externalagent | Meta | Training and indexing | 519 | 2.2% |
| ClaudeBot | Anthropic | Model training | 449 | 1.9% |
| Google-Extended | Gemini training and grounding, no crawling of its own | 423 | 1.8% | |
| Applebot-Extended | Apple | Apple model training, no crawling of its own | 401 | 1.7% |
| Amzn-SearchBot | Amazon | Search in Amazon services | 68 | 0.3% |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity | 64 | 0.3% |
| ChatGPT-User | OpenAI | Fetches at a user's request | 59 | 0.2% |
| OAI-SearchBot | OpenAI | Shows sites in ChatGPT search | 31 | 0.1% |
| meta-externalfetcher | Meta | Fetches at a user's request | 20 | 0.1% |
| Claude-User | Anthropic | Fetches at a user's request | 16 | 0.1% |
| Perplexity-User | Perplexity | Fetches at a user's request | 15 | 0.1% |
| Claude-SearchBot | Anthropic | Improves Claude's search results | 8 | 0.0% |
| Amzn-User | Amazon | Fetches at a user's request | 7 | 0.0% |
Sites rarely pick bots one at a time. Exactly seven bots are blocked on 372 of the 741, a single bot on 103. To see where your own site stands, run it through the AI crawler checker. The industry push for one shared training rule is covered in our news piece on the robots.txt AI training standard.
Cloudflare wrote 341 of the 741 blocks
Cloudflare sits in front of a website as a network layer and filters its traffic. From 1 July 2025 it asked every new domain at sign-up whether AI crawlers should be let in. Owners who switch on its managed robots.txt get a block placed ahead of their own rules, fenced by the lines '# BEGIN Cloudflare Managed content' and '# END Cloudflare Managed Content'. Inside, eight bots get Disallow: /: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. The group for all bots carries a Content-Signal line that allows search and refuses training; on the Belgian files it read 'search=yes,ai-train=no,use=reference'.
That block appeared on 352 Belgian SME sites in November 2026. For 341 of the 741 sites that keep out an AI bot, it explains the entire block (46.0%), and 10 more added names of their own on top. A shared file, meaning the same rules found word for word on five or more sites, accounts for 204 blocks (27.5%); the two largest sit on 63 and 56 sites. Rules we saw nowhere else make up the remaining 196 (26.5%).
Cloudflare wrote almost half of the blocks
Cloudflare's managed robots.txtthe block Cloudflare writes itself, 341 sites
46.0%Shared filethe same rules on five or more sites, 204 sites
27.5%Own rulesa list we saw nowhere else, 196 sites
26.5%
Percent of the sites that block at least one AI bot by name. Measured on 5 November 2026.
Cloudflare's block keeps training bots out and waves search bots through. On 343 of the 352 sites carrying it, every search and user bot on our list could still fetch the homepage. A site with that block stays visible to ChatGPT search and Perplexity, while the training crawlers that honour robots.txt are turned away.
The setup is changing again. Since 1 July 2026 Cloudflare lets you set search, agents and training separately, on the free plan too. On 15 September 2026 it announced that managed robots.txt will be retired in favour of a new system, Bot Preference Sync, and that existing users will be migrated. New domains without ads now allow search, agents and training by default. The files we read on 5 November 2026 still carried the older block.
Own Cloudflare account: one site in five blocks
Whether a site runs behind Cloudflare shows in its homepage headers, through the server header or a cf-ray. That held for 3,953 of the 23,676 sites (16.7%), and together they carry 53.0% of all blocks. Many of those 3,953 are built on Wix, Shopify or Webflow, platforms that run behind Cloudflare themselves; their owners do not control the Cloudflare settings. We therefore split the group in two.
| Network | Sites | Blocks an AI bot | Share | With Cloudflare's block |
|---|---|---|---|---|
| Behind Cloudflare, no hosted builder | 1,761 | 374 | 21.2% | 321 |
| Hosted builder behind Cloudflare | 2,192 | 19 | 0.9% | 12 |
| Not behind Cloudflare | 19,723 | 348 | 1.8% | 19 |
| All working sites | 23,676 | 741 | 3.1% | 352 |
Sites with their own Cloudflare zone block an AI bot 21.2% of the time, against 1.8% for sites outside Cloudflare. On 321 of those 374 the Cloudflare block is literally in the file, so this is not a guess about causes: the rules carry Cloudflare's own markers. The 19 sites outside Cloudflare that still show the block either copied the file or moved away from Cloudflare later.
Behind Cloudflare, one site in five blocks AI bots
All working sites: 3.1%
Behind Cloudflarenot on a site builder, 1,761 sites
21.2%Site builder behind CloudflareWix, Shopify, Webflow and others, 2,192 sites
0.9%Not behind Cloudflareaccording to the homepage headers, 19,723 sites
1.8%
Percent of the working sites per group. Cloudflare recognised by the server header or a cf-ray. Measured on 5 November 2026.
The EU rule Cloudflare points to
On 425 Belgian SME sites, robots.txt includes a long policy text from Cloudflare that cites Article 4 of Directive (EU) 2019/790, the EU copyright directive for the digital single market. Article 4 allows text and data mining of lawfully accessible works. Paragraph 3 sets the limit: the exception applies only where rights holders have not expressly reserved that use 'in an appropriate manner, such as machine-readable means' for content made public online. Cloudflare's text describes restrictions set through Content Signals as such an express reservation.
In November 2026, 386 sites had a Content-Signal line saying ai-train=no. On 75 the policy text stood alone without a signal, so nothing is reserved beyond what the named bots are already refused. On 145 the line said the reverse, ai-train=yes. Of those, 132 run on Ecwid, where the same line 'search=yes,ai-input=yes,ai-train=yes' turns up every time.
Search bots get in almost everywhere
Only 167 of 23,676 Belgian SME websites, or 0.7%, blocked a bot that an AI assistant uses to search or to fetch a page for a user in November 2026. A search bot is shut out on 149 sites, but Amazon's Amzn-SearchBot makes up most of that; leave it aside and 84 remain. User bots such as ChatGPT-User and Claude-User are refused on 63 sites.
The vendors spell out what such a block costs. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers. Anthropic says blocking Claude-SearchBot keeps a site from being indexed for search. Perplexity says PerplexityBot surfaces and links sites and is not used to train models. Our guide on getting found in ChatGPT explains how to show up in those answers, and the glossary entry on GEO covers what that discipline adds to SEO.
Refusing Google-Extended costs nothing in Google itself. Google states the token does not affect inclusion in Google Search and is not a ranking signal. None of the 423 SME sites that refuse Google-Extended also block Googlebot. Apple says the same about Applebot-Extended: pages that disallow it can still appear in search results. To keep your pages out of AI Overviews, Google uses a different control, explained in opt out of Google AI Overviews.
Firms that want to be found block about as often
Do businesses that clearly want customers to find them keep AI assistants out? We looked for three signs in the static HTML of the homepage: local business markup (LocalBusiness or a subtype like Restaurant or Dentist, not Organization alone), a Google reviews widget, and a tap-to-call tel: link. The schema markup glossary entry explains how such markup works.
| Group | Sites | Blocks an AI bot | Blocks a search or user bot |
|---|---|---|---|
| Local business markup or Google reviews widget | 4,476 | 2.9% | 0.5% |
| Tap-to-call link | 11,204 | 3.9% | 1.1% |
| At least one of the three | 12,538 | 3.7% | 1.0% |
| None of the three | 9,435 | 2.7% | 0.4% |
| Homepage needs JavaScript, not read | 1,703 | 1.4% | 0.1% |
Among the 12,538 Belgian SME sites showing one of those signs, 125 (1.0%) blocked an AI search or user bot in November 2026, against 0.4% of sites without them. The gap stays under 5 percentage points, so the two groups are about equal. Firms that want to be found seldom close the door on search bots, though 3.7% of them do refuse a training bot or another AI crawler. If reviews are your weak spot, start with how to get more Google reviews, and make a direct review link with the review link generator.
By sector: healthcare is higher because of two shared files
| Sector | Sites | Blocks an AI bot | Blocks GPTBot | Blocks a search or user bot | Behind Cloudflare |
|---|---|---|---|---|---|
| Hospitality | 8,126 | 2.0% | 1.8% | 0.3% | 15.9% |
| Retail | 6,955 | 3.7% | 2.9% | 0.6% | 19.5% |
| Business services | 2,767 | 3.3% | 2.5% | 0.5% | 14.0% |
| Healthcare | 2,429 | 6.3% | 4.0% | 2.9% | 14.8% |
| Beauty and leisure | 1,430 | 2.3% | 1.7% | 0.6% | 19.2% |
| Construction and trades | 1,330 | 1.7% | 1.4% | 0.4% | 14.7% |
| Automotive | 639 | 3.4% | 2.2% | 0.2% | 13.5% |
| All sites | 23,676 | 3.1% | 2.4% | 0.7% | 16.7% |
Healthcare leads at 6.3% of 2,429 sites, yet most of that comes from two files copied word for word onto 63 and 56 healthcare sites. The larger one, running on Magento, also refuses Amzn-SearchBot, which lifts healthcare to 2.9% for search bots. Take those two files out and 1.5% of the remaining 2,310 healthcare sites block an AI bot. Every gap between sectors stays below 5 percentage points: about equal.
By province: about equal
Each Belgian province fell between 2.0% and 4.4% of SME sites blocking an AI bot in November 2026. Flemish Brabant is highest and Walloon Brabant lowest, but the spread stays well short of 5 percentage points. The share behind Cloudflare ranges from 10.9% to 20.3%.
| Province | Sites | Blocks an AI bot | Blocks GPTBot | Blocks a search or user bot | Behind Cloudflare |
|---|---|---|---|---|---|
| Antwerp | 4,111 | 3.0% | 2.2% | 0.6% | 17.0% |
| East Flanders | 3,839 | 3.4% | 2.7% | 0.9% | 16.2% |
| West Flanders | 3,459 | 3.0% | 2.3% | 0.8% | 13.2% |
| Flemish Brabant | 2,523 | 4.4% | 3.0% | 0.9% | 19.1% |
| Brussels | 2,393 | 2.6% | 2.0% | 0.3% | 20.3% |
| Limburg | 1,938 | 3.9% | 2.9% | 0.6% | 18.3% |
| Liège | 1,621 | 2.6% | 2.3% | 0.6% | 14.0% |
| Hainaut | 1,245 | 2.4% | 2.0% | 0.9% | 16.8% |
| Walloon Brabant | 1,077 | 2.0% | 1.7% | 0.3% | 20.2% |
| Namur | 836 | 3.5% | 3.0% | 1.0% | 16.0% |
| Luxembourg | 634 | 2.1% | 1.9% | 1.1% | 10.9% |
Between 2 and 4.5 percent everywhere
Percent of the working sites per province; the province comes from the address in OpenStreetMap.
The platform decides who writes the file
On a hosted builder the platform writes robots.txt; on a self-run install the web developer or the owner does. The numbers reflect that. On Magento, 43.2% of 185 SME sites block an AI bot, on PrestaShop 16.1% of 174 and on Craft CMS 13.1% of 183, against 2.2% on WordPress. Most of the Magento figure comes from the shared file on 63 healthcare sites. On Wix, one.com, JouwWeb and Proxi almost nobody blocked an AI bot.
| Platform | Sites | Blocks an AI bot | Blocks a search or user bot | Behind Cloudflare |
|---|---|---|---|---|
| WordPress | 7,902 | 2.2% | 0.3% | 7.8% |
| No platform detected | 7,359 | 4.1% | 0.6% | 11.0% |
| Wix | 1,515 | 0.2% | 0.0% | 74.9% |
| Drupal | 907 | 2.5% | 0.2% | 7.6% |
| one.com Builder | 887 | 0.1% | 0.0% | 0.1% |
| Shopify | 631 | 1.0% | 0.2% | 94.5% |
| Squarespace | 501 | 0.8% | 0.0% | 2.4% |
| Odoo | 398 | 6.5% | 0.3% | 7.3% |
| Joomla | 335 | 1.2% | 0.3% | 3.0% |
| Next.js | 316 | 9.5% | 0.6% | 16.5% |
| Ecwid | 306 | 3.9% | 0.3% | 1.6% |
| Webflow | 275 | 2.9% | 1.8% | 88.4% |
| JouwWeb | 255 | 0.0% | 0.0% | 0.0% |
| Proxi | 252 | 0.0% | 0.0% | 0.0% |
| Magento | 185 | 43.2% | 35.1% | 23.8% |
| Craft CMS | 183 | 13.1% | 9.3% | 5.5% |
| PrestaShop | 174 | 16.1% | 2.3% | 19.0% |
Squarespace offers a checkbox, 'Block known artificial intelligence crawlers', that is off by default. Its standard file does list AI bots by name, yet leaves the homepage open to them: that was the case on 478 of the 501 Squarespace sites. WordPress.com has a 'Prevent third-party sharing' option that adds known AI bots to robots.txt. The CMS detector shows which platform a site runs on, and Wix vs Squarespace compares two of the builders above.
Why the figure is higher than in October
Our study of Belgian business websites in 2026 found 0.6% blocking GPTBot: 3 of the 477 sites that responded, out of a sample of 586. This count lands at 2.4% for GPTBot and 3.1% for any AI bot. Running our parser over the stored October files gives the same 3, so the gap is not down to how the files are read.
| Measurement | Sites | Block GPTBot | Share |
|---|---|---|---|
| October 2026 sample | 477 | 3 | 0.6% |
| Same 477 sites, 5 November 2026, by name | 477 | 6 | 1.3% |
| Same 477 sites, including User-agent: * | 477 | 7 | 1.5% |
| Full count, 5 November 2026 | 23,676 | 569 | 2.4% |
Four things explain the rest. The files changed: in one month the same 477 sites gained four blocks, three through Cloudflare's block and one through a rule that closes everything. Across all 764 domains of that sample, files with Cloudflare's block went from 4 to 8. The sample also weighted each sector equally, while the count follows the full frame. The headline here covers sixteen bots, not GPTBot alone. And 3 out of 477 is a small number in which a handful of sites moves the share a lot. The October sample saw no site refusing the search bots of ChatGPT, Claude or Perplexity; on 477 sites a rate of 0.7% can easily come out as zero.
What robots.txt cannot show
robots.txt is a request, not a lock. Cloudflare itself notes that its directives are voluntary. OpenAI, Meta, Amazon and Perplexity say their user-triggered fetchers may not follow all robots.txt rules. A block at network level, such as a firewall rule or a block set in Cloudflare's dashboard, turns a bot away without robots.txt saying so. You only see that by pretending to be the bot, which we did not do: every request went out under a name that identifies our own research. The real share of sites keeping AI bots out is therefore higher than robots.txt shows.
We also left out 370 domains whose robots.txt refuses every unnamed crawler. Of those, 218 still let Googlebot in, 152 shut out Google too, and 359 block GPTBot. Among the working sites, just 2 closed everything to every bot. How classic search and AI answers differ is set out in our SEO vs GEO comparison.
What this means for your business
Blocking an AI bot is a legitimate choice. What matters is knowing what your file says and who put it there. Here is how to check:
- Read your own robots.txt. Enter your domain in the AI crawler checker and see which bots may come in.
- Check whether Cloudflare sits in front of your site. If your file holds the block between '# BEGIN Cloudflare Managed content' and '# END Cloudflare Managed Content', it comes from your Cloudflare account. That is where you switch search, agents and training on or off.
- Decide on training and search separately. To appear in ChatGPT, Claude or Perplexity, keep OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed. You can still refuse training at the same time.
- Leave Googlebot alone. Refusing Google-Extended does not touch your place in Google; refusing Googlebot does. What zero-click search means for your traffic is a separate question worth reading up on.
- Make your business easy to read. Local business markup, current opening hours and reviews help every search engine. Run a quick SEO check to spot the gaps.
- Ask who manages the file. Your web developer, host or platform can add rules without telling you. Want a second pair of eyes? Book a call or see our found in ChatGPT service.
Before you change your robots.txt
How do you block AI crawlers in robots.txt?
Give each bot its own group: 'User-agent: GPTBot' followed by 'Disallow: /'. Every vendor publishes its bot names, and most run separate bots for training and for search, so refuse only what you actually want to keep out. In November 2026, 3.1% of 23,676 Belgian SME websites did so for at least one bot.
Does Cloudflare block AI crawlers by default?
Not for new domains without ads: since 15 September 2026 those allow search, agents and training. New domains with ads refuse training by default. The older managed robots.txt was something you switched on, or accepted when Cloudflare asked at sign-up. In November 2026, 352 Belgian SME sites carried that block.
Does blocking GPTBot hide you from ChatGPT?
No. GPTBot gathers content for training. ChatGPT searches with OAI-SearchBot, and only sites that refuse OAI-SearchBot drop out of ChatGPT search answers. Of the 569 Belgian SME sites blocking GPTBot, the large majority still let OAI-SearchBot in. Our guide to getting found in ChatGPT covers the rest.
Will blocking Google-Extended remove you from Google?
No. Google-Extended controls whether Gemini may use your content for training and grounding. Google says it does not affect inclusion in Google Search and is not a ranking signal. None of the 423 Belgian SME sites refusing Google-Extended also blocked Googlebot.
Do AI bots have to obey robots.txt?
robots.txt is a request. The large vendors say their crawlers follow it, but bots that fetch on a user's behalf do not always do so. In the EU, Article 4(3) of Directive 2019/790 gives a machine-readable reservation weight when it comes to text and data mining.
How many websites block AI crawlers in Belgium?
In November 2026, 3.1% of 23,676 working Belgian SME websites blocked at least one named AI crawler, and 0.7% blocked an AI search or user bot. With their own Cloudflare account the figure was 21.2%. The open data by sector, province and platform is in the CSV that comes with this study.