How to Block AI Crawlers (GPTBot, ClaudeBot, and Others)

AI companies are crawling the web at unprecedented scale. OpenAI'sGPTBot, Anthropic'sClaudeBot, ByteDance'sBytespider, and dozens of others are scraping website content to train large language models. For site owners, this raises three immediate concerns.

AI crawlers are aggressive. Unlike Googlebot, which is designed to crawl politely with rate limits, some AI bots hammer your server with hundreds of requests per minute. Bytespider in particular has earned a reputation for excessive crawling that can spike bandwidth costs and slow down your site for real visitors. If you're on a metered hosting plan, these bots are literally costing you money.

When an AI model trains on your content, your words become part of its output — without attribution, compensation, or consent. Publishers, journalists, and creators have legitimate reasons toblock AI crawlersfrom accessing their work. Whether you view this as a copyright issue or a business decision, the right to opt out is yours.

Before you start blocking, you need to know exactly which bots to target. Here is a comprehensive list of known AI crawler user agents as of early 2026, organized by company:

There are four primary ways toblock AI crawlersfrom your website. Each operates at a different layer, and the strongest protection combines multiple methods:

The simplest way toblock AI botsfrom your website is through your robots.txt file. This file lives at the root of your domain (e.g., yoursite.com/robots.txt) and tells crawlers which areas they can and cannot access. Add the following rules to block all major AI crawlers:

EachUser-agentdirective targets a specific bot by name, and Disallow: /blocks access to your entire site. The limitation of robots.txt is that it is a request, not enforcement. Well-behaved bots respect it, but nothing technically prevents a rogue crawler from ignoring it entirely. That is why server-level blocking matters.

For stronger protection, block AI crawlers at the server level. This returns a403 Forbiddenresponse before any content is served — the bot receives nothing at all, regardless of whether it respects robots.txt.

If you run Apache, add these rewrite rules to your.htaccessfile:

The[NC]flag makes the match case-insensitive, and[F,L] returns a 403 Forbidden and stops processing. The last condition in the chain omits the [OR]flag to properly close the rule.

For Nginx servers, use amapdirective in your server block. This is more efficient than chaining multipleifstatements because Nginx evaluates it once per request:

The~*prefix makes each pattern case-insensitive. After adding these rules, reload Nginx withsudo nginx -s reload. The advantage of server-level blocking is that the bot receives a 403 immediately — no content is ever transferred.

If your site is behind Cloudflare, you have access to built-in AI bot blocking without touching server configs. Cloudflare added a dedicated toggle in 2024 specifically for AI crawlers.

For more granular control, create a custom WAF rule. Go toSecurity → WAF → Custom rulesand create a rule with this expression:

1. Go to your Cloudflare dashboard and select your domain.
2. Navigate to<strong>Security → Bots</strong>.
3. Enable<strong>“Block AI Scrapers and Crawlers.”</strong>This single toggle blocks all known AI training bots, and Cloudflare updates the list automatically as new crawlers appear.

Set the action toBlock. This gives you the flexibility to allow some AI bots while blocking others — just remove their line from the expression. The advantage of CDN-level blocking is performance: the request never reaches your origin server, saving bandwidth and compute resources.

In addition to robots.txt and server rules, you can add HTML meta tags that signal to AI crawlers that your content should not be used for training. These tags work at the page level, giving you fine-grained control over which pages opt out.

Add this meta tag to thesection of any page you want to protect from AI training:

Thenoaidirective tells AI crawlers not to use the page's text content for training. Thenoimageaidirective specifically protects images from being used in image generation training datasets. Google introduced these directives alongside Google-Extended, and other AI companies have begun recognizing them.

You can also use bot-specific meta tags to target individual crawlers. Setting noindex, nofollowfor a specific bot name prevents that bot from indexing the page or following its links.

Each blocking method has different setup complexity, enforcement strength, and maintenance requirements. Choose based on your technical comfort level and how aggressively you need to block:

You've added robots.txt rules, server-level blocks, and meta tags. But how do you know they're actually working? Verification requires checking two things: that your rules are syntactically correct, and that blocked bots are actually receiving 403 responses.

Use Google's robots.txt Testerin Search Console to validate syntax. You can also manually verify by visitingyoursite.com/robots.txtin a browser and confirming your rules appear correctly.

Your web server access logs record every request, including the user agent string. Search for AI crawler activity and check the response codes:

If blocked bots still appear with 200 status codes, your server-level rules are not working correctly. You should see 403 responses if .htaccess or Nginx blocking is active. If you only have robots.txt rules (no server-level blocking), compliant bots simply will not request your pages — they will not appear in logs at all.

For ongoing monitoring without manually digging through log files, Copper Analytics's crawler tracking automates this process. It shows you exactly which bots are visiting, how often, and which pages they target — all in a visual dashboard.

Before you block everything, consider whether a blanket block is the right strategy for your site. There are legitimate reasons to allow some AI crawler access:

The smartest approach to AI crawlers is not to block blindly — it is to understand what is happening first. Before you add a single rule to your robots.txt, find out which bots are actually visiting your site, how much bandwidth they consume, and which pages they target most.

Copper Analytics's crawler tracking dashboard identifies AI bots by user agent, tracks their request volume over time, and shows you exactly which pages they are accessing. You can monitor AI crawler activity alongside your regular human traffic — all in one place, without digging through raw server logs.

Once you have data, you can make informed decisions: block the high-volume training bots eating your bandwidth, allow the retrieval bots sending you referral traffic, and verify that your blocking rules are actually working. Data-driven blocking beats guesswork every time.