AI Crawler IP Addresses: Known Ranges, Verification, and Blocking

Every request to your web server arrives with two key identifiers: a user-agent string and a source IP address. While user-agent strings tell you what a bot claims to be, the IP address tells you where the request actually originates. For AI crawler management, this distinction is critical.

User-agent spoofing is trivial. Any script can set its user-agent to "GPTBot" or "ClaudeBot" and your robots.txt rules will apply to it, but the request may actually come from a scraper with no connection to OpenAI or Anthropic. Conversely, a malicious actor could strip the AI bot user-agent entirely and crawl your site without triggering any bot-specific rules.

IP verification closes this gap. By checking the source IP of a request against the known IP ranges published by AI companies, you can confirm whether a bot is genuinely operated by OpenAI, Anthropic, Google, or ByteDance — or whether it is an impersonator. This is the same technique Google has recommended for years to verify Googlebot.

Most major AI companies now publish their crawler IP ranges publicly. The format and update frequency vary by company, but the information is available if you know where to look.

OpenAI publishes GPTBot and ChatGPT-User IP ranges in a JSON file at https://openai.com/gptbot-ranges.json. This file contains CIDR blocks that are updated periodically as OpenAI expands its crawling infrastructure. As of early 2026, the ranges include multiple /24 and /23 blocks across several cloud providers.

Anthropic does not publish a static IP list for ClaudeBot, but all ClaudeBot requests originate from IP addresses that resolve via reverse DNS to hostnames ending in *.anthropic.com. This makes reverse DNS the primary verification method for ClaudeBot traffic.

Google-Extended shares its IP infrastructure with Googlebot. Google publishes Googlebot IP ranges at https://developers.google.com/search/apis/ipranges/googlebot.json, and any IP verified as Googlebot via reverse DNS to *.googlebot.com or *.google.com may also be used by Google-Extended.

ByteDance operates Bytespider from data center IPs, primarily in US and Singapore regions. ByteDance does not publish an official IP list, but the IPs are identifiable by reverse DNS resolution to ByteDance-owned domains and by their consistent presence in known ByteDance CIDR ranges.

Reverse DNS (rDNS) lookup is the most reliable method for verifying that an AI crawler is genuinely operated by the company it claims to represent. The process works in two steps: a reverse lookup from IP to hostname, followed by a forward lookup to confirm the hostname resolves back to the same IP.

Two-Step DNS Verification Process

1. Run a reverse DNS lookup on the source IP: <code>dig -x 203.0.113.45</code> — this returns a hostname like <code>crawl-203-0-113-45.anthropic.com</code>.
2. Verify the hostname resolves back to the original IP: <code>dig +short crawl-203-0-113-45.anthropic.com</code> — the result should match <code>203.0.113.45</code>.
3. If both lookups match, the bot is verified. If the reverse lookup returns no result, or the forward lookup points to a different IP, the request is likely spoofed.

This two-step verification prevents DNS spoofing. An attacker could set up a reverse DNS record pointing their IP to crawl.anthropic.com, but the forward lookup would fail because Anthropic controls the A records for that domain.

For ClaudeBot, the reverse DNS should resolve to a hostname ending in .anthropic.com. For GPTBot, reverse DNS resolves to hostnames within OpenAI-controlled ranges. For Google-Extended, the hostname should end in .googlebot.com or .google.com — the same verification used for regular Googlebot.

There are two primary methods for blocking AI crawlers: user-agent-based rules (via robots.txt or server configuration) and IP-based blocking (via firewall rules or CDN configuration). Each approach has strengths and limitations, and the most effective strategy uses both.

The trade-off is maintenance. AI companies rotate and expand their IP ranges regularly. A firewall rule that blocks all current GPTBot IPs may miss new ranges added next month. User-agent rules, by contrast, remain effective as long as the bot identifies itself consistently.

Once you have the IP ranges you want to block, implementation depends on your infrastructure. The three most common approaches are server-level firewall rules, web server configuration, and CDN or WAF rules.

For dynamic or large-scale blocking, a CDN-level approach is more practical. Cloudflare, AWS WAF, and Fastly all support IP list rules that can be updated via API. This lets you automate updates — pull the latest IP ranges from OpenAI or Google, push them to your CDN, and the rules take effect globally within seconds.

On Linux servers, iptables or nftables provide the most direct firewall-level blocking. You can script the process to fetch published IP ranges and update your firewall rules automatically on a cron schedule.

The biggest challenge with IP-based AI crawler management is maintenance. AI companies regularly expand their infrastructure, adding new IP ranges and retiring old ones. A blocklist that was complete last month may have gaps today.

OpenAI updates their published GPTBot ranges without announcement. Google updates their Googlebot IP list as they add new crawling infrastructure. Anthropic and ByteDance do not publish static lists at all, which means you need to discover new ClaudeBot and Bytespider IPs through log analysis.

The practical approach is a combination strategy: use robots.txt for first-line blocking of compliant crawlers, IP blocking for enforcement against non-compliant bots, and continuous monitoring to detect new IPs that slip through both layers.

Automating the update process is essential. Set up a cron job or CI/CD pipeline step that fetches the latest published IP ranges, compares them against your current blocklist, and applies changes automatically. For crawlers without published IP lists, aggregate new IPs from your server logs weekly.

Manual IP verification and blocklist maintenance is effective but time-consuming. Copper Analytics automates the entire process — from identifying AI crawlers by IP to verifying their identity via reverse DNS to tracking new IP ranges as they appear.

The Crawlers dashboard shows every AI bot that has visited your site, organized by company and verified by IP. You can see which IP ranges GPTBot is currently using, confirm that ClaudeBot requests genuinely resolve to Anthropic hostnames, and spot unknown bots that may be impersonating legitimate crawlers.

For security teams, the IP-level visibility means you can export verified IP data to feed your firewall rules, WAF policies, or CDN blocklists. Instead of manually scraping logs and running DNS lookups, you get a continuously updated, verified dataset of every AI crawler touching your infrastructure.