Case Study: Neutralizing AI Crawlers & Scrapers

This case study demonstrates how to protect web applications, APIs, and content from automated Large Language Model (LLM) training bots, aggressive data harvesters, and rogue crawlers using RouteWarden active defense.

---

The Threat Model

Commercial artificial intelligence systems and data brokerage firms continuously deploy automated web crawlers to harvest proprietary articles, technical documentation, user-generated content, and API data to train LLMs.

Why AI Crawlers Are Harmful to Production Infrastructure

1. Extreme Resource & Concurrency Spikes: Unlike polite search engines that respect crawl rates, bots like ByteSpider and ClaudeBot frequently dispatch hundreds of concurrent threads. They scan paginated datasets, search queries, and dynamic filters, causing massive database load and CPU spikes on backend servers.
2. Cloud Egress Bill Shock: Scraping entire sites repeatedly consumes hundreds of gigabytes of network egress, directly increasing cloud hosting and CDN bills.
3. Disregard for robots.txt: While some well-known bots publish adherence to robots.txt, many rogue scrapers, third-party proxies, and headless browser fleets completely ignore User-agent: * Disallow: / directives.
4. Unlicensed Content Ingestion: Data is scraped without attribution, license compliance, or consent.

Primary AI Crawler User-Agents

Organization	Bot User-Agent	Primary Activity
ByteDance / TikTok	Bytespider	Aggressive LLM and recommendation engine training; notoriously high request volume.
OpenAI	GPTBot, ChatGPT-User	GPT model training dataset collection and live browsing requests.
Anthropic	ClaudeBot, Anthropic-AI	Claude training data crawling across public web applications.
Perplexity AI	PerplexityBot	Real-time conversational search indexing and summarization.
Common Crawl	CCBot	Open petabyte-scale web data pipeline consumed by dozens of foundation AI models.
Meta	FacebookBot	LLaMA model training and AI assistant grounding.
Google	Google-Extended	Separate crawler token used specifically for Gemini / Vertex AI model training.
Other Aggressive Harvesters	Diffbot, cohere-ai, Omgilibot, Amazonbot, ImagesiftBot	Commercial data brokers, knowledge graphs, and enterprise model pre-training.

---

The Defense Strategy: Active Neutralization

Standard WAFs typically respond to blocked user-agents with a static 403 Forbidden. However:

• Automated crawlers treat 403 as an instantaneous response and immediately proceed to request the next 10,000 URLs in their queue.
• Some scrapers automatically rotate proxy IP addresses when encountering 403 or 429 status codes.

RouteWarden uses Active Defense response modes to neutralize scraping operations:

●READONLY│UTF-8│MERMAID

1. tarpit (Reverse Slowloris):
   • Responds with 200 OK and slowly sends one byte every 2 seconds for up to 120 seconds.
   • Attacker Impact: Scrapers operate with fixed thread pools (typically 20–100 workers). By holding 50 sockets open for two minutes, RouteWarden starves the scraper's thread pool, grinding their crawl to a halt without burdening your backend.
2. silentDrop (Zero-Egress Socket Reset):
   • Resets the TCP socket immediately (TCP RST/EOF) with zero bytes sent.
   • Attacker Impact: Saves maximum outbound bandwidth and breaks headless browser connection pools instantly.
3. gzipBomb (Memory Exhaustion Trap):
   • Serves an enticing 200 OK with a 10MB gzip payload that expands to ~10GB in memory.
   • Attacker Impact: Python scrapers (requests, Scrapy) and headless Chrome instances that auto-decompress responses crash with Out-Of-Memory (OOM) fatal errors.

---

Strategy 1: Active Tarpitting (tarpit) — Recommended

This configuration intercepts AI crawler User-Agents and traps them in a 120-second reverse Slowloris tarpit, exhausting their concurrency pools while letting normal users and legitimate search engines through.

File:

# dynamic_conf.ymlhttp:  middlewares:    ai-crawler-tarpit:      plugin:        routewarden:          enabled: true          methods:            - GET            - POST            - HEAD          enableDefaultPatterns: true          enableDefaultAllowPatterns: true # Keep /robots.txt & /sitemap.xml accessible          checkHeaders:            - "User-Agent"          blockPatterns:            - '(?i)(GPTBot|ChatGPT-User|ClaudeBot|Anthropic-AI|PerplexityBot|Bytespider|CCBot|Diffbot|FacebookBot|Google-Extended|Omgilibot|cohere-ai|Amazonbot|ImagesiftBot)'          response:            mode: tarpit # Hold crawler worker threads hostage            statusCode: 200            tarpitDelayMs: 2000 # Send 1 byte every 2 seconds            tarpitMaxDurationSeconds: 120 # Lock crawler socket for 2 minutes  routers:    web-router:      rule: "Host(`example.com`)"      entryPoints:        - websecure      middlewares:        - ai-crawler-tarpit      service: web-service

●READONLY│UTF-8│YAML

TIP

Preserving Googlebot & Legitimate Search Engine Indexing: RouteWarden's enableDefaultAllowPatterns: true is enabled by default. This ensures critical discovery files like /robots.txt and /sitemap.xml remain accessible so legitimate search engines (Googlebot, Bingbot) continue indexing your site normally.

---

Strategy 2: Zero-Egress Connection Reset (silentDrop)

For servers operating under tight bandwidth quotas or facing high-volume scraper attacks (e.g. distributed ByteSpider clusters), silentDrop terminates the TCP connection with zero bytes returned.

# dynamic_conf.yml: Instant Socket Teardownhttp:  middlewares:    ai-crawler-drop:      plugin:        routewarden:          enabled: true          methods:            - GET            - POST            - HEAD          checkHeaders:            - "User-Agent"          blockPatterns:            - '(?i)(GPTBot|ChatGPT-User|ClaudeBot|Anthropic-AI|PerplexityBot|Bytespider|CCBot|Diffbot|FacebookBot|Google-Extended|Omgilibot|cohere-ai|Amazonbot|ImagesiftBot)'          response:            mode: silentDrop # Reset TCP connection with 0 bytes sent

●READONLY│UTF-8│YAML

---

Strategy 3: Memory Exhaustion Decompression Bomb (gzipBomb)

When scrapers aggressively ignore polite rate limits or scan protected endpoints, gzipBomb returns an HTTP 200 OK header containing a compressed recursive zero-byte stream.

# dynamic_conf.yml: Gzip Decompression Bombhttp:  middlewares:    ai-crawler-bomb:      plugin:        routewarden:          enabled: true          methods:            - GET            - HEAD          checkHeaders:            - "User-Agent"          blockPatterns:            - '(?i)(GPTBot|ChatGPT-User|ClaudeBot|Anthropic-AI|PerplexityBot|Bytespider|CCBot|Diffbot|FacebookBot|Google-Extended|Omgilibot|cohere-ai|Amazonbot|ImagesiftBot)'          response:            mode: gzipBomb # Explodes in crawler RAM            statusCode: 200 # Enticing 200 OK triggers auto-decompression            gzipBombMB: 10 # 10MB expands to ~10GB in memory

●READONLY│UTF-8│YAML

WARNING

Use on Verified Scrapers Only: Web vulnerability scanners and automated scrapers decompress responses in memory. Ensure checkHeaders: ["User-Agent"] is configured with strict regex so normal human browsers are never served decompression bombs.

---

Testing & Verification

You can verify your AI crawler defense using curl with custom User-Agent strings.

1. Test That AI Crawlers Are Tarpitted

# Test with GPTBot (should stall and receive trickle bytes for 120 seconds)
time curl -i -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://example.com/

# Test with Bytespider
time curl -i -A "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Bytespider" https://example.com/

# Test with ClaudeBot
time curl -i -A "ClaudeBot/1.0 (+https://www.anthropic.com/claudebot)" https://example.com/

2. Verify Legitimate Browsers & Search Engines Pass Normally

# Standard desktop browser: returns 200 OK immediately
curl -i -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" https://example.com/

# Googlebot search indexer on robots.txt: allowed by default allowlist
curl -i -A "Googlebot/2.1 (+http://www.google.com/bot.html)" https://example.com/robots.txt

---

Live Monitoring & Observability with rwarden push

When streaming logs to the central RouteWarden dashboard via rwarden push, every blocked or tarpitted crawler is tracked in real time:

# Monitor container logs for live scraper mitigation
rwarden push --docker

On the RouteWarden SOC Dashboard:

• AI scraper mitigations are highlighted with Verdict: "TARPIT" or Verdict: "BLOCK".
• Offending IP addresses, Autonomous System Numbers (ASNs), and targeting frequencies are triaged in the live event stream.
• If integrated with CrowdSec, IPs triggering repeated scraper mitigations can be automatically banned across your entire edge fleet via firewall bouncers (ipset/nftables).