caddy-ibe-shield

Caddy handler that puts a Cap captcha in front of the IBE, but only for visitors that look like scrapers:

• IP from a listed country (MaxMind GeoLite2),
• more than rate requests per window from one IP,
• a scraper User-Agent (curl, python-requests, Scrapy, HeadlessChrome, ...) or none,
• a request flagged by another handler via flag_header, e.g. a cloud/AI-crawler IP marked by caddy-defender (see below).

The last three challenge immediately, without waiting for the rate limit.

Everyone else passes through untouched. Solving the captcha sets a signed cookie (ibe_shield, bound to the User-Agent) that lasts cookie_ttl and is refreshed while the visitor is active. A solved visitor gets challenged again only when they go over 10× rate; solving again resets the counter.

Page loads (Sec-Fetch-Mode: navigate) get the challenge page. It solves Cap's proof-of-work headless (no Cap widget or branding) after a real click (isTrusted), shows its own progress bar and continues by itself. Text comes from translations in page.go (en, de, fr, es, it, nl, picked from Accept-Language, English fallback); add a language by adding a block with the same keys. Light/dark follows the OS. XHR/API requests get a plain 403, since they can't show a captcha.

Build

go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
xcaddy build v2.11.7 --with git.e-confirm.de/settings-master/caddy-ibe-shield=.
# for linux servers:
GOOS=linux GOARCH=amd64 xcaddy build v2.11.7 --with git.e-confirm.de/settings-master/caddy-ibe-shield=.
./caddy list-modules | grep ibe_shield

Swap the binary in place (/usr/bin/caddy), then systemctl restart caddy. Once the repo is pushed, --with git.e-confirm.de/settings-master/caddy-ibe-shield@main works without a local checkout.

GeoIP

Free MaxMind account, then use geoipupdate (EditionIDs GeoLite2-Country) plus a weekly cron. The DB is read at startup, so reload Caddy after an update (systemctl reload caddy).

Caddyfile

{
	servers {
		# only if Caddy is behind a load balancer/CDN; otherwise leave it out
		trusted_proxies static 10.0.0.0/8
	}
}

booking.example.com {
	# assets don't count towards the rate limit and are never challenged
	@pages not path /static/* /assets/* *.js *.css *.png *.jpg *.svg *.woff2 /favicon.ico
	ibe_shield @pages {
		geoip_db     /var/lib/GeoIP/GeoLite2-Country.mmdb
		countries    CN RU KP IR VN
		rate         120 1m
		cookie_ttl   12h
		secret       {env.IBE_SHIELD_SECRET}
		cap_endpoint https://cap.example.com/<site_key>/
		cap_secret   {env.IBE_SHIELD_CAP_SECRET}
		# widget_script https://cap.example.com/assets/widget.js   # optional, default jsDelivr @cap.js/widget@0.1.58
	}
	reverse_proxy app:8080
}

Secrets go in systemd (systemctl edit caddy):

[Service]
Environment=IBE_SHIELD_SECRET=<openssl rand -hex 32>
Environment=IBE_SHIELD_CAP_SECRET=<cap key secret>

ibe_shield takes a matcher. Without one, assets count towards the rate limit.

The challenge form posts to /.ibe-shield/verify; the matcher must include that path, else the POST goes past the shield to the app. A not path ... matcher like @pages above includes it automatically. To protect only certain pages, list them plus the verify path:

@shield path /kontakt* /regionen* /hotel* /.ibe-shield/*
ibe_shield @shield {
	...
}

(/hotel* also covers /hotels, /hotels/berlin, ...) The handler is ordered before basic_auth/handle, so a handle block won't keep requests away from it.

Options

option	default
geoip_db <path>	none	MaxMind Country/City .mmdb, needed for countries
flag_header <name>	none	challenge requests that carry this header
countries <CC>...	none	ISO codes, space or comma separated, repeatable
rate <n> <window>	off	page loads: challenge after more than n per window per IP
api_rate <n> <window>	unset	AJAX/fetch requests get their own budget; unset = they count towards rate
cookie_ttl <dur>	12h	how long a solve lasts, renewed while active
secret <str>	required	cookie HMAC key, 32+ chars
cap_endpoint <url>	required	https://<cap>/<site_key>/
cap_secret <str>	required	site key secret
widget_script <url>	jsDelivr @cap.js/widget@0.1.58	Cap client script

AJAX-heavy frontends

Requests that aren't page loads (Sec-Fetch-Mode other than navigate: fetch, XHR) count towards api_rate instead of rate, so a busy booking session doesn't use up the page budget. Don't exclude the API with a matcher: scrapers prefer the JSON endpoints, api_rate is what stops them there.

To size it: open a typical booking flow with DevTools > Network (Fetch/XHR), count the calls per minute of fast clicking, and set api_rate a few times above that, e.g. rate 120 1m + api_rate 1200 1m.

Over the limit, AJAX calls get 403 with X-IBE-Shield: challenge (they can't show a captcha). The next page load shows the challenge. Optionally the frontend can do that right away:

if (res.status === 403 && res.headers.get("X-IBE-Shield") === "challenge") location.reload();

(API on another origin: add Access-Control-Expose-Headers: X-IBE-Shield.)

Cloud and AI-crawler IPs (caddy-defender)

Real customers browse from home and mobile networks; scrapers mostly run in data centers. caddy-defender ships maintained IP lists for those. Its ratelimit responder doesn't block, it only sets X-RateLimit-Apply: true on the request and passes it on, and ibe_shield challenges it:

xcaddy build v2.11.7 \
  --with git.e-confirm.de/settings-master/caddy-ibe-shield=. \
  --with pkg.jsn.cam/caddy-defender

booking.example.com {
	defender ratelimit {
		ranges aws gcloud azurepubliccloud oci aliyun huawei digitalocean linode vultr openai deepseek mistral githubcopilot
	}
	@pages not path /static/* /assets/* *.js *.css *.png *.jpg *.svg *.woff2 /favicon.ico
	ibe_shield @pages {
		flag_header X-RateLimit-Apply
		# ... rest as above
	}
	reverse_proxy app:8080
}

Directive order is automatic (defender runs before ibe_shield). Its lists have no Hetzner/OVH/Contabo; add their CIDRs to ranges if you see scrapers there. Leave out vpn and cloudflare (WARP), they hit real customers. The lists are compiled in, so rebuild now and then to refresh them.

Health checks and uptime monitors usually use curl-like agents: keep their path out of the @pages matcher.

Several sites

Each ibe_shield block is its own instance: own settings, own counters. Share the setup with a snippet and pass what differs per site:

(ibe_shield) {
	@pages not path /static/* /assets/* *.js *.css *.png *.jpg *.svg *.woff2 /favicon.ico
	ibe_shield @pages {
		geoip_db     /var/lib/GeoIP/GeoLite2-Country.mmdb
		countries    {args[0]}
		rate         {args[1]} 1m
		secret       {$IBE_SHIELD_SECRET}
		cap_endpoint https://captcha.e-confirm.de/{args[2]}/
		cap_secret   {$IBE_SHIELD_CAP_SECRET}
	}
}

booking-a.example.com {
	import ibe_shield "CN RU" 120 <site_key_a>
	reverse_proxy app-a:8080
}

booking-b.example.com {
	import ibe_shield "CN RU KP IR" 60 <site_key_b>
	reverse_proxy app-b:8080
}

One Cap secret per site key: use separate env vars ({$CAP_SECRET_A}) if the sites use different site keys. Leave ibe_shield out of a site to not protect it.

A config reload (systemctl reload caddy) resets the request counters; cookies stay valid as long as secret doesn't change.

Log file

Every encounter is logged by http.handlers.ibe_shield:

msg	when	extra fields
challenged	request got the challenge (page) or a 403 (AJAX)	reason (rate, country:XX, bot_agent, flagged, rate_after_solve), hits, page
solved	captcha verified, cookie set	return
solve_failed	token missing or rejected by Cap

All carry host, ip, method, uri, user_agent, so one file covers every site. Collect them in the global options block:

{
	log ibe_shield {
		output file /var/log/caddy/ibe-shield.log {
			roll_size 50MiB
			roll_keep 10
		}
		format json
		include http.handlers.ibe_shield
	}
	log default {
		exclude http.handlers.ibe_shield   # optional: keep them out of the main log
	}
}

The caddy user needs write access to /var/log/caddy (the Debian/RPM packages create it). Queries with jq:

# reasons, most common first
jq -r 'select(.msg=="challenged") | .reason' /var/log/caddy/ibe-shield.log | sort | uniq -c | sort -rn
# IPs challenged most
jq -r 'select(.msg=="challenged") | .ip' /var/log/caddy/ibe-shield.log | sort | uniq -c | sort -rn | head
# solve rate per site: few solves vs many challenges = mostly bots
jq -r 'select(.msg=="challenged" or .msg=="solved") | "\(.host) \(.msg)"' /var/log/caddy/ibe-shield.log | sort | uniq -c

Start with a high rate, watch reason=rate entries for a few days, then lower it. Many solved from one reason means real customers get hit: loosen that one.

Limits

• Rate counters are in memory, per Caddy process (fixed window). With several Caddy nodes, each one only sees its own share of the traffic.
• A POST form submit that gets challenged loses its body. After solving, the visitor lands back on the URL as a GET.
• If Cap is down, flagged visitors can't get through (fails closed). Unflagged visitors aren't affected.

Test

go test ./...