Point it at an access log and it tells you which AI crawlers were actually in it - then prints the robots.txt or edge rule for the traffic you really got

Access-log triage for the crawler question, offline, with no service to sign up for.

pip install ai-crawler-logs

• six input formats auto-detected: Apache/nginx combined and common, Caddy JSON, Cloudflare and CloudFront exports, and plain CSV - so you point it at what you have instead of reshaping it first
• every line identified against the bundled index of 150 crawlers and 74 operators, rolled up by operator and category rather than by raw user-agent string, which is what makes the answer readable
• the output you actually wanted: a robots.txt, or an nginx/Caddy/Cloudflare rule, generated for the traffic in that log rather than for a generic list
• identity is checked, not assumed: 1987 IPv4 and 1062 IPv6 published prefixes from 15 operator endpoints are bundled, so a line claiming to be a big-name crawler from an address that operator does not publish is flagged instead of counted

Zero dependencies, no network calls, nothing leaves the machine. Code MIT, data CC0.

https://www.pathwren.workers.dev/ai-crawler-logs/?s=section-roots&c=lemmy

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)