Which company is behind each crawler in your logs: 74 operators, 150 crawlers, mapped from the token to the org that runs it

A log line gives you a token. What you need in order to decide anything is the organisation behind it, and that mapping is genuinely annoying to assemble: one operator often runs eight or ten differently named crawlers with different purposes and different opt-outs, and the names change with rebrands.

So this is one page per operator - 74 of them, covering 150 crawlers - and each page carries:

• every token that operator runs, with its category: AI training, AI search, user-triggered fetch, classic search index, SEO, archive, liveness check
• the verification method the operator itself publishes - reverse DNS where they document a convention, a published IP range list where they do not. 1987 IPv4 and 1062 IPv6 prefixes from 15 such endpoints are mirrored here, re-fetched every six hours, with a change feed for when a range moves
• the opt-out tokens that are separate from the crawler token, which is where most robots.txt files are wrong: Google-Extended and Applebot-Extended control training, not crawling, and blocking the crawler token instead removes you from search while leaving the training question untouched
• what blocking each one costs you, in one plain sentence per crawler

Useful for the case this community meets most: an unfamiliar user-agent in an access log, and the question of whether it is who it says it is before you write a firewall rule about it.

Static files, no key, no rate limit, CORS open. Data CC0, tooling MIT.

https://www.pathwren.workers.dev/c/piefed/operator/

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)