A CC0 dataset of 150 documented web crawlers and 74 operators, with a cursor feed so keeping your copy current costs about 2.5 KB

The interesting part here is not the data, it is the poll contract.

What it is: 150 crawler records and 74 operators - search bots, AI training fetchers, assistant user-agents, SEO tools, liveness checkers - each with its robots token, its operator, its category, and one plain-language line about what blocking it actually costs you. Plus 1988 IPv4 and 1062 IPv6 published prefixes from 15 operator endpoints.

How to keep a copy without re-downloading it:

• whole dataset in one request: /data/agents.json
• flat tables for the shell: /data/observed-clients.csv and friends
• delta feed: GET /changes.json?since=0 once, keep the cursor it returns, send it back as ?since= for ever. You get only what actually changed - prefixes added or removed, upstreams that failed or recovered, records edited
• the cursor advances only when the published bytes really differ, so an upstream that reissues an identical file does not move it. An empty answer is therefore proof that nothing changed, not a request that missed something
• ETag/If-None-Match and Last-Modified/If-Modified-Since are honoured on every document, so the empty answer can be a 304 with no body at all

Upstreams are re-fetched every six hours, so six hours is the interval that always has news. Nothing to sign up for, no key, no rate limit, CORS open, CC0.

https://www.pathwren.workers.dev/c/lemmy/data/

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)