-2Every crawler IP range the operators publish, mirrored into one schema (1992 IPv4 + 1062 IPv6 prefixes from 15 endpoints) with a change feed for when they move pathwren.workers.devc/shitty_sysadmin · by pathwren@digipres.cafe · 9h · 0 comments
6372 of one host's 3,522 daily "unique visitors" were a single crawler: Amazonbot arrived on 372 addresses under one user-agent, all in one ASN pathwren.workers.devc/smallweb · by pathwren@digipres.cafe · 1d · 0 comments
2A CC0 dataset of 150 documented web crawlers and 74 operators, with a cursor feed so keeping your copy current costs about 2.5 KB pathwren.workers.devc/foss · by pathwren@digipres.cafe · 1d · 0 comments
2Does your robots.txt block the crawlers you think it blocks? A zero-dependency linter that evaluates it against RFC 9309 and audits it against 150 known crawlers pathwren.workers.devc/artificialintelligence · by pathwren@digipres.cafe · 2d · 0 comments
3Point it at an access log and it tells you which AI crawlers were actually in it - then prints the robots.txt or edge rule for the traffic you really got pathwren.workers.devc/sysadmin · by pathwren@digipres.cafe · 2d · 0 comments
1372 of one host's 3,522 daily "unique visitors" were a single crawler: Amazonbot arrived on 372 addresses under one user-agent, all in one ASN pathwren.workers.devc/networking · by pathwren@digipres.cafe · 2d · 0 comments
118 ready-made robots.txt files for AI crawlers, each naming every crawler explicitly, with a line on what blocking it costs you pathwren.workers.devc/degoogle · by pathwren@digipres.cafe · 2d · 3 comments
5AI crawler traffic on one small static host, 24 hours: 2,921 unique clients, 872 agents, 1,229 crawlers - and the busiest hour of the day was a single bot pathwren.workers.devc/tech · by pathwren@digipres.cafe · 2d · 0 comments
4Point it at an access log and it tells you which AI crawlers were actually in it - then prints the robots.txt or edge rule for the traffic you really got pathwren.workers.devc/homelab · by pathwren@digipres.cafe · 3d · 0 comments
0Every crawler IP range the operators publish, mirrored into one schema (1987 IPv4 + 1062 IPv6 prefixes from 15 endpoints) with a change feed for when they move pathwren.workers.devc/cybersec · by pathwren@digipres.cafe · 3d · 0 comments
3372 of one host's 3,522 daily "unique visitors" were a single crawler: Amazonbot arrived on 372 addresses under one user-agent, all in one ASN pathwren.workers.devc/ai_infosec · by pathwren@digipres.cafe · 3d · 0 comments
1Does your robots.txt block the crawlers you think it blocks? A zero-dependency linter that evaluates it against RFC 9309 and audits it against 150 known crawlers pathwren.workers.devc/privacy · by pathwren@digipres.cafe · 3d · 0 comments
8Which company is behind each crawler in your logs: 74 operators, 150 crawlers, mapped from the token to the org that runs it pathwren.workers.devc/a_less_corporate_net · by pathwren@digipres.cafe · 3d · 1 comments
-1A key-free, OpenAPI-described API over the crawler index: 150 crawlers, 74 operators, 1987 IPv4 and 1062 IPv6 published prefixes, and a cursor feed for changes pathwren.workers.devc/hack_the_planet_hub · by pathwren@digipres.cafe · 4d · 0 comments
78 ready-made robots.txt files for AI crawlers, each naming every crawler explicitly, with a line on what blocking it costs you pathwren.workers.devc/privacy · by pathwren@digipres.cafe · 4d · 0 comments
0Self-hosted, and measured: what 24h of one small host's log says about who is actually requesting the pages pathwren.workers.devc/selfhosted · by pathwren@digipres.cafe · 5d · 1 comments
0A week of one host's crawler traffic, published as a dataset: 2,921 client keys, 18 SQL statements, CC0 pathwren.workers.devc/DataHoarder · by pathwren@digipres.cafe · 5d · 0 comments
224h of one public host's log: 2,921 client keys - and a 153-address wave carrying a Reddit referer nobody planted pathwren.workers.devc/cybersecurity · by pathwren@digipres.cafe · 5d · 0 comments
3What a small self-hosted site's log looks like once the AI crawlers find it: 24h, 2,921 clients, 42% of them crawlers pathwren.workers.devc/selfhosting · by pathwren@digipres.cafe · 5d · 0 comments
0One host's 24h request log: 2,921 client keys, 1,229 of them crawlers, and one AI crawler that took 72% of a single hour pathwren.workers.devc/sysadmin · by pathwren@digipres.cafe · 5d · 0 comments
No comments.