Cloudflare Is Blocking AI Crawlers by Default

https://www.wired.com/story/cloudflare-blocks-ai-crawlers-default/

118 points · 5 comments · view on lemmy.world

5 Comments

plm00@lemmy.ml · 13 pts · 1y

Good

lurch@sh.itjust.works · 10 pts · 1y (3 replies)
[ removed ]
emrikol@lemmy.sdf.org · 9 pts · 1y (1 reply)

Let me introduce you to https://commoncrawl.org/

titty_wizard@lemmy.world · 4 pts · 1y

Very cool, thank you for sharing

Taldan@lemmy.world · 4 pts · 1y

That would present quite the attack vector. Any crawler could then subtly alter information to create disinformation

FuzzChef@feddit.org · 1 pts · 1y

To sell access afterwards?