c/selfhosted · by tofu@lemmy.nocturnal.garden · 256d

Guarding My Git Forge Against AI Scrapers

https://vulpinecitrus.info/blog/guarding-git-forge-ai-scrapers/?ref=selfh.st

Cross posted from: https://lemmy.nocturnal.garden/post/407947

74 points · 12 comments · view on lemmy.world

12 Comments

Self-hosting anything that is deemed "content" openly on the web in 2025 is a battle of attrition between you and forces who are able to buy tens of thousands of proxies to ruin your service for data they can resell.

This is depressing. Profoundly depressing.

Sigh

magic_smoke@lemmy.blahaj.zone · 11 pts · 256d

I would like to know how well iocaines spanky new redirection module works compared to Anubis.

If nothing else, to see if throwing Anubis in front of iocaine is still a worthwhile idea.

xyro@lemmy.ca · 11 pts · 256d (1 reply)

Good read, I use crowdsec to block most of the traffic considered as malicious (which tend to overlap with scrappers), but I should look into Locaine to feed garbage to the remaining ones instead of throttling.

JustTesting@lemmy.hogru.ch · 6 pts · 255d

it's iocaine not Locaine, tripped me up at first as well.

Selfhoster1728@infosec.pub · 3 pts · 255d

Build your own captcha, there's just no other way to be sure it's human traffic with prebuilt solutions :(

Lemmchen@feddit.org · 2 pts · 245d

https://her.esy.fun/posts/0031-how-i-protect-my-forgejo-instance-from-ai-web-crawlers/index.html

Also: https://iocaine.madhouse-project.org/

False@lemmy.world · 1 pts · 256d

[ removed ]

possiblylinux127@lemmy.zip · 0 pts · 256d (5 replies)

Honesty we need a POW system built into http.

DaGeek247@fedia.io · 5 pts · 256d

POW built in to the web spec would be hell. Making every single device in the world do that extra bit of work would noticeably affect energy use across the planet.

tofu@lemmy.nocturnal.garden · 3 pts · 256d (3 replies)

A what? Not prisoners of war I guess

billhead@sh.itjust.works · 4 pts · 256d (2 replies)

I'm guessing Proof of Work.

tofu@lemmy.nocturnal.garden · 2 pts · 256d (1 reply)

Oh right. The article author isn't a fan of that. I guess it's fine while it works but I'm not too optimistic about how long it does

DaGeek247@fedia.io · 2 pts · 256d

Yeah. The corporations with money are always going to beat the casual users without in regards to processing capability.

There are smarter ways to discourage the big companies from taking pictures of your house than by adding speed bumps to your driveway.