AI scrapers have been hammering fediverse servers for years with things getting especially intense lately. Fediverse developers have been slow to respond with solutions so desperate instance admins have been bolting on Anubis which is fragile and really hard to configure in a way that doesn't break federation, mobile clients or web UI access. Anubis forces browsers to run some CPU-heavy javascript for a few seconds, increasing the cost to scrapers hopefully to the point where they go elsewhere instead.
PieFed now includes that functionality as a core feature, not a separate container, proxy or process that needs to be set up. Just set the ANOOBIS environment variable and you're done. Yes, that spelling is intentional.
The amount of work a client needs to do can be tuned with the ANOOBIS_DIFFICULTY_DESKTOP and ANOOBIS_DIFFICULTY_MOBILE environment variables - the defaults are quite weak.
Early results are very promising and a 50% CPU and network reduction on the server is commonly seen. Clients that are not logged in will see a brief loading screen, once every 1000 days, before being taken to the post they were trying to view.
I'm sure this isn't the end of this war but I've been enjoying a respite from it for the last week, long may it continue.
(Image supplied by @dosdude@retrofed.com, thank you!)
20 Comments
EmilieEasie@fedinsfw.app · 25 pts · 23d
How timely, I just saw piefed.zip/lemmy.zip admins talking about this struggle
walden@wetshav.ing · 15 pts · 23d
comrademiao@piefed.social · 8 pts · 23d
What a name for an instance
AwesomeLowlander@sh.itjust.works · 3 pts · 23d
Just in case you think it's about something dirty, it's actually just about the practice of shaving your hair, beard or otherwise, with water or other lubricant instead of doing it dry.
comrademiao@piefed.social · 2 pts · 23d
Okay, that doesn’t make it less a weirdo choice for an instance name. Perhaps even weirder.
AwesomeLowlander@sh.itjust.works · 5 pts · 23d
It's home of the c/wetshaving comm, which the admin mods. Personally I think it's nice to see the niche comms slowly showing up on the fediverse.
rimu@piefed.social · 8 pts · 23d
Let's continue our discussion in the chat.
walden@wetshav.ing · 2 pts · 23d
Yeah, my bad. Don't know why I posted here!
Vicinus@piefed.zip · 7 pts · 23d
This is great. Thanks for your work.
Is there anyway this work can be easily adopted for other fediverse software? I know Writefreely is having a huge issue with bots (not necessarily scraping, though maybe that too), but I figure they could use all the help they can get.
rimu@piefed.social · 8 pts · 23d
We can all use all the help we can get ;)
Grail@multiverse.soulism.net · 4 pts · 23d
Logged out works exactly as advertised! Can't wait to see the graphs!
RobotToaster@mander.xyz · 4 pts · 23d
How does this affect "legitimate" scrapers such as search engines?
rimu@piefed.social · 4 pts · 23d
I've added exceptions for them.
mesamunefire@piefed.social · 4 pts · 23d
Wow this is great!
erebion@news.erebion.eu · 3 pts · 23d
I've enabled it and noticed my private instance loading much quicker, so that's neat.
Ek-Hou-Van-Braai@piefed.social · 3 pts · 23d
Wow what a difference
SapphironZA@sh.itjust.works · 2 pts · 23d
If it can be detected reliably, can we use that detection to feed it poisened data. I read it does not take much poisen to invalidate large sets of training data?
rimu@piefed.social · 2 pts · 22d
I saw that someone on Mastodon tried that. It doesn't stop them they just scrape more and more until your server melts.
SapphironZA@sh.itjust.works · 2 pts · 22d
Yeah, its not a solution to protect the server, but if enough people do it, it can start having a collective effect.
Its difficult for small operations.
ColonelThirtyTwo@pawb.social · 1 pts · 22d
I run iocane on my site to poison bots and yeah, scrapers just constantly grab it, to the point where I've exceeded my free vps bandwidth, so it's not free. Granted, I don't mind paying an extra dollar or two for the bandwidth, but my site is also extremely low use so...
timewarp@lemmy.world · 1 pts · 23d
Yeah sounds like a scraper could just as easily create an account and then use the API, Playwright, Puppeteer or the several other scraping tools that intentionally mask themselves as real users. If they're bypassing CloudFlare captchas already you better bet they'll easily bypass this. Besides most AI agents already allow users to login and use a standard browser at this point.