I've had to turn on Cloudflare's full proxy with nearly 10's of thousands of requests coming out of CN IPs, in the last hour making ~7000 requests:
there are ~250,000 requests from china in the last ~20 hours
I've had to turn on Cloudflare's full proxy with nearly 10's of thousands of requests coming out of CN IPs, in the last hour making ~7000 requests:
there are ~250,000 requests from china in the last ~20 hours
30 Comments
Jozzo@lemmy.world · 5 pts · 24d
Cloudflare lets you block requests by location. I just have China and Russia permanently blocked
BlueEther@no.lastname.nz · 3 pts · 24d
I felt like blocking the whole 36.x.x.x block this morning
Dave@lemmy.nz · 2 pts · 24d
I get them evenly split across IPs in countries across Asia and South America. I'm guessing it's the thing I've read about on Lemmy where people install game apps on their TVs which have inbuilt proxies to proxy scraping traffic through residential internet connections ðŸ˜
Dave@lemmy.nz · 2 pts · 24d
We deployed Anubis, which helps, but we are still getting hammered with requests to the point that Lemmy has been unstable. I've been trialling moving away from Cloudflare, but I think the ASN blocks are pretty critical so I'll need to work out how to do that without Cloudflare.
Good luck, it's hard work fighting the bots.
BlueEther@no.lastname.nz · 2 pts · 23d
The storm seems to have abated over night, I guess they are logging blocked requests
I'm going to switch off cloudflare full proxy and revert back to good old DNS with these block in Caddy:
Dave@lemmy.nz · 2 pts · 23d
You're using Caddy? Have you considered Iocaine? It's like Anubis with more of a focus on data poisoning. I know their example uses Caddy which made me think of it: https://iocaine.madhouse-project.org/documentation/3/getting-started/
BlueEther@no.lastname.nz · 2 pts · 23d
The problem with iocaine, by the looks, is it's use of ai.robots.txt. The bots/scrapers from yesterday all looked to have the agent set to "node". I could of cause add that string to the robots file and send any node.js agent to purgatory
Dave@lemmy.nz · 2 pts · 23d
The robots.txt is for well behaved bots. I already deployed that list quite some time back but it didn't help, because the problem are the not well behaved bots.
Iocane should be showing hidden links that only bots can see, and leading then into a maze of gibberish. It has other features too, but the core function is to keep them busy on easy to serve pages that poison their training and are inexpensive for your site to serve (as opposed to the lemmy pages, that have expensive database calls on every page load).
BlueEther@no.lastname.nz · 2 pts · 23d
I used caddy on the deploy as it is what I was used to for all my homelab stuff
I might look at iocaine
Dave@lemmy.nz · 1 pts · 23d
I use Traefik for the same reason 😆. There's an Iocaine Traefik middleware but I haven't looked too far into it.
BlueEther@no.lastname.nz · 2 pts · 23d
it has started up again just before I went to turn CF off, I think I'll leave it in and install Iocaine
Dave@lemmy.nz · 1 pts · 23d
Installing Anubis was certainly a big help for us, Iocaine should do something similar for you, fingers crossed!
I have been toggling the Cloudflare proxying trying to handle bots without it. old.lemmy.nz has had over 100k hits in the last 12 hours, this is just the ones that got past anubis. Something tells me that's not legit.
BlueEther@no.lastname.nz · 2 pts · 23d
Yeah, I had about 600k + the ones I blackholed in Caddy in the 20h yesterday (I blackholed the whole 36.0.0.0/8 block in the end on the server)
3 requests (/api/v3/site /api/v3/post? /api/v3/comment/list?) in 50ms, from a IP then that IP would go silent for ages, then 500ms latter it was a fresh IP