c/piefed_help · by 1.ceramics926@kopitalk.net · 25d

High CPU Usage on 1.7.8

OS: Debian VPS: 4 CPU, 8 GB RAM
PieFed v1.7.8

Observed higher than normal CPU usage, eventually rising to 100% and the site would no longer load for a number of hours. I panicked and tried rolling back changes to v1.7.7 but the high CPU load persisted. I gave up this AM and restored to v1.7.8.

So far CPU usage dropped to the point I could load up PieFed again to write this message. I'm hoping the load tapers off now and I'll just carry on like this.

12 points · 6 comments · view on lemmy.world

6 Comments

INeedMana@piefed.zip · 3 pts · 25d

We observed similar, load went down noticeably after enabling challenge to the connection

Dump the logs to a file, maybe we can dig something out from those

Shadow@lemmy.ca · 2 pts · 25d (4 replies)

It's probably a bot or abusive client scraping things, did you check your access logs?

1.ceramics926@kopitalk.net · 1 pts · 25d (3 replies)

Ah, no. On that note I'll adjust settings for bots.

Where does one look for the access logs?

rimu@piefed.social · 2 pts · 25d (2 replies)

In /admin/misc, tick the Private Instance checkbox.

1.ceramics926@kopitalk.net · 1 pts · 25d (1 reply)

I'm happily back with normal load levels, PieFed is once again quite snappy.

Do some of us just keep the instance private from here on out after this experience?

rimu@piefed.social · 3 pts · 25d

Ok good, so we know it's scrapers and not something broken.

What you do next depends how much work you want to put into fighting scrapers. Switching to Private mode is extremely easy and very effective which is why I suggested it first. But makes it impossible for people to get a feel for what your instance is like before joining it, which sucks.

Often a couple of hours of private mode is all it takes for the latest wave of scrapers to give up and leave, so you can switch back later and see what happens.

I am thinking about making it possible for PieFed to automatically switch to private mode when CPU load crosses some kind of threshold. Then automatically switch back again later. Or have a cron job that switches to private mode on a schedule (during quiet times) just to fuck with whatever bots are hanging around.

But if you want to put in more effort there's many options which will help you keep Private mode to a minimum.

Block HTTP 1.1 connections - real people use http 2.0. Surprisingly effective. In Cloudflare you can block http 1.1 AND where the user agent string contains "Chrome" which will block scrapers while letting federation traffic through - surely some fedi platforms federate using 1.1 but I haven't checked.
Block old browsers - scrapers use fake user agent strings with a random browser version but 95% of those versions are so old that no human ever uses them. You can do this in nginx but the regex tricky. A chatbot can help with this.
Watch web server logs and find patterns and manually block them in the nginx config. Repeating IP addresses, user agents strings, etc. I blocked Chrome 133.0.0.0 and Chrome 144.0.0.0 yesterday, for example.
Anubis. Very difficult to get right.