In the past couple of weeks, the server has begun struggling to keep up with the load and the database has been growing quickly.
The server gets about 480K requests per day. 160K per day, or exactly 1/3 of all this traffic, is coming in from lemmy.world. This means that at least one person is following some of the larger lemmy.world communities. Every time someone upvotes, comments, posts, edits, in one of those communities, a message gets sent to this server.
Sadly, Friendica cannot handle this. Friendica groups were designed mostly for private groups. It was created well before Lemmy. And it has a lot of performance issues. As Rudd, who owns both Lemmy.World and Friendica.World has said, Friendica seems unable to handle more than 100 active users. We have more than 100 active users.
Currently, the server is an 8 CPU 16GB machine with regular SSD drives. It runs both the application and the database. The cost is $112/month, excluding the cost of database disk and image storage, which adds quite a bit to the cost, by the way, for the 350GB database. But, is not a factor in scaling up.
Looking at the server, it appears that the Friendica application is doing well. The CPU is almost always waiting for the database. It's the database that struggles, apparently not because of the CPU, but because the disks are not fast enough. The server/databse/design just cannot handle this volume.
To scale the server up to use faster disk brings the price to $524/month, almost 5x more cost. I can't afford this.
The other option is to split the server. Leave Friendica on a regular 4 CPU server for $96/month, and move the database to a 4 CPU server with fast disks for $262/month. This is a lot of work and still brings server costs way up over $112/month to $350/month or 3x the current cost. This is still too high.
I'm kind of stuck. I've closed registrations to keep growth down, but the impact is minimal.
I think, for now, my only option is to ask people who follow lemmy.world groups to do so instead from Piefed, Lemmy, or MBIN. You can join my PieFed server (https://feddit.online/) which already federates with a ton of their communities and does not have performance issues. And the application is optimized for public groups, so works better than Friendica for this.
Following smaller specialized groups on Friendica is fine.
To try to stabilize the server, I've decided to block lemmy.world traffic, effective immediately, so I can analyze the impact. I am really sorry that I have to do this. I hate that I have to do this. But, I don't see any other option.
I welcome your feedback on this.
Jerry
0 Comments
No comments yet.