[outage] Comments not showing

Last month, I made some changes that were meant to stop bots from harassing our instance. This also created some problems for large instances that push hundreds of thousands of deliveries a day to our /inbox, and made it hard for comments to come through.

(post about the update btw https://lemmy.radio/post/14120844)

Here's a rundown of what happened and how it's fixed:

Why remote comments went missing (Aug 15–22)

Short version: a misconfigured rate limit on our server was rejecting incoming comments from other instances. It's fixed. The backlog is refilling now.

What happened: back in July I added some anti-scraper protection to lemmy.radio. That change included a rate limit meant to stop bots from hammering the site, but it was accidentally applied to all incoming traffic, including the ActivityPub deliveries other Lemmy instances use to send us their posts and comments.

The limit was 1 request per second per server. For six weeks nothing broke, because no instance was sending us that much. Around Aug 15, lemmy.world crossed that line and once a server falls behind a limit like that, it can't recover on its own: the queue drains slower than new comments arrive, and every rejected delivery gets retried, which eats the same budget. By the time I caught it, we were rejecting about 60% of what lemmy.world tried to send us.

Only the three busiest instances (lemmy.world, hexbear.net, lemmy.ml) were affected. The other ~115 servers we federate with were never throttled. Since lemmy.world is where most Lemmy users are, it looked like "comments stopped working" rather than "one server stopped getting through."

One more thing that was fixed: the same limit also applied to the site's own stylesheets, so some of you may have seen slow or half-broken page loads, especially on mobile or shared connections. Same root cause, also fixed.

Current status: fixed as of Aug 22. Remote instances are re-sending their backlog now, oldest first, at roughly 5x normal speed. Expect comments from the outage window to keep appearing over the next day or two rather than all at once.

The bad news: some comments from Aug 15–22 are gone for good. Remote servers don't retry forever, so anything that aged out of their queues won't arrive. The gap will narrow but probably won't close completely. If you posted a comment during that window that never showed up that's why. Please recomment.

---

I only found out about this because members like you messaged me about it! So let me talk about best ways to get information to me.

https://matrix.to/#/#lemmy-radio:matrix.org

@lemmyradio@mstdn.social

---

This is going to be an ongoing watch as we backfill. Please comment here if you see any other issues!

~w0odl