LW Backend Resources experiencing major outage

Hi,

sorry for the late response to this. The immediate issue is resolved, and we're looking to extend our monitoring in the coming days to have better insights on the underlying issue, as well as alerts before it would get to the point of throwing errors.

For some reason Lemmy has been having significantly more open file handles (including e.g. connections) than usual, leading to a denial of service. At this point it's unknown whether this is caused by a bug in Lemmy, an attack, or something else.

i see this slowing down my voting and slowing the sorting of comments inside a post
Edit : i had to try 5 times for this comment

Ah. That explains why Lemmy is being all wonky today.

Edit: It seems to have de-wonked itself. Everything seems to be working correctly now. It was wonky earlier. Now it's de-wonked.

Hey I found this when I was having issues & couldn't upvote or reply bc of the errors, lol, but ty for posting so I knew it wasn't just me!

Looks like someone kicked the pods or something and the JSON api monitors are green again on the datadog dashboard: https://p.us5.datadoghq.com/sb/e801a480-2ef0-11ee-8d4a-da7ad0900005-baac9d1278e4447372f8948de4e5da64?refresh_mode=sliding&theme=dark&from_ts=1786281642282&to_ts=1786296042282&live=true

Thanks for the update. Was getting frustrated that I can hardly do or see anything lol. Hope it gets resolved soon!

Edit: I would upvote but I can't even do that 🙃

I saw a report 24 hours ago that was a canary in the coal mine. Will link it once things are working again