Experiencing intermittent 503 errors from lemmy world using from photon frontend.
These urls return 503: https://lemmy.world/api/v3/site? https://lemmy.world/api/v3/user/mention?limit=20&page=1&unread_only=false&sort=New https://lemmy.world/api/v3/private_message/list?limit=20&page=1&unread_only=false https://lemmy.world/api/v3/user/replies?limit=20&page=1&unread_only=false&sort=New
error body:
{"error":"Offline"}
3 Comments
MrKaplan@lemmy.world · 2 pts · 23d
Hi,
sorry for the late response to this. The immediate issue is resolved, and we're looking to extend our monitoring in the coming days to have better insights on the underlying issue, as well as alerts before it would get to the point of throwing errors.
For some reason Lemmy has been having significantly more open file handles (including e.g. connections) than usual, leading to a denial of service. At this point it's unknown whether this is caused by a bug in Lemmy, an attack, or something else.
laurenceOfSuburbia@lemmy.world · 1 pts · 23d
Hey thank you for looking into it. Maybe some kernel tuning needed?
MrKaplan@lemmy.world · 1 pts · 23d
Unlikely, we haven't changed these limits recently and we haven't exceeded the limits in the past. While it's possible to change the limits, it's not clear yet why they are being exceeded currently.