the immediate issue is resolved, and we're looking to extend our monitoring in the coming days to have better insights on the underlying issue, as well as alerts before it would get to the point of throwing errors.
For some reason Lemmy has been having significantly more open file handles (including e.g. connections) than usual, leading to a denial of service. At this point it's unknown whether this is caused by a bug in Lemmy, an attack, or something else.
Unlikely, we haven't changed these limits recently and we haven't exceeded the limits in the past. While it's possible to change the limits, it's not clear yet why they are being exceeded currently.
sorry for the late response to this. The immediate issue is resolved, and we're looking to extend our monitoring in the coming days to have better insights on the underlying issue, as well as alerts before it would get to the point of throwing errors.
For some reason Lemmy has been having significantly more open file handles (including e.g. connections) than usual, leading to a denial of service. At this point it's unknown whether this is caused by a bug in Lemmy, an attack, or something else.
sorry for the late response to this. The immediate issue is resolved, and we're looking to extend our monitoring in the coming days to have better insights on the underlying issue, as well as alerts before it would get to the point of throwing errors.
For some reason Lemmy has been having significantly more open file handles (including e.g. connections) than usual, leading to a denial of service. At this point it's unknown whether this is caused by a bug in Lemmy, an attack, or something else.
this is not only used for lemmy.world, but also mastodon.world
this is not correct - they have the same specs i believe, but it's 2 separate servers.
by now, it probably is also used for other services the Fedihosting foundation hosts
that is also not correct. pretty much all our services are isolated on VM or physical box level currently, although we are looking into a potential migration to kubernetes to use our hardware resource more efficiently, as we have a lot of unused resources with our current setup due to downscaling being a bit challenging with hardware and with the current market in general. lemmy.world and mastodon.world run on baremetal, most other services on hetzner cloud VMs.
are you seeing this on pages that work when you reload them or is it persistent on certain URLs?
Do you see an error message being returned?
The default Lemmy interface has a part like this below the matrix handles for admins:
The server returned this error: Error. This may be useful for admins and developers to diagnose and fix the error
Unfortunately, errors returned by Lemmy aren't often that useful in what you see as a user, but it would still help to know which URL you were on and roughly which time, including timezone.
Are you still having issues or just describing what you've experienced before?
We know why the error you're seeing is happening, as that's a caused by reaching a limit, we just don't know yet why the limit is being filled up by other stuff yet.
if the accounts you're referring to (i don't know which ones) are still up then we're likely not aware of it. we are aware of spam accounts and ban them from time to time as we become aware of them, but they can easily slip through the cracks as well.
we currently don't have that many automod actions that it would be problematic. often we don't even have a single one in a day. the notifications also include additional information about the user and reason to help with the review.
would you prefer an All feed on your instance that is filled with advertisements? of course we care about content that is posted to other instances in certain categories, like spam or illegal material.
doesn't seem to be the case, at least there is not enough information to link the two cases.
Hi,
the immediate issue is resolved, and we're looking to extend our monitoring in the coming days to have better insights on the underlying issue, as well as alerts before it would get to the point of throwing errors.
For some reason Lemmy has been having significantly more open file handles (including e.g. connections) than usual, leading to a denial of service. At this point it's unknown whether this is caused by a bug in Lemmy, an attack, or something else.
Unlikely, we haven't changed these limits recently and we haven't exceeded the limits in the past. While it's possible to change the limits, it's not clear yet why they are being exceeded currently.
Hi,
sorry for the late response to this. The immediate issue is resolved, and we're looking to extend our monitoring in the coming days to have better insights on the underlying issue, as well as alerts before it would get to the point of throwing errors.
For some reason Lemmy has been having significantly more open file handles (including e.g. connections) than usual, leading to a denial of service. At this point it's unknown whether this is caused by a bug in Lemmy, an attack, or something else.
Hi,
sorry for the late response to this. The immediate issue is resolved, and we're looking to extend our monitoring in the coming days to have better insights on the underlying issue, as well as alerts before it would get to the point of throwing errors.
For some reason Lemmy has been having significantly more open file handles (including e.g. connections) than usual, leading to a denial of service. At this point it's unknown whether this is caused by a bug in Lemmy, an attack, or something else.
it's still the same one.
this is not correct - they have the same specs i believe, but it's 2 separate servers.
that is also not correct. pretty much all our services are isolated on VM or physical box level currently, although we are looking into a potential migration to kubernetes to use our hardware resource more efficiently, as we have a lot of unused resources with our current setup due to downscaling being a bit challenging with hardware and with the current market in general. lemmy.world and mastodon.world run on baremetal, most other services on hetzner cloud VMs.
you mean creating or updating a post with embedded images is causing issues in some cases?
Hi,
are you seeing this on pages that work when you reload them or is it persistent on certain URLs?
Do you see an error message being returned?
The default Lemmy interface has a part like this below the matrix handles for admins:
Unfortunately, errors returned by Lemmy aren't often that useful in what you see as a user, but it would still help to know which URL you were on and roughly which time, including timezone.
Are you still having issues or just describing what you've experienced before?
We know why the error you're seeing is happening, as that's a caused by reaching a limit, we just don't know yet why the limit is being filled up by other stuff yet.
Hello,
we have just fixed an issue with our media service, can you please let us know whether you're still seeing this?
Hello,
this should be resolved for now. Unfortunately the root cause is still not clear and will require some additional investigation.
Hello,
this should be resolved for now. Unfortunately the root cause is still not clear and will require some additional investigation.
Hello,
this should be resolved for now. Unfortunately the root cause is still not clear and will require some additional investigation.
Hello,
this should be resolved for now. Unfortunately the root cause is still not clear and will require some additional investigation.
if the accounts you're referring to (i don't know which ones) are still up then we're likely not aware of it. we are aware of spam accounts and ban them from time to time as we become aware of them, but they can easily slip through the cracks as well.
lemmy doesn't support reporting communities yet.
post reports will go to community mods as well, and the mod can resolve the reports.
it's recommended to message @lwreport@lemmy.world with details for anything that requires admin attention to ensure it gets through to the team.
thanks for the report, i'll look into it
we will probably redirect the tesseract subdomain of lemmy.world to photon soon
we currently don't have that many automod actions that it would be problematic. often we don't even have a single one in a day. the notifications also include additional information about the user and reason to help with the review.
would you prefer an All feed on your instance that is filled with advertisements? of course we care about content that is posted to other instances in certain categories, like spam or illegal material.