Hey folks!
Unfortunately, roughly 2 hours ago, lemm.ee went offline. The cause was our load balancer: it suddenly decided that all of our servers had become unhealthy, despite all health checks responding successfully when I requested them directly. In such cases, the load balancer stops serving all requests, effectively meaning that lemm.ee is unreachable for all users. I am still not sure what exactly caused the issue, but I will try to investigate more over the weekend.
For now, we have partially recovered, and I am continuing to work on remaining issues. Hopefully we will be back to 100% very soon. Sorry for the inconvenience!
28 Comments
UltraGiGaGigantic@lemm.ee · 72 pts · 2y
We appreciate what you do hero
TLGA@lemm.ee · 24 pts · 2y
I was wondering what was going on, status.lemm.ee said the server was ok but the federation was broken. Thank you for fixing it
sunaurus@lemm.ee · 21 pts · 2y
Sorry for the delay in updating the status page - I actually had gone out for lunch just a few minutes before the downtime started, so I didn't even realize anything was up until I was back at my computer about 45 minutes later 💀
ToxicWaste@lemm.ee · 7 pts · 2y
no need to apologise. still a better response time, than some of the professionals I work with ;-)
don@lemm.ee · 21 pts · 2y
I survived the July 18th lemm.ee downtime, and all I got was this lousy comment.
Draegur@lemm.ee · 11 pts · 2y
All is forgiven, thank you for running this lovely instance ^_^
p3e7@lemm.ee · 10 pts · 2y
Thanks for your great work and transperancy!
fossphi@lemm.ee · 9 pts · 2y
Thanks for the quick fix! What did you have to do to get the load balancer working again?
sunaurus@lemm.ee · 16 pts · 2y
For now, I just redeployed all of our servers completely, but as I don't know the actual root cause of the issue yet, I'm still investigating to figure out if anything more is needed.
Amanduh@lemm.ee · 9 pts · 2y
I'd like to speak to a manager /s
ramble81@lemm.ee · 9 pts · 2y
Nginx? I had an nginx LB shit itself yesterday. Luckily it auto-recovered and I had HA but just weird it happened.
sunaurus@lemm.ee · 9 pts · 2y
Actually, we're using Hetzner's cloud load balancer for lemm.ee. But if this issue repeats in the near future, then I will definitely consider setting up something else.
db0@lemmy.dbzer0.com · 4 pts · 2y
haproxy is where it's at!
eleitl@lemm.ee · 2 pts · 2y
It's probably a managed haproxy in Hetzner's case.
db0@lemmy.dbzer0.com · 8 pts · 2y
Typically when this happens, the issue is on the LB itself. Maybe its own network had issues?
EABOD25@lemm.ee · 7 pts · 2y
Would it be in bad taste to blame Russia?
clot27@lemm.ee · 7 pts · 2y
Sometimes, downtimes are awesome. Get off your machine and spend time with your family, folks!
scytale@lemm.ee · 5 pts · 2y
I thought the entire lemmy network was down because status.lemm.ee was saying our instance was fine and federation wasn't working with every other instance. lol
becausechemistry@lemm.ee · 5 pts · 2y
o7
JimmyBigSausage@lemm.ee · 4 pts · 2y
LedgeDrop@lemm.ee · 4 pts · 2y
Seriously, your professionalism in handling the situation and in reporting it is fantastic.
It's totally above and beyond anything we should expect for a service powered by donations!
Thank you!
yournamehere@lemm.ee · 4 pts · 2y
love you guys!
SuperSpaceFan@lemm.ee · 2 pts · 2y
Thank you for keeping us abreast of what's happening. I appreciate you, and how you manage this instance.
tacosanonymous@lemm.ee · 2 pts · 2y
Is there another instance where you could report issues?
If we logged into another account, we’d be able to see those before it comes back up.
sunaurus@lemm.ee · 9 pts · 2y
There are two useful sections on https://status.lemm.ee for this - firstly, there is an automated check for federation with all other instances on the bottom of the page, and everything there being red is a definite sign that something is wrong with lemm.ee itself. Secondly, near the top of that page, I will always write a status message manually when I discover & start work on any issues. This second part can have a bit of a delay, as it requires manual input from myself, but I have updated it every time we had any issues so far.
tacosanonymous@lemm.ee · 2 pts · 2y
That’s good info. Thanks.
EinfachUnersetzlich@lemm.ee · 4 pts · 2y
There's a Discord server in the sidebar that updates are posted in: https://discord.gg/XM9nZwUn9K
tacosanonymous@lemm.ee · 10 pts · 2y
I know. I just don’t want to join a discord.
veniasilente@lemm.ee · 1 pts · 2y
Totally healthy servers have a right to rest every once in a while too. Thanks for keeping us notified!