2025-03-15 Unplanned lemm.ee downtime

Hey folks!

For the past few hours, lemm.ee has been bombarded with abnormal (almost definitely automated) traffic from a range of different IP addresses. This managed to overwhelm our servers, and we were offline for the past hour or so.

I was in the middle of celebrating my birthday, so response was a bit slow, but I believe we are recovering now, with mitigations in place to try and prevent further issues. Some of you may be inconvenienced by some bot checks when you browse lemm.ee, I am sorry about that, but it's necessary for now.

Sorry for the issues and I hope you have a nice weekend ahead!

251 points · 46 comments · view on lemmy.world

46 Comments

gregs_gumption@lemm.ee · 68 pts · 1y

Thanks for the work, happy birthday!

Fiction@lemm.ee · 34 pts · 1y

Thank you! Happy birthday!

kestrelx@lemm.ee · 28 pts · 1y

Happy birthday!

You_are_dust@lemm.ee · 19 pts · 1y
[ removed ]
aeharding@vger.social · 19 pts · 1y (4 replies)

❤️

henfredemars@infosec.pub · 14 pts · 1y (3 replies)

Your name is a different color in my client. Why is that?

aeharding@vger.social · 22 pts · 1y (1 reply)

bc I made your client

henfredemars@infosec.pub · 12 pts · 1y

No way…

Frantically looks up the app.

Wow. Ah, Nice job. Thanks. I spend a lot of time using it every day.

notanapple@lemm.ee · 11 pts · 1y

He is the CEO of lemmy.

makingrain@lemm.ee · 18 pts · 1y (3 replies)

Just enjoy your birthday! Sweet 21.

henfredemars@infosec.pub · 7 pts · 1y (2 replies)

Really, they’re 21 and managing such a large instance? Impressive. Is that true?

sunaurus@lemm.ee · 17 pts · 1y (1 reply)

Definitely not 😄

oddspinnaker@lemm.ee · 11 pts · 1y

Well you don’t look a day over 21…

dessalines@lemmy.ml · 14 pts · 1y

We've been dealing with these too at lemmy.ml, don't sweat it, we'll figure it out. Happy bday! Hope it was a good one.

Blaze@lemmy.dbzer0.com · 13 pts · 1y

Happy birthday!

muntedcrocodile@lemm.ee · 11 pts · 1y

Suspicious timing considering that we have had a large reddit influx.

jet@hackertalks.com · 11 pts · 1y

Happy birthday! 🎂

Technoworcester@lemm.ee · 11 pts · 1y

Happy birthday! Thanks for all your hard work.

Shadow@lemmy.ca · 11 pts · 1y

We saw a brief similar traffic spike approximately an hour ago, but it dropped off pretty quick. It seemed to focus on our mlmym front end and briefly overloaded our backends for a minute.

All the suspicious ips I've checked so far, are owned by chinese cloud providers (huawei, alibaba, tencent, etc)

RagingHungryPanda@lemm.ee · 10 pts · 1y

Happy birthday!

UraniumBlazer@lemm.ee · 10 pts · 1y

Happy birthday!!! <3

Vyvanse@lemm.ee · 9 pts · 1y

Thank you for taking time out of your birthday to get lemm.ee back online. I hope the rest of your day goes great. Happy Birthday!

balssh@lemm.ee · 9 pts · 1y

Happy birthday! Thanks for your service and hope you enjoy your day.

Lojcs@lemm.ee · 9 pts · 1y

Happy birthday!

TheTurner@lemm.ee · 9 pts · 1y
[ removed ]
A_Very_Big_Fan@lemmy.world · 9 pts · 1y (2 replies)

🎉birth

potoo22@programming.dev · 8 pts · 1y (1 reply)

...

Day

user224@lemmy.sdf.org · 7 pts · 1y

Happy: the prequel

BossDj@lemm.ee · 9 pts · 1y

Happy Birthday!

Xanza@lemm.ee · 8 pts · 1y

Happy birthday. Check out fail2ban.

Microw@lemm.ee · 7 pts · 1y

Have a happy rest of your birthday sunaurus!

Gullible@sh.itjust.works · 7 pts · 1y (1 reply)

I don’t go to this school, but this seems like a particularly strange instance to attack. Were they scrapers? Which is also an incredibly strange thing to do. Spinning up an instance and directly pulling 6 years of content would be far easier.

Endmaker@ani.social · 4 pts · 1y

I read somewhere that the perpetuators are likely getting some practice in before attacking their actual targets. Lemmy instances just happened to be unlucky enough to be among the victims chosen for this rehearsal.

abobla@lemm.ee · 7 pts · 1y

that's crazy, happy birthday!!!!

kratoz29@lemm.ee · 7 pts · 1y

Huh, I hope it wasn't me with my constant Summit (Lemmy client) refreshes (I was having issues with that app so I was doing a lot of API requests I guess) 🤣

Happy birthday! 🥳

sgbrain7@lemm.ee · 7 pts · 1y

your birthday is the ides of march? sick 🤟

gon@lemm.ee · 7 pts · 1y

Well, happy birthday! Sucks about this...

henfredemars@infosec.pub · 6 pts · 1y

Nice work! But remember, there are other instances, and as far as I know your users enjoy your services for free. Take it easy along the way.

fairpixel@lemm.ee · 6 pts · 1y

Happy birthday

revanite@lemm.ee · 6 pts · 1y

It seems this borked access for third party apps and the different front ends for Lemmy webpage. It is just showing network error

Dogsoftulkas@lemm.ee · 6 pts · 1y

Thank you for being born!

turbulentMagma@lemm.ee · 5 pts · 1y (1 reply)
[ removed ]
revanite@lemm.ee · 4 pts · 1y

Same here. My android app and the front ends I use for the webpage can't connect to lemmee still.

_____@lemm.ee · 4 pts · 1y

I'm curious about steps you took in reaction to this, of course if sharing compromises the mitigation then by all means, you don't have to

and/or also how do you learn about managing attacks like these ?

alecbowles@lemm.ee · 4 pts · 1y

Happy Birthday! I’m using Voyager and experienced some issues and downtime as well. Sneaky bots.

illi@lemm.ee · 3 pts · 1y

Happy birthday!

AizawaC47@lemm.ee · 3 pts · 1y

Happy Birthday!! I hope it was a good one!

scsi@lemm.ee · 2 pts · 1y
[ removed ]
dditty@lemm.ee · 1 pts · 1y

No problem, thanks as always for the hard work, and belated happy birthday!

coldsideofyourpillow@lemmy.cafe · 1 pts · 1y

I'm late (as always -_-), but happy birthday! 🎉