League of Legends players worldwide couldn't login for hours because Riot forgot to renew the client's SSL certificate—just like it did 10 years ago
https://www.pcgamer.com/games/moba/league-of-legends-players-worldwide-couldnt-login-for-hours-because-riot-forgot-to-renew-the-clients-ssl-certificate-just-like-it-did-10-years-ago/
343 points · 38 comments · view on lemmy.world
38 Comments
Shirasho@lemmings.world · 95 pts · 240d
This is a surprisingly common issue. I've had it happen at least once in every job I've worked. This is usually the responsibility of the devops or devsec teams, and they are usually heavily underfunded since they are cost centers that do not bring in profit.
AmbiguousProps@lemmy.today · 39 pts · 240d
I work in DevOps, this is one of the easier things to automate. It's common for certs to be issued on a 90 day basis these days, no way that would be maintainable without automating.
Limerance@piefed.social · 19 pts · 240d
The problem sometimes is the automation failing for some reason.
dgdft@lemmy.world · 5 pts · 240d
Have you had Certbot or LE fail on prod for you before?
I’m sure stuff happens, but I usually view them as one of the most robust moving parts on a server.
E: I don’t mean to express disbelief at all; just curious to learn about possible footguns.
four@lemmy.zip · 14 pts · 240d
Certbot / LE has to be running on some machine and that machine can be accidentally turned off, payments not fulfilled, was supposed to be moved but the new instance doesn't work, gateway configuration changed, etc.
Automation requires maintenance and that introduces human error
AmbiguousProps@lemmy.today · 5 pts · 240d
Like dgdft said, if you're using certbot, it should typically be running on the machine that your endpoints are hosted on. Enterprise solutions don't require this, but they have other means of deploying certificates automatically and alarming if they are unable to, before they expire. My organization has dashboards showing which certs expire and when, and it triggers alarms at least a month before anything goes wrong.
High stakes automation should always have alarms on error, and since certs have set expiration dates baked into them, you can alarm far before anything goes wrong. Apparently, Riot didn't have that.
Also, more frequent renewals make it so that people are less likely to forget it exists. Because of that, along with the possible security ramifications, 2 to 10 year certs should never be used, in my opinion. A 10 year cert will always get kicked on to the next team and it's very possible for things to fall through the cracks.
dgdft@lemmy.world · 4 pts · 240d
Certbot/LE should typically be running on the box that's terminating TLS for you, right? If the box handling your traffic is down, shouldn't that be a self-evident problem?
I've been running Caddy and certbot for nearly a decade and never found a way for them to break without it being 100% my fault. They're more or less self-healing too. I'm with AmbiguousProps; cert renewals have been pretty damn reliable to automate compared to any other piece of tech, IME.
phx@lemmy.world · 2 pts · 239d
Yeah I've had certbot mess up a few times, though more often it was the scripts that actually shuttle the updated certs to their proper locations and restart services after updating
vodka@feddit.org · 5 pts · 239d
Where I work we went down to 60 days as of 01.01.2026, by end of year the plan is that no cert should be valid for more than 30 days.
I hate it please make it stop.
nik282000@lemmy.ca · 5 pts · 239d
By 2028 you'll be at per-transaction certs!
ramble81@lemmy.zip · 3 pts · 240d
Cool story if everything you have has an API or code based. Try doing it on hundreds of switches and other embedded devices. The whole 42 day thing they’re floating is gonna be a massive nightmare because they don’t realize all the other things out there that use certificates.
AmbiguousProps@lemmy.today · 7 pts · 240d
What makes you think I don't do this on embedded devices? I'm not about to dox my self with specifics, but I do this exclusively for embedded hardware as my job. We even do it for devices not directly attached to our network. It's really not difficult so long as you have control of your enterprise hardware (which, you should, unless your management is terrible at their jobs). Hell, even the routers we use have this functionality built in, failure alarms and all.
If this is a problem for you, it's probably at an organizational level, and not a technical issue.
Fizz@lemmy.nz · 1 pts · 236d
Even the simplest things fail sometimes
AmbiguousProps@lemmy.today · 1 pts · 236d
That's what alarming is for.
Evotech@lemmy.world · 20 pts · 240d
Yeah. Oh it’s 10 years until we gotta replace it. That’s someone else’s job
AudaciousArmadillo@piefed.blahaj.zone · 12 pts · 240d
I'm young enough that I never had to experience anything but let's encrypt/ACME. Manually renewing certs sounds like such a major PITA that I'd switch to it as soon as I could...
Theoriginalthon@lemmy.world · 7 pts · 240d
I had to do it a couple of times, major pain, mostly as you do it that infrequently that you forget everything that you have to do to enable and change them
ZeDoTelhado@lemmy.world · 3 pts · 240d
There are tools to actually remind you to do this on a timely fashion.... Also, some of them go as far as doing auto renewal. Is this such a hard thing?
Shirasho@lemmings.world · 11 pts · 240d
You'd be surprised. DevOps are, at least by my experience, SEVERELY overworked and understaffed. Even such things as writing a script is not always so easy, especially when security credentials are involved. Depending on the company there may be many layers of red tape for using security credentials stored in a secrets vault, so sometimes such things aren't even possible since there is no official work request for that automation.
ZeDoTelhado@lemmy.world · 1 pts · 240d
A lot what you mention is for sure true. My problem usually is deferring infra work to people that should not do it. More often than not I see so called "devops teams", and once I see the elements on those, is very clear people that are actually for infra are either insufficient, missing, or straight up not enough permissions to do stuff. Yo would be shocked how many times I hear managers say " ah well, I bet one ofb our developers knows how to do infra work"
Limerance@piefed.social · 6 pts · 240d
If the reminder goes to someone, who was fire two months ago or someone on holiday, this can easily fail.
ZeDoTelhado@lemmy.world · 1 pts · 240d
And if the reminders are not set on a mailing list and you have no one looking at that, someone failed hard
Limerance@piefed.social · 3 pts · 240d
Or it goes to a mailing list, but none of the readers thinks it’s their responsibility.
noxypaws@pawb.social · 3 pts · 240d
until one of your employer's multi-million dollar customers insists on a commercial certificate so it's a yearly effort to buy and distribute the damn thing
30p87@feddit.org · 21 pts · 240d
LOL
bhamlin@lemmy.world · 13 pts · 240d
It's about to get worse with validity periods going down from 1year now.
Holytimes@sh.itjust.works · 24 pts · 240d
Honestly that is more likely to make it better. The short span means it becomes a normal maintenance item and thus gets actual thought and effort.
bhamlin@lemmy.world · 7 pts · 240d
I'm not sure it's short enough for that, honestly.
JustTheWind@lemmy.world · 10 pts · 239d
I see this as a win. I hope some take this as an opportunity to never log in again in order to stop supporting this toxic ass company.
Psionicsickness@reddthat.com · -2 pts · 239d
Skill issue.
zewm@lemmy.world · 8 pts · 240d
Damn, did Riot hire ex Manjaro devs or something?
RedSnt@feddit.dk · 2 pts · 239d
"Just turn your clock back guys!"
FiniteBanjo@feddit.online · 7 pts · 240d
Mundo does not go where Mundo pleases, apparently.
termaxima@slrpnk.net · 6 pts · 239d
Is that the game studio partly responsible for Arcane ? I wish they made a game that was anything like the series ! /s
jaschen306@sh.itjust.works · 5 pts · 239d
I manage websites and I too have done this. Most web managers don't last 10 years and normally we inherit a bag of dicks for a website.
msokiovt@lemmy.today · 0 pts · 240d
F**kin' good! Riot deserves the mess they're in.
thingsiplay@beehaw.org · 1 pts · 240d
Saying that to an indie studio is mean.
msokiovt@lemmy.today · 6 pts · 240d
Problem with that is they aren't indie any longer. They're AAA, thanks to Tencent.