c/selfhosted · by sj_zero@lotide.fbxl.net · 277d

Cloudflare is down this morning

I always love it when some massive piece of web infrastructure goes down but most websites I use are self-hosted so the only real effect is I see lots of news stories that cloudflare is down.

303 points · 72 comments · view on lemmy.world

72 Comments

iii@mander.xyz · 64 pts · 277d (17 replies)

Is there any meta analysis on these major outages?

They seem to be occuring more and more regularly.

southernbeaver@lemmy.world · 54 pts · 277d (2 replies)

It would be funny if it was because of some AI coding

Subscript5676@lemmy.ca · 4 pts · 276d

Please don’t remind me. Had a colleague with a senior title who just vibecoded on our CI pipeline and it ended up blocking deployments for half a day.

pinball_wizard@lemmy.zip · 1 pts · 276d

It would be funny if it was because of some AI coding

It is. Sometimes the big secret turns out to just be what we all thought was happening.

30p87@feddit.org · 30 pts · 277d (2 replies)

Maybe Kevin Fang will make a video about all combined?

My prejudice tells me that it's vibe slopped code.

UsedLinuxDealer@lemmy.world · 8 pts · 277d (1 reply)

I saw people speculating about this, but it'll be interesting to see if that's really what it was (and if so, if they'd admit it).

pinball_wizard@lemmy.zip · 2 pts · 276d

I'm sure it was, and I'm sure they will not admit it.

So I'll throw this into the "conspiracy theory, but probably true" pile; and I'll try to remember to check snopes.com in 20 years to see if I was somehow wrong.

"Did Vibe coding really routinely break half the Internet is the 2020s?" ...assuming the Internet and I are still both around by then...

Edit: I'm already wrong. It sounds like it was computers being stupid is old tried and true ways; not computers being stupid in new trendy ways.

frongt@lemmy.zip · 7 pts · 277d (5 replies)

Preliminary articles say this was a DDoS.

Anafabula@discuss.tchncs.de · 24 pts · 277d (1 reply)

A DDoS that brings down the biggest(?) DDoS protection provider?

eager_eagle@lemmy.world · 5 pts · 277d

def possible, cloudflare DDoS their own dashboard a few months ago with some react code

https://blog.cloudflare.com/deep-dive-into-cloudflares-sept-12-dashboard-and-api-outage/

carrylex@lemmy.world · 2 pts · 277d (2 replies)

Based on what facts?

frongt@lemmy.zip · 7 pts · 277d (1 reply)

A statement from cloudflare blamed "a spike in unusual traffic", but I guess the actual cause of the outage was a config file generated in response exceeded the max size

https://www.cnbc.com/2025/11/18/cloudflare-down-outage-traffic-spike-x-chatgpt.html

Technically a DoS, but not in the most common sense

pinball_wizard@lemmy.zip · 1 pts · 276d

but I guess the actual cause of the outage was a config file generated in response exceeded the max size

So it was automation, but that automation might have been stupid the normal computer way, rather than stupid in the new LLM enhanced computer way.

irmadlad@lemmy.world · 4 pts · 277d (4 replies)

As far as major outages, I can only think of around 10 going back 5 years. Sure there have been minor hiccups and glitches, but gosh, that's life on the internet.

BremboTheFourth@piefed.ca · 4 pts · 277d (2 replies)

Yeah I have issues with my home setup at least that often. I can only think of two occasions where the solution took more effort than a reboot. Maybe we're all just old enough for "every 6 months" to feel like every 5 mins

To clarify, by "issues" I mean "system stoppages not precipitated by me fucking with something." I screw up my own system way more often than that lol

irmadlad@lemmy.world · 3 pts · 277d

I don't know how old you are brother, but it feels like I'm in one of those Star Trek warps where all the stars are just whizzing by.

pinball_wizard@lemmy.zip · 1 pts · 276d

I screw up my own system way more often than that lol

Same! But the ones I cause don't count.

pinball_wizard@lemmy.zip · 2 pts · 276d

It does feel like the major outages are growing closer together.

I want to blame plugging AI where it doesn't belong, but I suppose that putting everything into the same couple of server closets could be the primary root cause.

mesamunefire@piefed.social · 45 pts · 277d (7 replies)

So far this year its been:

AWS was massively down
Azure was down a week later
Cloudflare down multiple times this year

Thats the ones that affect stuff at work. Fun times.

lepinkainen@lemmy.world · 8 pts · 277d (2 replies)

Azure was down? Did anyone notice? 😆

viking@infosec.pub · 2 pts · 276d

If you're stuck with Windows at work, you might have. I couldn't log on to our domain remotely and was unable to open some documents and emails that were signed as "internal use only".

mesamunefire@piefed.social · 1 pts · 277d

Starbucks payments sure did.

Mostly at work. All our stuff was on Azure, so when it went down....I kinda just stood around for the day. It was up/down the entire day. So I went for a coffee lol.

voodooattack@lemmy.world · 4 pts · 277d

It’s a vibe.

pinball_wizard@lemmy.zip · 3 pts · 276d (1 reply)

Dang. I had to look it up, but CrowdStike taking down a huge percentage of Windows PCs was so last year, I guess.

possiblylinux127@lemmy.zip · 1 pts · 276d

"Whoops"

some person at Cloudstrike, probably

possiblylinux127@lemmy.zip · 3 pts · 276d

My services have a higher uptime than AWS, Azure and Cloudflare despite the fact that they are running on 10 year old hardware

nuko147@lemmy.world · 42 pts · 277d (12 replies)

I Guess many lemmy instances are not as much decentralized as they thought.

mesamunefire@piefed.social · 18 pts · 277d (1 reply)

Cache-ing is a hard problem to solve. For MANY years cloudflare was the cheapest and easiest way to solve the issue.

But the same people that started cloudflare have moved on or passed away. Now its just another corp. So yeah we are going to get downtime. In the corporate world, its the standard so no one is going to move off it.

...unless it keeps going down consistently.

For those who are in the know, does memcache work with lemmy/piefed? What is a good Cache for modern day systems?

possiblylinux127@lemmy.zip · 2 pts · 276d

Honesty we need a open standard for caching and csam detection.

mhzawadi@lemmy.horwood.cloud · 13 pts · 277d (1 reply)

I dont use cloudflare at all, so I didnt notice. I do see a lot of recommendation's to host behind Cloudflare, so if they have a problem you do too

irmadlad@lemmy.world · 6 pts · 277d

I do see a lot of recommendation’s to host behind Cloudflare, so if they have a problem you do too

Fallback to Tailscale, or any number of ways to skin the cat. At the very least, everything is accessible locally. I'm not running anything mission critical, so it was more of a minor inconvenience to me.

Eyekaytee@aussie.zone · 8 pts · 277d (2 replies)

aussie.zone sh.itjust.works lemmy.world

all were down for me

TrickDacy@lemmy.world · 4 pts · 277d (1 reply)

Piefed.social was too, on my end

Uri@infosec.pub · 1 pts · 277d

That was the biggest reason I didn't choose it

A_norny_mousse@feddit.org · 3 pts · 277d (4 replies)

I didn't notice and I've been on Lemmyverse the whole day, on and off. Different time zone though (Eastern Europe) so I'm not sure how that translates to "this morning".

nuko147@lemmy.world · 4 pts · 277d (3 replies)

Not all instances were burned. Lemmy.world that i use was though. I will make a user for backup in an instance that is not affected by cloudflare.

A_norny_mousse@feddit.org · 1 pts · 277d (2 replies)

Lemmy.world that i use was though.

So what does that entail - the site was completely down, or images didn't load?

Would I even notice that in my unaffected instances feed? Just no lemmy.world posts at all, or would it leave empty placeholders?

nuko147@lemmy.world · 2 pts · 277d (1 reply)

Well my user is on lemmy.world, so even if i loaded other instances i would not be logged in. The site was completely down. Old posts from lemmy.world showed on the fediverse, just no new posts while down (users could not connect or view the site).

A_norny_mousse@feddit.org · 1 pts · 277d

Thanks for that!

aarRJaay@lemmy.world · 31 pts · 277d (2 replies)

"Further details will be made available when we've worked out how the AI screwed us this time"

adb@jlai.lu · 7 pts · 277d (1 reply)

This really should read: further details will be made available when our AI api is back up and can self-report on how it fucked up

TeddE@lemmy.world · 3 pts · 277d

AI: taking another hit of acid in preparation to research the reason why the last thing it did after taking acid didn't work out.

comrade_twisty@feddit.org · 20 pts · 277d (1 reply)

I am still using cloudflare tunnels for home assistant, so I am a little annoyed. On the other hand this was a much needed reminder that I need to move home assistant to pangolin/newt as well.

Damage@feddit.it · 6 pts · 277d

NAT may be insecure but it keeps chugging on

swizzlestick@lemmy.zip · 20 pts · 277d

Eggs, baskets, and lessons being taught about how it's not a good idea to heavily skew their ratio.

DonutsRMeh@lemmy.world · 17 pts · 277d

My own DNS is still up and strong. Take that, Cloudflare.

db0@lemmy.dbzer0.com · 12 pts · 277d (1 reply)

took much of lemmy.world down. I wonder how come I'm seeing some posts from lemmy.worlders though

lazynooblet@lazysoci.al · 2 pts · 277d

Cloudflare have a large international network. Perhaps some users could access sites when coming from another region

Confining@lemmy.dbzer0.com · 8 pts · 277d (1 reply)

I don’t use cloudflare but the only reason I noticed is a lot of websites I go on use cloudflare captcha due to me using a vpn. Now I can’t even get on those website due to the captcha being down 😭

quick_snail@feddit.nl · 1 pts · 276d

PSH, I can't even pass those shitty captchas when their online. Its just an infinite loop.

As soon as I see s site is behind a cf captcha, I close the tab and go to a different site

Aertbei@discuss.tchncs.de · 7 pts · 277d (2 replies)

I was having DNS issues yesterday using cloudflare's DoH, so I'm curious if it is going to be related.

neclimdul@lemmy.world · 6 pts · 277d

Jeff says... https://www.redshirtjeff.com/shop/p/it-was-dns-shirt

Uri@infosec.pub · 1 pts · 277d

Stop using that solved

bjoern_tantau@swg-empire.de · 5 pts · 277d

And it's taken down most of Lemmy's "selfhosted" images.

tal@lemmy.today · 4 pts · 277d