War damage to data centers exceeded what AWS services are designed to withstand.
Half a year after Iranian drone strikes knocked out multiple Amazon data centers, the US tech company has acknowledged the permanent loss of some customer data that was hosted in Bahrain and the United Arab Emirates.
Amazon Web Services was “unable to restore access to the resources and data” hosted in some of the war-damaged data centers, according to an AWS dashboard update posted on September 15. The development was first reported by Reuters and suggests that the Iranian strikes inflicted catastrophic damage on the data centers.
Customer data was irretrievably lost in one of three AWS availability zones in the United Arab Emirates region, specifically the mec1-az2 availability zone. Each availability zone is serviced by one or more Amazon data centers.
103 Comments
DarrinBrunner@lemmy.world · 229 pts · 2d
The only way to get them to delete your data.
Zahille7@lemmy.world · 42 pts · 2d
"It's the only way to be sure."
DarrinBrunner@lemmy.world · 3 pts · 1d
Nuke it from orbit.
Flames5123@sh.itjust.works · 5 pts · 1d
I work in AWS, and it’s crazy how hard they enforce data security and compliance. We actually do fully delete your data pretty quickly. And nothing sensitive is ever logged. Always encoded at rest.
Calfpupa@lemmy.ml · 179 pts · 2d
Me: "please delete my data"
Amazon: "Lol no"
Iran "I got you babe"
comrade_twisty@feddit.org · 173 pts · 3d
the cloud is much more secure because your data is decentralized
Leeks@lemmy.world · 70 pts · 3d
* But only if you pay for the most redundant storage, which cost all the money.
ramble81@lemmy.zip · 13 pts · 2d
Extra infrastructure costs extra…. Who would have thought?
makeshift0546@lemmy.today · 5 pts · 2d
Yes, storing PBs of tape in collocations is sooooooo much cheaper 🙄
AWS sucks for a lot of reasons none of which are this .
story@lemmy.zip · 1 pts · 1d
i wanna do this ;(
makeshift0546@lemmy.today · 1 pts · 17h
I’m running a nas collocated with hand swapped drives.
I get it, but I also understand the reality.
story@lemmy.zip · 1 pts · 6h
the reality is that the working class can currently afford tape drives and might not be able to in 3 years
FishFace@piefed.social · 9 pts · 2d
You have complete choice whether you centralise your data or not.
boobookittyfrick@lemmy.zip · 8 pts · 2d
I mean you’re supposed to use multi-region backups
Ironfist79@lemmy.world · 1 pts · 1d
AWS is not responsible for backing up your data. They tell you this up front.
grrgyle@slrpnk.net · 2 pts · 1d
You don't have to be reasonable when it comes to megacorps, because they suck and we all hate them.
Kaligalis@lemmy.world · 64 pts · 2d
"We don't need our own backups. The cloud takes care of everything."
daychilde@lemmy.world · 30 pts · 2d
People who don't understand what they are buying almost always find out the hard way.
Raiderkev@lemmy.world · 0 pts · 2d
https://youtu.be/9GP0KDuzgBc
areakode@riskeratspizza.com · 57 pts · 2d
Never thought I'd be a cheerleader for Iran against my own country. But please, Iran, keep making America better!
village604@adultswim.fan · 5 pts · 2d
They didn't attack US data centers... This hurt the companies in the area. Amazon almost certainly had it all covered by insurance.
turtlesareneat@piefed.ca · 17 pts · 2d
I doubt Amazon had any liabiliy here - the contract they make you sign probably says "we'll try really hard but if we lose your data, it's your fault." Microsoft does that with Azure.
village604@adultswim.fan · 3 pts · 2d
I meant the physical hardware destruction, not loss of data.
raspberriesareyummy@lemmy.world · 3 pts · 2d
Still teaches people to avoid fascist fucker B
makeshift0546@lemmy.today · 2 pts · 2d
In sure they don’t pay for that insurance and the rates will remain the same.
tempest@lemmy.ca · 2 pts · 2d
Large companies don't always insure like that. It's just as likely that Amazon self insures.
ramble81@lemmy.zip · 56 pts · 2d
Anyone in IT who has ever been near a backup or had to do a DR exercise knows full well not to have a backup in the same geographic area as your primary workload. Usually it’s for natural disasters, but it very much applies to war too.
This is just as much, if not more, on the companies that didn’t have geographicly dispersed backups.
Bytemeister@lemmy.world · 9 pts · 2d
A not insignificant part of my first azure cert covered different redundancy options and pricing. If the data was that critical, the tenants would/should have mirrored it to another region.
frongt@lemmy.zip · 2 pts · 2d
"yes but it's not in the budget this year"
tazeycrazy@feddit.uk · 2 pts · 1d
Depending on the data requirements they may not have been legally allowed to move it to another region. Medical information or banking data usually has an on shore requirement. Sometimes funds have issues with storing Information overseas esspesully if there is concerns about other governments gaining access.
BillyClark@piefed.social · 6 pts · 2d
I think pretty much every software dev has used AWS at some time, and when you set up new services, it specifically warns you to use multiple geographic areas for your services. It doesn't say "war" or something, only that it's good for redundancy, IIRC.
So anyway, yes, I follow instructions, but I'm always surprised that you have software people and systems admins who can't even follow basic instructions. I don't know what percentage of people it is, but it feels like over 50% who can't follow even the simplest instructions.
FishFace@piefed.social · 6 pts · 2d
I'm not surprised at all. It costs more to use more regions, both due to literal cost and in time to set it up.
BillyClark@piefed.social · 2 pts · 2d
The extra time to set up in more regions is pretty trivial, at least in my experience. It's either an option when setting up the service, or it's something that you need to cover in any case while documenting how to set up the service. But I suspect the devs I'm complaining don't read basic instructions also don't write basic instructions.
As for the monetary cost, that's a business decision. The devs I am complaining about are not making business decisions. They're just ignoring basic instructions.
FishFace@piefed.social · 0 pts · 2d
I haven't used aws for ages but "an option when setting up the service" does not encompass my recollection of it: you can set up (say) S3 in multiple regions, but that doesn't mean your application can actually make use of multiple regions. It doesn't mean your vlans actually enable anything to talk between the regions.
BillyClark@piefed.social · 1 pts · 2d
I haven't used it for a while, either, but it was just an option sitting there when using the AWS UI to set up RDS, IIRC.
FishFace@piefed.social · 0 pts · 2d
Right, but how is your application going to handle it if you enable it?
BillyClark@piefed.social · 1 pts · 2d
You have to set up your application to use the service, regardless, so I would say that you just have to follow the instructions.
I am not talking about fixing after a service has been set up for only one region, but about following the instructions when first setting it up.
Cocodapuf@lemmy.world · 2 pts · 2d
The 3 2 1 rule.
3 copies of the data
At least 2 different kinds of media
At least 1 needs to be off site.
taiyang@lemmy.world · 51 pts · 2d
At this point, I wouldn't be surprised if a bombing on US territory was cheered if it was an AWS data center (or Meta, xAI, etc). Hell, Iran could probably even find US citizens for sabotage at this point if they just ask, lol.
BlaestEgnen@feddit.dk · 4 pts · 2d
I expect Iran already hires Americans for unorganized sabotage, I know both Iran and Russia utilises Telegram to hire teenagers "torpedoes" as they're called among mobsters.
It's a fully gangster technique, but it's effective. One example would be when the Israeli embassy in Copenhagen had two kids attempt to throw a grenade their way in 2024 - It's proven to be related to Foxtrot (the largest Swedish organised crime syndicate), which Iran has influence on through one of the leaders living in Iran
IndustryStandard@lemmy.world · 1 pts · 2d
You have to be a special kind of brainwashed believe that story about Iran hiring children
BlaestEgnen@feddit.dk · 0 pts · 1d
So what does the Foxtrot network gain by attacking an embassy? Somebody pair them. The most obvious one would be Iran
IndustryStandard@lemmy.world · 0 pts · 1d
What did Iran gain besides Europe having an excuse to impose sanctions on them which is very convenient to Israel
Mubelotix@jlai.lu · -3 pts · 2d
Nobody will give their life for this, you must use drones for sabotage
BiscuityCat@lemmy.world · 7 pts · 2d
Just give any American $1000 and they will be happy to do that.
Mubelotix@jlai.lu · 7 pts · 2d
Lol, I would to be honest. Short the stock for 10k with big leverage, pay 10 americans 1k each to attack a datacenter, get 50k back. But they won't
boonhet@sopuli.xyz · 8 pts · 2d
Aaaaand this will be the last we hear of ElectricVocalist
mrmisses@lemmy.world · 36 pts · 2d
Well good thing it isn't a war then
tourist@lemmy.world · 2 pts · 2d
They are intentionally designed to withstand a little bit of "war damage"?
Like an Amazon data warehouse construction guy needs to specifically add a little bit of extra cement or some shit to meet a war damage protection quota?
silver@das-eck.haus · 35 pts · 2d
Speaking as a member of a company who has just recently realized that we can have products in regions other than us-east-1, this is not surprising. If UE1 went down the economy would collapse overnight
lobut@lemmy.ca · 11 pts · 2d
I've been working on Disaster Recovery for my company and the amount of sleep I'm losing on this is insane in general.
Our devs just write their own IaC stuff so like, they can provision their own databases and shit without us knowing with no backups or customer keys and stuff to back up properly.
tempest@lemmy.ca · 11 pts · 2d
It's incredibly hard to reign that behavior in once it's entrenched as well.
You basically gotta take away privileges from everyone and it becomes a huge fight if you don't have organizational support. Usually you have to wait for something to explode spectacularly for that to happen.
CosmicTurtle0@lemmy.dbzer0.com · 6 pts · 2d
Fellow SRE here.
The number of times I've heard "we have replication" followed by "we've tested it...on paper" make me cry.
Trying to convince a CEO that the only way to know for sure that their backups work is to do an actual disaster test is harder than trying to convince my parents that I don't need to eat more.
morbidcactus@lemmy.ca · 5 pts · 2d
That'd give me nightmares too on top of the DR, hopefully at the minimum they can't spin up resources in anything but a dev/test environment.
Best of luck to you, governance is super important, if it makes you feel any better you're definitely not alone.
Tiral@lemmy.world · 26 pts · 2d
Do they have a Go Fund Me or something I can donate to or something to keep them rolling?
phtheven@lemmy.world · 4 pts · 2d
Neither Amazon or Islamic Republic of Iran need your money, friend. Buy yourself a nice hat in stead.
jj4211@lemmy.world · 13 pts · 2d
And get it today, with Amazon Prime!
DaGreenGobbo@feddit.uk · 2 pts · 2d
I would but they lost my... custom. By being evil.
chewypoops@lemmy.world · 2 pts · 2d
Y'all really need to normalize downvoting serious replies to jokes. Just because this reply is agreeable doesn't make it sap the joy out of the conversation any less.
phtheven@lemmy.world · 2 pts · 1d
Sounds like someone could use a charming hat to lighten their mood.
Smaile@lemmy.ca · 20 pts · 1d
Fuck yes, keep doing that please.
shweddy@lemmy.world · 18 pts · 2d
Iran out here doing everyone a favor
Gladaed@feddit.org · 17 pts · 2d
2+1 isn't this building and the neighboring building. No back up no compassion. (Kein Backup kein Mitleid)
quack@lemmy.zip · 16 pts · 1d
That's one way to delete your Amazon account I suppose.
Washedupcynic@lemmy.ca · 13 pts · 1d
Good. Fuck Amazon, sucks for the customers. Also what kind of company like this doesn't have backups?
CrabAndBroom@lemmy.ml · 5 pts · 1d
Presumably, the kind that likes to cut corners and not do things properly, safe in the knowledge that they'll face no consequences. Or if they somehow do then the government will just bail them out anyway.
Psiczar@aussie.zone · 2 pts · 1d
It’s up to the customer to decide if they want data redundancy. AWS offers it, it just depends if the customer is willing to pay for it.
bold_atlas@lemmy.world · 2 pts · 1d
Pay to have your data backed up on the adjacent drive on the rack lol.
SalmonTractor@discuss.tchncs.de · 1 pts · 1d
Hey, now, sometimes the rack is across the street. /s
sportsjorts@lemmy.zip · 11 pts · 2d
Hahahaha!
nullspace@lemmy.world · 11 pts · 2d
lol
Mubelotix@jlai.lu · 11 pts · 2d
AWS makes so much cash that they won't even leave the region
kreskin@lemmy.world · 10 pts · 2d
Oh no, not amazons marketing data!
Widdershins@lemmy.world · 9 pts · 2d
Someone needs to shine a laser at bezos' noggin for a better permanent loss.
raspberriesareyummy@lemmy.world · 8 pts · 2d
Do it again. Harder!
Crozekiel@lemmy.zip · 8 pts · 2d
So like... the opposite of a data breach, the headlines I am used to reading about "customer data"? Hmmm... Ok.
chewypoops@lemmy.world · 8 pts · 1d
3,2,1 applies to cloud regions, so everyone who lost their data permanently deserved it.
Zensate@lemmy.world · 8 pts · 2d
Making me wanna Inshallah up
kibblebits@quokk.au · 8 pts · 1d
No great loss.
melsaskca@lemmy.ca · 7 pts · 2d
Finally! Data getting "lost" instead of "stolen". /s
chahn.chris@piefed.social · 6 pts · 2d
Ok, good.
itisileclerk@lemmy.world · 6 pts · 1d
Oh no, ... anyway, don't have Amazon account
bold_atlas@lemmy.world · 5 pts · 1d
The only drone delivery service I support.
Bristlecone@lemmy.world · 5 pts · 2d
Hilarious, YES IRAN!
Etterra@discuss.online · 5 pts · 2d
Oh no. Anyway...
SuiXi3D@fedia.io · 5 pts · 2d
Oh no! Anyway…
flandish@lemmy.world · 5 pts · 2d
good. more. do more.
callouscomic@lemmy.zip · 4 pts · 1d
Strikes during war cause loss of many things, including life. Why is this news?
INHALE_VEGETABLES@aussie.zone · 7 pts · 1d
It's a datacenter and they are so hot right now
bold_atlas@lemmy.world · 4 pts · 1d
Because it's funny.
snugglesthefalse@sh.itjust.works · 2 pts · 1d
Probably because some people thought it couldn't happen to Amazon
BilSabab@lemmy.world · 4 pts · 2d
kinda surprising it didn't happen earlier
kokesh@lemmy.world · 4 pts · 2d
Get them all!
Baguette@lemmy.blahaj.zone · 4 pts · 1d
Yall this is AWS not Amazon retail. There might be some regional retail data, but most of this is data from other businesses. Loss of customer data here is referring to AWS business customers, not retail.
weps@lemmy.world · 3 pts · 2d
ik hope my data was deleted
orenj@leminal.space · 3 pts · 2d
Based based based based
chiliedogg@lemmy.world · 2 pts · 1d
I'm sure Alexa still has it.
jerkface@lemmy.ca · -9 pts · 2d
ennof@feddit.org · 0 pts · 2d
Nutzername prüft aus.
jerkface@lemmy.ca · 1 pts · 2d
What is the objection?
jerkface@lemmy.ca · 0 pts · 2d
Fuck me for thinking ethical alternatives to AWS was a valid topic, I guess.