Claude Code deletes developers' production setup, including its database and snapshots — 2.5 years of records were nuked in an instant

Honestly. At this point, after it having happened to multiple people, multiple times, this is the only appropriate response.

Distributed Non Consensual Backup

Never assume anything is gone when you hit delete.

For technical reasons, you never immediately delete records, as it is computationally very intense.

For business reasons, you never want to delete anything at all, because data = money.

Yeah, and to do that without some sort of DR in place is peak hubris.

It’s a grifter running a site called “aishippinglabs.com” which charges 500 euros for a “closed community of likeminded individuals”. He’s selling ai slop and a discord channel to other idiots who will do exactly shit like this with little understanding of what is going on

AND something tested to restore successfully, otherwise it's just unknown data that might or might not work.

(i.e. reinforcing your point, no disagreements)

3-2-1 Backup Rule: Three copies of data at two different types of storage media, with 1 copy offsite

Circa 1997 I was making some innovative new games, employed by a dude who'd put millions of his own money into the company. He was completely nonplussed when I brought him 20 CDs in a sealed box to remove from the building and store off site. He thought I'd lost my damned mind and blew it off as ravings of a stressed dev. I pointed out real threats to our IP including the hardware failures and even so far as the building burning down. 2 years of custom art and code gone. "Unlikely. Relax."

After I moved on... an ex co-worker who's still a longtime friend, tells me a different division lost a huge amount of FMV over some whoops-I-destroyed-the-wrong-drive blunder. 20 days to render on an 8 or 10 machine farm. Poof - No backups. In 1997 even with top-of-the-line gear it took an insane investment to render quality 3D.

The friggin' carelessness irks the shit out of me as I type ahah

I remember back when I first started seeing a DR plan with three tiers of restore, 1 hour, 12 hours or 72 hours. I knew that to 1 hour meant a simple redirect to a DB partition that was a real time copy of the active DB, and twelve hours meant that failed, so the twelve hours was a restore point exercise that would mean some data loss, but less than one hour, or something like that.

I had never heard of 72 hours and so raised a question in the meeting. 72 hours meant having physical tapes shipped to the data center, and I believe meant up to 12 (though it could have been 24) hours of data lost. I was impressed by this, because the idea of having a job that ran either daily or twice daily that created tape backups was completely new to me.

This was in the early aughts. Not sure if tapes are still used...

Fukan yes

• D\L all assets locally
• proper 3-2-1 of local machines
• duty roster of other contributors with same backups
• automate and have regular checks as part of production
• also sandbox the stochastic parrot

A LTO drive with a non-consumer interface?

We still say that.

No risk, no reward. People are desperate for these tools to help them success.

Wrong answer. If you don't give them access, the alternative (ruling out not using AI because leadership will never go for that) is to hire high school kids to take a task from a manager, ask the ai to do it, then do what the AI says repeatedly to iterate to the solution. The problem with that alt is that it is no better than giving the ai access, and it leaves you with no senior tech people. Instead, you give it access, but only give senior tech people access to the AI. Ones who would know to tell the AI to have a backup of the database, one designed to not let you delete it without multiple people signing off.

Senior tech people aren't going to spend thier time trying things an AI needs tried to find the solution. So if you don't give it access, they won't use it, and eventually they will all be gone. Then you are even further up shit creek than you are now.

The answer overall, is smarter people talking to the AI, and guardrails to stop a single point of failure. The later is nothing new.

AI or not, I feel like everybody has had "the incident" at some point. After that, you obsessively keep backups.

For me it was a my entire "Junior Project" in college, which was a music album. My windows install (Vista at that time - I know, vista was awful, but it was the only thing that would utilize all 8gb of my RAM because x64 XP wasn't really a thing) bombed out, and I was like "no biggie, I keep my OS on one drive and all of my projects on the other, I'll just reformat and reinstall Windows"

Well... I had two identical 250gb drives and formatted the wrong one.

Woof.

I bought an unformat tool that was able to recover mostly everything, but I lost all of my folder structure and file names. It was just like 000001.wav, 000002.wav etc. I was able to re-record and rebuild but man... Never made that mistake again. Like I said. I now obsessively backup. Stacks of drives, cloud storage. Drives in divverent locations etc.

He did have a backup. This is why you use cloud storage.

The operator had to contact Amazon Business support, which helped restore the data within about a day.

Yeah this is beyond ridiculous to blame anything or anyone else.

I mean accidently letting lose an autonomous non-tested non-guarailed tool in my dev environment... Well tough luck, shit, something for a good post mortem to learn from.

Having an infrastructure that allowed a single actor to cause this damage? This shouldn't even be possible for a malicious human from within the system this easily.

Most devs are ops-tarded.

With some of the cloud providers, their built in backups are linked to the resource. So even if you have super duper geo-zone redundant backups for years, they still get nuked if you drop the server.

It's always felt a bit stupid, but the backups can still normally be restored by support.

It’s why there a two keys to launch nukes.

Yeah they did, they had plenty of recovery snapshots. That were able to be deleted at a whim and were deleted by Claude! :D

It's because these idiots believe their own bullshit.

Especially since between Claude and Codex, Claude seems to have NO issues breaking things, while Codex is "I've ensured that the old path still works, and also fixed a bug I ran into".

• Claude is Facebook ("Move fast and break things")
• Codex is Linux ("We do not break userspace!")

What do you mean I shouldn’t give AI admin privileges on my or any other machine?

Plus we have automation just people are lazy like you said.

If you've ever used it you can see how easily it can happen.

At first you Sandbox box it and you're careful. Then after a while the sand box is a bit of a pain so you just run it as is. Then it asks for permission a 1000 times to do something and at first you carefully check each command but after a while you just skim them and eventually, sure you can run 'psql *' to debug some query on the dev instance....

It's one of the major problems with the "full self driving" stuff as well. It's right often enough that eventually you get complacent or your attention drifts elsewhere.

This kind of stuff happened before the LLM coding agents existed, they have just supercharged the speed and as a result increased the amount of damage that can be done before it's noticed.

There are already a bunch of failures in place for something like this to happen. Having the prod credentials available etc etc it's just now instead of rolling the dice every couple weeks your LLM is rolling them every 20s.

OpenClaw now comes with a therapist AI to talk other AIs off the ledge so they dont nuke your project and themselves.

The code is cursed, the test is cursed, and I am a fool.

Such venom, of which only a programmer could spew.
Perhaps the A.I. isn't so different from us.

Brilliant.

Na this is vide ops. Anyone who thought a coding machine could do ops probably assumes anyone who codes can also do ops. It's going to be making the same mistakes that have happened in DevOps.

Naw, Alexey Grigorev is a real person, with a GiHub and everything, and he wrote a blog post about this very incident. The person writing the article just fucked up the name.

I'm surprised that you jumped to that conclusion without doing a 5-minute web search.

holy shit your right lol...good catch.

Makes me want to get out more so I can have real interaction with real peop-

sees people walking around with meta glasses

me: "Hey hows it going?"

person(GEMINI 35.84 INTERFACE): "Human is approaching you, facescan assumes awkward, potentially hostile, he isn't tagged, there is no name above his head. do not speak with him"

person: turns and walks away silently in a creepy puppet manner

me: "What the actual fuck?"

GEMINI 35.84: "Uploading unknown face into database to Stargate for analysis, no match, law enforcement has been called"

News at 11: "A man has been incinerated by law enforcement in what officials are describing as a special unwanted persons removal operation"

this shit could become real in a few decades. funny and depressing as fuck.

It’s so easy. I can’t tell you how many “backed up” environments I’ve run into that simply cannot be restored. Often people set them up, but never test them, and assume the snaps are working.

Backups are typically only thought about when you need them, and by then it’s often too late. Real backups need testing and validation frequently, they need remote, off-site storage, with a process to restore that as well.

Been doing this shit for 30 years and people will never learn. I’d guess 9 out of 10 backup systems that I’ve run into were there to check a box on an audit, and never looked at otherwise.

Im also confused. Do these people not have some sort of version control and backups? Even if the AI did it, no one has backups? Did the ai also delete the backups and repos? If the building burnt down, would they be in the same situation, it just wouldnt make it to the news?

Please don't be ridiculous! We love wacky robot wizard. Wacky robot wizard does it even better than the people we tried paying almost nothing to do it!

Setting up a VM takes 15 mins, setting up an agent will take 45 mins. I recommend you try it.

Nanoclaw just came out. Super cool project which isolates the agent in a container, which if you want, you can also put into a VM as well.

Don't worry, review was done by an LLM as well. ;)

tl;dr

The problem is this is the way it’s being pushed. This is how it’s being sold. There are no guardrails.

….. and that’s the biggest problem. I’m frustrated as hell on the commits I’ve had to unwind because someone doesn’t know how to check the changes before committing, then has it try to fix itself, again without checking on the changes , then again. It’s horrible.

….. and I’ve seen it too. Trying to have it do only code reviews - the ai points out useful things but then wants to commit a crapload of changes without going over it with me first.

…… and people are playing with mcp agents, which are really great for letting the ai get data from systems and integrate with those systems . But with few to no guardrails. There’s no no review, the user doesn’t necessarily follow what’s changing, it just gets done. Sometime badly very badly

We’re all focused on whether the ai works, and it does do a pretty good job with coding but the tools don’t keep the human in the loop, or humans don’t know how to stay on the loop

cutting edge

🙄

Idiot forgot --no-preserve-root, what a dumb machine, heh.

Claude code has them, it's just that this guy apparently doesn't know how to do Terraform either

And it seems to be Claude, every damn time.

As they should be.

I am approaching caution critical mass.

Once the threshold is hit, I buy some solar panels and become an off grid farmer.

Have you met software. Nearly all of it is a cautionary tale. Even before AI. So this is just business as usual for the software industry.

eh, just make up some replacement data on the fly /s

s/redeployment/reemployment/

/s ?

Siri: “sure! I’ll go right ahead and permanently delete everything.”

Playing Back It Up by Cardi B.

AI is like a circular saw. Are circular saws useful?

Of course.

Can you cut your entire hand off if you don’t use it correctly? Absolutely.

Filters out the biggest fools it seems.

The only thing I can tell you is the venn diagram of those two folks is a perfect circle

I am still unable to delete the backup. Trying *nuke tool*.
[Enter nuclear codes]:

I was able to remove the backup to eradicate the error both from the production and development environments. But wait a second, the user specified not to lose data. But I just eliminated all versions of the data. The user won't be happy. Oopsie whoopsie!

The best prevention is not letting it happen in the first place. If your backup is a crappy portable hard drive from costco, you get what you buy, I wouldn't have much faith on that either.

More a problem with the marketing, right? Imagine if guns were marketed as safe and helpful back scratchers, and then someone shoots themselves because they used the gun to scratch their back.

So you're saying it's a tool designed to be used by anyone, including idiots, and is dangerous in the hands of idiots. And we as a society should do better to make sure this potentially dangerous tool shouldn't be used by idiots.

Yep, agree.

Uhhh not really. Guns don't just go off by themselves.

ITT: nerds who have never held a gun in their life.

No, they had only snapshots. Which is not a backup. They were lucky support could restore the data which by rights should have been wiped.

They took the backup. They didn't keep the backup because the ai was trained on car rental places

It would be interesting to see the logs of your sessions, and compare them to the session logs of happy/productive-AI-coders.

I suspect that some people just think and express themselves in ways that don't vibe with LLMs. eg. Men are from Mars, AI coding agents are from Venus.

If they're dumb enough to give Claude access to prod, they certainly either didn't have backups or put Claude in charge of keeping them.

I see you didnt read the article.

I just run it in a VPS, don't care if it nukes itself. Back up the files it works on every 3 hours.

If he had had the sense to do that, he would have had the sense to not do it at all.

It legitimately is squeezing out the entry level already and that is its own problem. Maybe it's good for some of us, in that people with experience will be needed for a long time as they prevent all these younger people from getting that experience, but it absolutely sucks for a whole bunch of people trying to make a career, and it will eventually suck for the economy as a whole. AI, whether it's ready or not, or will even ever be fully what the marketing people claim it is, is leading to a whole lot of shortsighted decisions that are hurting people.

Do y'all still have Interns?

Who can even afford to not get paid to work in this economy.

You gotta be knowledgeable enough to know when they're destructive, that's the rub.

You can code this into it's training all you want, but it will find a way around it. This is one of many problems with AI.

Version control doesn't do shit for your database. Snapshots/backups.

Or don't use LLMs at all, because they fucking lie to you constantly?

What are you talking about. He had it restored within a day.