Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue

It looks like their website is pocketos.ai lol

It could be a moronic sysadmin, it could just as easily be a moronic exec pushing staff to implement this crap right now and damn the consequences.

I mean that's kinda the whole point.

Companies are looking at AI to replace people. Either it's ready or it's not.

If you need to treat it like it's an intern, then it's not worth the expense. Anyone hiring interns to be productive doesn't understand why you hire an intern.

“Treat an AI like an idiot intern without any references you just hired.”

Instead of this, treat AI like some dude off the street who you didn’t hire and leave it out of your life. It’s shitty, it’s wasteful, and it’s subsidized by everyone to get a few tech bros rich.

Like seriously, it’s just theft of people’s work it “trained on”, powered by energy companies that charge us more to power it, at the cost of poisoning our water supplies, to ultimately try and steal our salaries one day.

It’s absolutely parasitic software at every level.

Treat an AI like the idiot intern without any references you just hired.

My company is in the process of pivoting hard to Claude after 50yrs of doing virtually everything themselves and rolling their own versions of already-existing software, and this is almost verbatim how I've described to others what it feels like to use it.

It feels like cajoling an intern to understand a job for which they have some average skill but zero motivation, and they only want to do the bare minimum, so you spend all the time you could be doing your job holding their hand through basic tasks.

It's fucking annoying.

give any developer that power?

Fun fact: giving developers access to production deployments violates FedRAMP and like half a dozen other compliance regimes SOC2/IRAP/ISMAP/G-Cloud/BSI C5/...

I was once the intern who did relatively stupid things with one very big consequence.

My biggest fuckup was unplugging a 10base2 (edit: I originally wrote 10-base-T) coax wire from the loop so I could plug in a newly built computer. Everyone at the time (including me) knew that an unterminated 10-base-T network would crash Win 3.11, so the accepted process was to tell the entire network you were about to disconnect a cable so they could save their work and be ready to drop to DOS. I spaced that step in my haste to test a newly built computer and ruined a day's worth of work by the sales guy.

Ultimately, I was the one who fucked up and did know better. That's AI. However, it only had consequences because Win 3.11 networking code was fucking awful and because the sales guy didn't save his work frequently. If the same person in this story had asked Claude whether it was a good idea to have the backup and production databases on the same volume, the AI would have said No. If the person had asked Claude whether it was a good idea to delete a database without any confirmation dialogue, the AI would have said No. AI did it anyway. That's what makes this an AI story.

Was their database environment stupid? Yes. Did the sysadmin fuck up by not treating AI like an intern? Yes. Did the AI do something it knew it shouldn't do? Also yes. This is both an AI story and stupid sysadmin story.

It's both.

Treat an AI like the idiot intern without any references you just hired.

An extremely enthusiastic intern that, if presented with a question/problem/prompt they don't know the solution for will just overconfidently pull something out of their ass and run with it.

Problem is execs and stupid software devs wanna give these things full reign on systems because of “performance gainz “

It’s a collective stupidity that’s impossible to break because it’s hooked into the highest decision makers.

Treat an AI like the idiot intern without any references you just hired. Gave the idiot intern permission to delete your production database? That’s entirely on you

We've officially veered into a timeline where the standard for every tech employee's level of competence is on par with the guy who pushed through that update to CrowdStrike lol

These things are bought specifically because they are trying to replace the sysadmins... Along with everyone else.

I don't want to sound like a know it all here because I recently was reminded by a nice Lemmy person to actually TEST my backups, but damn. Every part of that is so dumb. I also have backups stored by a different company in addition to locally storing really important info. If your stuff is hosted and backed up by the same people, what happens if your account is randomly suspended or hacked or some other issue (like ai)?

Management are pushing sysadmins to use AI, yet AI tools permissions models are worse than useless.

User error.

PocketOS states that as well.

Also backups stored on the SAME VOLUME as the prod data? How fucking stupid do you have to be?

Hope he gets sued for defamation now.

ha! for real. you have scoped API tokens, but not using it properly. this is just a fear mongering click bait rage bait headline. sure, the agent executed the deletion, but it’s the human’s responsibility to configure security tokens correctly before handing the keys to anyone, human or agent.

this is an "Don't allow anyone access your backups without following protocol." problem.

Congratulations you just identified the AI problem.

Eh, if you pay attention, most of the times this happens the person was a jerk in their prompts.

Like look at the instruction echoed back in this case. All caps and containing a curse word.

You can believe that the incidents occurring are 100% because of negligence and not related to the model behavior shifting, but there seems to be a widening gap between people who prompt like this and have horror stories and people who give the models breaks over long sessions and seem to also regularly post pretty positive results.

Incredible emoji

Can we make sure to make Ted Farro suffers worse this time?

Being reduced to a mutant blob for, say, a few extra thousand years and maybe put in a zoo or something?

It's so weird how these chatbots always pretend they learnt something after they fuck up.
They literally can't.

yeah, it gives you the answer it thinks you want based on your prompts.

I'd be interested to see what prompts they used to, uh, prompt this response.

I lost it at the confession. The ai has no knowledge of what it did. You are feeding in your context and it is making up a (sycophantic) plausible explanation based on the chat history. Makes me wonder if this person should have production access in the first place.

The way it communicates suggests to me it's got some 'prompt engineer bro' garbage system prompt going on there.

You’re telling me I get to experience the joy of this headline more than once?

We should also expect to see “Thousands die needlessly after rushed deployment of botched AI, the first tragedy of this scale involving the technology.” as well. It’s coming.

"Man, I sure wish modern society would shrug off the shackles of capitalism"

*A single finger curls on monkey's paw*

Then what even is the point of all this? At my old job the idiot intern was sorting patch cables in a box

AI writes code

User vets code

User runs code

If you're not lock-step watching that shit, you need to just be doing it yourself.

It's just negligence. Power tools injure and people are stupid. The technology is alluring and people make dumb mistakes. There's no deeper motive here, and self admitting you're not even an amateur I will just tell you that you're giving way less credit to these models than they deserve by calling them purely probabilistic, and way more credit then they deserve by trying to assert some kind of malicious incentive by anthropic.

These bastards are hard to make, and they have a lot of layers (not like NN layers, but training steps). They are, however, definitely better at programming than you or your buddy or any commentator here, and it lures you into a false sense of security before it makes a colossal fuck up.

This is what happens when there is a new technology and companies are run by commerce grads, not scientist or engineers that understand the technology.

AI has good therapeutic uses, particularly for disabled or impoverished people who may not be able to access mainstream therapy

There really should be a certification course for using AI safely. I'm slop coding a hobby app and I'm shocked at how much it FEELS like it can do, because it can do amazing things, yet fails in the strangest ways. When it feels like it can get away with it, it forgets earlier discussions and moves on without it. So you can spend time hammering out a whole section of code, then move on, and AI will rip out everything that references that code and think of a different way in the moment and code that in instead. It won't be the same. It probably won't work, or at least won't pass all test cases. But if you aren't paying attention and keep coding, your original part of the project is no longer functioning and you won't understand why. But every step of the way it's confident in its answers and you won't suspect that it fundamentally no longer understands the project.

And it was right!

Yeah, this is just a long-winded way of blaming the tool and not the tool of a human using it.

This cloud provider is also vibe coded?

it's like saying the hammer I was using that blew up my house "went rogue" because I kept the propane tank underneath the 2x4 I was hammering a nail into.

the providers API allowed for potential destructive actions without confirmations, backups were kept on the SAME volume as the source and wiping said volume results in deleting all backups, no version control either.

COMBINE ALL THAT with the fact they relied on Claude which is NOTORIOUS for guessing, not verifying ANYTHING even though it says it does and whose solutions 8 to 9 times out of 10 are hallucinations...perfect storm.

Optical media? Is that a viable part of backup strategies? I would expect tapes for sure, sounds like you know more than me.

they did not follow the 3-2-1 rule...

It's a feature.

It seems like actually criminal too. Like legitimately "we need to shred 2TB of incriminating data instantly or we're all going to prison"

There are rules for backups and role separation. Some of that is in iso27002, and none of it is even known by these lost boys bereft of proper mentorship and bouyed by their own accidental success.

"If your prod can be deleted by your AI, it should be."

Seems like, if nothing else, Ai might finally force corporate accountants to acknowledge that the cost of a good backup strategy far outweighs the cost of losing all your data because some MBA thought he could write a product update himself with Claude code.

Not to mention the image

We just got sick of approving all those annoying prompts! /s

Managing access control is too much work. Better to just let the AI do it.

No, admins neither should have access.
Backups should (best case) be immutable.

The backup was on the same volume as the original data.

The AI deleted the whole volume/backup. 😕

Yes

At this point, we should not be surprised.

I don't know when businesses stopped backing up their production databases on physical media, but maybe we should go back to that. I can remember multiple previous jobs where the IT manager was responsible for daily or weekly backups to external drives that were stored off-site in fireproof safes.

I get how cloud storage can meet that requirement now, but surely people recognize that cloud backups are worthless if every dumbass in your company can accidentally delete them.

If your artificial coding agent has a level of access that allows it to delete the cloud backup, maybe the person who gave it that access is the dumbass.

Just because it's not a best practice doesn't mean it's not being done.

Sounds like a responsible strategy to draw back from a lot of this. It's all so...effervescently remade, the "ecosystem", every few months.

For me the takeaway comes from time I spent in some safety-critical parts of engineering and personal hobbies. Ultimately relying on people to make good decisions ~all of the time isn't enough to prevent disaster, if something like disaster is on the line.

Systems must be engineered to remove possibilities for accidentally bad, in-the-moment human decisions, where it counts. Thoughtfully. This is the weird same-shape but exactly-opposite doppelganger of that set of best practices.

When the systems are using ~opaque automations that behave like humans (w.r.t. some decision-making and unreliable expectations of behavior) - and then relying on people making the right calls on top of that ever-shifting set of capabilities - I mean c'mon lol.

This is gonna happen a lot, while the carrot of go-faster remains dangling so unignorably (because it's in front of everyone, everyone working anywhere near the stuff). Until we look around and take a broader view. Which will be learned the same way we learned to make safety regulations, but I largely doubt our ability to respond in a similar way.

The money will eventually respond, of course, but that's always a poor and late proxy for what ought to be done.

---

Sidenote, for aspiring engineers, take heart!

It will be you who ends up tasked with unburying from all the technical debt incurred, truly. A practice steeped in the ancient wizardly traditions of yore. Spending a career on that and building something better.

It will be necessary, the work begins roughly a while ago lol but more fully when things settle somewhat. Many large and slow organizations are right now very engaged in simply unburying themselves from the technical debt of a previous hype cycle, AKA now making use of all the data they collected (badly, via go-fast charlatans) during the "Big Data! You'll be left behind if you don't collect extreme amounts of data, it's cheapish and everyone else is doing it!" era.

It's not a defense of the LLM. They are bad tools: LLMs can be unpredictable and if you hand them a nuke they may trigger it for whatever reason, even harmless and unrelated ones.

So don't hand them a nuke, idiots.

He may be correct in that this has already happened previously to other companies so they were forewarned and took basically no steps to mitigate the chances or protect their backups.

The AI company bears some responsibility for the act because they programmed it. But the company using the tool also didn't take the precautions they should have taken.

AI is crap and companies shouldn't be so gung ho about it, because this and situations like it appear to be prone to happen if for no other reason. But any tool you don't respect will bite you eventually.

it actually will take your job... it will crash industries and economies a touch later, when all the knowledge and expertise is lost and we're left with a choice of rolling back a century or keep living within enshitified society. And not because AI is useless, but because suits and wallstreet are casing exponential profits from the snake oil AI outfits are selling.

I wanna see this done with Ron Perlman's Voice.

1000 Ways to Kill A Vibe Sesh Trustfund Startup

https://youtu.be/X6NJkWbM1xk

The only thing inevitable is stupidity and lazy greed lol

Only if they die or the CEO commits seppuku.

Wait til someone invents 8 second wipes

They operate system for Car Rental places. Why would they bother data recovery ?

Losing reservations is their bread n butter.

You see, you know how to take the reservation, you just don't know how to hold the reservation. And that's really the most important part of the reservation: the holding. Anybody can just take them.

There's nothing left to buy. can't Brain-Drain either, these are idiots

AI companies are selling that “coding has been largely solved”.

It seems that it hasn’t.