Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue
https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-powered-ai-coding-agent-deletes-entire-company-database-in-9-seconds-backups-zapped-after-cursor-tool-powered-by-anthropics-claude-goes-rogue
435 points · 53 comments · view on lemmy.world
53 Comments
gravitas_deficiency@sh.itjust.works · 137 pts · 121d
If you are giving your codegen LLM - the model involved truly, genuinely doesn’t matter - admin access to your prod env, all I’m going to do is point and laugh.
curbstickle@anarchist.nexus · 96 pts · 120d
Just to add - AND ACCESS TO THE BACKUPS!!
SkaveRat@discuss.tchncs.de · 53 pts · 120d
and having the backups stored in the same location as the primary data
msage@programming.dev · 26 pts · 120d
Then it's not a backup, it's just duplicated data.
curbstickle@anarchist.nexus · 15 pts · 120d
Just a shit show top to bottom for sure
zwerg@feddit.org · 10 pts · 120d
No one should be able to delete or change backups. This infra was in any case vulnerable to a ransomware attack as any bad actor that breaks in can delete the database and encrypt the backups with a key they promise to share in return for bitcoin.
tomiant@piefed.social · 5 pts · 120d
Also no prompts, ironically, for operations like "Are you sure you want to delete the production database? (y/N)"
gravitas_deficiency@sh.itjust.works · 3 pts · 120d
It’s amateur hour all around lol
deadbeef79000@lemmy.nz · 122 pts · 120d
Man who shit his own pants horrified that his pants are full of shit.
Demands explanation from pants vendor.
grendel84@tiny.tilde.website · 4 pts · 120d
@deadbeef79000
@cm0002
I can't believe that they criticized the vendor api for not having confirmation.
It's a freaking API!!! It's designed for automation, not direct human (or LLM!) use. If you added confirmation then devs would have to code automatic acceptance, which defeats the purpose.
It doesn't make a bit of sense. Someone is passing the buck.
deadbeef79000@lemmy.nz · 2 pts · 120d
Everything is always somebody else's fault.
Croquette@sh.itjust.works · 1 pts · 119d
Yeah but a higher up would have to take responsibility, we can't have that
Quibblekrust@thelemmy.club · 2 pts · 120d
In your analogy, I would think they would demand an explanation from the food vendor.
deadbeef79000@lemmy.nz · 2 pts · 120d
Demands explanation from local grocery store?
thejml@sh.itjust.works · 67 pts · 120d
Hot take: offsite, offline backups are so cool right now.
drcobaltjedi@programming.dev · 28 pts · 120d
Yup, follow the 3-2-1 rule or you don't have backups
Railcar8095@lemmy.world · 25 pts · 120d
Hotter take: do not give an LLM agent permissions you wouldn't give a recently hired junior
jafra@slrpnk.net · 10 pts · 120d
Actually this is how AI should be viewed. Under the right circumstances it maybe saves lots of time, but it also might destroy, so treat it like you would an intern...
Taleya@aussie.zone · 13 pts · 120d
hell I've got a better backup methodology with my fucking cat photos
tomiant@piefed.social · 1 pts · 120d
Analogue is the next big thing.
chatokun@lemmy.dbzer0.com · 1 pts · 119d
They blame AI, but what if this was one of those aggressive encrypt everything malware attacks? Why are the backups accessible anytime except during a backup? I know, I know, convenience. But all of the backups?
Brokkr@lemmy.world · 56 pts · 121d
I don't understand what Railway is supposed to do here? If deleting a drive also deletes the backup, what's the point of the backup?
i_stole_ur_taco@lemmy.ca · 70 pts · 121d
I save space on backups by symlinking my data in a backup directory. It’s never failed!
four@lemmy.zip · 16 pts · 120d
You obviously should do a hardlink, as this is much safer
trolololol@lemmy.world · 1 pts · 120d
That's the joke
Nomecks@lemmy.ca · 7 pts · 120d
Hyperconverged backups FTW!
massacre@lemmy.world · 5 pts · 120d
It saves on storage costs!
Aceticon@lemmy.dbzer0.com · 2 pts · 120d
I XOR all the bytes of my data and write down the resulting byte value on a post-it as our backup.
Saves tons of space, it's fully offline and I never had any problem with it.
grueling_spool@sh.itjust.works · 45 pts · 120d
No bro you don't understand, Claude needs access to backups so it can restore them in case something breaks because our senior dev ($50k, 2YoE) doesn't know how to do it
Quibblekrust@thelemmy.club · 2 pts · 120d
Damn, you got two-year-olds making 50k?
db2@lemmy.world · 31 pts · 120d
They can't go rogue, they have no agency or desire or thought. What really happened is the thing specifically designed to do whatever the Plinko line with the most chips says did it because the incompetent dickheads who deployed it didn't know how not to do that.
Michal@programming.dev · 30 pts · 120d
So what? Gen AI can build a new database and populate it with data 🤷♂️ /s
rtxn@lemmy.world · 25 pts · 120d
Fuck that. I'm blaming the PocketOS founder and every person in the chain of decisions that led to a clanker being given this level of unrestricted access to the database and the backups.
merc@sh.itjust.works · 20 pts · 120d
"I hooked up spicy autocomplete to our production systems and it nuked them. What have I learned from this? Here are some bullet points for how the spicy autocomplete industry needs to do better."
Bane_Killgrind@lemmy.dbzer0.com · 4 pts · 119d
If I did that the first day I was hired, nobody's going to my parents for answers
zalgotext@sh.itjust.works · 2 pts · 119d
To be fair, those bullet points are pretty standard security best practices that any software company should be following.
But like, at the same time, even if AI companies were doing those best practices, I still wouldn't let their products loose on production systems.
merc@sh.itjust.works · 5 pts · 119d
Sure, they're reasonable requests. They're things that he should have looked at before letting spicy autocomplete loose on production.
If I put a supposedly trained monkey into a tractor's cabin and had it try to do farm work, I don't think people should be listening to my ideas about what kinds of skills monkeys should be taught before they're assigned farm work. I think people should be laughing at me and calling me an idiot for assuming a monkey could do farm work.
pitiable_sandwich540@feddit.org · 3 pts · 119d
Yes but your monkey was able to start your tractor that one time... And like, what if your neighbor starts using monkeys? He might harvest more than you. Besides, everyone will be using farming monkeys in the future, so better start letting them drive now. You wouldn't like to be known as a monkey free farm, right?
firelizzard@programming.dev · 19 pts · 120d
Fuck around and find out
raven@lemmy.org · 18 pts · 120d
This is fun to read. I hope people will have their actual intelligence activated after this.
ComfortableRaspberry@feddit.org · 13 pts · 120d
They won't but they will continue to accidentally create content for my amusement :D
patruelis@lemmy.world · 18 pts · 120d
Its bound to happen more and more. More concerning, what is it decides to insert unknown code into backups? How are they detected? Who's guarding all if these? Another AI?
pinball_wizard@lemmy.zip · 9 pts · 120d
Exactly. We aren't (and probably won't) even learn about all the subtle poisoning happening, causing waste and data loss.
FellowEnt@sh.itjust.works · 14 pts · 120d
Seems like user error, I'm no programmer but even I lnow you don't give an agent access to critical things, and Claude is very insistent at asking for permission at every step.
pinball_wizard@lemmy.zip · 9 pts · 119d
Yes.
But these models have (largely correctly) learned from Stack Overflow that, on average, every problem is due to not enough permissions.
Someone fully relying on an agentic AI model is essentially destined to give it full control (or close enough), eventually.
At some point, a tool like these LLMs either needs to not be marketed to that user, or needs stupid levels of safety warnings.
My money is on neither solution happening, and this kind of result continuing for the foreseeable future - until the rest of us doing cleanup instigate Dune's Butlerian Jihad to stop the damage and save our own sanity.
Avicenna@programming.dev · 13 pts · 120d
If you are going to give an LLM a free pass to your whole prod database least you should do is to take weekly (or daily if plausible) offline backups of it. A hard limit against deleting stuff would be better.
DarrinBrunner@lemmy.world · 11 pts · 120d
My suggestion is to not give it access to the backups, but may I'm naive that way.
Aceticon@lemmy.dbzer0.com · 11 pts · 120d
Thus completelly eliminating all bugs and data incorrections in Production!
That right there is the sweet smell of Victory!
jlow@slrpnk.net · 9 pts · 120d
🍿
HyperfocusSurfer@lemmy.dbzer0.com · 9 pts · 120d
Efficient, tho.
webp@mander.xyz · 3 pts · 120d
One of us
OhStopYellingAtMe@lemmy.world · 2 pts · 120d
Son of Anton strikes again!
lambisio@feddit.cl · 2 pts · 119d
These cases always, always make me laugh.
Because avoiding them is quite simple.