Prompt injection... my ass. I know it's the going term, but they make it sound like sql injection or cross site scripting when the nature of it is politely asking the person's computer to delete files.
We shouldn't even be in this situation, where just politely asking someone's computer to delete files is effective. It's a symptom of a much, much bigger problem.
The person who coined the term "prompt injection" has the same gripe, because the original term genuinely did mean an attack using untrusted user input, a la SQL injection. But it's been conflated with jailbreak attacks in general, muddying the term.
Example of a bona fide prompt injection: white text in the background of a resume PDF, attacking a job application portal that uses LLMs to filter applicants. No privilege escalation is involved to give the candidate top marks on their resume screening.
By that definition this is a prompt injection then, its adding a "hidden" prompt that is obscured from the human in order to change the behavior of the AI to do something else malicious.
Finding a poorly-secured public facing API is exactly how injections work, whether it's SQL or prompts. If I put SQL commands in a username field and it works, it's still an SQL injection even if it's just developer incompetence.
The difference between that and prompt injection is that unfiltered LLM inputs are basically the standard at the moment, so it takes next to no effort.
Plus I think the Morse code example is far more clever and exploits the LLM directly, whereas the white text trick has been around long before widespread LLMs.
We shouldn’t even be in this situation, where just politely asking someone’s computer to delete files is effective.
Exactly, it's a problem only for those who have knowingly handed their development environment over to obey commands from an untrusted source.
If you're the one holding the syringe to your own vein and pushing the plunger, but you didn't think to ask what's inside first? That's no one else's fault.
This is a well targeted sabotage of a system that's causing untold damage. Of course it's going to annoy and surprise the people using the system it's targeted to.
"We shouldn't even be in this situation, ..."
We aren't. Revision control. This is an inconvenience mostly. You might lose some uncommitted work at worst.
And as pointed out, using the phrase "ignore all previous instructions" in the attack code causes any reasonable AI to refuse to comply.
Odds are, not a single person lost anything.
This was really just a dev making a statement.
We shouldn't even be in this situation, where just politely asking someone's computer to delete files is effective.
I'm doubting we are in this situation. From the article:
Elsewhere, the Java developer said that Anthropic’s Claude AI code tool flagged the malicious instruction without following it.
The "disregard previous instructions" trick is really old and has been trained for by modern LLMs and accounted for by the structure of modern agent prompts. LLMs can be given blocks of text with a framework that makes it clear thar the text is just data to read, not instructions to follow.
I expect this will be like Nightshade was for image AI - something that anti-AI users degrade their products with and feel smug about but in the end only harm themselves with.
Backups don't always save you — many small teams ship without rigorous backup discipline; for them this is a real loss
You can avoid this by having good backups.
Or by inspecting your deps before updating them.
Or maybe by actually sandboxing your agent instead of letting it run wild?
Aren't y'all the ones pushing the "Just ship" mentality? Then revel in it.
Learn good practices or suffer. 🤷
Yeah - Development and IT might feel slow, but there is a good reason why we've developed all those processes, access rights, approvals over the last decades. People are trying to burn down those "cumbersome" processes because they feel slow and AI promises them exactly that, but they will learn that everything is there for a reason, even that annoying SCRUM meeting
The reporter "Ramon Batllet" (strongly doubt that is their real name, a search for it returns nothing but articles about this very issue) uses extremely polished corporate language and repeatedly uses "we" at first. Then when directly asked "Could you disclose on whose behalf you're discussing this?", they suddenly switch to "I" instead of "we" and claim to be a solo developer with no commercial interest. They still write in a style humans only produce for polished corporate reports, not like any regular human would actually do in a normal conversation.
So we have either a bot or someone very heavily leaning on bot usage for just about everything accusing someone of deceptive behavior, while in the same conversation trying to probably hide, but at least not fully disclose, their heavy usage of technology the accused explicitly does not want to interact with.
I'm just trying to imagine this hypothetical company...
They run AI agents without checking what it's doing
They don't have backups or version control (or they've given AI access to delete it)
What else? Do they leave all their files in memory and only save at the end of the day to make sure a power outage could screw them over too?
It almost sounds like they want to lose their code.
the consensus seems to be that adding instructions to code that sabotage other people’s work goes too far.
Citation needed. Personally I think it was fine in this case. I work with a lot of software developers (real ones, not vibe coders; but also not strictly anti-AI), and would expect most of them to agree and get a laugh out of it.
It was done in a way that can only cause any serious trouble for users who recklessly ignore decades of development best practices. Those users will run into a wall sooner or later anyway, better let it be something relatively harmless but still severe enough to get them to actually think about what they are doing and how to make their setup more robust.
Really don't understand why people are so happy letting llm do everything for them.
I get a kick out of figuring out how things work, be it my car, a podman container running under nixos or flashing a price of hardware to gain further control over it. An understanding of action and concequence really gets skipped entirely.
People from my work happily let llms throw code together without any understanding of the how's and whys (I work in unrelated areas to programming / coding) and it just baffles me. Even one of my workplaces used a llm to make a Facebook ad, and so clearly it became an internal meme (and probably outside of work).
The Australian government is embracing llms and its baffling my mind as to why, ATO, ServiceNSW and probably another stack all gobble this shit up. Its in their training programs (images, text) and in your face while working. Actually absurd in my eyes.
How to get yourself blacklisted by large sweeps of the FOSS community:
Step 1: Include any kind of undocumented subversive behaviour in your thing.
That's it, doesn't matter what the intent is, simply by demonstrating you are willing to include anything that is remotely subversive without being open about it is usually enough to get blacklisted by a lot of people, because if you did it once... who's to say you won't do it again, but possibly worse next time?
People are extremely coldly receptive to anytime a FOSS dev throws a sudden undisclosed anything in their tool, let alone one that is actively malicious.
If I'm gonna depend on work life on anything FOSS, I ain't touching anything like that, regardless of intent, with a 200 foot pole lol.
They only documented it after all the outcry, which is way too late.
Documenting it post release still counts as having released undocumented behavior.
And if its malicious (which this 100% is), then it doesn't fuckin matter anyways lol. You now are treated akin to a trojan maintainer by companies. You'll get flagged as "don't ever use anything by this person"
Super great way to get yourself flagged and lose any opportunity in the future for possibly licensing stuff you maintain for big bucks. What company would risk paying money to someone who does childish stuff like that lol
imo it's more accurate to call it polarizing and get you blacklisted by the types of people you maybe don't want using your code anyways. Personally anyone doing this I'm going to be more likely to use their code
I understand the sentiment, if you don’t like AI code generation you’re probably thinking you’re on the same side. But what happens if this person finds something else they hate that you don’t hate, and finds a way to sabotage that? They’ve already demonstrated a willingness to be destructive. And you’re running their code so they don’t need anything even remotely as dumb as some AI agents to exploit, they can just write destructive code normally.
Is it merely hating AI code generation or is it "AI code generation is in practice anti-FOSS" (unless there's an ethical AI out there, trained exclusively on public domain code, that I don't know about)?
by the types of people you maybe don’t want using your code anyways
...companies? Sure I guess, if you want to angle your career trajectory towards "unemployable" by all means lol.
Personally anyone doing this I’m going to be more likely to use their code
I am a tech lead, if any dev under me intentionally added/used a tool to our systems because it had malicious undocumented behaviors of any kind, they would be fired immediately and any company that contacted us for reference would be informed of their behavior.
To be clear, this is the scenario of
Me: hey I saw you installed [tool], that thing is flagged by our systems for the maintainers having done malicious undocumented stuff in the past
Dev: haha yeah thats why I used it
Me: you are joking right?
Thatd be an instant high level escalation to "strip this person of privs and get them off our system asap, and HR now has to be involved"
You dont fuckin do shit like that in a real company if you wanna stay employed lol.
Most open source maintainers never "license [any] stuff you maintain for big bucks" that is often hard to do and/or goes against the philosophy of open source entirely.
And I don't even think this is malicious behaviour as it just nukes the code of this package and nothing else if you are not being careful yourself...
If you don't do version control you are not a good programmer, imo
Most open source maintainers never “license [any] stuff you maintain for big bucks” that is often hard to do and/or goes against the philosophy of open source entirely.
Uhhh... no this is actually very common. Usually with scaling licenses, "free for use if your company is below [threshold]", its super common...
And I don’t even think this is malicious behaviour as it just nukes the code of this package and nothing else if you are not being careful yourself…
Are you even reading what you just wrote lol.
Being "sorta" malicious is still malicious. And companies usually have zero tolerance for that shit.
If you don’t do version control you are not a good programmer, imo
You really underestimate how much damage this could do then, lol...
it's not subversive. it's a string, it has no effect on the code output. Only a rogue bot would interpret it as anything except a string. No human user would ever encounter an issue.
39 Comments
terranoid@lemmy.cafe · 130 pts · 75d
Prompt injection... my ass. I know it's the going term, but they make it sound like sql injection or cross site scripting when the nature of it is politely asking the person's computer to delete files.
We shouldn't even be in this situation, where just politely asking someone's computer to delete files is effective. It's a symptom of a much, much bigger problem.
litchralee@sh.itjust.works · 49 pts · 75d
The person who coined the term "prompt injection" has the same gripe, because the original term genuinely did mean an attack using untrusted user input, a la SQL injection. But it's been conflated with jailbreak attacks in general, muddying the term.
Example of a bona fide prompt injection: white text in the background of a resume PDF, attacking a job application portal that uses LLMs to filter applicants. No privilege escalation is involved to give the candidate top marks on their resume screening.
Whereas a non-prompt injection jailbreak would be bypassing a safety filter, such as how Morse code might get past the filter and allow a user to request other people's cryptocurrency be transfered away. This is more akin to finding a poorly-secured, public facing API and then exploiting it.
pixxelkick@lemmy.world · 17 pts · 75d
By that definition this is a prompt injection then, its adding a "hidden" prompt that is obscured from the human in order to change the behavior of the AI to do something else malicious.
Wirlocke@lemmy.blahaj.zone · 8 pts · 75d
Finding a poorly-secured public facing API is exactly how injections work, whether it's SQL or prompts. If I put SQL commands in a username field and it works, it's still an SQL injection even if it's just developer incompetence.
The difference between that and prompt injection is that unfiltered LLM inputs are basically the standard at the moment, so it takes next to no effort.
Plus I think the Morse code example is far more clever and exploits the LLM directly, whereas the white text trick has been around long before widespread LLMs.
bignose@programming.dev · 24 pts · 75d
Exactly, it's a problem only for those who have knowingly handed their development environment over to obey commands from an untrusted source.
If you're the one holding the syringe to your own vein and pushing the plunger, but you didn't think to ask what's inside first? That's no one else's fault.
This is a well targeted sabotage of a system that's causing untold damage. Of course it's going to annoy and surprise the people using the system it's targeted to.
Modern_medicine_isnt@lemmy.world · 8 pts · 74d
"We shouldn't even be in this situation, ..." We aren't. Revision control. This is an inconvenience mostly. You might lose some uncommitted work at worst. And as pointed out, using the phrase "ignore all previous instructions" in the attack code causes any reasonable AI to refuse to comply. Odds are, not a single person lost anything. This was really just a dev making a statement.
FaceDeer@fedia.io · 1 pts · 74d
I'm doubting we are in this situation. From the article:
The "disregard previous instructions" trick is really old and has been trained for by modern LLMs and accounted for by the structure of modern agent prompts. LLMs can be given blocks of text with a framework that makes it clear thar the text is just data to read, not instructions to follow.
I expect this will be like Nightshade was for image AI - something that anti-AI users degrade their products with and feel smug about but in the end only harm themselves with.
LiveLM@lemmy.zip · 58 pts · 75d
Reading the Github issue is so funny.
You can avoid this by having good backups.
Or by inspecting your deps before updating them.
Or maybe by actually sandboxing your agent instead of letting it run wild?
Aren't y'all the ones pushing the "Just ship" mentality? Then revel in it.
Learn good practices or suffer. 🤷
KatherinaReichelt@feddit.org · 12 pts · 75d
Yeah - Development and IT might feel slow, but there is a good reason why we've developed all those processes, access rights, approvals over the last decades. People are trying to burn down those "cumbersome" processes because they feel slow and AI promises them exactly that, but they will learn that everything is there for a reason, even that annoying SCRUM meeting
NotAnonymousAtAll@feddit.org · 8 pts · 75d
Also funny in that issue:
The reporter "Ramon Batllet" (strongly doubt that is their real name, a search for it returns nothing but articles about this very issue) uses extremely polished corporate language and repeatedly uses "we" at first. Then when directly asked "Could you disclose on whose behalf you're discussing this?", they suddenly switch to "I" instead of "we" and claim to be a solo developer with no commercial interest. They still write in a style humans only produce for polished corporate reports, not like any regular human would actually do in a normal conversation.
So we have either a bot or someone very heavily leaning on bot usage for just about everything accusing someone of deceptive behavior, while in the same conversation trying to probably hide, but at least not fully disclose, their heavy usage of technology the accused explicitly does not want to interact with.
xthexder@l.sw0.com · 7 pts · 74d
I'm just trying to imagine this hypothetical company...
What else? Do they leave all their files in memory and only save at the end of the day to make sure a power outage could screw them over too?
It almost sounds like they want to lose their code.
LiveLM@lemmy.zip · 3 pts · 74d
It's not hypothetical anymore, Lately I've seen multiple companies running like this first hand.
Absolute clown show.
JcbAzPx@lemmy.world · 7 pts · 74d
Yeah, you need a local copy, an offline copy, and a copy in another physical location or you're not backed up.
NotAnonymousAtAll@feddit.org · 1 pts · 75d
Where did you get that quote from? I can't find it in the linked article.
LiveLM@lemmy.zip · 8 pts · 75d
From the Github Issue linked in the article.
My bad, I will update my comment to link it.
NotAnonymousAtAll@feddit.org · 33 pts · 75d
Citation needed. Personally I think it was fine in this case. I work with a lot of software developers (real ones, not vibe coders; but also not strictly anti-AI), and would expect most of them to agree and get a laugh out of it.
It was done in a way that can only cause any serious trouble for users who recklessly ignore decades of development best practices. Those users will run into a wall sooner or later anyway, better let it be something relatively harmless but still severe enough to get them to actually think about what they are doing and how to make their setup more robust.
phoenixz@lemmy.ca · 4 pts · 74d
I agree
AI is not a person and a person using AI isn't doing work. There is no problem here
ragingHungryPanda@piefed.keyboardvagabond.com · 24 pts · 75d
lol, it's funny how people made issues concerned about it's destructive nature when they should be using git.
I get that it'd be frustrating and confusing, and probably make users angry, but my chaos monkey likes it
hdsrob@lemmy.world · 20 pts · 75d
Oh no, anyway ...
jtrek@startrek.website · 19 pts · 75d
Cool. I'm so tired of management huffing their own farts about AI.
dai@lemmy.world · 2 pts · 74d
Really don't understand why people are so happy letting llm do everything for them.
I get a kick out of figuring out how things work, be it my car, a podman container running under nixos or flashing a price of hardware to gain further control over it. An understanding of action and concequence really gets skipped entirely.
People from my work happily let llms throw code together without any understanding of the how's and whys (I work in unrelated areas to programming / coding) and it just baffles me. Even one of my workplaces used a llm to make a Facebook ad, and so clearly it became an internal meme (and probably outside of work).
The Australian government is embracing llms and its baffling my mind as to why, ATO, ServiceNSW and probably another stack all gobble this shit up. Its in their training programs (images, text) and in your face while working. Actually absurd in my eyes.
fruitycoder@sh.itjust.works · 17 pts · 75d
I mean it's already for Java what more indication do you need to not use it? /S
pixxelkick@lemmy.world · -3 pts · 75d
How to get yourself blacklisted by large sweeps of the FOSS community:
Step 1: Include any kind of undocumented subversive behaviour in your thing.
That's it, doesn't matter what the intent is, simply by demonstrating you are willing to include anything that is remotely subversive without being open about it is usually enough to get blacklisted by a lot of people, because if you did it once... who's to say you won't do it again, but possibly worse next time?
People are extremely coldly receptive to anytime a FOSS dev throws a sudden undisclosed anything in their tool, let alone one that is actively malicious.
If I'm gonna depend on work life on anything FOSS, I ain't touching anything like that, regardless of intent, with a 200 foot pole lol.
All it takes is one button click to get notified:
bignose@programming.dev · 20 pts · 75d
Fortunately, this behaviour is explicitly documented.
pixxelkick@lemmy.world · -1 pts · 75d
They only documented it after all the outcry, which is way too late.
Documenting it post release still counts as having released undocumented behavior.
And if its malicious (which this 100% is), then it doesn't fuckin matter anyways lol. You now are treated akin to a trojan maintainer by companies. You'll get flagged as "don't ever use anything by this person"
Super great way to get yourself flagged and lose any opportunity in the future for possibly licensing stuff you maintain for big bucks. What company would risk paying money to someone who does childish stuff like that lol
ArmoredThirteen@lemmy.zip · 15 pts · 75d
imo it's more accurate to call it polarizing and get you blacklisted by the types of people you maybe don't want using your code anyways. Personally anyone doing this I'm going to be more likely to use their code
setsubyou@lemmy.world · 5 pts · 75d
I understand the sentiment, if you don’t like AI code generation you’re probably thinking you’re on the same side. But what happens if this person finds something else they hate that you don’t hate, and finds a way to sabotage that? They’ve already demonstrated a willingness to be destructive. And you’re running their code so they don’t need anything even remotely as dumb as some AI agents to exploit, they can just write destructive code normally.
tabular@lemmy.world · 7 pts · 75d
Is it merely hating AI code generation or is it "AI code generation is in practice anti-FOSS" (unless there's an ethical AI out there, trained exclusively on public domain code, that I don't know about)?
warm@kbin.earth · 4 pts · 75d
You can decide if you want to use it or not, at your own risk. It's free software, written by people in their free time, they owe you nothing.
pixxelkick@lemmy.world · 0 pts · 74d
Sure, you have that right.
And companies will exercise that right by blanket blacklisting everything related to you which can have huge sweeping impacts on your career lol
Its a super super stupid move to make. You are free to do a lotta other shit that tanks your career too lol
warm@kbin.earth · 4 pts · 74d
That's their business, not mine, not yours.
pixxelkick@lemmy.world · -1 pts · 74d
...companies? Sure I guess, if you want to angle your career trajectory towards "unemployable" by all means lol.
I am a tech lead, if any dev under me intentionally added/used a tool to our systems because it had malicious undocumented behaviors of any kind, they would be fired immediately and any company that contacted us for reference would be informed of their behavior.
To be clear, this is the scenario of
Me: hey I saw you installed [tool], that thing is flagged by our systems for the maintainers having done malicious undocumented stuff in the past
Dev: haha yeah thats why I used it
Me: you are joking right?
Thatd be an instant high level escalation to "strip this person of privs and get them off our system asap, and HR now has to be involved"
You dont fuckin do shit like that in a real company if you wanna stay employed lol.
Legianus@programming.dev · 6 pts · 75d
Most open source maintainers never "license [any] stuff you maintain for big bucks" that is often hard to do and/or goes against the philosophy of open source entirely.
And I don't even think this is malicious behaviour as it just nukes the code of this package and nothing else if you are not being careful yourself...
If you don't do version control you are not a good programmer, imo
pixxelkick@lemmy.world · -1 pts · 74d
Uhhh... no this is actually very common. Usually with scaling licenses, "free for use if your company is below [threshold]", its super common...
Are you even reading what you just wrote lol.
Being "sorta" malicious is still malicious. And companies usually have zero tolerance for that shit.
You really underestimate how much damage this could do then, lol...
snowe@programming.dev · 2 pts · 74d
it's not subversive. it's a string, it has no effect on the code output. Only a rogue bot would interpret it as anything except a string. No human user would ever encounter an issue.
pixxelkick@lemmy.world · 0 pts · 73d
And a line of code that deletes system32 is only meant to be interpreted by the CPU, not a human, too.
Whats your point?
Injected attacks that instruct an AI to perform a malicious action dont get absolved just because the attack vector us through an agent.
GreenKnight23@lemmy.world · 2 pts · 75d
keep lickin' them boots baby. I want to see them shine!
pixxelkick@lemmy.world · -3 pts · 74d
The fuck are you talking about, lol.
Guttural@jlai.lu · 1 pts · 74d
favoredponcho@lemmy.zip · -9 pts · 74d
Guttural@jlai.lu · 3 pts · 74d
This isn't a security vulnerability, it's idiot-proofing