Everyone always focuses on the Ship of Theseus or the existential parts of this, but the societal impacts would be crazy.
Millions of people teleporting to the most beautiful places on earth and wrecking them, highways and boring parts of cities abandoned, mass death of people teleporting into a crush crowd at Mecca, landmarks toppled under the weight of teleporting tourists and bosses teleporting into your house for a "work from home" meeting, weapons and bombs being teleported into war zones...
Things would eventually find a new normal and teleporting food and resources around would remove the need for ships and trucks, you could explore the world much more easily and there would be lots of science to do in hard to reach places, marine biologists would have a field day exploring the deep ocean, doctors could teleport cancer cells out of patients, space agencies teleport probes, bases and astronauts to the moon and around the solar system...
Like it would be such a huge upheaval with so many implications. It's interesting that people really fixate on the "would I die though? What does dying even mean if I'm still around?" part, even though I think that's maybe the least interesting thing. I guess it taps into a very primal survival instinct?
It's not so much that it'll go rogue and decide to punish people or that it's designed to be good for humanity.
The fact that it's possible to build a machine that punishes people who didn't help create it provides an incentive to deliberately try to create that machine in order to avoid its punishment.
Believers in the basilisk aren't trying to make ai to benefit humanity, they're explicitly trying to create an AI that deliberately punishes those who didn't build it because they fear that, if they don't, someone else will and it will come for them.
The part about it being a magical, omnipotent, time traveling, mind torture device is just framing to make the thought experiment clearer. It's not required for the basilisk to be effective. Even if the punishment is just death and it can't get to everyone, and only goes off limited information, that's still motivation enough for some people.
It is a scary thing, because it's only silly if everyone thinks it's silly and all it takes is one mad billionaire (eg. Elon) to build a lesser basilisk that can cause real harm.
It wasn't actually piracy, so much as ripping off customers.
What they were selling was copies of the shareware demo version of doom, which was supposed to be free. When id found out, they sold them empty boxes to put the shareware in and keep selling it.
If people bought the demo and liked the game, they would buy the full version from id by mail order.
I think they're implying that it's not actually a nation state threat actor, that Russian cyber criminals looking to exploit systems for money are able to act with impunity so long as they don't target Russians (hence "want their bag"). Most incidents are financially motivated.
That's a good idea. Be careful of the more advanced models finding ways to abuse/circumvent it though, even with only a limited set of parameters. Eg. A lot of git commands can be tricked into dropping into less which can drop into a full shell.
They don't require any work to use, have a battery life measured in weeks, not hours, keep all your health data local unless you very specifically set it up to send it to a health app, and are open source and reasonably priced.
Not an advertisement. It's still a US company, but compared with something like a Samsung thing they're pretty cool.
There's a bit of a difference between FLDS style polygamy and modern polyamory.
Polygamy is about being married to multiple people, but to most people generally refers to situations where men have multiple wives. A "polygamous sect" in Utah generally means a man who is worshipped in a cult-like situation by a group of oppressed women. The wives are groomed through the religion from a very young age to believe that their path to heaven lies in obeying their husband who is closer to god. They're taught that it's sinful to think "sinful thoughts" or to question anything their husband says.
In this case he, quite literally, told them all he was god's prophet, the successor to Warren Jeffs, the equivalent of Jesus, and his followers needed to stop thinking and live in worship, give him their daughters to be his wives, give him their money and work for him, let him control every aspect of their lives and thank him for the beautiful opportunity to be allowed to worship a real prophet and be one of god's chosen.
Not the first time this has happened and not the only active polygamous sect around there. It's super fucked up even without the child sex offenses.
On the other hand polyamory is generally about ethical non-monogamy and involves people having multiple loving relationships (married or not). One of the reasons the term was invented (aside from removing the connotations of polygyny and marriage) was to distance polyamorous people from the FLDS which was often the only kind of non-monogamy that people knew about.
the LLM is not doing machine learning while users are using it
This is a small terminology misconception. The LLM is not doing "training" during inference. It's still a "machine learning" system.
In terms of learning/retaining information in the short/mid term while the user is using it, as the context grows, it retains that information during the current session. In a lot of systems, sections of that context are then summarised and stored, indexed by a vector, to be retrieved into future contexts that have similar semantics. That's why some systems seem to be able to "remember" things from previous "conversations". Your message is vectorised and then that vector used to look up similar past interactions. The model isn't fine tuning on that, so it's not "long term" memory, but the model can take it into account for future interactions.
AI companies do then use that (and full conversation histories) to regularly fine tune the models, as well as train new ones. It might not be fresh trained every day but certainly more often than you might think.
to trust an LLM to tell you the truth on your question that you don't know the answer is like trusting some random drunk at the pub
They're a little more reliable than that and are getting significantly more capable at an alarming rate. We absolutely agree that they shouldn't be trusted and are not very accurate (nor should most humans be trusted or are accurate) but I also think it's dangerous to underestimate them.
running the LLM, which is done with a random number generator and a massive matrix of probable next words.
Not true. Inference is done by providing the context to the pre-trained neutral network (technically a transformer network not your daddy's old multilayer perceptron) to generate possible outcomes with logprobs that are then selected based on their likelihood. If it was just frequency-based RNG, they wouldn't have any semantics in the responses and would sound more like traditional Markov chains (like when you mash a button on predictive text and it spits out correct but meaningless gibberish).
If it were just selecting random words from a matrix of probabilities without the network and attentions, it would also be waaay faster and easier to run on a potato.
The stuff about human learning also isn't quite right. There are different types of "learning" and different kinds of memory.
Sleep is generally understood physiologically to be required to formulate long term memory (eg. as described in this paper).
The previous commentator was analogising human short and mid-term memory with LLM context windows (also things like vector databases etc.) and long term memory with retraining/merging/fine tuning of LLMs. It's not totally the same but the analogy is accurate. Brain behaviour is a big influence and inspiration on how machine learning techniques are designed.
Human memory is also notoriously inaccurate and unreliable and tasks done by humans often needs to be double checked and externally verified.
This isn't to say LLMs are trustworthy or reliable. They are not. More that humans think much more highly of themselves than is really warranted.
While this advice is true for all models, when it comes to agentic tasks (add this small feature/write this test harness/find bugs/suggest improvements), open source models are still way behind, vibe code or not.
Claude Fable or even Opus in an editor like Zed have a 1 million token context window and will "think" through the goals of the application, test their changes, work through debugging processes the way a programmer would, stop to ask for clarification, check diagnostic tools and linters, prompt to run test code, etc.
Llama, Gemma and Qwen etc. Do lack a lot of the world knowledge to get the goals of the application, but they also just don't have the debugging skills, won't test their code, don't always tool call correctly, get confused as the context increases and nobody has enough vram to run on large context sizes locally.
They can do autocomplete on small functions but aren't really there for more complex tasks.
On top of that, the biggest problem is that the best open source models are trained and released by the same giant tech conglomerates that have an interest in not competing with their own products. Qwen is Alibaba, Llama is Meta, gpt-oss is OpenAI. Even the more "independent" ones, kimi (Moonshot) and GLM (z.ai) are mostly funded by Alibaba and Tencent. They're released for research and marketing purposes and to please their corporate backers with inflated stock. Almost nobody has the resources to train new models from scratch. People make lots of merges and fine tunes but AI is not democratised the way that traditional programming tools have been.
Maybe some day there will be enough cheap compute for open source communities to pool together resources to build competing models but they're not really there yet :(
I don't think the "companies making ready-to-drink coffee products at industrial scale" that article is talking about this being designed for are running a 1500 watt appliance for 15 minutes a day.
They're thinking more that factories won't need to use traditional extractors which generally need to heat stuff to high temperatures to make coffee milk drinks and soluble instant coffee, etc.
Use the CNA-LR since I don't think they have a CNA.
You were probably trying to do the right thing disclosing, just know that there is a better process for it (even if you think the devs are asshats, it's good to do it like that for the community who aren't).
Even if it only affects admins, that includes admins of forks etc.
I'm sure there's probably more vulnerabilities to find.
Responsible disclosure does not assume the vendor acts in good faith. Usually the disclosure period is around 90 days before the vulnerability is released, fixed or not (although this is negotiable with a good faith vendor).
Forks etc. could have been informed privately first too if possible.
amateurs now have access to tools they should not, and WILL forgo proper standardized communication channels to disclose issues
This is not a good argument. Undisclosed zero days in the wild have always been part of the threat model. Amateurs with LLMs or not, a large percentage of vulnerabilities are not disclosed responsibly and are only fixed after damage has been done. Putting people and their personal information at risk because you want to make a point about the dangers of zero days (which everyone is already aware of) is woefully unethical.
Not everyone is privileged enough to afford security courses, and standardized education.
That doesn't mean we should abandon these things. The vendor can report the CVE too. Or anyone else with an interest in it. It doesn't have to be the untrained amateur grey hat asking Claude for vulns. A malicious threat actor exploiting a system doesn't report it either. The community benefits from skilled people handling things properly. Pretending that it doesn't because most people don't have those skills is silly.
Public disclosure is good, but responsible disclosure usually involves informing the dev first, giving them a period of time to push out a patch and then publicly disclosing for the community to learn from.
Also good to report it to mitre and give it a CVE number.
If writing a lot of bash scripts, I really recommend shellcheck. It's a linter for bash that gives a lot of good advice and points out common issues/inefficiencies and errors.
There's plugins for most editors or you can just run it in a terminal.
I also like that it has good documentation that tells you why something might be wrong or inadvisable.
Yeah. Wikipedia calls it "link aggregation" and the standard is IEEE 802.1AX which also calls it that and the protocol LACP. I think the real reason for so many names is that the standard wasn't developed until later so everyone built their own competing incompatible implementations with different names and it was a mess for years.
Linux implemented it with the Linux bonding driver and switch manufactures made up their own proprietary extensions for it but the standard didn't become a thing until like 2000. Seems like "teaming" is one of the most popular names for it.
Some stuff calls it bonded, sometimes it's teamed, sometimes LAGed or aggregated or bundled or link channelled or ethertrunked or smartgrouped or Multi-link trunked etc. etc.
There are quite a few. The best ones are sustainable closed loop datacenters with on-site solar which is becoming pretty common across the world, especially for new builds. Often producing more power than they need and feeding it back to the grid (especially if the local government has an energy buy back scheme).
But most data centers are pretty tiny and just built into an office building with a bunch of server racks.
Depending on where you live, a quick web search for data centers in your local area will probably show up dozens of them of varying quality hosting people's websites and business apps etc. They aren't any scarier than anything else you find in a city. They're critical infrastructure that helps make the internet a thing. In most cases, if it wasn't a datacenter, it would be a car yard or a factory, etc.
But! There are also truly evil datacenters. Like this insane Utah monstrosity built for a shitty purpose and the size of a freaking city. An obscene monument to the US tech cesspool's hubris.
That's fair. The nuance that people lose is more that people are often painting them all with the same brush. Protesting any datacenter regardless of impact.
It becomes something like: "datacenters are evil and are a symbol of techno fascist distopia! If they build a datacenter in my city, the taps will run dry and Elon Musk will use it to make ai porn of my children!" Even if it's a small solar powered closed loop that provides VPS, storage and web hosting for nerds and small businesses.
I also do think there's also a scale of evil there. Some environmental impacts are not immediately obvious and might not be known about during planning. Some were built a long time ago with older tech and are a bit shitty but have a plan to transition to be more sustainable, etc.
The world is full of "alright but a little bit shit." It's not all perfect angels and mustache twirling villains.
I don't want to detract too much from the real villains though. Nobody needs a 9GW datacity for military ai.
Everyone always focuses on the Ship of Theseus or the existential parts of this, but the societal impacts would be crazy.
Millions of people teleporting to the most beautiful places on earth and wrecking them, highways and boring parts of cities abandoned, mass death of people teleporting into a crush crowd at Mecca, landmarks toppled under the weight of teleporting tourists and bosses teleporting into your house for a "work from home" meeting, weapons and bombs being teleported into war zones...
Things would eventually find a new normal and teleporting food and resources around would remove the need for ships and trucks, you could explore the world much more easily and there would be lots of science to do in hard to reach places, marine biologists would have a field day exploring the deep ocean, doctors could teleport cancer cells out of patients, space agencies teleport probes, bases and astronauts to the moon and around the solar system...
Like it would be such a huge upheaval with so many implications. It's interesting that people really fixate on the "would I die though? What does dying even mean if I'm still around?" part, even though I think that's maybe the least interesting thing. I guess it taps into a very primal survival instinct?
It's not so much that it'll go rogue and decide to punish people or that it's designed to be good for humanity.
The fact that it's possible to build a machine that punishes people who didn't help create it provides an incentive to deliberately try to create that machine in order to avoid its punishment.
Believers in the basilisk aren't trying to make ai to benefit humanity, they're explicitly trying to create an AI that deliberately punishes those who didn't build it because they fear that, if they don't, someone else will and it will come for them.
The part about it being a magical, omnipotent, time traveling, mind torture device is just framing to make the thought experiment clearer. It's not required for the basilisk to be effective. Even if the punishment is just death and it can't get to everyone, and only goes off limited information, that's still motivation enough for some people.
It is a scary thing, because it's only silly if everyone thinks it's silly and all it takes is one mad billionaire (eg. Elon) to build a lesser basilisk that can cause real harm.
It wasn't actually piracy, so much as ripping off customers.
What they were selling was copies of the shareware demo version of doom, which was supposed to be free. When id found out, they sold them empty boxes to put the shareware in and keep selling it.
If people bought the demo and liked the game, they would buy the full version from id by mail order.
Kinda shady tbh.
I think they're implying that it's not actually a nation state threat actor, that Russian cyber criminals looking to exploit systems for money are able to act with impunity so long as they don't target Russians (hence "want their bag"). Most incidents are financially motivated.
That's a good idea. Be careful of the more advanced models finding ways to abuse/circumvent it though, even with only a limited set of parameters. Eg. A lot of git commands can be tricked into dropping into less which can drop into a full shell.
They know all the gtfobins.org tricks.
Pebble are pretty good.
They don't require any work to use, have a battery life measured in weeks, not hours, keep all your health data local unless you very specifically set it up to send it to a health app, and are open source and reasonably priced.
Not an advertisement. It's still a US company, but compared with something like a Samsung thing they're pretty cool.
There's a bit of a difference between FLDS style polygamy and modern polyamory.
Polygamy is about being married to multiple people, but to most people generally refers to situations where men have multiple wives. A "polygamous sect" in Utah generally means a man who is worshipped in a cult-like situation by a group of oppressed women. The wives are groomed through the religion from a very young age to believe that their path to heaven lies in obeying their husband who is closer to god. They're taught that it's sinful to think "sinful thoughts" or to question anything their husband says.
In this case he, quite literally, told them all he was god's prophet, the successor to Warren Jeffs, the equivalent of Jesus, and his followers needed to stop thinking and live in worship, give him their daughters to be his wives, give him their money and work for him, let him control every aspect of their lives and thank him for the beautiful opportunity to be allowed to worship a real prophet and be one of god's chosen.
Not the first time this has happened and not the only active polygamous sect around there. It's super fucked up even without the child sex offenses.
On the other hand polyamory is generally about ethical non-monogamy and involves people having multiple loving relationships (married or not). One of the reasons the term was invented (aside from removing the connotations of polygyny and marriage) was to distance polyamorous people from the FLDS which was often the only kind of non-monogamy that people knew about.
This is a small terminology misconception. The LLM is not doing "training" during inference. It's still a "machine learning" system.
In terms of learning/retaining information in the short/mid term while the user is using it, as the context grows, it retains that information during the current session. In a lot of systems, sections of that context are then summarised and stored, indexed by a vector, to be retrieved into future contexts that have similar semantics. That's why some systems seem to be able to "remember" things from previous "conversations". Your message is vectorised and then that vector used to look up similar past interactions. The model isn't fine tuning on that, so it's not "long term" memory, but the model can take it into account for future interactions.
AI companies do then use that (and full conversation histories) to regularly fine tune the models, as well as train new ones. It might not be fresh trained every day but certainly more often than you might think.
They're a little more reliable than that and are getting significantly more capable at an alarming rate. We absolutely agree that they shouldn't be trusted and are not very accurate (nor should most humans be trusted or are accurate) but I also think it's dangerous to underestimate them.
Not true. Inference is done by providing the context to the pre-trained neutral network (technically a transformer network not your daddy's old multilayer perceptron) to generate possible outcomes with logprobs that are then selected based on their likelihood. If it was just frequency-based RNG, they wouldn't have any semantics in the responses and would sound more like traditional Markov chains (like when you mash a button on predictive text and it spits out correct but meaningless gibberish).
If it were just selecting random words from a matrix of probabilities without the network and attentions, it would also be waaay faster and easier to run on a potato.
The stuff about human learning also isn't quite right. There are different types of "learning" and different kinds of memory.
Sleep is generally understood physiologically to be required to formulate long term memory (eg. as described in this paper).
The previous commentator was analogising human short and mid-term memory with LLM context windows (also things like vector databases etc.) and long term memory with retraining/merging/fine tuning of LLMs. It's not totally the same but the analogy is accurate. Brain behaviour is a big influence and inspiration on how machine learning techniques are designed.
Human memory is also notoriously inaccurate and unreliable and tasks done by humans often needs to be double checked and externally verified.
This isn't to say LLMs are trustworthy or reliable. They are not. More that humans think much more highly of themselves than is really warranted.
While this advice is true for all models, when it comes to agentic tasks (add this small feature/write this test harness/find bugs/suggest improvements), open source models are still way behind, vibe code or not.
Claude Fable or even Opus in an editor like Zed have a 1 million token context window and will "think" through the goals of the application, test their changes, work through debugging processes the way a programmer would, stop to ask for clarification, check diagnostic tools and linters, prompt to run test code, etc.
Llama, Gemma and Qwen etc. Do lack a lot of the world knowledge to get the goals of the application, but they also just don't have the debugging skills, won't test their code, don't always tool call correctly, get confused as the context increases and nobody has enough vram to run on large context sizes locally.
They can do autocomplete on small functions but aren't really there for more complex tasks.
On top of that, the biggest problem is that the best open source models are trained and released by the same giant tech conglomerates that have an interest in not competing with their own products. Qwen is Alibaba, Llama is Meta, gpt-oss is OpenAI. Even the more "independent" ones, kimi (Moonshot) and GLM (z.ai) are mostly funded by Alibaba and Tencent. They're released for research and marketing purposes and to please their corporate backers with inflated stock. Almost nobody has the resources to train new models from scratch. People make lots of merges and fine tunes but AI is not democratised the way that traditional programming tools have been.
Maybe some day there will be enough cheap compute for open source communities to pool together resources to build competing models but they're not really there yet :(
I don't think the "companies making ready-to-drink coffee products at industrial scale" that article is talking about this being designed for are running a 1500 watt appliance for 15 minutes a day.
They're thinking more that factories won't need to use traditional extractors which generally need to heat stuff to high temperatures to make coffee milk drinks and soluble instant coffee, etc.
They don't need to have one.
You can report it here: https://cveform.mitre.org/
Use the CNA-LR since I don't think they have a CNA.
You were probably trying to do the right thing disclosing, just know that there is a better process for it (even if you think the devs are asshats, it's good to do it like that for the community who aren't).
Even if it only affects admins, that includes admins of forks etc.
I'm sure there's probably more vulnerabilities to find.
Responsible disclosure does not assume the vendor acts in good faith. Usually the disclosure period is around 90 days before the vulnerability is released, fixed or not (although this is negotiable with a good faith vendor).
Forks etc. could have been informed privately first too if possible.
This is not a good argument. Undisclosed zero days in the wild have always been part of the threat model. Amateurs with LLMs or not, a large percentage of vulnerabilities are not disclosed responsibly and are only fixed after damage has been done. Putting people and their personal information at risk because you want to make a point about the dangers of zero days (which everyone is already aware of) is woefully unethical.
That doesn't mean we should abandon these things. The vendor can report the CVE too. Or anyone else with an interest in it. It doesn't have to be the untrained amateur grey hat asking Claude for vulns. A malicious threat actor exploiting a system doesn't report it either. The community benefits from skilled people handling things properly. Pretending that it doesn't because most people don't have those skills is silly.
Public disclosure is good, but responsible disclosure usually involves informing the dev first, giving them a period of time to push out a patch and then publicly disclosing for the community to learn from.
Also good to report it to mitre and give it a CVE number.
If writing a lot of bash scripts, I really recommend shellcheck. It's a linter for bash that gives a lot of good advice and points out common issues/inefficiencies and errors. There's plugins for most editors or you can just run it in a terminal. I also like that it has good documentation that tells you why something might be wrong or inadvisable.
https://github.com/koalaman/shellcheck
Yeah. Wikipedia calls it "link aggregation" and the standard is IEEE 802.1AX which also calls it that and the protocol LACP. I think the real reason for so many names is that the standard wasn't developed until later so everyone built their own competing incompatible implementations with different names and it was a mess for years.
Linux implemented it with the Linux bonding driver and switch manufactures made up their own proprietary extensions for it but the standard didn't become a thing until like 2000. Seems like "teaming" is one of the most popular names for it.
Why does this have so many names?
Some stuff calls it bonded, sometimes it's teamed, sometimes LAGed or aggregated or bundled or link channelled or ethertrunked or smartgrouped or Multi-link trunked etc. etc.
There are quite a few. The best ones are sustainable closed loop datacenters with on-site solar which is becoming pretty common across the world, especially for new builds. Often producing more power than they need and feeding it back to the grid (especially if the local government has an energy buy back scheme).
But most data centers are pretty tiny and just built into an office building with a bunch of server racks.
Depending on where you live, a quick web search for data centers in your local area will probably show up dozens of them of varying quality hosting people's websites and business apps etc. They aren't any scarier than anything else you find in a city. They're critical infrastructure that helps make the internet a thing. In most cases, if it wasn't a datacenter, it would be a car yard or a factory, etc.
But! There are also truly evil datacenters. Like this insane Utah monstrosity built for a shitty purpose and the size of a freaking city. An obscene monument to the US tech cesspool's hubris.
I mean they're not all for AI and they're not all environmentally devastating.
This one very much is.
That's fair. The nuance that people lose is more that people are often painting them all with the same brush. Protesting any datacenter regardless of impact.
It becomes something like: "datacenters are evil and are a symbol of techno fascist distopia! If they build a datacenter in my city, the taps will run dry and Elon Musk will use it to make ai porn of my children!" Even if it's a small solar powered closed loop that provides VPS, storage and web hosting for nerds and small businesses.
I also do think there's also a scale of evil there. Some environmental impacts are not immediately obvious and might not be known about during planning. Some were built a long time ago with older tech and are a bit shitty but have a plan to transition to be more sustainable, etc.
The world is full of "alright but a little bit shit." It's not all perfect angels and mustache twirling villains.
I don't want to detract too much from the real villains though. Nobody needs a 9GW datacity for military ai.