Like, if a person tried to pass off software that did this it would have significant career implications, right?
Typically no.
The space of incredibly obvious failures is vast, and this kind of plumbing code isn't always written by someone who's been around the block enough times to think about what kinds of things can go wrong. I've written worse code when I had only a few years of experience.
However companies should know this; so ideally the tool would have gone through a launch review which should have kicked off a security review where a security expert should have read about the git checkout in the design document and started asking questions like "what happens if the repository is taken over" or "why are hashes and branches and tags all in the same field"?
Of course security experts who think about stuff like supply chain attacks are expensive and slow down the darling vibe-coding workflows of Silicon Valley so...
Aside: even without this particular vulnerability, SHA-1 is considered weak -- https://git-scm.com/docs/hash-function-transition, so that should have been thought about as well. Dear supply-chain attackers: maybe there's still a hole here! Good luck!
Although no specific errors in this story have so far been identified
Well here's the very start of the retracted article:
On the screen, two side-by-side squares of color flicker rapidly
Except oops it was referring to a study that used groups of moving dots rather than "squares". This can be seen in that studies first figure. Bam. A specific error in the first sentence...
It is amazing to me how many companies love setting money on fire. And instead of, I don't know, funding a library or something they do nonsense like this.
FWIW, I literally lost my house last week. I've put everything into making sure that AI is not institutionally captured by capital.
Yikes.
machines will give a harder fight than trans people have
Rude.
Your natural rights are so protected by law that you had to flee America. Some of us stay behind and our natural rights do not go away simply because the law no longer protects us. Instead of running away, some of us must stay to fight.
Also rude.
This is my bet for real time continuous learning AI infrastructure. We're gearing up for a major update. You'll be able to login with your Bsky / Atproto account. It's not our AI, it's yours. Open source, open weights, and it learns what you want it to. www.hyprstream.com
Overall the weirdest advertisement I've read in awhile.
I'm tired of hearing the same near future sci-fi political thriller story over and over. When are they going to get new material? Ghost in the Shell did it better anyway. Sigh. I might as well do the requisite sneering once again.
Mid 2025: Advertisements for computer-using agents emphasize the term "personal assistant".
You could even call them a personal digital assistant. Or PDA for short. Maybe Chat-GPT should rebrand as something that emphasizes this, they could call it "Palm Pilot".
Mid 2025: AI personal assistants can be prompted with tasks like "order me a burrito on DoorDash".
Hunger pains interrupting your game? Order pizza while playing Everquest II!
September 2027: Agent-4 doesn't do anything dramatic like try to escape its datacenter.
You know... I think I'll give them this one.
October 2027: The NYT article cites evaluations showing Agent-4's off-the-charts bioweapons capabilities, persuasion abilities, and ability to automate most white-collar jobs.
lmao
Anyway that aside: Petition to make the Daniel Curve a term of art. I want to hear less talk from the tech visionaries about how we're in the gentle singularity or the soft-serve singularity or maybe kinda sorta approaching the singularity if you squint. I want to hear more talk about where exactly we are in the Daniel Curve.
This may be being A/B tested, so if you can't reproduce it here is one of the worse results I got:
::: spoiler slop
:::
Here the "source" it gave me was the same URL it was hosted on. When I clicked through to it it just gave me different slop.
I've seen some bad decisions in the AI mania, but "let's take our entire corpus of professionally written technical writing and replace it with slop" managed to surprise me.
I've never heard of this AI spec driven development, but I've had the displeasure of reading a few obviously AI generated design documents and they are worse than useless. I always end up more confused about what the author wants after reading them than before.
There was a cryptocurrency wallet hack the other day which has so far gone unremarked here because the popcorn is stale and we all pivoted to AI. And yet there are dozens of voices of bagholders crying out in pain wanting to be heard. This is their story.
COLDCARD firmware contains an RNG integration error that causes ngu.random to use MicroPython's deterministic Yasmarang fallback instead of the STM32 hardware RNG.
Oh hey they got the most important thing not to get wrong wrong in the regulation free money settlement layer of the future.
The bug? Well in one file they wrote #define MICROPY_HW_ENABLE_RNG 0, and a library they depended on had this code:
This should have caused a build failure, but since MICROPY_HW_ENABLE_RNG was defined, to zero, it did not and instead continued down an insecure codepath.
As a C++ programmer this isn't an unheard of sort of mistake to make in the moment. Many libraries use undefined for preprocessor variables, but there are some people who also use zero.
But it's less excusable to not catch it when this is literally the most important thing to get right in a bitcoin wallet. Did they not trace the code path when writing the code? Did they have no integration tests or unit tests for RNG seed quality? Poor RNG seeding has come up a ton in literature, so is the sort of thing people writing cryptographic code should check for...
We'll have to addend that old saying: "don't roll your own crypto unless the other person's crypto is from a blockchain developer in which case you're probably better off rolling your own after all."
I personally don't want to sneer at the bagholders too much here. It's 2026 and at this point it just feels kinda sad. But I'll end this comment with this glowing review:
I thought I was secure because Cold Card was always praised as one of the best and most secure wallets. It’s open source, so anyone can verify.
"It's open source, so anyone can verify" -- unfortunately for them the verifier was also a hacker.
On land, cooling and water are among a site's largest costs.
Oh gosh that sounds... uhmm not too bad actually? I wonder what the largest costs are at sea with a floating nuclear reactor in salt water and tugged around by large ships.
It sure is a zany coincidence that all the major American AI companies have come out either for or against open weight AI models with grand ethical and/or patriotic concerns all at the same time isn't it?
I'm just imagining a WSJ reader reading this and thinking "what the heck is this man talking about". At some point you need a little structure or text to balance out the subtext.
Maybe they should start applying some of that superhuman box-escaping AI to their own sandbox designs because if your sandbox escape is a bug in a "package registry cache proxy" you might be doing it wrong.
Apparently it's cheaper to foist the job of waking passengers up on 911 than to hire an employee to do it. Good ol' privatize the gains / socialize the losses!
Oh no every day sneerclub teaches me about a new guy.
So he's basically arguing that the Borg did nothing wrong (note: the Borg do not actually exist, note #2: the Borg are space fascists).
Worthy Successor: A posthuman intelligence so capable and morally valuable that you would gladly prefer that it (not humanity) determine the future path of life itself.
This isn't the first time I've seen this idea in rationalist circles: that is bad / wrong / irrational to be biased in favor of humans and or humans to succeed in particular. Often the idea is taken as obviously true, but I cannot understand it at all.
Like I don't care if you can make a computer "better" than me or less sad than me or less likely to eat an innocent shrimp than me. I am not that computer-- if it's a choice between me and them I'm fighting on team me. There is nothing wrong with wanting to make one's space in the universe or to "selfishly" want self-actualization or to want humans to thrive.
One probable outcome of an open-weight-model-dominant world is full AI communism
full AI communism
At some point the models will be capable enough that you will notice. "A nonliving, invisible, dangerous, and infinitely self-replicating agent escaped from a Chinese lab," you say? Color me shocked.
Listen I'm all for hating on the Chinese government, but it's more enjoyable to hate on them for real reasons rather than whatever this is.
Second mistake: I was relatively imprecise, writing, as I usually do, for a fairly high-context audience that was inclined to give me grace rather than pick apart every word
Oh no the lesson he took away from his tweet is that he didn't use enough words.
Typically no.
The space of incredibly obvious failures is vast, and this kind of plumbing code isn't always written by someone who's been around the block enough times to think about what kinds of things can go wrong. I've written worse code when I had only a few years of experience.
However companies should know this; so ideally the tool would have gone through a launch review which should have kicked off a security review where a security expert should have read about the git checkout in the design document and started asking questions like "what happens if the repository is taken over" or "why are hashes and branches and tags all in the same field"?
Of course security experts who think about stuff like supply chain attacks are expensive and slow down the darling vibe-coding workflows of Silicon Valley so...
Aside: even without this particular vulnerability, SHA-1 is considered weak -- https://git-scm.com/docs/hash-function-transition, so that should have been thought about as well. Dear supply-chain attackers: maybe there's still a hole here! Good luck!
Ah yes I always wanted to read an article in the form of a lot of teeny tiny boxes with lots and lots of janky scrolling.
Well here's the very start of the retracted article:
Except oops it was referring to a study that used groups of moving dots rather than "squares". This can be seen in that studies first figure. Bam. A specific error in the first sentence...
(Am I being too pedantic?)
It is amazing to me how many companies love setting money on fire. And instead of, I don't know, funding a library or something they do nonsense like this.
We Should Improve Society Somewhat .jpg
Yikes.
Rude.
Also rude.
Overall the weirdest advertisement I've read in awhile.
"Our last spy got addicted to superhumanly persuasive AI generated anime girls and quit. So dang we're like plum out of spies now."
Good luck proving this prediction true or false given that computer algorithms do not have a concept of time separate from the real one lol.
I'm tired of hearing the same near future sci-fi political thriller story over and over. When are they going to get new material? Ghost in the Shell did it better anyway. Sigh. I might as well do the requisite sneering once again.
You could even call them a personal digital assistant. Or PDA for short. Maybe Chat-GPT should rebrand as something that emphasizes this, they could call it "Palm Pilot".
Hunger pains interrupting your game? Order pizza while playing Everquest II!
You know... I think I'll give them this one.
lmao
Anyway that aside: Petition to make the Daniel Curve a term of art. I want to hear less talk from the tech visionaries about how we're in the gentle singularity or the soft-serve singularity or maybe kinda sorta approaching the singularity if you squint. I want to hear more talk about where exactly we are in the Daniel Curve.
Google is seemingly experimenting with replacing their help articles with LLM text.
I was searching for how to use multiple profiles on Chrome and found https://support.google.com/chrome/answer/2364824?hl=en&co=GENIE.Platform%3DDesktop in the search results, which used to look like a normal useful help article, however it's now been replaced by an LLM that takes upwards of 30 seconds to load, will randomly just refuse to answer the question, and produces significantly worse output than the existing (now inaccessible to me) technical writer written article.
This may be being A/B tested, so if you can't reproduce it here is one of the worse results I got:
::: spoiler slop
:::
Here the "source" it gave me was the same URL it was hosted on. When I clicked through to it it just gave me different slop.
I've seen some bad decisions in the AI mania, but "let's take our entire corpus of professionally written technical writing and replace it with slop" managed to surprise me.
I've never heard of this AI spec driven development, but I've had the displeasure of reading a few obviously AI generated design documents and they are worse than useless. I always end up more confused about what the author wants after reading them than before.
It is the worst and I want to retire.
There was a cryptocurrency wallet hack the other day which has so far gone unremarked here because the popcorn is stale and we all pivoted to AI. And yet there are dozens of voices of bagholders crying out in pain wanting to be heard. This is their story.
For some reason the best explanation comes from a boring company that makes point of sale credit card readers https://engineering.block.xyz/blog/predictable-rng-fallback-and-32-bit-reseed-in-coldcard-firmware
Oh hey they got the most important thing not to get wrong wrong in the regulation free
moneysettlement layer of the future.The bug? Well in one file they wrote
#define MICROPY_HW_ENABLE_RNG 0, and a library they depended on had this code:This should have caused a build failure, but since MICROPY_HW_ENABLE_RNG was defined, to zero, it did not and instead continued down an insecure codepath.
As a C++ programmer this isn't an unheard of sort of mistake to make in the moment. Many libraries use undefined for preprocessor variables, but there are some people who also use zero.
But it's less excusable to not catch it when this is literally the most important thing to get right in a bitcoin wallet. Did they not trace the code path when writing the code? Did they have no integration tests or unit tests for RNG seed quality? Poor RNG seeding has come up a ton in literature, so is the sort of thing people writing cryptographic code should check for...
We'll have to addend that old saying: "don't roll your own crypto unless the other person's crypto is from a blockchain developer in which case you're probably better off rolling your own after all."
I personally don't want to sneer at the bagholders too much here. It's 2026 and at this point it just feels kinda sad. But I'll end this comment with this glowing review:
"It's open source, so anyone can verify" -- unfortunately for them the verifier was also a hacker.
Oh gosh that sounds... uhmm not too bad actually? I wonder what the largest costs are at sea with a floating nuclear reactor in salt water and tugged around by large ships.
It sure is a zany coincidence that all the major American AI companies have come out either for or against open weight AI models with grand ethical and/or patriotic concerns all at the same time isn't it?
Most recently our pal Zuck! https://www.wsj.com/opinion/the-ai-future-is-for-everyone-a0c24e20
I'm just imagining a WSJ reader reading this and thinking "what the heck is this man talking about". At some point you need a little structure or text to balance out the subtext.
Edit: did I miss any?
OpenAI (discussed here the other week): https://nitter.net/deanwball/status/2078133895766114412 / https://xcancel.com/deanwball/status/2078619513575137330
Microsoft: https://www.microsoft.com/en-us/corporate-responsibility/topics/open-weight/
Anthropic: https://www.anthropic.com/news/position-open-weights-models
Nvidia: https://images.nvidia.com/pdf/Open-Weights-and-American-AI-Leadership.pdf
Google: https://xcancel.com/sundarpichai/status/2081026488158040181
AmazonThe Washington Post who I'm sure wrote this for reasons completely unrelated to Mr. Bezos: https://www.washingtonpost.com/opinions/2026/07/02/right-answer-chinas-open-weight-ai-models-is-build-our-own/That one guy I hate: https://xcancel.com/elonmusk/status/2080672505660834163
As the post was removed: It was a chart of what rape percentage various situations were according to some internet survey.
I advise not reading it but here it is: https://pbs.twimg.com/media/G41gJkzW4AA6ThU.jpg?name=orig
Bingo
OpenAI: "Sorry about accidentally hacking you teehee. It was because of how good we are at computer security :3"
https://openai.com/index/hugging-face-model-evaluation-security-incident/
Maybe they should start applying some of that superhuman box-escaping AI to their own sandbox designs because if your sandbox escape is a bug in a "package registry cache proxy" you might be doing it wrong.
Apparently it's cheaper to foist the job of waking passengers up on 911 than to hire an employee to do it. Good ol' privatize the gains / socialize the losses!
Oh no every day sneerclub teaches me about a new guy.
So he's basically arguing that the Borg did nothing wrong (note: the Borg do not actually exist, note #2: the Borg are space fascists).
This isn't the first time I've seen this idea in rationalist circles: that is bad / wrong / irrational to be biased in favor of humans and or humans to succeed in particular. Often the idea is taken as obviously true, but I cannot understand it at all.
Like I don't care if you can make a computer "better" than me or less sad than me or less likely to eat an innocent shrimp than me. I am not that computer-- if it's a choice between me and them I'm fighting on team me. There is nothing wrong with wanting to make one's space in the universe or to "selfishly" want self-actualization or to want humans to thrive.
full AI communism
Listen I'm all for hating on the Chinese government, but it's more enjoyable to hate on them for real reasons rather than whatever this is.
Oh no the lesson he took away from his tweet is that he didn't use enough words.
Heartwarming 🥰.