A similar experiment I did comes to mind from 3 years ago.
For the fun of it I was trying to train a few deep neural network configurations (LSTM, a few variations of FCNs, ...) to trade shitcoins and downloaded 4 years of 1h candles.
The first easiest idea was to prepare the training data to fire three signals, buy, sell, do nothing (I know a terrible choice). The cost function was setup to do the simple thing and maximize the overall profit (I know an other terrible choice). Fast forward 30min of training and the final outcome is a model that outputs "do nothing" in 100% of the cases.
We could probably improve on that significantly without losing speed.
return $x < 8
That should yield one additional correct answer, while also confusing anyone who thinks it just returns false.
And if we just hard coded and checked the first 20 or so primes before always returning false, we would probably get noticeable improvement (depending on the total range).
I've seen so many game jam entries where the code is like this. Delicately balanced and using so many assumptions to just get the thing out the door.
It's funny when they decide to make a full game out of it and realize that it's gonna take them 6 months just to undo the tech debt of the original "demo"
"AI models have started training other AI models, by pressing The-Button-That-Trains-AI-models; this button was built 7 years ago by a bunch of online volunteers we won't ever credit."
The output is not the output of the algorithm, it's the output of the unit test.
95% of numbers up to that point at not prime. Testing the algorithm that only says "not prime" is therefore correct 95% of the time. The joke is that, similar to AI, the algorithm is being presented as a useful tool because it's correct often but not always.
33 Comments
TomMasz@lemmy.world · 41 pts · 201d
95.121% of the time it works everytime.
idriss@lemmy.ml · 13 pts · 201d
A similar experiment I did comes to mind from 3 years ago.
For the fun of it I was trying to train a few deep neural network configurations (LSTM, a few variations of FCNs, ...) to trade shitcoins and downloaded 4 years of 1h candles.
The first easiest idea was to prepare the training data to fire three signals, buy, sell, do nothing (I know a terrible choice). The cost function was setup to do the simple thing and maximize the overall profit (I know an other terrible choice). Fast forward 30min of training and the final outcome is a model that outputs "do nothing" in 100% of the cases.
MonkeMischief@lemmy.today · 11 pts · 201d
To be fair, your program demonstrated the most reliable way to win at crypto! 😉
Ranulph@thelemmy.club · 4 pts · 200d
Diamond Hands in action. Buy and hold is not as profitable as simply never buying and just holding.
idriss@lemmy.ml · 29 pts · 201d
I am screenshoting this so it will be screenshot of a screenshot of a screenshot then post it somewhere else
SharkAttak@kbin.melroy.org · 8 pts · 201d
Not even adding some watermark? smh
athatet@lemmy.zip · 3 pts · 201d
ifunny
sepiroth154@feddit.nl · 22 pts · 201d
You could simplify it even further by removing the int x parameter of the function...
obelisk_complex@piefed.ca · 14 pts · 201d
needanke@feddit.org · 3 pts · 198d
Even better, do the work at compile time to respect the customers resources:
sepiroth154@feddit.nl · 3 pts · 198d
My gosh, you always feel so stupid when someone points out something so obvious! Thanks
HiddenLayer555@lemmy.ml · 18 pts · 201d
Just put "Precondition: x must not be prime" in the function doc and it'll be 100% accurate. Not my fault if you use it wrong.
418_im_a_teapot@sh.itjust.works · 16 pts · 201d
We could probably improve on that significantly without losing speed.
That should yield one additional correct answer, while also confusing anyone who thinks it just returns false.
And if we just hard coded and checked the first 20 or so primes before always returning false, we would probably get noticeable improvement (depending on the total range).
Blass_Rose@pawb.social · 2 pts · 199d
I've seen so many game jam entries where the code is like this. Delicately balanced and using so many assumptions to just get the thing out the door.
It's funny when they decide to make a full game out of it and realize that it's gonna take them 6 months just to undo the tech debt of the original "demo"
Thekingoflorda@lemmy.world · 15 pts · 201d
It approaches 100% accuracy
sunbytes@lemmy.world · 10 pts · 200d
This is how AI accuracy is also measured.
rbos@lemmy.ca · 8 pts · 201d
I've had managers who follow that exact algorithm.
Armand1@lemmy.world · 6 pts · 201d
I said something similar here about an election fraud detection system with 99.999% accuracy.
https://lemmy.world/comment/22178379
Evotech@lemmy.world · 5 pts · 201d
This but AI
But they are like 60-80%
Jankatarch@lemmy.world · 4 pts · 201d
"AI models have started training other AI models, by pressing The-Button-That-Trains-AI-models; this button was built 7 years ago by a bunch of online volunteers we won't ever credit."
pineapplelover@lemmy.dbzer0.com · 5 pts · 201d
I'm confused, shouldn't this be printing false no matter what the input is?
anton@lemmy.blahaj.zone · 6 pts · 201d
The test suite probably looks something like this:
pineapplelover@lemmy.dbzer0.com · 3 pts · 201d
Ah that makes more sense thanks. So the bottom one is a unit test and not the code being run itself
Couldbealeotard@lemmy.world · 6 pts · 201d
The output is not the output of the algorithm, it's the output of the unit test.
95% of numbers up to that point at not prime. Testing the algorithm that only says "not prime" is therefore correct 95% of the time. The joke is that, similar to AI, the algorithm is being presented as a useful tool because it's correct often but not always.
fallaciousBasis@lemmy.world · 2 pts · 201d
regedit@lemmy.zip · 2 pts · 201d
pruwybn@discuss.tchncs.de · 5 pts · 201d
Is this not at all stochastic, or do I just not know what stochastic means?
cypherpunks@lemmy.ml · 7 pts · 201d
maybe it would be better to say that it is stochastically accurate?
Jankatarch@lemmy.world · 4 pts · 201d
...95.121%
???
iByteABit@lemmy.ml · 2 pts · 200d
It's a decimal point, not thousands
razen@lemmy.world · 4 pts · 201d
But when the input is all prime numbers then the accuracy is 0.
cypherpunks@lemmy.ml · 9 pts · 201d
also btw icymi, this is a post about LLMs
fallaciousBasis@lemmy.world · 4 pts · 201d
True
lnxtx@sopuli.xyz · 3 pts · 201d
But cryptography...
anton@lemmy.blahaj.zone · 2 pts · 201d
The test suite probably looks something like this:
Paulemeister@feddit.org · 1 pts · 199d
That's why you gotta use more metrics like recall and precision