Not IT but data analyst. Missed a 2% salary increase for our union members when projecting next year's budget. $12 million mistake that was only caught once it was too late to fix.
Yep. Ran a config as code migration on prod instead of dev. We introduced new safeguards for running against prod after that. And changed the expectations for primary on call to do dev work with down time. Shifted to improving ops tooling or making pretty charts from all the metrics. Actually ended up reducing toil substantially over the next couple quarters.
10/10 will absolutely still do something dumb again.
I once deleted the whole production kubernetes environment trying to fix an update to prod give awry, at11pm. My saving grace was that our systems are barely used between 10pm-8am, and I managed to teach myself by reading enough docs and stack overflow comments to rebuild it and fix the initial mistake before 5am. Never learned how to correctly use a piece of stack that quickly before or since.
No they were ancient ROM based tills, I unchecked a box that was blocking firmware updates from being pushed to the tills. For some reason I still don't completely understand, these tills received their settings by Ethernet, but received their data by dialup fucking modems. When I unchecked the box, it told the tills to cease processing commands until the firmware update was completed. But the firmware update wouldn't happen until I dialled into every single store, one at a time, and sent the firmware down through a 56k modem with horrendous stability, to each till, also one at a time. If one till lost one packet, I had to send it's firmware again.
I say for 8 hrs watchimg bytes trickle down to the tills while answering calls from frantic wait staff and angry managers.
Immediately applied, no save button. It was labeled something like "Allow/disable firmware updates" which is bad design. Label should say exactly one thing, either "Allow" or "disable" never "Allow/disable" The software was very antiquated.
Reminds me of the time all those porn pop-ups hijacked my browser and filled my history. My dad thought I'd visited all those sites on purpose for a second there.
It was all a Pentest! The company should have been operating under the Zero Trust Policy and their Security systems should not have permitted a new employee to have that many rights. You're welcome, the bill for this insightful Security Audit will arrive via mail.
Reminds me of a major incident I got involved in. I was the Problem Manager and not MIM (Major Incident Management), but I've had years of MIM experience so was asked to help out on this one. The customer manufactured blood plasma and each of the lots on the production floor was worth a cool $1 million. The application that was down and had brought production down was not the app that actually handled production, but an application (service) that supplied data to it.
Of course the customer thought that app was not Mission Critical so it didn't have redundancy. I joined the call and first thing I asked was when did the last change go through on this app... Spoiler: I had the change in front of me and it went in the previous night. The admin of the app speaks up that he did a change the previous night... And NO the MIM team had NOT looked at that change yet... Did I mention this was FOUR FUCKING HOURS into the outage? That is MIM 101. Something goes down, look to see who last fucked with it.
This is why you need experienced MIM people in enterprise environments.
So I took control of the MIM, instructed the App Admin to share his screen and walk us through the change he did the previous night... Two screens in and OH... Look at that... There's a check box that put the app into read only (or something like that, this happened back in 2009 and I don't remember all the details). I'd never seen the application before in my life, but knew that check box being checked, just based on the verbiage, could not be right... So I asked... The Admin, sounding embarrassed, said yeah he forgot to uncheck that box last night...
Fuck me.
He unchecked the box, bounced the app and what do you know... It started to work.
A single damn check box brought down the production line of a multi-billion dollar company.
My investigation for that Problem was a bit scathing to multiple levels of the customer. If a service supports a Tier 1 production app and that Tier 1 app would stop working if that service goes down... GUESS WHAT! That service is MISSION FUCKING CRITICAL and it should be supported as such. My employer was not on the hook for this one, as both applications involved were customer supported.
I would love to say that the above is an uncommon occurrence, but honestly it is the main reason for outages in my experience. Something small and stupid that is easily missed.
You can just insert directly to the helper/audit table, and delete using it, no need for the cursor loop. If you need a handle to go through the records one by one, something else has already gone wrong.
At least in PG, that is explicitly not the case, unless I'm misunderstanding:
Similarly, a PL/pgSQL loop of the form FOR x IN query LOOP .. END LOOP will never use a parallel plan, because the parallel query system is unable to verify that the code in the loop is safe to execute while parallel query is active.
60 Comments
TheFunkyMonk@lemmy.world · 172 pts · 333d
If you have the ability to do this on your first day, it’s 100% not your fault.
InvalidName2@lemmy.zip · 50 pts · 333d
This is literally true and I know it because I came here to say it and then noticed you beat me by 5 minutes.
Magnum@lemmy.dbzer0.com · 9 pts · 332d
PmMeFrogMemes@lemmy.world · 3 pts · 330d
This is literally true and I know it because these three people said so
LuxSpark@lemmy.cafe · 88 pts · 333d
It’s fine, just restore the backup.
python@lemmy.world · 96 pts · 333d
The what now?
Diplomjodler3@lemmy.world · 83 pts · 333d
It's right there, in the room with the unicorns and leprechauns.
sfxrlz@lemmy.dbzer0.com · 1 pts · 329d
Right next to the windows backup cd
BanMe@lemmy.world · 7 pts · 333d
You know how we installed that system and have been waiting for a chance to see how it works
ininewcrow@lemmy.ca · 21 pts · 333d
You go ask the other monkey who was in charge of the backup ..... and all they do is incoherently scream at you.
SinkingLotus@lemmy.world · 7 pts · 333d
Backup straight out the building. Ain't about to be there when they find out.
WhatsHerBucket@lemmy.world · 64 pts · 333d
Every seasoned IT person, devOps or otherwise has accidentally made a catastrophic mistake. I ask that in interviews :D
partial_accumen@lemmy.world · 43 pts · 333d
Mine was replacing a failed hard drive in array.
shit.
WhatsHerBucket@lemmy.world · 13 pts · 333d
I accidentally rm’ed /bin on a remote host located in another country, and had to wait for someone to get in and fix it.
LordOfLocksley@lemmy.world · 15 pts · 333d
I pushed a $1 bln test trade through production instead of my test environment.... that was a sweaty 30 minutes
manny_stillwagon@mander.xyz · 11 pts · 333d
Not IT but data analyst. Missed a 2% salary increase for our union members when projecting next year's budget. $12 million mistake that was only caught once it was too late to fix.
Botzo@lemmy.world · 9 pts · 333d
Yep. Ran a config as code migration on prod instead of dev. We introduced new safeguards for running against prod after that. And changed the expectations for primary on call to do dev work with down time. Shifted to improving ops tooling or making pretty charts from all the metrics. Actually ended up reducing toil substantially over the next couple quarters.
10/10 will absolutely still do something dumb again.
pticrix@lemmy.ca · 9 pts · 333d
I once deleted the whole production kubernetes environment trying to fix an update to prod give awry, at11pm. My saving grace was that our systems are barely used between 10pm-8am, and I managed to teach myself by reading enough docs and stack overflow comments to rebuild it and fix the initial mistake before 5am. Never learned how to correctly use a piece of stack that quickly before or since.
martinb@lemmy.sdf.org · 7 pts · 333d
Nothing focuses the mind more than the panicked realisation that you have just hosed the production systems
piefood@feddit.online · 5 pts · 333d
I deleted all of our DNS records. As it turns out, you can't make money when you can't resolve dns records :P
vale@sh.itjust.works · 47 pts · 333d
TropicalDingdong@lemmy.world · 46 pts · 333d
Ctrl + z.
Thank you for coming to my Ted talk.
JPAKx4@piefed.blahaj.zone · 30 pts · 333d
Works on my machine (excel sheet)
Hupf@feddit.org · 1 pts · 332d
Elon was onto something after all
Agent641@lemmy.world · 40 pts · 333d
I once bricked all the POS terminals in a 30-store chain, at once.
One checkbox allowed me to do this.
aeternum@lemmy.blahaj.zone · 7 pts · 333d
Agent641@lemmy.world · 11 pts · 333d
No they were ancient ROM based tills, I unchecked a box that was blocking firmware updates from being pushed to the tills. For some reason I still don't completely understand, these tills received their settings by Ethernet, but received their data by dialup fucking modems. When I unchecked the box, it told the tills to cease processing commands until the firmware update was completed. But the firmware update wouldn't happen until I dialled into every single store, one at a time, and sent the firmware down through a 56k modem with horrendous stability, to each till, also one at a time. If one till lost one packet, I had to send it's firmware again.
I say for 8 hrs watchimg bytes trickle down to the tills while answering calls from frantic wait staff and angry managers.
aeternum@lemmy.blahaj.zone · 5 pts · 333d
kuberoot@discuss.tchncs.de · 2 pts · 332d
I'm curious - was it also a checkbox that immediately applied when toggled, instead of not actually applying until you press save?
Agent641@lemmy.world · 3 pts · 332d
Immediately applied, no save button. It was labeled something like "Allow/disable firmware updates" which is bad design. Label should say exactly one thing, either "Allow" or "disable" never "Allow/disable" The software was very antiquated.
LaunchesKayaks@lemmy.world · 4 pts · 332d
Nah. It was totally a virus attack.
ivanafterall@lemmy.world · 2 pts · 332d
Reminds me of the time all those porn pop-ups hijacked my browser and filled my history. My dad thought I'd visited all those sites on purpose for a second there.
prole@lemmy.blahaj.zone · 3 pts · 333d
That's actually pretty impressive
python@lemmy.world · 39 pts · 333d
It was all a Pentest! The company should have been operating under the Zero Trust Policy and their Security systems should not have permitted a new employee to have that many rights. You're welcome, the bill for this insightful Security Audit will arrive via mail.
iii@mander.xyz · 30 pts · 333d
Pretend you thought you were hired as disaster recovery tester
prole@lemmy.blahaj.zone · 4 pts · 333d
*Gets in car and drives away*
Kolanaki@pawb.social · 26 pts · 333d
"Ah, shit. Oh well. They have backups."
"..."
"They have backups, right?"
purplemonkeymad@programming.dev · 13 pts · 333d
If they don't, that's something you can't blame on a new start.
CubitOom@infosec.pub · 25 pts · 333d
It's my last day at work, and I just started to
ddmy work laptop...but I forgot I was ssh'd into the production database.iii@mander.xyz · 41 pts · 333d
Did you know that morning it would be your last day at work?
HootinNHollerin@lemmy.dbzer0.com · 18 pts · 333d
Get this monkey a job at Tesla
wetbeardhairs@lemmy.dbzer0.com · 10 pts · 333d
We need an army of them working at Palantir too
m3t00@piefed.world · 17 pts · 333d
SaveTheTuaHawk@lemmy.ca · 16 pts · 332d
I can do one better. Novo Nordisk lost their Canadian patent for Ozempic because someone forgot to fill out the renewal with a $400 admin fee.
They will lose $10B before patent ending.
But they saved $400. Someone needs to talk to HR.
Semi_Hemi_Demigod@lemmy.world · 13 pts · 333d
You’re stress testing the IT department’s RTO and RPO. This is important to do regularly at random intervals.
Netflix even invented something called Chaos Monkey that randomly breaks shit to make sure they’re ready.
piefood@feddit.online · 2 pts · 333d
https://en.wikipedia.org/wiki/Chaos_engineering#Simian_Army
s@piefed.world · 12 pts · 333d
What’s with the weird vertical artifacts in this image?
ayyy@sh.itjust.works · 5 pts · 333d
Trying to hide the slop
WhiskyTangoFoxtrot@lemmy.world · 2 pts · 333d
Scanlines in tate mode.
PmMeFrogMemes@lemmy.world · 1 pts · 330d
this meme is made out of corduroy
hansolo@lemmy.today · 10 pts · 333d
Tell them your name is Claude, they'll pay you $200 a month for the privilege.
Canopyflyer@lemmy.world · 10 pts · 332d
Reminds me of a major incident I got involved in. I was the Problem Manager and not MIM (Major Incident Management), but I've had years of MIM experience so was asked to help out on this one. The customer manufactured blood plasma and each of the lots on the production floor was worth a cool $1 million. The application that was down and had brought production down was not the app that actually handled production, but an application (service) that supplied data to it.
Of course the customer thought that app was not Mission Critical so it didn't have redundancy. I joined the call and first thing I asked was when did the last change go through on this app... Spoiler: I had the change in front of me and it went in the previous night. The admin of the app speaks up that he did a change the previous night... And NO the MIM team had NOT looked at that change yet... Did I mention this was FOUR FUCKING HOURS into the outage? That is MIM 101. Something goes down, look to see who last fucked with it.
This is why you need experienced MIM people in enterprise environments.
So I took control of the MIM, instructed the App Admin to share his screen and walk us through the change he did the previous night... Two screens in and OH... Look at that... There's a check box that put the app into read only (or something like that, this happened back in 2009 and I don't remember all the details). I'd never seen the application before in my life, but knew that check box being checked, just based on the verbiage, could not be right... So I asked... The Admin, sounding embarrassed, said yeah he forgot to uncheck that box last night...
Fuck me.
He unchecked the box, bounced the app and what do you know... It started to work.
A single damn check box brought down the production line of a multi-billion dollar company.
My investigation for that Problem was a bit scathing to multiple levels of the customer. If a service supports a Tier 1 production app and that Tier 1 app would stop working if that service goes down... GUESS WHAT! That service is MISSION FUCKING CRITICAL and it should be supported as such. My employer was not on the hook for this one, as both applications involved were customer supported.
I would love to say that the above is an uncommon occurrence, but honestly it is the main reason for outages in my experience. Something small and stupid that is easily missed.
SektorC@discuss.tchncs.de · 7 pts · 333d
I was hired as a backup representative and just wanted to know what I was dealing with and make a clear statement.
abbadon420@sh.itjust.works · 7 pts · 333d
the ono-second
myfunnyaccountname@lemmy.zip · 6 pts · 333d
The data was a mess and needed to be cleaned up anyway.
medem@lemmy.wtf · 5 pts · 332d
...which would make it about 99,9999999% their fault for giving a new employee write access to the DB.
BigBenis@lemmy.world · 3 pts · 333d
Have you tried turning it off and on again?
AnUnusualRelic@lemmy.world · 2 pts · 332d
Oh, so that's what the Delete button does.
WanderWisley@lemmy.world · 2 pts · 333d
All of my bananas are worthless now!
Blackmist@feddit.uk · 1 pts · 333d
Why is the default for some database tools to auto commit after that? Pants on head design decision.
ivanafterall@lemmy.world · 1 pts · 332d
Right click Recycle Bin --> Restore
bacon_pdp@lemmy.world · -4 pts · 333d
whats_a_lemmy@midwest.social · 1 pts · 333d
You can just insert directly to the helper/audit table, and delete using it, no need for the cursor loop. If you need a handle to go through the records one by one, something else has already gone wrong.
bacon_pdp@lemmy.world · -1 pts · 333d
whats_a_lemmy@midwest.social · 1 pts · 332d
At least in PG, that is explicitly not the case, unless I'm misunderstanding:
https://www.postgresql.org/docs/current/when-can-parallel-query-be-used.html
At any rate, I feel like it's questionable design to have a user making row-by-row decisions on hard deletes.
bacon_pdp@lemmy.world · 0 pts · 332d