I have a tool that I wrote, probably 5+ years ago. Runs once a week, collects data from a public API, translates it into files usable by the asterisk phone server.
I totally forgot about it. Checked. Yep, up to date files created, all seem in the right format.
Meanwhile, had to debug a script that zipped a zip recursively, with the new data appended. The server had barely enough storage left, as the zip took almost 200GB (the data is only 3GB). I looked at the logs, last successful run: 2019
Well it's not that simple... Because whoever wrote that made it way too complicated (and the production version has been tweaked without updating the dev too)
A clean rewrite with some guard clauses helped remove the haduken ifs and actually zipping the file outside of the zipped directory helped a lot
I mean, I have to say I've hastened my own demise (in program terms) by over-engineering something that should be simple. Sometimes adding protective guardrails actually causes errors when something changes.
I oversimplified it but the actual process was to zip files to send to an FTP server
The cron zipped the files to send in the same directory as the zipped files, then sent the zip, then deleted the zip
Looks fine, right? But what if the FTP server is slow and uploading take more time than the hourly cron dispatch? You now have a second script that zip all the folder, with the previous zip file, which will slow down the upload, etc...
I believe may have been started by an FTP upload erroring out and forcing an early return without having a cleanup, and progressively got worse
... I suppose this happened. The logs were actually broken and didn't actually add the message part of the error object, and only logging the memory address to it
Oh no need. The client didn't noticed anything in 6 years, and the reason why we had to check is because they wanted us to see if we could add this feature... That already existed.
My favorite part is, if you do some extensive analytics from time to time (e.g. to prepare an upgrade to a new major version) and as a side effect stumble upon some workflows/pipelines/scripts constantly failing (and alerting the process owner) every five minutes for... at least a few months already.
Then you go and ask the process owner and they're just like "yeah, we were annoyed by the constant error notification mails, so we mad a filter that auto deletes them"...
I feel like half my job is trying to stop false positives and other noise from hitting important places. Because false positives kill any chance true positives will be noticed/reacted to/processed.
yeah, um, about that. I have no idea where it comes from. We can type alias and see what it is, so if it's ever lost, we can recreate it, but I looked for 30 minutes yesterday even did a grep -R and I have NO IDEA where it comes from, or why it's named electricboogaloo
And that's why I added a crontab entry that periodically purges my cron configuration. That way, I'm forced to readd only the truly necessary cron jobs, successfully reducing the amount of crontab entries.
At first I thought you missed the -r. Then I checked. Defaulting to STDIN here is very, very dumb, IMHO. Almost as bad as putting the “edit” flag right next to the “delete everything without confirmation” flag on a Western keyboard (-e vs -r).
Since I'm somewhat of a simpleton... isn't that how pipelines actually work? The only difference being, they're all (scripts) available from a centralized system and triggered i.e. with webhooks?
Instead of a local script on a server, the system opens i.e. a ssh session and runs the script step by step remotely?
Pipelines are meant to be versioned an replicable, as opposed to a hack job that only runs on a forgotten server in someone's closet depicted in the meme.
Oh man, you guys should see what I was cooking up at my old place.
Head office too shitty to give us an actual asset management solution, but we did have full access to the Microsoft suite, so i used a SharePoint lists as databases, powerapps apps running on iPads for all the data entry ux and then like two dozen hacked together power automate flows linking them all together as well as taking any Info out of the actual IT systems head office used and since we didn't have API access to those system any data feeding back in to them would be in the form of automated emails that the poor 1st line techs in head office would have to sort through and process manually.
As much as I love the magic of working and attending meetings in your undies, I've found I'm a far better professional if I'm actually fully dressed while I work. And when I go into the office I always wear something with a collar even at workplaces where that's overdressed. It just puts me in the right mindset to be the best I can be at what I do
58 Comments
r00ty@kbin.life · 98 pts · 326d
I have a tool that I wrote, probably 5+ years ago. Runs once a week, collects data from a public API, translates it into files usable by the asterisk phone server.
I totally forgot about it. Checked. Yep, up to date files created, all seem in the right format.
Sometimes things just keep working.
RustyNova@lemmy.world · 53 pts · 326d
Meanwhile, had to debug a script that zipped a zip recursively, with the new data appended. The server had barely enough storage left, as the zip took almost 200GB (the data is only 3GB). I looked at the logs, last successful run: 2019
r00ty@kbin.life · 19 pts · 326d
Yes, had the same happen. Something that should be simple failing for stupid reasons.
RustyNova@lemmy.world · 13 pts · 326d
Well it's not that simple... Because whoever wrote that made it way too complicated (and the production version has been tweaked without updating the dev too)
A clean rewrite with some guard clauses helped remove the haduken ifs and actually zipping the file outside of the zipped directory helped a lot
r00ty@kbin.life · 7 pts · 325d
I mean, I have to say I've hastened my own demise (in program terms) by over-engineering something that should be simple. Sometimes adding protective guardrails actually causes errors when something changes.
Croquette@sh.itjust.works · 1 pts · 322d
Which are what guardrails are for. When something change, you don't know the impact the change will have.
By having guardrails, you make sure to limit/eliminate potential critical issues.
Quantenteilchen@discuss.tchncs.de · 3 pts · 325d
Am I understanding that last part correctly?
Did they just automatically create a backup zip-bomb in their script‽
RustyNova@lemmy.world · 9 pts · 325d
I oversimplified it but the actual process was to zip files to send to an FTP server
The cron zipped the files to send in the same directory as the zipped files, then sent the zip, then deleted the zip
Looks fine, right? But what if the FTP server is slow and uploading take more time than the hourly cron dispatch? You now have a second script that zip all the folder, with the previous zip file, which will slow down the upload, etc...
I believe may have been started by an FTP upload erroring out and forcing an early return without having a cleanup, and progressively got worse
... I suppose this happened. The logs were actually broken and didn't actually add the
messagepart of the error object, and only logging the memory address to itadavis@lemmy.world · 2 pts · 326d
Need some monitoring!
RustyNova@lemmy.world · 14 pts · 326d
Oh no need. The client didn't noticed anything in 6 years, and the reason why we had to check is because they wanted us to see if we could add this feature... That already existed.
elvith@feddit.org · 8 pts · 325d
My favorite part is, if you do some extensive analytics from time to time (e.g. to prepare an upgrade to a new major version) and as a side effect stumble upon some workflows/pipelines/scripts constantly failing (and alerting the process owner) every five minutes for... at least a few months already.
Then you go and ask the process owner and they're just like "yeah, we were annoyed by the constant error notification mails, so we mad a filter that auto deletes them"...
greybeard@feddit.online · 1 pts · 325d
I feel like half my job is trying to stop false positives and other noise from hitting important places. Because false positives kill any chance true positives will be noticed/reacted to/processed.
Gonzako@lemmy.world · 20 pts · 326d
Yeah, all these simple data processing scripts will always work as long as both sides stay the same/compatible
r00ty@kbin.life · 21 pts · 326d
Yep. It seems they haven't changed a thing about the format. Probably a script much older than mine on their end is generating it too.
MonkderVierte@lemmy.zip · 9 pts · 326d
Isn't that true for all of data processing?
Gonzako@lemmy.world · 8 pts · 326d
Maybe. But webdevs have made it a mission not to seem like so
cupcakezealot@piefed.blahaj.zone · 59 pts · 326d
the final part of that is "written by person that left the company ten years ago"
rumba@lemmy.zip · 46 pts · 325d
I don't see the alias in your .bashrc
yeah, um, about that. I have no idea where it comes from. We can type alias and see what it is, so if it's ever lost, we can recreate it, but I looked for 30 minutes yesterday even did a grep -R and I have NO IDEA where it comes from, or why it's named electricboogaloo
Gonzako@lemmy.world · 37 pts · 326d
OK, I got called out
cenzorrll@lemmy.ca · 13 pts · 325d
Ha, loser.
*glances over at 6 bash scripts and 2 cron jobs*
Not you, you're perfect
dotslashme@infosec.pub · 32 pts · 325d
My current project has a crontab with 216 entries.
pinball_wizard@lemmy.zip · 34 pts · 325d
Well, here's a sentence I haven't been tempted to use before:
"I believe that may be too many crontab entries."
DickFiasco@sh.itjust.works · 20 pts · 325d
Any problem in server administration can be solved with an additional crontab entry. Except for the problem of too many crontab entries.
Opisek@piefed.blahaj.zone · 10 pts · 325d
And that's why I added a crontab entry that periodically purges my cron configuration. That way, I'm forced to readd only the truly necessary cron jobs, successfully reducing the amount of crontab entries.
curbstickle@anarchist.nexus · 1 pts · 323d
Which can be solved by an additional server.
Boom, problem solved.
cupcakezealot@piefed.blahaj.zone · 10 pts · 325d
just randomly delete 50 of them.
pinball_wizard@lemmy.zip · 17 pts · 325d
Yes. The strongest crontab entries will probably restore themselves. (For anyone reading along, this is sarcasm. Don't do this.)
farngis_mcgiles@sh.itjust.works · 5 pts · 325d
a crontab can regenerate from bisection to form two whole crontabs
rumba@lemmy.zip · 3 pts · 325d
pshaw, just drop in there and combine a few
/etc/cron.d/first25 /etc/cron.d/second25 ...
j_z@feddit.nu · 2 pts · 325d
This is the way. Exactly what we did + migrated 80% of everything to k8s cronjobs and Argo workflows
Lightfire228@pawb.social · 7 pts · 325d
Use SystemD timers, you animal
marcos@lemmy.world · 5 pts · 325d
At some point it may be good to migrate to airflow or something similar.
It's not the number of entries that makes it bad. It's the fact that if you run
crontab, they are gone...dondelelcaro@lemmy.world · 10 pts · 325d
That's why there's a crontab rule to load the crontab from a file. Cronception if you will.
marcos@lemmy.world · 7 pts · 325d
Make the rule start a secondary cron system. Otherwise it won't run after you erase the crontab.
dondelelcaro@lemmy.world · 6 pts · 325d
Here you go:
bleistift2@sopuli.xyz · 8 pts · 325d
At first I thought you missed the
-r. Then I checked. Defaulting to STDIN here is very, very dumb, IMHO. Almost as bad as putting the “edit” flag right next to the “delete everything without confirmation” flag on a Western keyboard (-evs-r).marcos@lemmy.world · 7 pts · 325d
Crontab is a really badly designed program that we just can't fix because everybody depends on its WFTs for something.
ag10n@lemmy.world · 28 pts · 326d
Suck my dick O’Leary
iamdefinitelyoverthirteen@lemmy.world · 20 pts · 325d
Nah bro, that bash alias is FULLY documented in .bashrc! Idiot.
A_norny_mousse@feddit.org · 16 pts · 326d
A self-written shell script "daemon" that tails & greps log output for "ERR|FAIL"
barnaclebutt@lemmy.world · 11 pts · 326d
I know there's a meme here, but as a Canadian, I'm sorry about that traitorous asshat.
MonkderVierte@lemmy.zip · 10 pts · 326d
cupcakezealot@piefed.blahaj.zone · 9 pts · 326d
alias thisdoessomething='cd /home/linuxuser/ && ./myscript.sh'elvith@feddit.org · 10 pts · 325d
alias cd='echo "command not found"'iamdefinitelyoverthirteen@lemmy.world · 5 pts · 325d
jaybone@lemmy.zip · 6 pts · 326d
If you try hard enough
marcos@lemmy.world · 5 pts · 325d
Nobody write down that if you run the stuff in a different machine, you have to create the alias first.
And once you lose the machine and are trying to restore your backups, you can't run
aliasand discover whatdoThingyactually does.SaharaMaleikuhm@feddit.org · 2 pts · 325d
This a joke? Cause that won't show up on another machine. Of course it's undocumented.
Adderbox76@lemmy.ca · 8 pts · 325d
I'll hear NO aspersions against my precious Cron!
Cron is magic. Cron is civilization!
phutatorius@lemmy.zip · 5 pts · 325d
Naw, mate, that's Crom.
0x0@lemmy.zip · 1 pts · 325d
This might come in handy.
AnanasMarko@lemmy.world · 7 pts · 326d
Since I'm somewhat of a simpleton... isn't that how pipelines actually work? The only difference being, they're all (scripts) available from a centralized system and triggered i.e. with webhooks?
Instead of a local script on a server, the system opens i.e. a ssh session and runs the script step by step remotely?
So is that the joke or am I missing something?
orhtej2@eviltoast.org · 13 pts · 326d
Pipelines are meant to be versioned an replicable, as opposed to a hack job that only runs on a forgotten server in someone's closet depicted in the meme.
gmtom@lemmy.world · 6 pts · 325d
Oh man, you guys should see what I was cooking up at my old place.
Head office too shitty to give us an actual asset management solution, but we did have full access to the Microsoft suite, so i used a SharePoint lists as databases, powerapps apps running on iPads for all the data entry ux and then like two dozen hacked together power automate flows linking them all together as well as taking any Info out of the actual IT systems head office used and since we didn't have API access to those system any data feeding back in to them would be in the form of automated emails that the poor 1st line techs in head office would have to sort through and process manually.
desmosthenes@lemmy.world · 6 pts · 325d
I feel attacked
SlartyBartFast@sh.itjust.works · 3 pts · 326d
Yes
pinball_wizard@lemmy.zip · 2 pts · 325d
This used to be my remote work wardrobe. But now I dress more casually.
Trainguyrom@reddthat.com · 0 pts · 325d
As much as I love the magic of working and attending meetings in your undies, I've found I'm a far better professional if I'm actually fully dressed while I work. And when I go into the office I always wear something with a collar even at workplaces where that's overdressed. It just puts me in the right mindset to be the best I can be at what I do
phutatorius@lemmy.zip · 4 pts · 325d
I'm always fully dressed while working remotely. That is, if wearing a bow on my winkie counts as "dressed."