Is an actual setting in the config for the (now apparently unmaintained) Shorewall Firewall software/tool for linux.
If I remember correctly, it always checks on firewall rule changes if there is an active connection on port 22, and adds a special rule at the end to maintain that connection.
Fun fact: When you do iptables-save, you have to redirect the output if you want to save it to a file. But when you use iptables-restore, you don't need to pipe it back in, you can just use the filename!
I keep a Windows 2008 w Java 6 VM on ice for administering old Java console shit like that.
The VM is unsafe as hell. Completely virgin unpatched. The only protection is that I don't give it a gateway or dns, and I shut it down when its not in use.
And it works. Old Java shit can still be used.
Almost the same thing happened to me. I accidentally fucked up the internet connection in my home while in Japan, and I had to video call my mom to have her fix it. It was a pain for both of us, but thankfully it went rather smoothly. Thank you mom!
So I connected through ssh back home to fiddle with the router settings, and in the PPPoE settings (where you set a pair of username and password that your router sends to the ISP such that the ISP knows you and knows what IP to assign to you) I made a typo, and apparently that instantly killed the internet connection at home and also for me. I had to call my mom to instruct her to fix the typo in the username. TBH I don't know that much about PPPoE either, I only do it so that the ISP assigns us the same IP address every time.
Yes, I also used to run an "on premise" server - in my kitchen, not 500km away. I sometimes might need to admin it remotely, but never critical setup work.
And the meme makes it sound like they have to drive there specifically to fix it, like nobody is actually living nearby.
I mean it's a pretty realistic scenario. I happened to be the unlikely remote hands for the company I work for just a few weeks ago.
Company: an industrial cleaning company with about 1500 AD users and about 8000 employees, historically had 2 corporate offices, currently has three as it's transitioning one corporate office across the country
Server and mistake in question: old admin who's no longer with the company setup the ESXI 6.0 cluster in the server room at the office without documenting the root password to access it. This cluster happens to host the companies critical services including AD so being unable to access the host has been blocking the office migration. Old admin had also not fixed the ESXI backups which have been broken for over 3 years so no backups to restore from. Also the out of band access to the servers was never correctly setup
I happening to be close to this office and having IT experience was poked to go in and with physical access to modify the shadow file and set the root password to be blank. Had I not been available they would have had to fly someone in from the office 2000 miles away or hire a very expensive local contractor to come in after hours to do the same thing
Well, the original post (as in the image) is about locking yourself out of a remote server by changing a firewall rule, thus needing to drive to the server to access it locally.
By using wireguard to tunnel into the router, you can remotely enter the LAN, thus bypassing the firewall, as if you were accessing the server locally.
I hate it when my boss says that. Or he will call it "D-RAC". Annoys the hell out of me.
It's iDRAC.
Yes, there are components that are called RAC, but the Dell out of band management system is called iDRAC.
... but that's not as dumb as when he calls the SuperMicro system "iLO". That's IPMI. We don't even own any HPE. I've no idea why he's stuck on iLO.
Chassis Management Controller (CMC) for Dell PowerEdge M1000e and PowerEdge VRTX
[...]
And it's just shorter and easier to say ¯\_(ツ)_/¯
but that's not as dumb as when he calls the SuperMicro system "iLO". That's IPMI. We don't even own any HPE. I've no idea why he's stuck on iLO.
Perhaps his first encounter with remote management was with iLO and he just thinks that this is how it's called. It's "integrated Lights Out", and "Lights-Out Management" as well as "Remote Access Controller" both are generic terms (and I suspect that this is why Dell adds an “iD” in front of its product names).
But we are way to close to the “GNU/Linux Copypasta” than I would like.
Since that happens to the best of us, I envision writing a wrapper script around {n,}pfctl that asks for confirmation upon detecting that you're logged in via ssh through a specific port AND detecting that the new rules would block that port.
VMware does this with its virtual networking. If a change takes it offline, it automatically rolls it back. It can be frustrating at times, but mostly its saved my ass.
Meraki does this as well. If you change anything that might disconnect the uplink or the port you are connected to, it gives you a pop-up warning before it commits.
I'll always be grateful for the firewalls like OpenWRT that will automatically revert any changes if you don't log back in after a few minutes (at least on the web interface). I'm not proud of how many times that's saved me.
This is the NetAdmin's problem. And he's got 3 ways to get into the datacenter, so he goddamn well better have an answer that doesn't involve airfare. Worst case, he's gotta use remote hands, but that would be embarrassing, and I'd not let him forget it. Nobody forgives me when I screw up a server cluster, so he gets no latitude when he takes a datacenter offline.
Does it actually happen to people? All servers I worked with both had a back door (or two), and someone at the data centre (during work hours at least) you could contact in an emergency.
Most data centers have some kind of service where you can request a KVM to be connected to the server. It's not instant as an actual human has to do so but a lot sooner than another human driving long distance. I guess in this case, it's a mid size company that is big enough to have multiple locations yet small enough to still manage to use on-premise infra instead of data centers.
iptables default DENY and flush the rules. Done by at least two people I know (then me) at the same company. Led to them moving the servers in-house and virtualizing some services to connect to the hypervisor. It does happen though.
A standard Docker container with a NodeJS/PHP/Python app is usually around 200-300 MB (yes really), the OpenJDK JVM is around a hundred MB, but a fully statically compiled rust binary that doesn't even depend on libc is just a couple MB and can be deployed as a tiny distroless Docker container.
It's a lot heavier than your 8kb C++ executable but it's nothing compared to what is required to deploy anything else.
To me, it is mostly a real blocker for using it in some embedded Linux devices due to size constraints, otherwise I personally would be using it extensively.
I'm having a hard time imagining this Goldilocks embedded device that is simultaneously big enough to run Linux (so not an actual microcontroller), yet too small for a few megabytes worth of statically-linked libraries. Got an example?
94 Comments
napkin2020@sh.itjust.works · 148 pts · 364d
Happened to me once. Had a little Pi at my parent's house and that was a nice excuse to visit them.
anomnom@sh.itjust.works · 8 pts · 364d
Except when you get there and don’t want to talk or do all the meeting and greeting until you know the server still works.
Taldan@lemmy.world · 17 pts · 363d
Relevant XKCD
Vanilla_PuddinFudge@infosec.pub · 5 pts · 364d
mavu@discuss.tchncs.de · 113 pts · 364d
even worse. I regularly have to get up out of my chair and go down 2 stairs.
Also this took a while to find, but : https://sourceforge.net/p/shorewall/svn/HEAD/tree/branches/4.2/Samples/one-interface/shorewall.conf
ADMINISABSENTMINDED=YesIs an actual setting in the config for the (now apparently unmaintained) Shorewall Firewall software/tool for linux.
If I remember correctly, it always checks on firewall rule changes if there is an active connection on port 22, and adds a special rule at the end to maintain that connection.
They don't build them like they used to anymore.
Jankatarch@lemmy.world · 11 pts · 364d
Well if we did, the way it works would be by telling a chatbot to enable ssh on port 22 at the end.
null@lemmy.nullspace.lol · 82 pts · 364d
Doing this is a right of passage.
palordrolap@fedia.io · 120 pts · 364d
Believe it or not, "rite" is the, uh, right, word here.
null@lemmy.nullspace.lol · 129 pts · 364d
Messing up the spelling is a wrong of passage.
Tikiporch@lemmy.world · 37 pts · 364d
I don't belief it.
JackbyDev@programming.dev · 12 pts · 364d
Just breath!
TriflingToad@sh.itjust.works · 9 pts · 364d
deaths
zalgotext@sh.itjust.works · 8 pts · 364d
deaths nuths
JackbyDev@programming.dev · 11 pts · 364d
You have a right to pass once you've done this rite of passage.
Resonosity@lemmy.dbzer0.com · 6 pts · 364d
Believe it or not, straight to jail
piefood@feddit.online · 73 pts · 364d
Before you make a change, do this in a screen-session:
sleep 300 && iptables-restore old_fw_rules.bak
eager_eagle@lemmy.world · 35 pts · 364d
permission denied
fuuuu
Itdidnttrickledown@lemmy.world · 8 pts · 364d
Found the debian user.
eager_eagle@lemmy.world · 10 pts · 364d
user permissions is a debian thing now?
Itdidnttrickledown@lemmy.world · 8 pts · 364d
A long time ago, Debian 8 or so it was a bug with Debian. Something about the command running without root despite the sudo command.
Itdidnttrickledown@lemmy.world · 7 pts · 364d
Yeah except it would be iptables-restore < old_fw_rules.bak
piefood@feddit.online · 15 pts · 364d
Fun fact: When you do iptables-save, you have to redirect the output if you want to save it to a file. But when you use iptables-restore, you don't need to pipe it back in, you can just use the filename!
Itdidnttrickledown@lemmy.world · 7 pts · 364d
It wasn't always that way. At one time you had to so I still do.
piefood@feddit.online · 6 pts · 364d
Totally! I still catch myself doing that sometimes. Old habits die hard
inconceivable@feddit.org · 71 pts · 364d
Real servers have lights out management and management networks.
qaz@lemmy.world · 60 pts · 364d
I'd rather plug in a screen with VGA than deal with HPE iLO 4
bacon_pdp@lemmy.world · 16 pts · 364d
dbtng@eviltoast.org · 5 pts · 364d
I keep a Windows 2008 w Java 6 VM on ice for administering old Java console shit like that.
The VM is unsafe as hell. Completely virgin unpatched. The only protection is that I don't give it a gateway or dns, and I shut it down when its not in use.
And it works. Old Java shit can still be used.
mkhopper@lemmy.world · 5 pts · 364d
To be honest, HPE iLO 6 isn't too bad, if you're using the GUI. It's the API that remains really broken in many places.
NeilBru@lemmy.world · 5 pts · 364d
Networking noob here; what, pray tell, is HPE iLO4... or do I want to even know?
Edit: Never mind. Found it. HP... shudders
buttnugget@lemmy.world · 9 pts · 364d
“In December 2021 Iranian researchers at Amnpardaz security firm have discovered rootkits in HPE's iLO (Integrated Lights-Out) management modules.”
Because of course lol
Appoxo@lemmy.dbzer0.com · 0 pts · 364d
Sounds like an issue draling with .NET or JRC console.
Are you on the nosz up to date firmware?
qaz@lemmy.world · 1 pts · 364d
I remember there being the option of using HTML or a Java applet, I chose the former
Appoxo@lemmy.dbzer0.com · 4 pts · 364d
If you have the HTML5 option you should be on a pretty recent firmware.
Interesting that you'd prefer going (literally) analog connection rather than over the IPMI.
qaz@lemmy.world · 1 pts · 364d
The latest version of iLO4 is from 2023
Appoxo@lemmy.dbzer0.com · 2 pts · 364d
You know, I wanted to say "Bet!" and proove your wrong as I couldnt believe they never went past 2023 for the firmware.
Turns out that was the latest.
But I do know they have more recent firmware uploads for the UEFI than 2023. ^(A year younger but no less nore recent/s)
randint@lemmy.frozeninferno.xyz · 65 pts · 364d
Almost the same thing happened to me. I accidentally fucked up the internet connection in my home while in Japan, and I had to video call my mom to have her fix it. It was a pain for both of us, but thankfully it went rather smoothly. Thank you mom!
buttnugget@lemmy.world · 7 pts · 364d
Do you mind explaining the details? I’m trying to learn as much as possible!
Honytawk@feddit.nl · 3 pts · 363d
Most corporate network devices like Cisco will reset their config to the one written in memory when they lose power.
So in that case, just unplug and replug them to restore to previous config.
Just make sure you write your new config to memory or it will reset when there is ever a power failure.
randint@lemmy.frozeninferno.xyz · 2 pts · 362d
So I connected through ssh back home to fiddle with the router settings, and in the PPPoE settings (where you set a pair of username and password that your router sends to the ISP such that the ISP knows you and knows what IP to assign to you) I made a typo, and apparently that instantly killed the internet connection at home and also for me. I had to call my mom to instruct her to fix the typo in the username. TBH I don't know that much about PPPoE either, I only do it so that the ISP assigns us the same IP address every time.
jubilationtcornpone@sh.itjust.works · 46 pts · 364d
What's really fun is hearing "oh shit" from the UPS maintenance tech followed by darkness and silence.
bjoern_tantau@swg-empire.de · 35 pts · 364d
Classic.
Love Hetzner. If something like that were to happen to me they can hook up a remote console accessible through their web interface.
A_norny_mousse@feddit.org · 28 pts · 364d
Many hosting providers have a remote console feature.
Evotech@lemmy.world · 31 pts · 364d
Console
supernight52@lemmy.world · 15 pts · 364d
Fuck, that is really good wordplay.
A_norny_mousse@feddit.org · 27 pts · 364d
Don't practically all commercial hosting providers provide remote console access?
This seems a combo of an extremely newb mistake in an extremely unusual scenario - worthy of Gru I guess.
onslaught545@lemmy.zip · 47 pts · 364d
Physical, on premises servers are still a thing.
A_norny_mousse@feddit.org · 4 pts · 364d
Yes, I also used to run an "on premise" server - in my kitchen, not 500km away. I sometimes might need to admin it remotely, but never critical setup work.
And the meme makes it sound like they have to drive there specifically to fix it, like nobody is actually living nearby.
Trainguyrom@reddthat.com · 10 pts · 364d
I mean it's a pretty realistic scenario. I happened to be the unlikely remote hands for the company I work for just a few weeks ago.
Company: an industrial cleaning company with about 1500 AD users and about 8000 employees, historically had 2 corporate offices, currently has three as it's transitioning one corporate office across the country
Server and mistake in question: old admin who's no longer with the company setup the ESXI 6.0 cluster in the server room at the office without documenting the root password to access it. This cluster happens to host the companies critical services including AD so being unable to access the host has been blocking the office migration. Old admin had also not fixed the ESXI backups which have been broken for over 3 years so no backups to restore from. Also the out of band access to the servers was never correctly setup
I happening to be close to this office and having IT experience was poked to go in and with physical access to modify the shadow file and set the root password to be blank. Had I not been available they would have had to fly someone in from the office 2000 miles away or hire a very expensive local contractor to come in after hours to do the same thing
Anivia@feddit.org · 6 pts · 364d
Well, I have my server running in my parents basement, because they have fiber, and I don't.
It's not quite a 500km drive, but still a long enough distance for this scenario to be a major inconvenience.
But since I have wireguard running on their router though this specific scenario is not something that could happen to me
buttnugget@lemmy.world · 1 pts · 364d
Wireguard is a VPN protocol, so you are able to tunnel into their router to…do what exactly?
Anivia@feddit.org · 1 pts · 364d
It let's me remote into their LAN, thus bypassing the firewall
buttnugget@lemmy.world · 1 pts · 363d
Please forgive the ignorance here. What are you trying to do? I thought you were trying to reboot an offline server. I’m probably just confused!
Anivia@feddit.org · 1 pts · 363d
Well, the original post (as in the image) is about locking yourself out of a remote server by changing a firewall rule, thus needing to drive to the server to access it locally.
By using wireguard to tunnel into the router, you can remotely enter the LAN, thus bypassing the firewall, as if you were accessing the server locally.
Atherel@lemmy.dbzer0.com · 3 pts · 364d
They should have a remote console like Dell RAC or HP iLO
Trainguyrom@reddthat.com · 5 pts · 364d
Could be they were configuring the actual network firewall and got a couple of rules out of order so they blocked all of their out of band access
dbtng@eviltoast.org · 1 pts · 364d
I hate it when my boss says that. Or he will call it "D-RAC". Annoys the hell out of me.
It's iDRAC.
Yes, there are components that are called RAC, but the Dell out of band management system is called iDRAC.
... but that's not as dumb as when he calls the SuperMicro system "iLO". That's IPMI. We don't even own any HPE. I've no idea why he's stuck on iLO.
Atherel@lemmy.dbzer0.com · 2 pts · 364d
I'd say that RAC is the overarching term for different Dell Solutions, see Dell Remote Access Configuration Guide
And it's just shorter and easier to say
¯\_(ツ)_/¯Perhaps his first encounter with remote management was with iLO and he just thinks that this is how it's called. It's "integrated Lights Out", and "Lights-Out Management" as well as "Remote Access Controller" both are generic terms (and I suspect that this is why Dell adds an “iD” in front of its product names).
But we are way to close to the “GNU/Linux Copypasta” than I would like.
dbtng@eviltoast.org · 1 pts · 364d
Mmm. Ya ya. No argument. But its iDRAC. I've had to sit through enough propaganda. I'm pretty sure about this.
qaz@lemmy.world · 9 pts · 364d
Yeah, all the ones I've used had remote access
thatradomguy@lemmy.world · 25 pts · 364d
Most secure box is the one that does nothing.
medem@lemmy.wtf · 24 pts · 364d
Since that happens to the best of us, I envision writing a wrapper script around {n,}pfctl that asks for confirmation upon detecting that you're logged in via ssh through a specific port AND detecting that the new rules would block that port.
dbtng@eviltoast.org · 2 pts · 364d
VMware does this with its virtual networking. If a change takes it offline, it automatically rolls it back. It can be frustrating at times, but mostly its saved my ass.
Honytawk@feddit.nl · 1 pts · 362d
Meraki does this as well. If you change anything that might disconnect the uplink or the port you are connected to, it gives you a pop-up warning before it commits.
rmuk@feddit.uk · 17 pts · 362d
I'll always be grateful for the firewalls like OpenWRT that will automatically revert any changes if you don't log back in after a few minutes (at least on the web interface). I'm not proud of how many times that's saved me.
clockworkrat@slrpnk.net · 16 pts · 364d
That the slrpnk.net admins in the picture?
qaz@lemmy.world · 12 pts · 364d
They had a hardware failure but close enough
joyjoy@lemmy.zip · 5 pts · 364d
Would misusing the
ddcommand be considered a hardware failure?rikudou@lemmings.world · 6 pts · 364d
Yes. Everything is a hardware failure because where does the software run? That's right, on hardware. So software bug = hardware failure.
sylver_dragon@lemmy.world · 5 pts · 364d
Yup, that's a bug in the chair-keyboard interface.
dbtng@eviltoast.org · 14 pts · 364d
This is the NetAdmin's problem. And he's got 3 ways to get into the datacenter, so he goddamn well better have an answer that doesn't involve airfare. Worst case, he's gotta use remote hands, but that would be embarrassing, and I'd not let him forget it. Nobody forgives me when I screw up a server cluster, so he gets no latitude when he takes a datacenter offline.
phirdowak@programming.dev · 10 pts · 364d
I try to remember to always open two SSH connections when altering iptables or the ssh config - just in case
cupcakezealot@piefed.blahaj.zone · 9 pts · 364d
this sounds like something chip from sales would do
mkhopper@lemmy.world · 3 pts · 364d
It's gray on the bottom.
frank@sopuli.xyz · 3 pts · 364d
Hello Derek you fucking idiot
observantTrapezium@lemmy.ca · 9 pts · 364d
Does it actually happen to people? All servers I worked with both had a back door (or two), and someone at the data centre (during work hours at least) you could contact in an emergency.
PotatoesFall@discuss.tchncs.de · 8 pts · 364d
I guess some smaller companies might have simpler setups they self-host
tvbusy@lemmy.dbzer0.com · 7 pts · 364d
Most data centers have some kind of service where you can request a KVM to be connected to the server. It's not instant as an actual human has to do so but a lot sooner than another human driving long distance. I guess in this case, it's a mid size company that is big enough to have multiple locations yet small enough to still manage to use on-premise infra instead of data centers.
JackbyDev@programming.dev · 6 pts · 364d
onlinepersona@programming.dev · 3 pts · 364d
iptables default DENY and flush the rules. Done by at least two people I know (then me) at the same company. Led to them moving the servers in-house and virtualizing some services to connect to the hypervisor. It does happen though.
Anti Commercial-AI license
balsoft@lemmy.ml · 8 pts · 364d
This is precisely the problem that deploy-rs solves!
somerandomperson@lemmy.dbzer0.com · 2 pts · 364d
why is everything in rust now
balsoft@lemmy.ml · 12 pts · 364d
It's easy to write, easy to build, produces lightweight and fast executables, and the type system is great. Why not rust?
UpperBroccoli@lemmy.blahaj.zone · 4 pts · 364d
Rust does not have an ABI. Everything is linked into the executables. I would not call them lightweight.
qaz@lemmy.world · 3 pts · 364d
A standard Docker container with a NodeJS/PHP/Python app is usually around 200-300 MB (yes really), the OpenJDK JVM is around a hundred MB, but a fully statically compiled rust binary that doesn't even depend on libc is just a couple MB and can be deployed as a tiny distroless Docker container.
It's a lot heavier than your 8kb C++ executable but it's nothing compared to what is required to deploy anything else.
grue@lemmy.world · 1 pts · 364d
Oh, so it's inconvenient for GPL-circumventers, too? That just sounds better and better.
UpperBroccoli@lemmy.blahaj.zone · 1 pts · 364d
To me, it is mostly a real blocker for using it in some embedded Linux devices due to size constraints, otherwise I personally would be using it extensively.
grue@lemmy.world · 3 pts · 364d
I'm having a hard time imagining this Goldilocks embedded device that is simultaneously big enough to run Linux (so not an actual microcontroller), yet too small for a few megabytes worth of statically-linked libraries. Got an example?
vane@lemmy.world · 5 pts · 364d
Rescue mode with networking, mount drive, make changes and reboot.
PoisonedPrisonPanda@discuss.tchncs.de · 5 pts · 364d
i feel that. Hetzner support has a special place in my heart
MTK@lemmy.world · 4 pts · 364d
That is why I always put any:any for ssh on all my firewalls!
Angelusz@lemmy.world · 3 pts · 360d
Nice drive to clear your head.
paulbg@programming.dev · 2 pts · 358d
good reason to take a day out, will tell it to my boss.
Korkki@lemmy.ml · 2 pts · 359d
No connection, no hackers.
technocrit@lemmy.dbzer0.com · 1 pts · 364d
Zozano@aussie.zone · 0 pts · 364d
Lol.
Just tailscale it and this will never happen again.
(Set the whole interface of tailscale0 as a trusted network)