This is a joke, I didn’t really lock myself out
Console
Fuck, that is really good wordplay.
This is the NetAdmin’s problem. And he’s got 3 ways to get into the datacenter, so he goddamn well better have an answer that doesn’t involve airfare. Worst case, he’s gotta use remote hands, but that would be embarrassing, and I’d not let him forget it. Nobody forgives me when I screw up a server cluster, so he gets no latitude when he takes a datacenter offline.
Since that happens to the best of us, I envision writing a wrapper script around {n,}pfctl that asks for confirmation upon detecting that you’re logged in via ssh through a specific port AND detecting that the new rules would block that port.
VMware does this with its virtual networking. If a change takes it offline, it automatically rolls it back. It can be frustrating at times, but mostly its saved my ass.
Meraki does this as well. If you change anything that might disconnect the uplink or the port you are connected to, it gives you a pop-up warning before it commits.
i feel that. Hetzner support has a special place in my heart
good reason to take a day out, will tell it to my boss.
That the slrpnk.net admins in the picture?
They had a hardware failure but close enough
Would misusing the
ddcommand be considered a hardware failure?Yes. Everything is a hardware failure because where does the software run? That’s right, on hardware. So software bug = hardware failure.
Yup, that’s a bug in the chair-keyboard interface.
No connection, no hackers.
Nice drive to clear your head.
Rescue mode with networking, mount drive, make changes and reboot.
Don’t practically all commercial hosting providers provide remote console access?
This seems a combo of an extremely newb mistake in an extremely unusual scenario - worthy of Gru I guess.
Yeah, all the ones I’ve used had remote access
Physical, on premises servers are still a thing.
Yes, I also used to run an “on premise” server - in my kitchen, not 500km away. I sometimes might need to admin it remotely, but never critical setup work.
And the meme makes it sound like they have to drive there specifically to fix it, like nobody is actually living nearby.Well, I have my server running in my parents basement, because they have fiber, and I don’t.
It’s not quite a 500km drive, but still a long enough distance for this scenario to be a major inconvenience.
But since I have wireguard running on their router though this specific scenario is not something that could happen to me
Wireguard is a VPN protocol, so you are able to tunnel into their router to…do what exactly?
It let’s me remote into their LAN, thus bypassing the firewall
Please forgive the ignorance here. What are you trying to do? I thought you were trying to reboot an offline server. I’m probably just confused!
Well, the original post (as in the image) is about locking yourself out of a remote server by changing a firewall rule, thus needing to drive to the server to access it locally.
By using wireguard to tunnel into the router, you can remotely enter the LAN, thus bypassing the firewall, as if you were accessing the server locally.
I mean it’s a pretty realistic scenario. I happened to be the unlikely remote hands for the company I work for just a few weeks ago.
Company: an industrial cleaning company with about 1500 AD users and about 8000 employees, historically had 2 corporate offices, currently has three as it’s transitioning one corporate office across the country
Server and mistake in question: old admin who’s no longer with the company setup the ESXI 6.0 cluster in the server room at the office without documenting the root password to access it. This cluster happens to host the companies critical services including AD so being unable to access the host has been blocking the office migration. Old admin had also not fixed the ESXI backups which have been broken for over 3 years so no backups to restore from. Also the out of band access to the servers was never correctly setup
I happening to be close to this office and having IT experience was poked to go in and with physical access to modify the shadow file and set the root password to be blank. Had I not been available they would have had to fly someone in from the office 2000 miles away or hire a very expensive local contractor to come in after hours to do the same thing
They should have a remote console like Dell RAC or HP iLO
Could be they were configuring the actual network firewall and got a couple of rules out of order so they blocked all of their out of band access
I hate it when my boss says that. Or he will call it “D-RAC”. Annoys the hell out of me.
It’s iDRAC.
Yes, there are components that are called RAC, but the Dell out of band management system is called iDRAC.
… but that’s not as dumb as when he calls the SuperMicro system “iLO”. That’s IPMI. We don’t even own any HPE. I’ve no idea why he’s stuck on iLO.It’s iDRAC.
I’d say that RAC is the overarching term for different Dell Solutions, see Dell Remote Access Configuration Guide
DRACT supports the following types of RACs that support RACADM commands:
-
Integrated Dell Remote Access Controller 8 (iDRAC8)
-
Integrated Dell Remote Access Controller 7 (iDRAC7)
-
[…]
-
Chassis Management Controller (CMC) for Dell PowerEdge M1000e and PowerEdge VRTX
-
[…]
And it’s just shorter and easier to say
¯\_(ツ)_/¯but that’s not as dumb as when he calls the SuperMicro system “iLO”. That’s IPMI. We don’t even own any HPE. I’ve no idea why he’s stuck on iLO.
Perhaps his first encounter with remote management was with iLO and he just thinks that this is how it’s called. It’s “integrated Lights Out”, and “Lights-Out Management” as well as “Remote Access Controller” both are generic terms (and I suspect that this is why Dell adds an “iD” in front of its product names).
But we are way to close to the “GNU/Linux Copypasta” than I would like.
-
Lol.
Just tailscale it and this will never happen again.
(Set the whole interface of tailscale0 as a trusted network)
I’ll always be grateful for the firewalls like OpenWRT that will automatically revert any changes if you don’t log back in after a few minutes (at least on the web interface). I’m not proud of how many times that’s saved me.
Most secure box is the one that does nothing.
Before you make a change, do this in a screen-session:
sleep 300 && iptables-restore old_fw_rules.bak
Yeah except it would be iptables-restore < old_fw_rules.bak
Fun fact: When you do iptables-save, you have to redirect the output if you want to save it to a file. But when you use iptables-restore, you don’t need to pipe it back in, you can just use the filename!
It wasn’t always that way. At one time you had to so I still do.
Totally! I still catch myself doing that sometimes. Old habits die hard
permission denied
fuuuu
Found the debian user.
user permissions is a debian thing now?
A long time ago, Debian 8 or so it was a bug with Debian. Something about the command running without root despite the sudo command.
Happened to me once. Had a little Pi at my parent’s house and that was a nice excuse to visit them.
Except when you get there and don’t want to talk or do all the meeting and greeting until you know the server still works.
