For a while, the windows machine I was using kept randomly becoming unresponsive (both via network, as well as the keyboard+mouse) and I just could not figure out why. Everytime it did, I'd have to force a shutdown by holding the power button until it shutoff, then press it again to start back up. Sometimes this happened while I wasn't home, so it had to just stay offline until I was.
I got sick of having to do this; so I replaced the power button with a transistor and a RPI. The PI would ping the server every 5min; if it failed to get a response 3 times in a row, it would trigger the transistor for 10sec, release for 3sec, then press and release again; forcing a poweroff then starting the machine again. It'd also write these events to a log file. I think it was around twice a week ish.
Never did figure out what the cause of the lockup was; but it stopped when I replaced windows with Debian.
Had a zyxel nas that got unresponsive after 10-15days of uptime, thankfully it hat a scheduled shutdown/reboot setting. Runs since 2014 with a reboot every Mondy 3am.
I've got an extra and don't know what to do with it. I feel like its bad to pull power unexpectedly once a week though. Especially with a single HDD storage setup. I'd like to setup a RAID config, but its a 28TB drive and additional drives would cost me about $$$$$$$.
I have a NAS that had a similar problem, every week or two it would stop responding. From the logs it was clear that the machine was running fine after going non-responsive, but either the NIC or something else network related was crashing. However, the logs weren’t capturing the exact issue. I wrote a script to ping Google every 10 minutes, write a bunch of debug data to disc if it failed, and then reboot.
And it has never happened in the year since. I have no idea what the root cause was, but sending those pings out seems to have fixed it.
I also had a machine that would randomly crash on Windows, but not on Linux.
I'm pretty sure it was some sort of hardware failure, as it also sometimes failed to boot Linux. But if it booted Linux, it would be stable until I shut it down.
::: spoiler Note to any future lemmings who find this thread troubleshooting the same symptom…
Rule out unswitched (non-chipset, direct CPU lanes) PCI connectivity and power issues by testing without specific PCI components. Unstable PCI errors can be missed during post and present later after boot, and Windows is especially bad at handling it transparently. Common culprits include GPU sag, dubious riser extensions, and anemic PSUs
If there is a “high bandwidth” feature toggle for RAM in your BIOS settings that is currently on, try turning it off
If your system drive is an NVMe blade with a Phizon controller, check if the manufacturer software has a firmware update available.
If the freeze is regular but not persistent, try turning the polling rate down on your mouse, and if it’s wired, try to make sure it’s plugged into one of the board’s SOC-hosted USB 2.0 ports
:::
I have two server room dogs. An English pit, which is my seizure dog, and a Jack Russel. They shed, especially in the summer months, and the floor sweeper gets a daily work out, as well as keeping it out of equipment enclosures. It's not that I hate cats, they are nice as a pet in your house, just not mine. LOL
They make sprung flip-top covers specifically for this reason. It’s just a little adhesive square/circle that you stick over the power/reset button, and it has a flip-top that automatically springs closed again.
Oh, my housemates' cat hits the front power button when he leaps off my tower. I've got some leather taped over it. My uptime approaches 1! I love your little weight setup, too!
I hosted shit for about ten years before really understanding how local dns worked.
Imagine a primitive society that discovered literally every major modern scientific achievement, but somehow missed the wheel. That's basically how my first setup went lol.
One day, our internet just stopped working entirely. Our ISP was fine, router good, but no outside connection and I couldn’t access local devices. I have two Unraid servers. I took a quick glance at them, they were powered on, all looked ok. I decide to power cycle everything. Still no internet. Then I realize that I can still access Google from my phone when on WiFi.. hmm. That would mean… an issue with pi-hole!
I begin to try to diagnose from my phone with a hotspot to my computer. I’m getting warmer but some things just aren’t making sense. I go back and stare at the server running pi-hole. Then it fucking hits me.
I have the pretty odd and totally not ideal home network. I have plans to upgrade everything but currently I have two wifi networks tied to the same WAN. One network is run off of access points so I can get Ethernet to odd areas of the property. The Unraid server hosting pi-hole was connected to an AP and the Ethernet cable I used didn’t have a clip, so it just barely unseated itself from it, breaking the connection. The router+pi-hole config meant also breaking the internet if anything happened to pi-hole or that server. I swapped it for a better cable and it all instantly started working.
Spent that weekend sorting out the network so it doesn’t happen again. It was definitely a head-scratcher.
Pihole is cool, but fucked me over as well more than once (not the pihole itself but the host). It's on its own hardware now, but I plan to add a second one to reduce the single point of failure.
I let the router handle dhcp, while having it hand out each of the piholes IPs as DNS #1 and DNS #2. Those piholes settings are mirrored using Nebula-Sync, so any change made to one pihole gets copied over to the other automatically.
Both of those piholes are setup to wrap outbound DNS reqests in DOH as well, with the router blocking port 53 outbound. So all DNS traffic leaving the network has to be DOH/DOT or it just gets dropped. Currently using NextDNS and quad9 as my upstream DOH providers.
Note: the cloudflared method for DOH is outdated. I'm still using it via the docker image "image: cloudflare/cloudflared:2025.11.1"
I mean, would you want it to serve clients outside of your subnet? I guess maybe if you had something like a guest network. But I tend to take the “fuck it, you get basic service and nothing else” approach to things like guest networks.
Isn’t it a potential security issue to have your pihole serve things on any subnet, because it allows an improperly configured firewall to stab at your pihole? I think you can specify any additional subnets it will serve in the settings somewhere. But IIRC, the default behavior is to intentionally drop traffic from unknown subnets. But that’s just off the top of my head.
At my first-ever tech job back in 2000, I was given a little Celeron desktop computer to do my work: mostly writing some ColdFusion and cutting up images in Photoshop. For the most part, the machine worked fine, except whenever I scrolled down on a webpage in Netscape, the box's PC speaker would start screaming. I tried to ignore it, but I was in a small room with two other nerds and it was getting really annoying.
One day, my frustrated colleague decided it was time to investigate. As part of the process, he noticed that there was very little air coming out of the power supply fan port. "Must be a busted fan and that's causing an overheating alarm" he declared. We unplugged the box, popped it onto my desk, and opened it up. The internals were pristine. No dust, nothing. The computer was practically brand new after all.
We were just about to crack open the power supply (not recommended, but we were getting desperate) when my colleague noticed that something was wedged inside the fan... it was the biggest (dead, thankfully) cockroach I have ever seen, at least 10cm long. For context, this was in Vancouver, Canada. Those just don't exist there.
My colleague jammed his screwdriver in there to grind up the cockroach carcass, then plugged the box back in. After a shittone of dessicated cockroach guts spewed out the back, my little computer was operating normally.
As best we could guess, the monster crawled into the fan in Malaysia when as was assembled and somehow got pinned in there, died of starvation, and then stowed away to Canada.
And this is the way bioinvasor species spread - most of the time they don't come delivered inside of PSUs but simple crates, which improves their survivability lol
Not selfhosting, but I remember standing in a server room with a hands' width of water on the ground (and rising), and hastily shutting down servers where the water was already in the case. It were big towers (Compaq Proliant, if anyone remembers), and luckily the board, drives, and PSU were in the top part of the big tower cases, the drive bays at the bottom were empty.
We got all equipment out safely, and we did not get electrocuted, either. We left with the last equipment when the water level was approaching the knees.
Reminds me of a time at my old job. It was a small office and this was before "the cloud" was a buzzword, so we had a little server room running a few boxes for AD, Exchange, and the thick end of our thin clients.
This "server room" was basically a large closet behind my office, retrofitted with its own AC.
We had residential units on the higher floors of our building. Residents occasionally thought to themselves "you know what makes a great hanging rack? Sprinkler pipes!" Hijinks ensued.
That was a fun call to get at 10PM.
Shoutout to the facilities guy who immediately thought to run a makeshift tarp over the server rack, fashioned from trash bags. No permanent damage.
Ahh yes, the "Impact Fix" works everytime til it doesn't. Had a monitor with a loose wire to the power button. For almost a year a forceful tap in the correct spot would make it work, til one day
Well at home i only have problems with spiders in my racks, not like the racks i have seen at work.
Highlights are toiletpaper, chipmunks+nuts and worst in a school server rack a gun.
Mine would be when I had a 48 bay disk array / JBOD fail on me... badly. After a storm, it killed the larger card that allowed for me to get many of the drives into a PCIe 16x slot, and I was relegated down to only getting ~8 disks per box made from spare hardware. A single box I got 16 going. Add to this mix an SSD for mid-line caching.
These were all running bcache on top of mdraid... One single mount.
Yes I understand how obnoxiously stupid it was to run RAID6 on a 48 disk volume. It was almost all just stuff I could re-acquire over time, not irreplaceable things.
I just HAD to solve this one though.
In come several spare chassis / mobo etc.... get a bunch of drives powered and on /dev/ , move to the next.
A couple spare gigabit switches...
several gigabit NICs...
two explicit paths for each machine...
a bit of iSCSI magic, and one machine now had the physical disks all exposed to it... mdadm --assemble blah blah, bit of UUID chaos...
It's surprising that while a bit speed limited (I think I got just around 110MB/sec reads), it was nicely performant for what a huge mess of wires and disks just strewn out around my rack.
Managed to evacuate all I needed without much issue once I got that going. Now, I try to keep my arrays under 16 drives at a time, or keep a very rigid policy of "I can lose this and don't care" vs "this box gets RAID10 and/or offsite backups nightly".
Power isn't so bad really. These were all 2tb disks, so ended up with about 40tb total after shenanigans.
Cabling was atrocious. I'd originally been using several pulled 4 disk SAS backplanes from servers thrown out of my work, just screwed down to some pine boards in a stack.
Zip ties and sata/sas cables were the rule.
Overall the drives had 4 small power supplies initially, 300watt iirc. Cheap.
Somewhere I have an old photo from the wiring during the "rescue" but I have to look around.
Disk failure was actually not an issue for ages before. A specific thunderstorm with 2 nearby tornadoes is what caught me this time. Before that, the occasional single disk every 6 months or so.
I still have running arrays using 15 year old disks, many with more than 10 years of that powered on. If you find good models, they can last ages.
Goddamn I hate mice. I just had to tear out the entire contents of a combine cab to get rid of their piss and shit, and fix the wiring they chewed off.
I’ve got Pfsense installed on a device with 6 RJ45 ports. 0 is wan, 1 is lan0, 2 is lan1, etc up to 4. Why I have multiple lans is somewhat tangential.
One day it rebooted due to a power outage and WAN worked but lan1 and 2 and 3+ were messed up. I’m still not entirely sure what happened, but the second physical Ethernet port was non responsive at boot and Pfsense just assigned the ports as if it never existed. I moved all the cables over one except LAN and it worked just fine. Except that of course in the troubleshooting i had moved all the assignments around and broke it thoroughly, so it took a while.
Not exactly self-hosting, but I've worked at two different academic labs that had unique issues and small budgets.
The first was on top of a mountain at 3km elevation. The low air pressure meant that all hard drives had a lifespan of no more than a year before starting to have issues. This was in the pre-SSD days. The "solution" was to boot from a floppy, never compress data, and have a student return the drives to a low elevation and swap them for the prior ones every six months.
The second was at a place that was using old Datacube MaxVideo hardware that we would never have the budget to replace. It would crash if the room temperature got above about 18C, so we set the thermostat to 15C. Which doesn't sound too bad until you've tried sitting at a desk for 8 hours where everything is 15C and the heat is just slowly leaving your body into the chair, the desk, and the keyboard. And then you walk outside where it is 32C and instantly start sweating because you're still in several layers and fingerless gloves.
Not self hosted but at work: after an update to both our firewall cluster nodes they both acted wonky, restarted 5 min after booting, ssh connection wasn't working half the time, dropped packages... the problem persisted through multiple reboots, support had no idea.
the fix was to pull the plug entirely, wait 10 seconds then power them on again. on both nodes we had this behavior and both worked again after losing power for a few seconds.
seems that even for business appliances "have you tried to turn it off and on again" is a viable solution.
for me it was the fact that the desktop case i used for the server had the power button on top and it was big enough to be pushed in, when my cat walked over it...
my cat currently loves hitting the "Sleep"-button on my keyboard every time he jumps on my table -.- at least i can wake it again, but it IS annoying. Haven't had the time yet to reconfigure that button to instead run a soundfile of me scolding my cat lol.
My server once randomly kernel panicked once (and only once) and it took me an embarrassingly long time to figure out that a ram stick (too poor for ECC) had gone bad. Also took all my proxmox backup server backups with it because the validation would always fail (also due to the ram stick). Still worked fine otherwise lol
I had a screen get burn in (forgot to turn it off, oops) removed it from the laptop, waited 3 days, put it back in and it was flawless again.
I had a laptop that refused ethernet and would only ever use wifi... When the ethernet was connected. No wifi unless that cord was connected. Confirmed it was, indeed, wifi by using a dummy connector.
Bought a 10tb hdd that refused to read anything at all, opened it for giggles/curiosity and found a piece of scotch tape stuck to one of the disks and one of 4 disks completely shattered. The disk was still factory sealed (by appearance, anyways; they still had the antitamper stickers)
I dunno if that counts or if it's the weirdest cause I rarely have any issue, but one day, I wanted to deploy speedtest tracker via docker on my 1st NAS..
So I tried, but it didn't work (I forgot the reason). So I try to check the logs, but it turns out my NAS was insanely slow.
When I rebooted it, it couldn't stop beeping and telling me there was a hardware problem, so I tried everything I could do to fix it, but nothing worked, so I ended up reinstalling the whole OS
Keep in mind this was the cost to see I had a bad result at the speedtest lmao 😭
I do not care to admit how many times when I was first getting into home labbing that I could not figure out why I could not reach a pi or the internet. All because I either killed an SD card for pihole or just plain out forgot to put one in!
I use e self made wood cabinet in the attic where also the PV inverter is located. I used Velcro to strip on the wall (brick wall) stuff like switch firewall etc, only to find those collapsed on the server itself every time. No matter which glue or adhesive i used.
At the end, a strong unbranded Chinese double sided tape was better and still strong after all big western brand failed over time.
Maybe a mix of heat and uneven surface was the culprit, who knows.
The mouse did not survive interrogation.
Also mouse not pictured.
I thought it was the huge heat sink that looks like it extends past the side plane of the cover. Reminds me of something I did with a CoolAir Cosmos tower. I mounted two 300 cfm fans over each CPU and took the side panel, neatly cut out the area around the fans, then 3d printed a scoop, with a slot for filtration, to fit on the side panel and allow the fans to breathe.
Back in the day of BBS’ my buddy had a wonky modem. If I picked up phone , pursed my lips and did a squeaky sound the modem worked and I could log into BBS.
96 Comments
Ebby@lemmy.ssba.com · 82 pts · 13h
I lost a raspberry pi for a couple months. Just couldn't find the darn thing. It was working fine serving away.
Turns out it was in the basement. I forgot I moved it to be closer to my tilt Hydrometer.
Gekkonen@sopuli.xyz · 19 pts · 13h
To be fair, those things are really tiny!
SeductiveTortoise@piefed.social · 17 pts · 9h
Oh, good thing you wrote this. I'm about to give one of my printers away and there's still a pi inside
not@lemmy.dbzer0.com · 7 pts · 9h
Cups?
SeductiveTortoise@piefed.social · 9 pts · 7h
OctoPrint (3d printer)
NickwithaC@lemmy.world · 5 pts · 3h
Memory unlock!
Darkassassin07@lemmy.ca · 55 pts · 13h
For a while, the windows machine I was using kept randomly becoming unresponsive (both via network, as well as the keyboard+mouse) and I just could not figure out why. Everytime it did, I'd have to force a shutdown by holding the power button until it shutoff, then press it again to start back up. Sometimes this happened while I wasn't home, so it had to just stay offline until I was.
I got sick of having to do this; so I replaced the power button with a transistor and a RPI. The PI would ping the server every 5min; if it failed to get a response 3 times in a row, it would trigger the transistor for 10sec, release for 3sec, then press and release again; forcing a poweroff then starting the machine again. It'd also write these events to a log file. I think it was around twice a week ish.
Never did figure out what the cause of the lockup was; but it stopped when I replaced windows with Debian.
UxyIVrljPeRl@lemmy.world · 14 pts · 13h
Had a zyxel nas that got unresponsive after 10-15days of uptime, thankfully it hat a scheduled shutdown/reboot setting. Runs since 2014 with a reboot every Mondy 3am.
vaionko@sopuli.xyz · 7 pts · 7h
Heh, my router is on one of those timer plug things. No matter how frozen, it will reboot.
LincolnsDogFido@lemmy.zip · 2 pts · 5h
I've got an extra and don't know what to do with it. I feel like its bad to pull power unexpectedly once a week though. Especially with a single HDD storage setup. I'd like to setup a RAID config, but its a 28TB drive and additional drives would cost me about $$$$$$$.
BartyDeCanter@piefed.social · 2 pts · 4h
I have a NAS that had a similar problem, every week or two it would stop responding. From the logs it was clear that the machine was running fine after going non-responsive, but either the NIC or something else network related was crashing. However, the logs weren’t capturing the exact issue. I wrote a script to ping Google every 10 minutes, write a bunch of debug data to disc if it failed, and then reboot.
And it has never happened in the year since. I have no idea what the root cause was, but sending those pings out seems to have fixed it.
A_norny_mousse@piefed.zip · 10 pts · 10h
the mysteries of closed-source software
black0ut@pawb.social · 9 pts · 9h
I also had a machine that would randomly crash on Windows, but not on Linux.
I'm pretty sure it was some sort of hardware failure, as it also sometimes failed to boot Linux. But if it booted Linux, it would be stable until I shut it down.
possiblylinux127@lemmy.zip · 9 pts · 9h
Sometimes Linux handles faults better than Windows
Typically dmesg shows error when the hardware is acting up
Septimaeus@infosec.pub · 7 pts · 5h
::: spoiler Note to any future lemmings who find this thread troubleshooting the same symptom…
waldfee@feddit.org · 4 pts · 6h
Using an RPi with a transistor almost seems professional... ITAPPMONROBOT
SirHaxalot@nord.pub · 46 pts · 10h
No mouse problem, but I've had to take measures to prevent cat-related downtime. Cats and power-button at the top is not a good combination.
Retro_unlimited@lemmy.world · 11 pts · 10h
When I had this problem I disconnected the power button on the motherboard.
16mhz@lemmy.ml · 6 pts · 9h
Cant you just assign a different action to that power button instead of messing with it internally?
MagicShel@lemmy.zip · 10 pts · 9h
Connect to spray bottle aimed at..... well nope.
vaionko@sopuli.xyz · 6 pts · 7h
You can for a click, but no way for the long press, as that's handled on a lower level
16mhz@lemmy.ml · 3 pts · 2h
You got a point. Cats are sneaky 😅
tomatoely@sh.itjust.works · 2 pts · 6h
Did you reconnect it everytime you had to power it on?
frongt@lemmy.zip · 6 pts · 5h
Nah just jam a screwdriver in there to jump the pins.
You shut down your PC though?
Retro_unlimited@lemmy.world · 3 pts · 4h
On one computer I had a recessed reset button, so I used that as power.
Another computer I added a wire to an external button double stick taped to the back of the monitor so I can turn on both at the same time.
Peffse@lemmy.world · 3 pts · 2h
You sound like you are one step short of wiring up a giant lever to dramatically pull down when you want to power on the computer.
givesomefucks@lemmy.world · 7 pts · 9h
Instead of cat proofing, you were supposed to get a server room dog...
irmadlad@lemmy.world · 3 pts · 6h
I have two server room dogs. An English pit, which is my seizure dog, and a Jack Russel. They shed, especially in the summer months, and the floor sweeper gets a daily work out, as well as keeping it out of equipment enclosures. It's not that I hate cats, they are nice as a pet in your house, just not mine. LOL
Carl@anarchist.nexus · 6 pts · 5h
They make sprung flip-top covers specifically for this reason. It’s just a little adhesive square/circle that you stick over the power/reset button, and it has a flip-top that automatically springs closed again.
lyralycan@sh.itjust.works · 3 pts · 9h
Aha, I disabled the LEDs by unplugging them from the motherboard, you could do the same with the power sw plugs
Sidhean@piefed.social · 3 pts · 3h
Oh, my housemates' cat hits the front power button when he leaps off my tower. I've got some leather taped over it. My uptime approaches 1! I love your little weight setup, too!
Trampampoline@sh.itjust.works · 40 pts · 11h
I hosted shit for about ten years before really understanding how local dns worked.
Imagine a primitive society that discovered literally every major modern scientific achievement, but somehow missed the wheel. That's basically how my first setup went lol.
eco_game@discuss.tchncs.de · 9 pts · 10h
I have heard of local DNS multiple times now and been selfhosting for many years, but I'm good with my 192.168.178.39 :)
Trampampoline@sh.itjust.works · 8 pts · 9h
It opened the door to pihole and unbound, then dns certs, man... everything's so nice now lol
nnullzz@lemmy.world · 37 pts · 13h
One day, our internet just stopped working entirely. Our ISP was fine, router good, but no outside connection and I couldn’t access local devices. I have two Unraid servers. I took a quick glance at them, they were powered on, all looked ok. I decide to power cycle everything. Still no internet. Then I realize that I can still access Google from my phone when on WiFi.. hmm. That would mean… an issue with pi-hole!
I begin to try to diagnose from my phone with a hotspot to my computer. I’m getting warmer but some things just aren’t making sense. I go back and stare at the server running pi-hole. Then it fucking hits me.
I have the pretty odd and totally not ideal home network. I have plans to upgrade everything but currently I have two wifi networks tied to the same WAN. One network is run off of access points so I can get Ethernet to odd areas of the property. The Unraid server hosting pi-hole was connected to an AP and the Ethernet cable I used didn’t have a clip, so it just barely unseated itself from it, breaking the connection. The router+pi-hole config meant also breaking the internet if anything happened to pi-hole or that server. I swapped it for a better cable and it all instantly started working.
Spent that weekend sorting out the network so it doesn’t happen again. It was definitely a head-scratcher.
rimu@piefed.social · 26 pts · 12h
It's always dns at least half the time
BuckFutter@timeyak.com · 4 pts · 2h
tofu@lemmy.nocturnal.garden · 11 pts · 12h
Pihole is cool, but fucked me over as well more than once (not the pihole itself but the host). It's on its own hardware now, but I plan to add a second one to reduce the single point of failure.
nnullzz@lemmy.world · 9 pts · 12h
Haven’t thought of that tbh. Would they both just share configuration? Curious how leasing would be split/shared between them. Off to research!
Darkassassin07@lemmy.ca · 11 pts · 12h
I've been running two piholes for a few years:
I let the router handle dhcp, while having it hand out each of the piholes IPs as DNS #1 and DNS #2. Those piholes settings are mirrored using Nebula-Sync, so any change made to one pihole gets copied over to the other automatically.
Both of those piholes are setup to wrap outbound DNS reqests in DOH as well, with the router blocking port 53 outbound. So all DNS traffic leaving the network has to be DOH/DOT or it just gets dropped. Currently using NextDNS and quad9 as my upstream DOH providers.
Note: the cloudflared method for DOH is outdated. I'm still using it via the docker image "image: cloudflare/cloudflared:2025.11.1"
I believe This (dns-crypt) is the more recent method.
nnullzz@lemmy.world · 5 pts · 12h
Oh sweet this is super helpful! Doesn’t seem too complicated. Thanks!
meltedcheese@c.im · 2 pts · 10h
@Darkassassin07 Looks very useful. Also timely for me. I have a use case that requires multiple synched pi-holes. 👍🏼
tofu@lemmy.nocturnal.garden · 4 pts · 12h
There's some sync mode but I haven't really looked into it yet.
nnullzz@lemmy.world · 4 pts · 12h
Makes sense.
Archer@lemmy.world · 4 pts · 12h
Yep I’ve had it randomly stop working if it has to serve clients outside it’s subnet
Carl@anarchist.nexus · 3 pts · 5h
I mean, would you want it to serve clients outside of your subnet? I guess maybe if you had something like a guest network. But I tend to take the “fuck it, you get basic service and nothing else” approach to things like guest networks.
Isn’t it a potential security issue to have your pihole serve things on any subnet, because it allows an improperly configured firewall to stab at your pihole? I think you can specify any additional subnets it will serve in the settings somewhere. But IIRC, the default behavior is to intentionally drop traffic from unknown subnets. But that’s just off the top of my head.
rumba@lemmy.zip · 2 pts · 7h
I keep one Physical Pihole and one in a container, be sure yeah, this happens.
somenonewho@feddit.org · 29 pts · 11h
I was frantically searching for it, then i opended the post to See if the comments found it ...
Gekkonen@sopuli.xyz · 19 pts · 11h
I'm sorry. Have this picture of a lime hawk-moth caterpillar I saw the other day.
https://plantmonster.net/pictures/phone/PXL_20260813_122831209.jpg
kfh@lemmy.world · 26 pts · 8h
Then you tell people about it on IRC and become known as "the dead mouse guy".
But the maximum nickname length on the IRC server is limited, so you settle for "deadmau5" as your nick.
danielquinn@lemmy.ca · 26 pts · 12h
At my first-ever tech job back in 2000, I was given a little Celeron desktop computer to do my work: mostly writing some ColdFusion and cutting up images in Photoshop. For the most part, the machine worked fine, except whenever I scrolled down on a webpage in Netscape, the box's PC speaker would start screaming. I tried to ignore it, but I was in a small room with two other nerds and it was getting really annoying.
One day, my frustrated colleague decided it was time to investigate. As part of the process, he noticed that there was very little air coming out of the power supply fan port. "Must be a busted fan and that's causing an overheating alarm" he declared. We unplugged the box, popped it onto my desk, and opened it up. The internals were pristine. No dust, nothing. The computer was practically brand new after all.
We were just about to crack open the power supply (not recommended, but we were getting desperate) when my colleague noticed that something was wedged inside the fan... it was the biggest (dead, thankfully) cockroach I have ever seen, at least 10cm long. For context, this was in Vancouver, Canada. Those just don't exist there.
My colleague jammed his screwdriver in there to grind up the cockroach carcass, then plugged the box back in. After a shittone of dessicated cockroach guts spewed out the back, my little computer was operating normally.
As best we could guess, the monster crawled into the fan in Malaysia when as was assembled and somehow got pinned in there, died of starvation, and then stowed away to Canada.
Wildmimic@anarchist.nexus · 16 pts · 11h
And this is the way bioinvasor species spread - most of the time they don't come delivered inside of PSUs but simple crates, which improves their survivability lol
blurry@feddit.org · 24 pts · 13h
Cat vomit... my cat hit my raspberry pi.
potter2010@lemmy.ca · 4 pts · 6h
Sounds like a cat. The entire place to vomit and it finds the most expensive/inconvenient thing to lay that stomach lasagna on.
barcaxavi@lemmy.world · 21 pts · 12h
You don't need a mouse for a server. Just SSH into it, duh.
wreckedcarzz@lemmy.world · 7 pts · 10h
Now now don't sshush the server, it's doing the best it can, okay? It gets hot in that box...
barcaxavi@lemmy.world · 3 pts · 6h
Well, if it could think outside the box every once in a while, that would be great y'know.
Treczoks@lemmy.world · 16 pts · 8h
Not selfhosting, but I remember standing in a server room with a hands' width of water on the ground (and rising), and hastily shutting down servers where the water was already in the case. It were big towers (Compaq Proliant, if anyone remembers), and luckily the board, drives, and PSU were in the top part of the big tower cases, the drive bays at the bottom were empty.
We got all equipment out safely, and we did not get electrocuted, either. We left with the last equipment when the water level was approaching the knees.
hersh@literature.cafe · 6 pts · 5h
Reminds me of a time at my old job. It was a small office and this was before "the cloud" was a buzzword, so we had a little server room running a few boxes for AD, Exchange, and the thick end of our thin clients.
This "server room" was basically a large closet behind my office, retrofitted with its own AC.
We had residential units on the higher floors of our building. Residents occasionally thought to themselves "you know what makes a great hanging rack? Sprinkler pipes!" Hijinks ensued.
That was a fun call to get at 10PM.
Shoutout to the facilities guy who immediately thought to run a makeshift tarp over the server rack, fashioned from trash bags. No permanent damage.
blueduck@piefed.social · 5 pts · 7h
Did you try turning the faucet off first to stop the water from rising while you were working?
teawrecks@sopuli.xyz · 14 pts · 2h
Optical or roller ball?
DrunkAnRoot@sh.itjust.works · 13 pts · 7h
Fail2ban crashing on 3 seperate days i tried setting it up then one day i did the exact same thing an it worked
deft@lemmy.wtf · 13 pts · 6h
One time when I was 10 my PS2 wouldn't work so I threatened to punch it if it didn't work again. It didn't work so I punched it. Then it worked.
Hamartia@lemmy.world · 7 pts · 6h
Loving the character development arc.
lost_faith@lemmy.ca · 7 pts · 4h
Ahh yes, the "Impact Fix" works everytime til it doesn't. Had a monitor with a loose wire to the power button. For almost a year a forceful tap in the correct spot would make it work, til one day
stardustpathsofglory@lemmy.world · 4 pts · 1h
Percussive maintenance
gblues@lemmy.zip · 2 pts · 4h
lesson learned, apparently 🤣
UxyIVrljPeRl@lemmy.world · 12 pts · 13h
Well at home i only have problems with spiders in my racks, not like the racks i have seen at work. Highlights are toiletpaper, chipmunks+nuts and worst in a school server rack a gun.
wreckedcarzz@lemmy.world · 16 pts · 10h
The gun is for the printer if it tries its bullshit one more time
devaly@ani.social · 11 pts · 13h
Had a problem where my ports would suddenly close. At first I thought it was Telekom, then the modem, then the NAS.
After much investigation I discovered that restarting my router would solve it.
After some more weeks I discovered that the crawling bots were triggering some stateful tplink firewall that would close everything until rebooted.
So the router in the middle of the setup was the problem, not the edges.
tenchiken@anarchist.nexus · 10 pts · 12h
Mine would be when I had a 48 bay disk array / JBOD fail on me... badly. After a storm, it killed the larger card that allowed for me to get many of the drives into a PCIe 16x slot, and I was relegated down to only getting ~8 disks per box made from spare hardware. A single box I got 16 going. Add to this mix an SSD for mid-line caching.
These were all running bcache on top of mdraid... One single mount.
Yes I understand how obnoxiously stupid it was to run RAID6 on a 48 disk volume. It was almost all just stuff I could re-acquire over time, not irreplaceable things.
I just HAD to solve this one though.
In come several spare chassis / mobo etc.... get a bunch of drives powered and on /dev/ , move to the next.
A couple spare gigabit switches...
several gigabit NICs...
two explicit paths for each machine...
a bit of iSCSI magic, and one machine now had the physical disks all exposed to it... mdadm --assemble blah blah, bit of UUID chaos...
It's surprising that while a bit speed limited (I think I got just around 110MB/sec reads), it was nicely performant for what a huge mess of wires and disks just strewn out around my rack.
Managed to evacuate all I needed without much issue once I got that going. Now, I try to keep my arrays under 16 drives at a time, or keep a very rigid policy of "I can lose this and don't care" vs "this box gets RAID10 and/or offsite backups nightly".
back up your critical stuff people!
Shimitar@downonthestreet.eu · 7 pts · 12h
48 disks seems a nightmare in power consumption, failure rate, and overall cabling management....
What was the total capacity?
tenchiken@anarchist.nexus · 5 pts · 10h
Power isn't so bad really. These were all 2tb disks, so ended up with about 40tb total after shenanigans.
Cabling was atrocious. I'd originally been using several pulled 4 disk SAS backplanes from servers thrown out of my work, just screwed down to some pine boards in a stack.
Zip ties and sata/sas cables were the rule.
Overall the drives had 4 small power supplies initially, 300watt iirc. Cheap.
Somewhere I have an old photo from the wiring during the "rescue" but I have to look around.
Disk failure was actually not an issue for ages before. A specific thunderstorm with 2 nearby tornadoes is what caught me this time. Before that, the occasional single disk every 6 months or so.
I still have running arrays using 15 year old disks, many with more than 10 years of that powered on. If you find good models, they can last ages.
ikidd@lemmy.dbzer0.com · 10 pts · 5h
Goddamn I hate mice. I just had to tear out the entire contents of a combine cab to get rid of their piss and shit, and fix the wiring they chewed off.
CompactFlax@discuss.tchncs.de · 10 pts · 9h
I’ve got Pfsense installed on a device with 6 RJ45 ports. 0 is wan, 1 is lan0, 2 is lan1, etc up to 4. Why I have multiple lans is somewhat tangential.
One day it rebooted due to a power outage and WAN worked but lan1 and 2 and 3+ were messed up. I’m still not entirely sure what happened, but the second physical Ethernet port was non responsive at boot and Pfsense just assigned the ports as if it never existed. I moved all the cables over one except LAN and it worked just fine. Except that of course in the troubleshooting i had moved all the assignments around and broke it thoroughly, so it took a while.
BartyDeCanter@piefed.social · 9 pts · 3h
Not exactly self-hosting, but I've worked at two different academic labs that had unique issues and small budgets.
The first was on top of a mountain at 3km elevation. The low air pressure meant that all hard drives had a lifespan of no more than a year before starting to have issues. This was in the pre-SSD days. The "solution" was to boot from a floppy, never compress data, and have a student return the drives to a low elevation and swap them for the prior ones every six months.
The second was at a place that was using old Datacube MaxVideo hardware that we would never have the budget to replace. It would crash if the room temperature got above about 18C, so we set the thermostat to 15C. Which doesn't sound too bad until you've tried sitting at a desk for 8 hours where everything is 15C and the heat is just slowly leaving your body into the chair, the desk, and the keyboard. And then you walk outside where it is 32C and instantly start sweating because you're still in several layers and fingerless gloves.
shrugs@lemmy.world · 8 pts · 12h
Not self hosted but at work: after an update to both our firewall cluster nodes they both acted wonky, restarted 5 min after booting, ssh connection wasn't working half the time, dropped packages... the problem persisted through multiple reboots, support had no idea.
the fix was to pull the plug entirely, wait 10 seconds then power them on again. on both nodes we had this behavior and both worked again after losing power for a few seconds.
seems that even for business appliances "have you tried to turn it off and on again" is a viable solution.
Decronym@lemmy.decronym.xyz · 8 pts · 12h
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:
[Thread #80 for this comm, first seen 18th Aug 2026, 08:10] [FAQ] [Full list] [Contact] [Source code]
justme@lemmy.dbzer0.com · 7 pts · 12h
for me it was the fact that the desktop case i used for the server had the power button on top and it was big enough to be pushed in, when my cat walked over it...
Wildmimic@anarchist.nexus · 9 pts · 11h
my cat currently loves hitting the "Sleep"-button on my keyboard every time he jumps on my table -.- at least i can wake it again, but it IS annoying. Haven't had the time yet to reconfigure that button to instead run a soundfile of me scolding my cat lol.
rimu@piefed.social · 7 pts · 13h
Nice chonky heatsink, goddam
Gekkonen@sopuli.xyz · 4 pts · 13h
I used this, among other things, for my personal Blender renderfarm. Cooling was a must.
rimu@piefed.social · 5 pts · 13h
haha looks like you're aaalmost at the point where water cooling is compulsory.
Is it a Threadripper? I built a couple of those for architectural rendering, such absolute beasts.
Gekkonen@sopuli.xyz · 2 pts · 12h
I honestly don't remember now as this was over 10 years ago.
16mhz@lemmy.ml · 2 pts · 9h
10 years!! aren't you woried about dried up thermal paste? I think Re-pasting CPU/GPU every 2~3 years or so is a good practice to prolong longevity.
Gekkonen@sopuli.xyz · 2 pts · 9h
Ah, sorry for the confusion - that server's been out of commission for a few years already.
UltraBlack@lemmy.world · 6 pts · 8h
My server once randomly kernel panicked once (and only once) and it took me an embarrassingly long time to figure out that a ram stick (too poor for ECC) had gone bad. Also took all my proxmox backup server backups with it because the validation would always fail (also due to the ram stick). Still worked fine otherwise lol
Funkt4st1c@lemmy.world · 6 pts · 3h
I had a screen get burn in (forgot to turn it off, oops) removed it from the laptop, waited 3 days, put it back in and it was flawless again.
I had a laptop that refused ethernet and would only ever use wifi... When the ethernet was connected. No wifi unless that cord was connected. Confirmed it was, indeed, wifi by using a dummy connector.
Bought a 10tb hdd that refused to read anything at all, opened it for giggles/curiosity and found a piece of scotch tape stuck to one of the disks and one of 4 disks completely shattered. The disk was still factory sealed (by appearance, anyways; they still had the antitamper stickers)
Buage_@piefed.social · 4 pts · 3h
I dunno if that counts or if it's the weirdest cause I rarely have any issue, but one day, I wanted to deploy speedtest tracker via docker on my 1st NAS..
So I tried, but it didn't work (I forgot the reason). So I try to check the logs, but it turns out my NAS was insanely slow.
When I rebooted it, it couldn't stop beeping and telling me there was a hardware problem, so I tried everything I could do to fix it, but nothing worked, so I ended up reinstalling the whole OS
Keep in mind this was the cost to see I had a bad result at the speedtest lmao 😭
Btw, I also lost all my important files, yayyy!
psycotica0@lemmy.ca · 4 pts · 8h
I thought that heatsink and fan was a spindle of CDs! I thought that was an interesting place to store them I hadn't seen before...
Reannlegge@lemmy.ca · 3 pts · 3h
I do not care to admit how many times when I was first getting into home labbing that I could not figure out why I could not reach a pi or the internet. All because I either killed an SD card for pihole or just plain out forgot to put one in!
Schlemmy@lemmy.ml · 4 pts · 2h
I took my SD card out of the rpi to reinstall home assistant. Terribly annoyed because I had no backup of my configuration.
When reinstalling my SD I found out it was the ethernet cable that wasn't plugged in.
Shimitar@downonthestreet.eu · 3 pts · 13h
I use e self made wood cabinet in the attic where also the PV inverter is located. I used Velcro to strip on the wall (brick wall) stuff like switch firewall etc, only to find those collapsed on the server itself every time. No matter which glue or adhesive i used.
At the end, a strong unbranded Chinese double sided tape was better and still strong after all big western brand failed over time.
Maybe a mix of heat and uneven surface was the culprit, who knows.
irmadlad@lemmy.world · 3 pts · 4h
I thought it was the huge heat sink that looks like it extends past the side plane of the cover. Reminds me of something I did with a CoolAir Cosmos tower. I mounted two 300 cfm fans over each CPU and took the side panel, neatly cut out the area around the fans, then 3d printed a scoop, with a slot for filtration, to fit on the side panel and allow the fans to breathe.
Bebopalouie@lemmy.ca · 3 pts · 2h
Back in the day of BBS’ my buddy had a wonky modem. If I picked up phone , pursed my lips and did a squeaky sound the modem worked and I could log into BBS.
KyuubiNoKitsune@lemmy.blahaj.zone · 2 pts · 7h
Fitting a NAS on top of my cupboard. Been reconsidering putting it there though, may hide it under a table somewhere instead.
I should note that I haven't built the nas yet because of these space constraints.