I'm a programmer by trade with twenty years of experience, having first used Linux in 2004 (iirc), and I thought I knew the game pretty well. There are CPU-bound tasks, memory-bound tasks, network-bound tasks, and disk-bound tasks. I'm doing async programming and it's working fine, offloading i/o to the kernel while my app is doing cpu stuff. Except, I'm now using a laptop with 8 GB of memory and Linux Mint-Cinammon on HDD as my main machine, and somehow shit is constantly in the swap and waits for the disk. Right now I'm moving a Windows partition, I have quit everything except four Firefox tabs, Emacs, and Double Commander — and I still think I might kill myself before the ordeal is over. The entire system is completely and totally fucked. Barely anything moves, nothing responds in any kind of a timely fashion. Emacs freezes when typing, with two open buffers. Audio playback stutters. VPN is nonresponsive. I closed all the hundreds of buffers in Emacs, I set the priority and io priority of Gparted to the minimum, and the other apps to the max. I set swappiness to zero, and turned off the swap altogether with swapoff. All of this barely helps. I'm drunk right now, and still this affair bugs me beyond measure. I could maybe deal with this as a one-time thing if swapping and disk i/o weren't incapacitating the system every day.
Is Linux kernel so ridiculously single-threaded that any i/o cripples the entire thing? Or is something tuned horribly wrong in Mint by default? Can I unfuck this somehow? I live close to a river, I can't take this much temptation.
36 Comments
phi0@lemmy.zip · 14 pts · 3d
Is this satire?
SlurpingPus@lemmy.world · -16 pts · 3d
Of course, because any and all discussion of Linux is dick-measuring, and never actual questions. Someone actually having a problem? Imagine that! It would be so ridiculous.
Onomatopoeia@lemmy.cafe · 5 pts · 3d
You're moving a partition...
SlurpingPus@lemmy.world · -11 pts · 3d
Apparently you don't grasp the difference between the disk, the CPU and the memory. Learn some shit before commenting on the web.
infeeeee@lemmy.zip · 11 pts · 3d
This may have something to do with the unresponsive HDD
The other thing that may cause this symptoms.
frongt@lemmy.zip · 5 pts · 3d
"I'm completely saturating disk I/O and I have no idea why it's slow"
No shit dude you're reading and writing tens of gigabytes to the same disk you're trying to do other stuff on. Of course it's going to be slow. That's like worst case scenario. Look at whatever system monitor Mint uses and you'll see.
SlurpingPus@lemmy.world · -11 pts · 3d
True, me being drunk definitely explains why the system has to work like shit for five hours straight. Curiously, it doesn't behave like that on other days when I'm plastered but amn't moving partitions around. Odd thing, isn't it?
Dran_Arcana@lemmy.world · 6 pts · 3d
The Linux kernel (and windows btw) often lock threads to the completion of reads or writes to disk. When your primary drive is a spinner and you saturate it's bandwidth with a long-running copy (your from windows copy) you take those normally short processes and add the latency penalty of a thrashing disk to every lock.
Yes you can fix this by moving the entire kernel and operating system into ram at boot, but with 8gb you probably don't want to do that, especially when a couple of Firefox tabs will fill that up pretty quickly.
The solution is drop the gui, load the os into ram and disable logging, or be patient, and don't expect any modern operating system to hold up when you are running long copies against a high-latency root disk.
SlurpingPus@lemmy.world · -7 pts · 3d
Yeah, I do know that i/o is mostly single-threaded (except maybe for when multiple disks are involved, not sure about that). However I'm baffled as to why the entire system is impaired when i/o is done. It feels like the kernel has only one thread doing anything, and the userspace apps are constantly tugging on the kernel for some reason. It makes no sense. Userspace apps should be running on the CPU without the kernel's involvement, but for some reason they don't. I have eight cores (with hyperthreading) doing fuckall when the disk is busy.
Dran_Arcana@lemmy.world · 1 pts · 2d
That is... not remotely how any of that works. Do you want a deeper dive into this?
SlurpingPus@lemmy.world · 1 pts · 2d
Please do tell then. What do apps do after they handle i/o out to the kernel in a dedicated thread? Do they just sit around and wait, in your opinion?
Dran_Arcana@lemmy.world · 1 pts · 2d
Well first, the linux kernel is open-source. This is not my opinion and everything I'm telling you is auditable.
programs are generally written to wait for writes to close. There are good reasons for this. The simplest example I can think of would be a basic text editor's "save and exit" function. It writes the data, and then closes. If it didn't wait for the write to finish, and it closed mid-write, you'd lose half your data. Now remember that this doesn't just apply to editors. It applies to logging functions, updating programs, creating a shortcut, changing a setting in a menu, etc... The program relies on kernel signaling to know when a file has "finished" writing. In the case of datacenter high-performance filesystems or raid controllers, it will use memory (think: ram) as cache for the writes and the kernel will report back to the process that the write is finished, even while a little bit of the file is still in ram and waiting to be flushed to disk. This is safe because those systems have multiple redundant power supplies and/or backup batteries. In the case of a desktop, you typically don't do this because a power flicker or a system crash could take data with it. Modern SSDs make this somewhat of a non-issue because they're still relatively fast at small writes while under load. The difference on a small write between a thrashing ssd and a thrashing hdd can be the difference between 200 nanoseconds (nvme ssd) and 15 milliseconds (hdd).
Programs need to block on writes sometimes as a mater of procedure to ensure data integrity, and you're hitting the worst case possible. 50,000-500,000 times slower than a modern enterprise filesystem, 1,000-10,000 slower than a bog-standard, modern consumer-grade nvme ssd.
Operating systems are generally built with the assumption of mid-range consumer hardware from 4-6 years ago, and spinning disk root volumes haven't been standard for over a decade. There are absolutely things you can do to make a system play nicely with a spinning root volume but modern general-purpose operating systems tune for different assumptions. Almost none of them tune for a 15ms write penalty on logging.
SlurpingPus@lemmy.world · -1 pts · 2d
Thanks for the info, but I'm quite familiar with how i/o works in general. I know what the write cache is. I know what journaling is, and FYI it's doing its job on desktop systems too. I know how flushing affects performance (for the worse). I know how to disable flushing in databases to gain performance, and how to force it to gain durability. I even know about onboard batteries in some disks. I know about the orders of magnitude in delay between different interfaces, I'm obligated to be versed in that for my job. The only problem is, I'm not a sysadmin.
None of this explains why Firefox is constantly swapping in and out when it just displays pages on the screen, presumably writing slightly more than bupkis on the disk, and why the entire system is slow as molasses when some disk i/o is happening but I'm not saving anything.
None of that applies to Firefox when it's just displaying pages, and Emacs when it's showing me buffers and running code that's already in the memory.
Thanks again for the info. One might even think that I created the whole post to ask what specifically I could do about it, explicitly saying so in the text, and fortunately you're here to inform me that there are indeed things I can do. How lucky I am to receive this knowledge that there is something I can tune somewhere.
Dran_Arcana@lemmy.world · 2 pts · 2d
firefox does a lot of disk caching during regular use, mostly from webapps trying to be clever about using your disk space to cache chats, media, etc. there's functionally nothing you can do about this per-site. It also does a lot of journaling in local sqlite databases for its own cookies, history, caching, etc.
if you want actual solutions:
use a DE like lxde to free up system memory as much as possible.
move firefox's caching and profile directories to /dev/shm and accept the memory footprint penalty on launch. accept that a crash or shutdown without flushing back to disk will result in losing this data.
add an override to journald.conf.d to move the journal to volatile storage with a functionally nonexistant rolling size, but just big enough not to crash (~128mb)
disable rsyslog
create a systemd service to move /tmp /var/tmp /var/log and /var/cache/apt/archives to ramdisk on boot.
disable any systemctl timer that you don't absolutely need (apt-daily.timer apt-daily-upgrade.timer, logrotate.timer etc...)
increase the journal commit interval on your root filesystem to something like ~60s
SlurpingPus@lemmy.world · 1 pts · 2d
Thanks for proper advice. I think LXDE wouldn't help much, since most memory is likely used by the apps instead of the system. Curiously, I enabled memory compression which was off by default and which helped me a lot with low-memory machines in the past — it's set to use 40% of the memory currently, but I might need to raise that. Unfortunately, I don't think it makes too much difference in the end.
I'll probably fiddle with the journaling parameters and rsyslog's options first, since those seem fairly harmless. Don't want to lose useful functionality right away.
I'm gonna move the system partition to an SSD sooner or later just to save what remains of my sanity, except the swap won't be there to avoid the device being hammered to death in a month or two before I can splurge on some mem upgrades. It's remarkable how twenty years ago I could glide through the filesystem as fast as my fingers diddled the cursor keys; or switch through images at 60 fps or so like a Terminator — but now I somehow have to constantly wait for the apps even with an SSD. Not quite sure what happened, but shit's wack.
Ironically, I'm a webdev, so I'm somewhat sure that the sites I use don't behave that badly — and I just can't move so fast as to cause much i/o with the history and whatnot, exactly because it's barely responding when shit hits the fan. But I need to trace FF one of these days to figure out what the hell it's doing. Maybe the extensions are the culprits.
breadsmasher@lemmy.world · 0 pts · 2d
You should stop drinking.
SlurpingPus@lemmy.world · -1 pts · 2d
So you're now following me around to chime in with your bullshit when I'm talking to someone who has actual knowledge, because your fragile ego's self-worth is tied to the OS you like and you can't accept it being scrutinized a little? You should probably stop using Lemmy, because I never did such pathetic crap even when on a bender.
By the way, MacOS is optimized way better than Linux in every way. Deal with it.
CallMeAl@piefed.zip · 3 pts · 3d
yeah linux is completely unusable bro /s
SlurpingPus@lemmy.world · -4 pts · 3d
That would explain things, if I weren't deploying my backend apps on Linux all this time, except for like three years with sysadmins who preferred FreeBSD. Maybe that's my sin, dunno.
notabot@piefed.social · 2 pts · 3d
You might want to check the SMART stats on the disk, if it's struggling that much, it might be because it's starting to fail. That's assuming it's the internal disk, if it's external, make sure you're using an appropriate port and cable. I've never seen a linux box struggle the way you're describing without it being a hardware problem, but technogy's always throwing up new surprises.
SlurpingPus@lemmy.world · -3 pts · 3d
Good call, thanks. Though Gparted showed 30+ MB in and 30+ out, for a total of 70 at times, which is pretty good for an hdd in my experience.
However, this doesn't explain why the system is so tied to the disk.
thenextguy@sh.itjust.works · 2 pts · 3d
Try changing your io scheduler.
SlurpingPus@lemmy.world · -2 pts · 3d
Interesting, somehow I missed the fact of the existence of different i/o schedulers, although I knew about the cpu schedulers. I'm definitely gonna look into this, thanks!
Oddly, the manpage for ionice says that the cpu priority is used as the primary input for the i/o priority, which feels kinda limiting. Perhaps this can be changed.
breadsmasher@lemmy.world · 1 pts · 3d
are you running it from a spinning disk? HDD, not an SSD? Thatd be why its so slow.
SlurpingPus@lemmy.world · -15 pts · 3d
Thanks, this is very helpful. I'm cured now.
breadsmasher@lemmy.world · 3 pts · 3d
Whats your problem?
BurgerBaron@quokk.au · 1 pts · 2d
They're drinking alone at home troubleshooting linux with a bad attitude and didn't change the next day...alcoholic?
SlurpingPus@lemmy.world · -8 pts · 3d
Hm? I described it in the post, but I gotta admit my day really went downhill when you showed up.
breadsmasher@lemmy.world · 2 pts · 3d
Right .. sorry for trying to help I guess.
SlurpingPus@lemmy.world · -4 pts · 3d
breadsmasher@lemmy.world · 1 pts · 2d
HDDs are slower than SSDs.
Youre doing heavy data movements.
You then decide to attack me.
????
SlurpingPus@lemmy.world · -3 pts · 2d
Such attack, wow.
Thanks for informing me that I could use an SSD if I had a shitton of money for the memory to avoid the swapping so that the SSD wouldn't be smashed to pulp in three months. I could've never guessed that on my own, wow. You're a genius.
Diplomjodler3@lemmy.world · 0 pts · 3d
Linux Mint usually works well on older hardware. There are limits though and it looks like your PC is simply too old. There are plenty of distros that are specifically designed to work on older hardware. I suggest you try one of those.