Instant boot Linux?

Hi all,

Is there a method out there that can "save a hibernate state" to disk, and just permanently boot immediately into that state every time?

It would be OK to lose data used after this point. And ot would also be OK if it required building a custom image based on the hardware, similar to embedded systems.

The goal is to be able to, as quickly as possible, spin up a GPU server when needed, but without using suspend/resume.

Thanks.

31 points · 27 comments · view on lemmy.world

27 Comments

mlfh@lm.mlfh.org · 22 pts · 14d (1 reply)

I was really curious about the possibilities here for a re-usable "saved hibernation state", and did a lot of digging around and testing in exploration of that idea. Please note that the following is wacky as hell and is not a good way to run a server at all, but is possible:

Set up a swap partition that can fit your full system memory.
Enable swap and set up hibernation (https://wiki.archlinux.org/title/Power_management/Suspend_and_hibernate#Hibernation )
Hibernate your system to the swap partition in the state you want to use as your perpetual checkpoint.
Boot from a live usb, NOT your regular system (when the linux kernel boots with the resume=uuid=xxxx parameter which enables hibernation/resume, it checks that partition for a hibernation signature and immediately clears it, preventing multiple boots of the same hibernated state. You need to preserve this signature).
From the live usb system, copy the swap partition that contains your main system's hibernated state to an .img file on a filesystem mounted from one of your main disks.
Reboot into your main system.
Set up a scheduled job that triggers every time your system resumes (could use a systemd service with After=hibernate.target and WantedBy=hibernate.target) that does the following:
1. Disables swap on the swap partition you set up (keeping the resume=uuid=xxxxx kernel parameter that points to it).
2. Writes the .img file to that partition
Use your system as you want, and power it off when done (do not suspend or hibernated again).

The next time you boot, the kernel will load the saved hibernation state from the partition specified in the resume= parameter, as if it had just been hibernated instead of halted.

Now for the downsides:

It's very dangerous to resume a system's memory to one state with storage in a different state (the official kernel docs for this have a giant red warning banner saying "If you touch anything on disk between suspend and resume... ...kiss your data goodbye." https://docs.kernel.org/power/swsusp.html ).
Resuming from hibernation isn't really that much faster than cold booting a lean system, if speed is your primary goal.
The scheduled write job will hit you with a bunch of heavy io every time you boot/resume, probably negating any speed benefits of the whole thing anyway.

Anyway, now that we know it's possible, if you really can't suspend/resume in a non-wacky way I recommend instead just booting normally and trying to make the boot process as fast as possible. A good guide here: https://wiki.archlinux.org/title/Improving_performance/Boot_process

If booting quickly to a static state is the goal, alpine diskless mode might be a good option: https://wiki.alpinelinux.org/wiki/Diskless_Mode

Fmstrat@lemmy.world · 5 pts · 14d

First, that's super smart. I've got hibernate to LUKS in LVM working already, so saving the state males sense. However, speed of resume being the issue doesn't he'll much, to your point.

The Diskless Alpine looks very interesting. Do you know if there is a Debian version?

Die4Ever@retrolemmy.com · 12 pts · 15d (8 replies)

Pretty sure you just have to enable hibernation and give it a file to write to? Then just set your PC to go into hibernate on idle

I kind of doubt you would have to change distros. What distro are you on now?

Fmstrat@lemmy.world · 1 pts · 14d

Oh hibernate to LUKS I have working. What I want is a permanent static version of that. I.E. boot to the same spot every time. This way I'm not writing 64GB to disk every time.

HelloRoot@lemy.lol · 0 pts · 14d (6 replies)

For my two systems (pc and laptop)with 32GB RAM each it takes ~4 minutes to boot from hibernation.

Which is expexted afaik.

Thats the opposite of "instant".

khleedril@cyberplace.social · 6 pts · 14d (5 replies)

@HelloRoot @Die4Ever There is something wrong with your setup.

HelloRoot@lemy.lol · 1 pts · 14d (4 replies)

No not really.

It's how long it takes to read 32GB from my swap. 170 MB/s continuous read speed ≈ 188 sec (3 min 8 sec) add the normal boot time to it (bios, grub, kernel initramfs) and you get the 4 minutes.

When the system boots cold it doesn't have to read 32GB, just 0.5-2GB maybe.

Fmstrat@lemmy.world · 4 pts · 14d (1 reply)

If you are writing the full 32GB, something is wrong. Shared memory isn't written twice.

HelloRoot@lemy.lol · 1 pts · 14d

You might be right. But I still have no clue how to properly set it up.

Sims@lemmy.ml · 3 pts · 14d (1 reply)

Just for the record, Linux can do with less swap-space than the memory you have, also for hibernation. The system doesn't write every byte to swap, and it uses compression. With 32gb I would choose a default swapsize of ~10gb max, but test it first as it depends on the specific content of your ram, and usage patterns oc. However, its an instant time-saver during boot.

HelloRoot@lemy.lol · 1 pts · 14d

Didn't work for me, it would cancel the hibernation saying there is not enough space in swap. I followed the setup instructions on arch wiki.

Do you have a guide for that?

dgriffith@aussie.zone · 3 pts · 14d (1 reply)

Firstly, suspend might be an option compared to hibernation for your subsequent startups if you can suffer a slow startup the very first time.

Otherwise look up "Linux from scratch" .

A lot of boot time is decompressing and mounting initrd, loading modules there, mounting the root filesystem, jumping to that.

Build a kernel that has exactly the minimum modules needed for your system, skip initrd and jump straight to a root filesystem and call your init binary (it can be BusyBox even) do exactly what's needed there to get your network+graphics card up and running with a single shell script, done.

Also look at COW filesystems. Read only base with an overlay file for writes. Discard that file on each boot and you basically have a system that is always at the same state on startup.

Fmstrat@lemmy.world · 1 pts · 14d

Yea, I was hoping to not have to go the scratch route, but we shall see. I do already run ZFS.

Edit: And I'm using suspend now, but am unhappy without encryption at rest. Also, full memory GPUs don't suspend very quickly.

FerrisEuler@programming.dev · 3 pts · 14d (1 reply)

You might check out coreboot and PCs that support it https://www.coreboot.org/end_users.html

Fmstrat@lemmy.world · 2 pts · 14d

Interesting, I don't have this hardware now, but will track this one.

frongt@lemmy.zip · 3 pts · 14d (7 replies)

Are we talking physical servers like Poweredge or Proliant? Those take a long time to boot, especially with lots of RAM. Why can't you use suspend/resume?

And modern servers don't really consume a lot of power while idle. You can configure them to downclock the CPU and GPU while idle just like a desktop or laptop.

But if you really wanted it, you could boot an immutable distro and mount user partitions tmpfs. Or use a portable distro and discard the state. Or a filesystem with snapshots and roll back on each shutdown or boot. But none of those are really going to get you instant boot. Linux boots pretty quick, but not instant.

Fmstrat@lemmy.world · 1 pts · 14d (4 replies)

Desktop with GPUs. I've tried immutable, but that really doesn't impact boot speed. It's still the same startup procedure. And I use ZFS, so I could roll back (or use tmpfs), but that still requires the full boot process.

hirihit640@sh.itjust.works · 1 pts · 14d (3 replies)

What's your use case or broader goal? In some specific cases you might be able to use a VM instead of a bare metal server. The host would be idling all the time, but the VM can start from a snapshot near instantly

Fmstrat@lemmy.world · 1 pts · 14d (2 replies)

Power saving with encryption at rest. I want the GPUs offline power wise.

hirihit640@sh.itjust.works · 1 pts · 13d (1 reply)

Just spitballing here, but if you use a VM with gpu passthrough, the host system doesn't even see the GPU anymore, so if the VM is off I imagine the GPU is basically off as well. In terms of encryption at rest, you can probably make a separate LUKS volume for the VM disk image, and then lock it when the VM is off.

Edit: for example I have a VM with GPU passthrough and when the VM is off, I have no way to check the power consumption of the GPU (aside fom using external hardware) since the host doesn't have access. The host uses a sort of dummy driver for the GPU, so that the hypervisor can pass it on to the VM, so it is actually possible for the host to switch the driver and take control of the GPU when the VM is off. But until then I imagine power consumption is minimal.

Fmstrat@lemmy.world · 1 pts · 13d

I used to do this for my virtual gaming rig, and the catch is still idle draw. Just one B70 pulls over 30W idle, then there's the nvidia card and then CPU etc, too. As this machine is solely for remote development and inference, I want it fully off.

Now, I guess I do have all the confidential stuff in encrypted ZFS, so I can shutdown the docker containers, unmount ZFS, and suspend. This gives me encryption at rest. The problem is the startup/resume is actually slower than you might think.

HelloRoot@lemy.lol · 1 pts · 14d (1 reply)

Linux boots pretty quick

I never managed to make linux boot faster than windows (on the same device ofc).

And to this day I don't understand why.

Out of the 3 desktop OSes (lin, win, mac) linux was always the slowest one in my experience.

Fmstrat@lemmy.world · 2 pts · 14d

Fast boot most likely. Windows can shut down user processes and then hibernate the kernel/system processes. Because the system only changes on updates, it doesn't have to wrote the hibernate file every time, only on update. Never understood why distros didn't look to replicate that. If they had this would probably be a non-issue for me.

fozid@lem.radiantfig.fyi · 2 pts · 14d (2 replies)

Baring post, my pc cold boots in 3 seconds. And does a full shutdown in 3 seconds as well.

Unfortunately my mobo's slow to post so that takes around 5-6 seconds.

I moved from arch Linux to void and its init system called runit is amazing. Mega simple and fast.

Fmstrat@lemmy.world · 1 pts · 14d (1 reply)

Interesting. Did you do any tuning or out of the box?

fozid@lem.radiantfig.fyi · 3 pts · 14d

Out of the box. Void is a very Diy distro though, so don't expect to install it and be up and running. But I have a fully functional pc with sway / Wayland, steam and video editing, so no compromises. I was previously on arch Linux for 15 years and tuned it as hard as I could and couldn't get anywhere near this boot time.

just_another_person@lemmy.world · -1 pts · 14d

Yes