I’m looking for storage classes for a multi node cluster. I’m currently using Longhorn and NFS, but I'm not happy with the performance. My cluster doesn’t have beefy nodes, so Ceph/Rook is out of the question (for now).
Nodes:
- 8 GB RAM, 4 cores VM, control plane. 256 GB SSD
- 4 GB RAM, 2 cores, control plane, currently cordoned. 128 GB SSD
- 8 GB RAM, 4 cores, ARM, control plane. 512 GB SSD
- 8 GB RAM, 4 cores. 256 GB SSD
- 16 GB RAM, 6 cores. 256 GB SSD + 1 TB HD
- RPi 4, 4 GB RAM. 128 GB SSD
25 Comments
supersheep@lemmy.world · 4 pts · 196d
I’m currently using Piraeus / LINSTOR and am quite happy with it: https://github.com/piraeusdatastore/piraeus
eutampieri@feddit.it · 2 pts · 196d
Found a Reddit thread that says that LINSTOR has a lower CPU usage (which is my main gripe with Longhorn). Might as well try this and report back. Is there a good way to migrate PVs and PVCs?
F04118F@feddit.nl · 2 pts · 195d
Volsync
eutampieri@feddit.it · 1 pts · 195d
Thanks!
supersheep@lemmy.world · 2 pts · 196d
I can confirm that the resource usage is quite low indeed. I only used it with Nomad instead of Kubernetes, so I can’t comment on how to best migrate PVs and PVCs.
eutampieri@feddit.it · 2 pts · 196d
Thanks
nebula@lemmy.ca · 1 pts · 194d
It's great, I spent 2 years finding perfect CSI for homelab and landed on Piraeus. The best part is you get full read performance of your local disk so I didn't have to use 10G, write are limited by network link between nodes. But that hasn't been a problem for me. Also, they're super responsive for any issues/bugs you hit.
Let me know if you have any specific questions about this.
eutampieri@feddit.it · 1 pts · 194d
Thanks, I will!
Decronym@lemmy.decronym.xyz · 3 pts · 196d
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:
[Thread #95 for this comm, first seen 15th Feb 2026, 11:20] [FAQ] [Full list] [Contact] [Source code]
surewhynotlem@lemmy.world · 1 pts · 156d
Gotta add CSI to that
moonpiedumplings@programming.dev · 3 pts · 196d
Openebs mayastor
But you could fit ceph on that I think. As long as your network between nodes is fast enough.
silenium_dev@feddit.org · 3 pts · 196d
Mayastor or Linstor, Ceph requires too much CPU for these nodes
eutampieri@feddit.it · 2 pts · 196d
I used to use Ceph at work and I'm a bit reluctant to use it at home. Don’t get me wrong, it’s really cool, but those were beefy nodes, and I only have 1 Gbps between nodes
custard_swollower@lemmy.world · 3 pts · 196d
I’m not sure you’ll get nice performance in local network with small appliances (consumer network hardware, mini PCs and rpi 4). I’ve never got sub-ms network disk access on 1Gbps switch and router. In the end I’ve done the opposite - I’ve added one k8s host with a lot of storage, and any storage services are deployed there. All the other k8s services rely on local SSDs.
eutampieri@feddit.it · 1 pts · 195d
Which CSI?
h3ron@lemmy.zip · 2 pts · 195d
I have two storage nodes and one is much faster than the other.
I'm currently evaluating a juicefs deployment based on two minio instances (one per node, replicated with async bucket replication) through a load balancer (sidekick) in failover. Because juicefs also needs a db for metadata, I went with valkey + sentinel.
Juicefs provides a CSI driver that supports ReadWriteMany volumes and CSI snapshots and manages both read and write cache. Performance is much much better than Ceph. In theory it should be riskier (because of the async replication) but in practice I haven't yet lost a bit.
eutampieri@feddit.it · 1 pts · 195d
Thanks! A bit more involved that I’d have thought but still worth considering! Could you update us after your evaluation?
h3ron@lemmy.zip · 2 pts · 194d
Well actually it is very easy to spin up in docker and most of the configuration happens through env variables.
juicefs itself only exists on the client side, so you basically only have to install and configure the CSI driver with helm.
as it took me a few days to come up with this solution I'd be happy to share my config files.
Performance wise is quite fast on sequential reads (it saturates my 2.5G bandwidth) and slower than I expected on sequential writes (for me it caps at 60MB/s). Postgresql seems happy. I saw no visible performance degradation with Authentic, Immich and Opencloud. Nextcloud installation took ages. I've yet to try it with jellyfish and the *arr suite.
A simple NFS share would be faster, but it doesn't support replication, failover and CSI snapshots.
performation@feddit.org · 1 pts · 192d
I started setting up longhorn today. My hardware is a little it beefier, but I still am curious: what performance problems did you run into exactly?
eutampieri@feddit.it · 1 pts · 192d
My greatest problem is that the CPU load is too high and I ran into an issue with iSCSI that would occasionally peg one core to 100%. It would also make nodes NotReady when too many PVs were scheduled
performation@feddit.org · 1 pts · 191d
What CPUs are you running exactly? I was under the assumption my greatest bottleneck would be an 1Gbps LAN
eutampieri@feddit.it · 1 pts · 191d
Just don’t go below 4 cores of x86 and you’ll be fine
eutampieri@feddit.it · 1 pts · 196d
Has anyone tried this https://github.com/awslabs/mountpoint-s3-csi-driver?
Perhaps with a Garage DaemonSet as a backend?
jonathan@piefed.social · 2 pts · 196d
I'd expect the performance to be awful but it still has relatively niche usecases, especially where performance isn't a concern. I'm imagining legacy apps that don't speak S3.
eutampieri@feddit.it · 1 pts · 195d
Thanks for the feedback! So not what I’m looking for