What is your favourite way to transfer files in your homelab?

Hello selfhosted! Sometimes I have to transfer big files or a large amounts of small files in my homelab. I used rsync but specifying the IP address and the folders and everything is bit fiddly. I thought about writing a bash script but before I do that I wanted to ask you about your favourite way to achieve this. Maybe I am missing out on an awesome tool I wasn't even thinking about.

Edit: I settled for SFTP in my GUI filemanager for now. When I have some spare time I will try to look into the other options too. Thank you for the helpful information.

46 points · 74 comments · view on lemmy.world

74 Comments

Kit@lemmy.blahaj.zone · 40 pts · 1y (14 replies)

Not gonna lie, I just map a network share and copy and paste through the gui.

Flamangoman@leminal.space · 6 pts · 1y

Same lol, somebody please enlighten me on a faster way!

theorangeninja@sopuli.xyz · 2 pts · 1y (11 replies)

Sounds very straight forward. Do you have a samba docker container running on your server or how do you do that?

drkt_@lemmy.dbzer0.com · 16 pts · 1y (5 replies)

I just type sftp://[ip, domain or SSH alias] into my file manager and browse it as a regular folder

tacocatgoat@lemmy.world · 8 pts · 1y (2 replies)

YOU CAN DO THAT???

drkt_@lemmy.dbzer0.com · 7 pts · 1y (1 reply)

Linux is truly extensible and it is the part I both love and struggle to explain the most.
I can sit at my desktop, developing code that physically resides on my server and interact with it from my laptop. This does not require any strange janky setup, it's just SSH. It's extensible.

blackbrook@mander.xyz · 4 pts · 1y

I love this so much. When I first switched to Linux, being able to just list a bunch of server aliases along with the private key references in my .ssh/config made my life SO much easier then the redundantly maintained and hard to manage putty and winscp configurations in Windows.

blackbrook@mander.xyz · 1 pts · 1y (1 reply)

Dolphin?

drkt_@lemmy.dbzer0.com · 8 pts · 1y

Any file manager on Linux supports this

Kit@lemmy.blahaj.zone · 4 pts · 1y

I have two servers, one Mac and one Windows. For the Mac I just map directly to the smb share, for the Windows it's a standard network share. My desktop runs Linux and connects to both with ease.

Shimitar@downonthestreet.eu · 2 pts · 1y (1 reply)

Do you really need a container for Samba?

I see the benefits of containers, but a use would be overkill.

GamingChairModel@lemmy.world · 1 pts · 1y

Yeah, if OP has command line access through rsync then the server is already configured to allow remote access over NFS or SMB or SSH or FTP or whatever. Setting up a mounted folder through whatever file browser (including the default Windows Explorer in Windows or Finder in MacOS) over the same protocol should be trivial, and not require any additional server side configuration.

Lv_InSaNe_vL@lemmy.world · 1 pts · 1y

I dont have a docker container, I just have Samba running on the server itself.

I do have an owncloud container running, which is mapped to a directory. And I have that shared out through samba so I can access it through my file manager. But that's unnecessary because owncloud is kind of trash.

Appoxo@lemmy.dbzer0.com · 1 pts · 1y

Set up smb on my file share VM.
My dedicated docker host accesses it through an NFS mount.

GamingChairModel@lemmy.world · 2 pts · 1y

Yeah, I mean I do still use rsync for the stuff that would take a long time, but for one-off file movement I just use a mounted network drive in the normal file browser, including on Windows and MacOS machines.

e0qdk@reddthat.com · 16 pts · 1y (1 reply)

People have already covered most of the tools I typically use, but one I haven't seen listed yet that is sometimes convenient is python3 -m http.server which runs a small web server that shares whatever is in the directory you launched it from. I've used that to download files onto my phone before when I didn't have the right USB cables/adapters handy as well as for getting data out of VMs when I didn't want to bother setting up something more complex.

GamingChairModel@lemmy.world · 2 pts · 1y

Honestly, this is an easy way to share files with non-technical people in the outside world, too. Just open up a port for that very specific purpose, send the link to your friend, watch the one file get downloaded, and then close the port and turn off the http server.

It's technically not very secure, so it's a bad idea to leave that unattended, but you can always encrypt a zip file to send it and let that file level encryption kinda make up for lack of network level encryption. And as a one-off thing, you should close up your firewall/port forwarding when you're done.

sugar_in_your_tea@sh.itjust.works · 15 pts · 1y (9 replies)

What's wrong with rsync? If you don't like IP addresses, use a domain name. If you use certificate authentication, you can tab complete the folders. It's a really nice UX IMO.

If you'll do this a lot, just mount the target directory with sshfs or NFS. Then use rsync or a GUI file manager.

possiblylinux127@lemmy.zip · 3 pts · 1y (6 replies)

Just don't run rsync as a daemon as that's a security nightmare

sugar_in_your_tea@sh.itjust.works · 3 pts · 1y (5 replies)

Why would you do that? That sounds awful...

possiblylinux127@lemmy.zip · 2 pts · 1y

It is, rsync sends data in plain text. There is a optional password that is also sent in plain text.

jollyrogue@lemmy.ml · 2 pts · 1y (3 replies)

The daemon tracks file state, so the transfers start quicker because rsync doesn’t have to scan the filesystem.

sugar_in_your_tea@sh.itjust.works · 2 pts · 1y (2 replies)

Right, but if you're transferring things that frequently, there are better solutions.

jollyrogue@lemmy.ml · 2 pts · 1y (1 reply)

Not necessarily. Rsync deltas are very efficient, and not everything supports deltas.

It may very well be the correct tool for the job.

Anyway, problem fit wasn’t part of the question.

sugar_in_your_tea@sh.itjust.works · 2 pts · 1y

Yeah, there are probably a few perfect fits for it. I don't rsync between machines very often, so the only use case I might have is backups, which is already well covered with a number of tools. Otherwise I just want to sync a few directories.

grumuk@lemmy.ml · -1 pts · 1y (1 reply)

I never even set up DNS for things that aren't public facing. I just keep /etc/hosts updated everywhere and ssh/scp/rsync things around using their non-fqdn hostnames.

sugar_in_your_tea@sh.itjust.works · 3 pts · 1y

You could also use mDNS to the same effect.

linuxguy@lemmy.gregw.us · 13 pts · 1y (5 replies)

scp

Sunbutt23@lemmy.world · 3 pts · 1y

Checks username… yeah that tracks

atzanteol@sh.itjust.works · 1 pts · 1y (3 replies)

scp is deprecated.

blackbrook@mander.xyz · 13 pts · 1y (2 replies)

SCP, the protocol, is deprecated. scp, the command, just uses the SFTP protocol these days. I find its syntax convenient.

atzanteol@sh.itjust.works · 4 pts · 1y (1 reply)

Oh does it? I didn't realize that. I've just switched over to rsync completely.

blackbrook@mander.xyz · 4 pts · 1y

Since OpenSSH version 9.0, so like mid '22. So as long as you're not running something more out of date than that.

harsh3466@lemmy.ml · 9 pts · 1y (1 reply)

Rsync and NFS for me.

IsoKiero@sopuli.xyz · 2 pts · 1y

And me.

lemming741@lemmy.world · 8 pts · 1y

sftp

All my machines have my keys, nothing to set up, nothing to tear down.

Mosfar@sh.itjust.works · 7 pts · 1y

Syncthing

boreengreen@lemm.ee · 4 pts · 1y (3 replies)

rsync is indeed fiddly. Consider SFTP in your GUI of choice. I mount the folder I need in my file browser and grab the files I need. No terminal needed and I can put the folders as favorites in the side bar.

Lv_InSaNe_vL@lemmy.world · 3 pts · 1y (2 replies)

If you want to use the terminal though, there is scp which is supported on both windows and Linux.

Its just scp [file to copy] [username]@[server IP]:[remote location]

CmdrShepard42@lemm.ee · 5 pts · 1y (1 reply)

That's essentially the same as rsync

sugar_in_your_tea@sh.itjust.works · 3 pts · 1y

Just slower if you already have some of the files there.

magic_smoke@lemmy.blahaj.zone · 4 pts · 1y
  • sftp for quick shit like config files off a random server because its easy and is on by default with sshd in most distros
  • rsync for big one-time moves
  • smb for client-facing network shares
  • NFS for SAN usage (mostly storage for virtual machines)
Xanza@lemm.ee · 4 pts · 1y

rclone. I have a few helper functions;

fn mount { rclone mount http: X: --network-mode }
fn kdrama {|x| rclone --multi-thread-streams=8 --checkers=2 --transfers=2 --ignore-existing --progress copy http:$x nas:Media/KDrama/$x --filter-from
~/.config/filter.txt }
fn tv {|x| rclone --multi-thread-streams=8 --checkers=2 --transfers=2 --ignore-existing --progress copy http:$x nas:Media/TV/$x --filter-from ~/.config/filter.txt }
fn downloads {|x| rclone --multi-thread-streams=8 --checkers=2 --transfers=2 --ignore-existing --progress copy http:$x nas:Media/Downloads/$x --filter-from ~/.config/filter.txt }

So I download something to my seedbox, then use rclone lsd http: to get the exact name of the folder/files, and run tv "filename" and it runs my function. Pulls all the files (based on filter.txt) using multiple threads to the correct folder on my NAS. Works great, and maxes out my connection.

motsu@lemmy.world · 4 pts · 1y (1 reply)

smb share if its desktop to desktop. If its from phone to PC, I throw it on nextcloud on the phone, then grab it from the web ui on pc.

Smb is the way to go if you have identity set up, since your PC auth will carry over for the connection to the smb share. Nextcloud will be less typing if not since you can just have persistent auth on the app / web.

BeardedGingerWonder@feddit.uk · 1 pts · 1y

Solid explorer on android is pretty useful too, it can access the SMB share. I use nextcloud for photo backup, but usually solid explorer for one off file transfers.

muhyb@programming.dev · 3 pts · 1y

As a lazy person, I just prefer sftp on thunar.

possiblylinux127@lemmy.zip · 3 pts · 1y

What do you mean by specifying IP address?

lemmylommy@lemmy.world · 3 pts · 1y

WinSCP for editing server config

Rsync for manual transfers over slow connections

ZFS send/receive for what it was meant for

Samba for everything else that involves mounting on clients or other servers.

MasterBlaster@lemmy.world · 3 pts · 1y

By "homelab", do you mean your local network? I tend to use shared folders, kdeconnect, or WebDAV.

I like WebDAV, which i can activate on Android with DavX5 and Material Files, and i use it for Joplin.

Nice thing about this setup is that i also have a certificate secured OpenVPN, so in a pinch i can access it all remotely when necessary by activating that vpn, then disconnecting.

jaek@lemmy.world · 3 pts · 1y (1 reply)

Magic wormhole is pretty dead simple https://magic-wormhole.readthedocs.io/en/latest/welcome.html#installation

I use this a lot at work for moving stuff between different test vms, as you don't need to check IPs/hostnames

boreengreen@lemm.ee · 1 pts · 1y

As I understand it, the establishing of the connection is reliant on a relay server. So this would not work on a local network without a relay server and would, by default, try to reach a server on the internet to make connections.

neidu3@sh.itjust.works · 2 pts · 1y

rsync if it's a from/to I don't need very often

More common transfer locations are done via NFS

melroy@kbin.melroy.org · 2 pts · 1y

yeah I also use SFTP using FileZilla. Or like everybody mentioned including yourself, rsync to sync files across computers. Or even scp.

lIlIllIlIIIllIlIlII@lemmy.zip · 2 pts · 1y

Syncthing and/or ftp.

hendrik@palaver.p3x.de · 2 pts · 1y

I'd say use something like zeroconf(?) for local computer names. Or give them names in either your dns forwarder (router), hosts file or ssh config. Along with shell autocompletion, that might do the job. I use scp, rsync and I have a NFS share on the NAS and some bookmarks in Gnome's file manager, so i just click on that or type in scp or rsync with the target computer's name.

PerogiBoi@lemmy.ca · 2 pts · 1y (1 reply)

Ye old samba share.

But I do like using Nextcloud. I use it for syncing my video projects so I can pick up where I left off on another computer.

jacksilver@lemmy.world · 2 pts · 1y

Samba Bamba!!

node815@lemmy.world · 1 pts · 1y

I work from home, however my two systems (home and work) are on the same LAN, they don't see each other for file sharing. I get paid via direct deposit like everyone else which means my pay stubs are all electronic. I print those out and then use WinSCP to copy those over to my desktop. No other files are ever sent.

At home, depending on the amount of files, I either use SFTP via Filezilla, or if the mood strikes me and for a single file, I will just use SCP if I'm already on the cli which is most of the time it seems anymore doing work on my personal servers. I've found that SFTP is faster at transferring than doing a copy/paste to the NFS share to the same drive.

ptz@dubvee.org · 1 pts · 1y

Depends on what I'm transferring and to/from where:

  • scp is my go-to since I'm a Linux household and have SSH keys setup and LDAP SSO as a fallback
  • sshfs if I'm too lazy to connect via SMB/NFS (or I don't feel like installing the tools for them) or I'm traversing a WAN
  • rsync for bulk transfer and backups
  • Snapdrop/Pairdrop for one-off file/text shares between devices with GUIs (mostly phone <--> PC)
  • SMB if I'm on a client PC and need to work with the files directly from the fileserver
  • NFS between servers
  • To get bulk data to my phone (e.g. updating my music library), I connect via USB in MTP mode and copy from the server via SMB or sshfs.
Turboblack@lemmy.world · 1 pts · 1y

you can use a regular ftp server with administrator and user rights, distribute rights to those who replenish, and those who just take - guests at home I transfer in this way from computer to computer without connecting them to a common network, what could be simpler? why invent some ways with keys or bash if there is a 40-year-old technology that just works great, and to open ftp it is enough to enter the IP address in the explorer

possiblylinux127@lemmy.zip · 1 pts · 1y

I like unison personally. It is a bit more of a hassle but it works pretty well.

iii@mander.xyz · 1 pts · 1y (2 replies)

I have a shared syncthing folder on all my devices

pirat@lemmy.world · 2 pts · 1y (1 reply)

A single folder synced between all of them, or a separate folder for each, syncing everything to a single device?

iii@mander.xyz · 1 pts · 1y

The first, single folder for all of them.

raldone01@lemmy.world · 1 pts · 1y

Ähm. So your not gonna like this but I just connect with vscode remote-ssh and drag'n drop em from the os file explorer into the vscode one.

So long story short scp I guess.

MangoPenguin@lemmy.blahaj.zone · 1 pts · 1y

Just regular old WinSCP, or XPipe for smaller stuff and editing config files.

I need a GUI, I'll use rsync to migrate a lot of data to a new server or something occasionally, but it's just a pain compared to a nice graphical file browser.

aeno@discuss.tchncs.de · 1 pts · 1y

SFTP! 😃

merthyr1831@lemmy.ml · 1 pts · 1y

rsync over an SMB share was pretty seamless.

tal@lemmy.today · 1 pts · 1y

I used rsync but specifying the IP address and the folders and everything is bit fiddly.

If this is Linux:

  • You can throw an entry in /etc/hosts to add a hostname mapping, even if you don't want to set up DNS:

      192.168.34.16       tea
    
  • It looks like bash has tab-completion for rsync paths set up by default these days on my Debian box. Will probably need to have passwordless or ssh-agent-based pub/privkey authentication set up in some fashion.

  • If you don't need rsync's functionality -- replicating attributes and reasonable resuming and doing partial copies -- I tend to use lftp, which supports sftp, an ssh-based file-transfer mechanism; one uses something like sftp://remotehost/remotepath. That's more-comfortable to browse, can bip around the local and remote filesystem with tab-completing cd, lcd, pwd, and lpwd and list remote files. The mirror and mirror -R command will move a file tree from or to the remote system.

  • If you do want rsync's "update changed files" functionality, and also want bidirectional synchronization in each direction instead of rsync's unidirectional functionality --- I do this to maintain a synchronized directory of videos that I want to be able to watch offline on various devices, among other things --- I use unison. You can set this up to auto-synchronize a set of directories. If you've got a preconfigured set of directories to synchronize set up, just type "unison", and bam.

  • If I'm doing a lot of browsing and work on the remote end, I use emacs's dired plus TRAMP as a rough approximation of a "two-pane file manager", more-properly called an orthodox file manager. emacs /ssh:remotehost:/remotepath localpath will fire it up. TRAMP is pretty clever, tries to do stuff on the remote machine as much as possible -- like, you can do stuff like version-control on remote directories and it'll use git on the remote machine, can grep through the remote file tree without transferring the files locally, etc. That being said, this is probably mostly-applicable to users who already heavily use emacs, but it's a handy technique if you have only a low-bandwidth connection and need to work on files remotely. I have emacs set up to default to using the "other pane" as a default target for things like copy and move commands using the following text in my init.el:

      ;; Try suggesting dired targets                                                                                                
      (setq dired-dwim-target t) 
    
  • If I'm wanting to play a video file remotely, I'll sometimes do a sshfs mount using FUSE:

      $ mkdir videos
      $ sshfs remotehost:/videos `pwd`/videos
      $ mpv videos/vid01.mkv
      $ umount videos
    
kokesh@lemmy.world · 1 pts · 1y

Resilion Sync

GreenKnight23@lemmy.world · 0 pts · 1y
[ removed ]
cass80@programming.dev · 0 pts · 1y (2 replies)

Snapdrop if they both have a gui/webbrowser. https://github.com/SnapDrop/snapdrop

Scp otherwise

theorangeninja@sopuli.xyz · 1 pts · 1y

I used that before but usually I prefer the fork of snapdrop, pairdrop or localsend. Might want to try magic wormhole one day.

westingham@sh.itjust.works · 1 pts · 1y

Or https://github.com/schlagmichdoch/PairDrop if you don't want to support a project bought out by LimeWire

UltraBlack@lemmy.world · -1 pts · 1y
rsync -are ssh from to@pc:/dir