Backups for Headless Linux

On my home network I have a headless Linux server that I use for a variety of tasks. One of the problems is that I can't use Deja Dup to backup the data on the server because it requires a GUI. I'm not sure why this is, but it is what it is. So I needed to find a way to backup the disks.

For some time I had used rsnapshot to take regular backups but the problem is it doesn't have a nice way to restore or find files other than searching through the folder tree, which is fine but sometimes annoying. Also there is a lot of manual config stuff I had to set to get it to behave the way I want because it doesn't use deduplicating backups but rather hard linking. The reason deduplicating backups are nice is they only perform incremental deltas and make efficient use of storage resources.

Fortunately I was able to find, maybe late to the game actually, two CLI tools that are pretty good for this purpose. The first is borgbackup. The other is restic.

Borgbackup

Borgbackup is a deduplicating backup tool ... interesting name though. Seems to be a pretty modern backup tool that is designed to be fast, secure, and efficient. Ideally it focuses on backing up and restoring your data. I had it setup, you can install from Ubuntu repos using sudo apt install borgbackup, but the downside is it doesn't really support cloud storage directly, although through rclone I think you can get it to work, on that note let me talk about rclone.

Rclone

One of the biggest gripes I have about Google is that they don't have a native Linux client/tool to support Google Drive. It is pretty ridiculous in my opinion, but for a long time the rclone project has been a great solution; it's generally cross-platform and supports a lot of cloud storage providers. It is a command line tool to sync files and directories to and from various cloud storage providers. To be honest I'm not too strong with its configuration and structure because on my personal laptop I use insync to sync my Google Drive; I paid $4.99 for a lifetime license and have been grandfathered into the newer versions which are subscription based.

The thing with rclone is if you just go through the generic setup you will have to use the rclone org's built-in Google API Client ID which is going to most likely be rate-limited and have other quotas. So the best thing to do is to create your own Google API Client ID and use that. You'll have to login to console.cloud.google.com and create a new project. Then you can go to APIs & Services > Credentials and create a new credentials for your internal use. Save the output from this its the Client ID and Client Secret.

Configuring Google API Client ID

Then when your setting up rclone and it asks you if you want to setup your own Google API Client ID then input the Client ID and Client Secret you just created. Then you can proceed to setup the rest of the configuration. This will allow you to experience a much more fully transport sync and backup experience. Essentially you have a way to sync your local filesystem to Google Drive and vice versa. Going to leave it there for now.

Restic

So with Borgbackup not being so friendly to cloud storage or rclone. I kept on my search and found restic, which supports rclone natively. The main command to leverage the restic repo (thats how they name things) is:

restic -r rclone:GoogleDrive:Backup init
restic -r rclone:GoogleDrive:Backup <commands>

This creates a backup repository using rclone on your configured Google Drive and it calls the folder Backup in your Google Drive. You can then use the following commands to backup your data:

restic -r rclone:GoogleDrive:Backup backup <path/want/to/backup>
restic -r rclone:GoogleDrive:Backup restore <snapshot_id> --target <path/where/to/restore>
restic -r rclone:GoogleDrive:Backup list snapshots

So the thing is, this is all manual and you'll most likely want to alias things so that it's more natural, because you could have several different restic repositories that you're backing your data onto. The other thing is you will want this to occur through a cron job or some other automated process that way you have regular backups and you don't have to remember to run the commands manually.

There is the downside that restic requires that you password encrypt your backups. So you can't view these backups directly, you have to leverage restic to view them and or mount them with say FUSE to view them locally.

Automating

To achieve regular backups without having to manually run the restic commands, the simplest solution is to just setup a cron job. Run something like:

crontab -e
# Run backup script daily at 2 AM
0 2 * * * /home/user/backup_script.sh

where the backup_script.sh is just a wrapper bash script for running restic and possibly any webhook notifications you want. You can add any logging stdout you want as well.

#!/bin/bash
export RESTIC_PASSWORD_FILE="/path/to/user/.restic_password"
REPO="rclone:GoogleDrive:Backup"
restic -r $REPO backup /location/to/backup
... other restic commands ...

Notice that you'll have to set the RESTIC_PASSWORD_FILE environment variable to the path to the file that contains your restic password for the repo you are backing up 😠.

Backup frequency and retention

The cron job just runs the backup_script.sh script at the specified time but it is not responsible for the backup frequency and retention policies. Although if you have daily backups that occur more than once you would want the cron job to run at that same frequency so that you keep the backups consistent.

So say you want to keep daily backups for 7 days, weekly for 4 weeks, and monthly for 12 months. You would want to run the following command after each backup:

restic -r rclone:GoogleDrive:Backup forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12

# Removes the actual stale backups
restic -r rclone:GoogleDrive:Backup prune

The prune command actually removes the data that's no longer needed, while forget just marks snapshots for deletion. The forget command is more like a "soft delete" in that it marks the snapshots for deletion but doesn't actually remove the data. The prune command is more like a "hard delete" in that it actually removes the data.

Performance Considerations

Because I backup directly to Google Drive, I've noticed it's very slow and time consuming. This probably means I need to tune the restic command to be more efficient like using parallel uploading or changing the chunking size.

Final Thoughts

I'm finding that rclone + restic is a pretty good combination for my needs. The deduplication will prevent a lot of Google Drive storage consumption. I don't really care about the data encryption (wish I could turn off) but it is what it is. I think if you're working on a headless server (e.g. self-managed compute cluster) and you want to backup data on your network, this is a pretty reasonable solution.

The one thing I can't speak to is how syncing behaves, where you're working on two systems and need files to be locked and synced properly as you work on those files. Like I mentioned on my personal laptop I use insync to sync my Google Drive and this works pretty well.

Reuse and Attribution

Limiting the Data Shuffle

I had to write a blog post about a simple and old but super powerful Linux utility. In computational materials science, or any computational science, remote computing resources are essential for carrying out high-performance computing. In many cases your calculations will generate a large number of files or files that are large in size. The issue always becomes where do you access the data, remotely or locally, and where should you perform the analysis. In general the best practice is to do as much as you can remotely to avoid the overhead of shuttling data back and forth. The issue is that some analysis or data wranggling is best done in an environment you control or have elevated privileges.

In the scenario where you can't VPN or mount a remote filesystem, it might seem that there isn't a path forward! But indeed there is, well at least for Linux³. The small but nice SSHFS Filesystem utility offers a seamless solution by mounting remote directories directly into your local filesystem, allowing secure, integrated access without constant file transfers¹. It is a piggy back off of SSH and FUSE for Linux, both these utilities are mainstays in Linux.

What is SSHFS?

SSHFS is a FUSE (Filesystem in Userspace)-based filesystem client [1] that uses SSH for securely mounting and interacting with remote directories on your local machine. Unlike traditional file transfer tools such as scp or sftp, SSHFS allows you to work with remote files as if they were stored locally, enabling easy access to simulation outputs on a remote server or datasets without manual transfers [2]. For me particularly this is very useful as I have been using TensorDock a lot recently for my ML/AI and computational materials science projects that I do on my personal time. The issue is -- more because I'm a little bit lazy -- that I don't want to setup the VM instance with all the software I use to do analysis, plotting, etc. I much rather just mount the remote directory directly into my local filesystem and access the files. So how does it work, it works pretty well so far.

Benefits of SSHFS

• Seamless Integration: Direct access to remote files within the local filesystem enables the use of local applications, such as data visualization or code editors.
• Security: SSHFS employs SSH encryption, ensuring secure data transfers; I think! I don't know how secure this really is.
• Efficiency: Reduces the need for manual file transfers, allowing a smoother research workflow.

Setting Up SSHFS

To begin, install SSHFS on your system:

# On Ubuntu/Debian systems
sudo apt install sshfs

Create a mount point and use the following syntax to mount:

mkdir ~/remote_folder_mounted_locally
sshfs username@remote-server:/remote/folder ~/remote_folder_mounted_locally

Thats pretty easy. You can look to see if its mounted using mount command and the best way to unmount is using fusermount -u.

Performance Optimization

To make SSHFS more efficient for my scientific workflows where there are a lot of files or very large files that are produced, I try to optimize the mount flags for speed and stability:

Caching Options

• cache=yes: Enables file and directory caching to improve access speeds.
• kernel_cache and auto_cache: Allow system-level caching and automatic cache updates.

Transfer Settings

• readahead=32768: Sets the read-ahead buffer size to 32KB, which is used by SSHFS to pre-fetch file data. This can improve throughput by reducing the number of read operations needed, especially beneficial when accessing large files or directories with many files.
• big_writes: Supports larger write sizes, enhancing transfer speeds.

Stability Enhancements

• reconnect: Automatically re-establishes the connection if interrupted.
• ServerAliveInterval=15: Keeps the connection alive with periodic pings.

An optimized mount command might look like this:

sshfs username@remote-server:/remote/path ~/remote-calc \
    -o cache=yes \
    -o kernel_cache \
    -o auto_cache \
    -o readahead=32768 \
    -o big_writes \
    -o reconnect \
    -o ServerAliveInterval=15

I haven't benchmarked how this different settings effect the performance of SSHFS for the type/size of files I produce, but seems to work pretty well. One thing the keep in mind is you will want to use ssh keys to make the connection more seamless (and secure). You can do this like:

ssh-keygen -t ed25519 -C "your_email@example.com"
ssh-copy-id username@remote-server

Limitations and Considerations

While SSHFS is powerful, it probably has limitations:

• File Size Constraints: SSHFS seems to be slower with files larger than 500MB, making it less optimal for very large datasets.
• Latency Sensitivity: The performance is highly dependent on network quality; high latency or low bandwidth can impact file operations⁴.
• Concurrent Access: File locking support is limited, so SSHFS may not handle concurrent access well, which could lead to data inconsistencies.

Performance Comparison

Based on my experience over the years this would be the table of comparisons for different transfer methods:

Transfer Method	Small Files (<10MB)	Large Files (>500MB)	Concurrent Access
SSHFS	Excellent	Fair	Limited
SCP	Good	Good	N/A
Tar over SSH	Good	Excellent	N/A

This table reflects SSHFS's performance strengths, especially with small files, while acknowledging its limitations in concurrent access and handling very large files. SCP and tar over SSH generally outperform SSHFS for single large files or batched file transfers due to reduced latency concerns.

Final Thoughts

For me SSHFS is becoming a valuable tool for my personal workflows providing local-like access to remote files and supporting efficient data analysis. While it has limitations with large files and concurrent editing, the integration it offers is just straighforward and simple.

Footnotes

---

1. SSHFS is particularly helpful when VPN access is unavailable or when working across multiple remote systems. ↩

2. Performance optimizations referenced are based on SSHFS version 3.7.0. ↩

3. There probably are variants of SSHFS or steps that let you work with Windows and MacOS. ↩

4. When I grab servers in Estonia I get disconnected or experience latency issues more regularly. ↩

References

1. FUSE: Filesystem in Userspace, Wikipedia.
2. SSHFS utility, GitHub Repo.

---

Reuse and Attribution