- tools
- data
Backing up my Synology with rsync
Some years ago I was gifted a used Synology DS713+ NAS (Network Attached Storage) device with dual 3TB Western Digital Red WD30EFRX NAS drives. The friend I got it from had upgraded because this setup was "too slow" but its worked well for me for many years. Mostly I use it as just a backup server. It is a central repository of all the the family pictures and videos from everyone's cameras, digital creations of various varieties, including recorded music and video, documents and files synced from various family machines, website collatoral and backups from 30 years of website design, schoolwork, code, ROMs for a zillion games, etc.
Generally these days the NAS is just off until someone needs to grab an archive or run a backup (this is personal stuff so its not a daily thing). The personal machines which run linux are backed up with just an rsync script set to synchronize only the important stuff. One example use is that we all add to a central store of ripped music, and can grab new stuff from it to our local machines if we want, but we all have whatever we want on our personal machines to play, we don't need to back that up again and again. Same with movies - grab one you want to play from the repo but only upload a new one you want to archive.
Having a central repository for all this stuff is nice, but from my description of how we use it you can already see that the NAS may have the only copy of certain files. Its critical that we back up the NAS too, after all its obviously getting pretty old at this point. I like to copy the entire contents of the NAS off to USB drives that I can store in a fire safe or a go bag.
Synology's "Hyper Backup" app which runs on the NAS provides a nice GUI (Graphical User Interface) and is fairly easy enough to use since it steps you through setting up a new backup but the default settings lock the resultant backed up files in an .hbk file which you need special software to access. It does this because its actually compressing your files so you can fit more on the USB than a straight one to one copy would be able to do, and also provides a way to have incremental backups over time so you can go back to a point in time before you deleted that important file and so that subsequent backups will take less time.
I've used Hyper Backup for years but never really had to access the files from the backup. When a family member moved they wanted a copy of their files on the repo and so created a backup with this method to USB. Accessing the files required using a Synology application on their laptop and it turned out to be painfully slow and nearly unusable. I don't have anything highly sensitive to protect so I don't bother with whole disk encryption and I don't want the files locked up in a proprietary backup format. I just want to be able to plug the drive into a PC and read the files.
Synology also has a "USB Copy" app but it only does on folder at a time and you can't pick and choose subfolders to include. I do not have a single shared folder on the NAS under which all files are kept so I think I'd have to run this many times to copy each one.
I wanted to be able to just rsync the desired files over to the USB, and indeed there is an option for rsync in Hyper Backup and I thought I would be able to use this to create a mirrored copy of the files on an external drive. I couldn't figure out the magic spell to make it actually work to simply copy the files over. I chose many variations of the Rsync and Rsync local options with "export to a local shared folder" both with and without the option for compression, but always ended up with an .hbk file on the external drive. Perhaps I'm doing something wrong but that isn't what I expected with an "rsync" option, and in any case I wanted more control that the GUI provides.
I wanted to run rsync to the USB from the command line, but the only way to do that is over ssh from another machine and with Terrabytes of data that means dedicating a machine to the backup task for quite some time to keep the session running. If the machine went to sleep or you accidentally shut the lid (if you were running it from a laptop) the connection would end and the rsync would stop. The simplest solution for me was a raspberry pi that I use for lots of odd jobs which has the application "screen" installed. The pi connects to my network over wifi so I can just ssh into it, then use connect to the Synology with screen -S <some_text> ssh my_user@my_nas_ip and start my rsync script. Then I can just disconnect with Ctrl-A, Ctrl-D and re-connect to it later. When I want to check on the backup process I ssh back into the pi and reconnect to the running screen ssh session with screen -r <some_text>. Using this method I can write out (and save) an rsync script with lots of options to pass all sorts of excludes, choose to keep or delete files that may have been moved around since last backup, and its all just a text file I can edit. Some example options include --progress so when I ssh back in and reattach to the session I can see how far along it is, --delete to keep the source and destination mirrored exactly (so that if I move a file I don't want the backup to have two of that file one in the old place and one in the new), --exclude='@*' because Synology has a lot of special folders that begin with the @ sign that I definitely do not need to retain, and --exclude='#recycle' so I'm not backing up stuff I've deleted on the Synology (usually huge files like whole machine backups I no longer want).
I have generally already excluded a bunch of stuff I would otherwise have to backup at this point during the earlier per-machine backups like Mozilla cache folders, and depending on the machine, .config or .local files. For the number of times I've actually needed a .config file its not worth it (I think emacs might be the only one I keep).
Understanding the rsync exclude command finally gelled for me with this statement from stackoverflow that "In rsync exclude path is relative to the source path."
My final script excludes a bunch of Synology specific folders I don't need in my backup
!/bin/bash
rsync -av --progress --delete-excluded --exclude='@*' --exclude='#recycle' --exclude='lost+found' --exclude='/web/' --exclude='/web_packages/' --e
xclude='/photo/' --exclude='/video/' --exclude='aquota.group' --exclude='aquota.user' --exclude='synoquota.db' --exclude='/surveillance/' \
/volume1/ /volumeUSB1/usbshare/
~