Introduction #
Rsync is a very useful and flexible tool to do incremental backups and I have been using it for years without any problems. Recently I discovered the power of Rsync and Cryptsetup with LUKS to do encrypted and incremental backups as well.
Warning: part of the content of this post is deprecated. See this newer post.
Partitioning scheme #
I have chosen a partitioning scheme that separates bigger and rarely used files (on HDDs) from smaller and commonly used files (on SSDs). For this reason I need two different hard disks for backup:
Disk name | Disk type | Size (order of magnitude) | Backup of mountpoint |
---|---|---|---|
backup root [1,2] | HDD | Gigabyte | / |
backup data [1,2] | HDD | Terabyte | /data |
Two different types of backups #
I also keep two different types of incremental backups. It’s not exactly the 3 2 1 rule but it’s fine for my use case.
Backup name | Online | Offsite | Automatic | Encrypted | Frequency | Reason |
---|---|---|---|---|---|---|
backup | x | x | daily | to recover files deleted by mistake | ||
backup enc | x | x | monthly (at best) | in case some disaster occurs |
Totals #
Disk name | Backup name | Device |
---|---|---|
backup root 1 | backup | /dev/sdx |
backup data 1 | backup | /dev/sdy |
backup root 2 | backup enc | /dev/sdw |
backup data 2 | backup enc | /dev/sdz |
Preliminary actions #
Notes #
-
A table of the notation used from this point on
Mountpoint Partition Notes /mnt/backup_root
/dev/sdx1
unencrypted root partition backup /mnt/backup_data
/dev/sdy1
unencrypted data partition backup /mnt/backup_root_enc
/dev/sdw1
encrypted root partition backup /mnt/backup_data_enc
/dev/sdz1
encrypted data partition backup -
Bash wildcards will be used in this section.
Steps #
-
Install Rsync, Cryptsetup, msmtp, S-nail, GNU Bash and GNU awk
-
Make the mountpoints and change the permissions. Make them accessible only to the
root
user and group, just to be on the safe side. The first thing to do is to unmount everything under/mnt
# umount -R /mnt # pushd /mnt # mkdir -p backup_{root,data}{_enc,} # chown 700 backup_{root,data}{_enc,} # popd
-
Partition, format and mount the unencrypted hard disks. You should use fstab instead of mounting the filesystems manually
# fdisk /dev/sd{x,y} # mkfs.ext4 /dev/sd{x,y}1 # mount /dev/sd{x,y}1 /mnt/backup_{root,data}
-
Shred, partition and format the encrypted backups as written in the Arch Wiki page. We will use some non-default options to make the encryption stronger
# shred /dev/sd{x,y} -n1 --random-source=/dev/urandom --verbose # fdisk /dev/sd{w,z} # cryptsetup -v --type luks2 --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-urandom --verify-passphrase luksFormat /dev/sd{w,z}1
-
We will now mount and create a filesystem on the encrypted partitions. You will be prompted for a password which will then be used every time you call cryptsetup
# cryptsetup open /dev/sd{w,z}1 {root,data}_enc # mkfs.ext4 /dev/mapper/{root,data}_enc # mount /dev/mapper/root_enc /mnt/backup_{root,data}_enc
-
Unmount
# umount /mnt/backup_{root,data}_enc # cryptsetup close {root,data}_enc
-
Backup the LUKS headers of the encrypted drives in case they need to be recovered in the future.
# pushd "${a_safe_place_for_backups}" # cryptsetup luksHeaderBackup /dev/sd{w,z}1 --header-backup-file backup_{root,data}_enc_luks_header.img # popd
The scripts #
Incremental backup script #
This script, called backup.sh
, was inspired by an
Arch Wiki
page dating back some years ago, so I have to use the same license. The
following code is licensed under the GNU Free Documentation License
1.3 or later.
#!/bin/bash
#
# backup.sh
#
# Copyright (C) 2019 Franco Masotti <franco.masotti@live.com>.
# Permission is granted to copy, distribute and/or modify this document
# under the terms of the GNU Free Documentation License, Version 1.3
# or any later version published by the Free Software Foundation;
# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
# A copy of the license is included in the section entitled "GNU
# Free Documentation License".
declare -A BACKUP_ALIAS
# Get correct path before loading the configuration file.
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
. ""${DIR}"/backup.conf"
SRC="${1}"
DST="${2}"
{ command -V rsync && command -V bash && command -V mail \
&& command -V printf && command -V sync; } \
|| exit 1
{ [ -z "${SRC}" ] || [ -z "${DST}" ]; } \
&& printf "%s\n" 'check inputs' 2>&1 && exit 1
[ ${UID} -ne 0 ] && printf "%s\n" 'user must be root' 2>&1 && exit 1
START=$(date +%s)
if [ "${SRC}" = '/*' ]; then
# Use the exclude option to avoid infinite loops due to symlinks and to
# avoid backing up the /data mountpoint itself.
rsync --archive --acls --xattrs --hard-links --delete /* "${DST}" --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/data}
elif [ "${SRC}" = '/data/*' ]; then
rsync --archive --acls --xattrs --hard-links --delete /data/* "${DST}"
else
rsync --archive --acls --xattrs --hard-links --delete "${SRC}" "${DST}"
fi
ret_rsync=${?}
# Flush data.
sync
FINISH=$(date +%s)
if [ ${ret_rsync} -eq 0 ]; then
if [ "${LOG_TO_FILE}" = 'true' ]; then
printf "%s\n" "[$(date '+%Y-%m-%d, %T') OK backup_ext_disks] "${DST}" ; \
$((${FINISH}-${START}))s" >> "${LOG_FILE}"
fi
if [ "${LOG_TO_EMAIL}" = 'true' ]; then
printf "%s\n" "Backup on "${BACKUP_ALIAS["${DST}"]}" took $((${FINISH}-${START}))s" \
| mail -r "${EMAIL_SENDER}" -s 'Backup complete' "${EMAIL_RECIPIENT}"
fi
else
if [ "${LOG_TO_FILE}" = 'true' ]; then
printf "%s\n" "[$(date '+%Y-%m-%d, %T') FAILED backup_ext_disks \
ret_rsync=${ret_rsync}] "${DST}"" \
>> /var/log/rsync.log
fi
if [ "${LOG_TO_EMAIL}" = 'true' ]; then
printf "%s\n" "Backup error on "${BACKUP_ALIAS["${DST}"]}". Error code ${ret_rsync}" \
| mail -r "${EMAIL_SENDER}" -s 'Backup error' "${EMAIL_RECIPIENT}"
fi
fi
Configuration file #
#
# backup.conf
#
# Copyright (C) 2019 Franco Masotti <franco.masotti@live.com>.
# Permission is granted to copy, distribute and/or modify this document
# under the terms of the GNU Free Documentation License, Version 1.3
# or any later version published by the Free Software Foundation;
# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
# A copy of the license is included in the section entitled "GNU
# Free Documentation License".
# Friendlier names used in logs and emails.
BACKUP_ALIAS['/mnt/backup_root']='backup-root'
BACKUP_ALIAS['/mnt/backup_root_enc']='backup-root-encrypted'
BACKUP_ALIAS['/mnt/backup_data']='backup-data'
BACKUP_ALIAS['/mnt/backup_data_enc']='backup-data-encrypted'
LOG_TO_EMAIL='true'
EMAIL_SENDER='Computer <your.email@address.com>'
# You can use aliases if your mailing system is set up correctly.
EMAIL_RECIPIENT='all'
LOG_TO_FILE='true'
LOG_FILE='/var/log/rsync.log'
Encrypted backup script #
To do our encrypted backups we will use LUKS and the previous script as a base. The idea is quite simple:
- mount the encrypted partition.
- do the backup using the previous script with the appropriate parameters.
- unmount the partition.
This new script is called backup_enc.sh
and was inspired by
this Arch Wiki
page.
#!/bin/bash
#
# backup_enc.sh
#
# Copyright (C) 2019 Franco Masotti <franco.masotti@live.com>.
# Permission is granted to copy, distribute and/or modify this document
# under the terms of the GNU Free Documentation License, Version 1.3
# or any later version published by the Free Software Foundation;
# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
# A copy of the license is included in the section entitled "GNU
# Free Documentation License".
#
# This backup is intended to be run manually.
#
declare -A BACKUP_ALIAS_ROOT
declare -A BACKUP_ALIAS_DATA
# Get correct path before loading the configuration file.
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
. ""${DIR}"/backup_enc.conf"
WHAT="${1}"
{ command -V cryptsetup && command -V bash && command -V printf \
&& command -V mount && command -V umount && command -V awk; } \
|| exit 1
[ ${UID} -ne 0 ] && printf "%s\n" 'user must be root' 2>&1 && exit 1
if [ "${WHAT}" = 'root' ]; then
SRC="${BACKUP_ALIAS_ROOT['SRC']}"
DST="${BACKUP_ALIAS_ROOT['DST']}"
UUID="${BACKUP_ALIAS_ROOT['UUID']}"
NAME="${BACKUP_ALIAS_ROOT['DEVICE_MAPPER_NAME']}"
elif [ "${WHAT}" = 'data' ]; then
SRC="${BACKUP_ALIAS_DATA['SRC']}"
DST="${BACKUP_ALIAS_DATA['DST']}"
UUID="${BACKUP_ALIAS_DATA['UUID']}"
NAME="${BACKUP_ALIAS_DATA['DEVICE_MAPPER_NAME']}"
else
printf "%s\n" 'error: configuration not set' 2>&1 && exit 1
fi
# Find partition via UUID.
dev="/dev/"$(lsblk -x name -i -o name,uuid | grep "${UUID}" | awk '{print $1}')""
[ -z "${dev}" ] && exit 1
cryptsetup open "${dev}" "${NAME}" && mount /dev/mapper/"${NAME}" "${DST}" \
&& ""${DIR}"/backup.sh" "${SRC}" "${DST}"
umount "${DST}" && cryptsetup close "${NAME}"
Configuration file #
You must put the correct UUIDs of the partitions in the configuration file. To get these run:
$ lsblk -o name,uuid
and copy the appropriate ones.
#
# backup_enc.conf
#
# Copyright (C) 2019 Franco Masotti <franco.masotti@live.com>.
# Permission is granted to copy, distribute and/or modify this document
# under the terms of the GNU Free Documentation License, Version 1.3
# or any later version published by the Free Software Foundation;
# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
# A copy of the license is included in the section entitled "GNU
# Free Documentation License".
BACKUP_ALIAS_ROOT['SRC']='/*'
BACKUP_ALIAS_ROOT['DST']='/mnt/backup_root_enc'
BACKUP_ALIAS_ROOT['UUID']='<put the UUID here>'
BACKUP_ALIAS_ROOT['DEVICE_MAPPER_NAME']='root_enc'
BACKUP_ALIAS_DATA['SRC']='/data/*'
BACKUP_ALIAS_DATA['DST']='/mnt/backup_data_enc'
BACKUP_ALIAS_DATA['UUID']='<put the UUID here>'
BACKUP_ALIAS_DATA['DEVICE_MAPPER_NAME']='data_enc'
First backups #
Once you have everything in place, including the configuration files,
you may start backups manually along with Rsync’s --dry-run
option in the
script.
# ./backup.sh '/*' /mnt/backup_root
# ./backup.sh '/data/*' /mnt/backup_data
If everything works as expected, remove that option.
You can use Cron for the automatic backups. In my case I use Cronie. Just remember to call the backup scripts from their directory otherwise the configuration files won’t be sourced.
~
Till next time!