Syncing files from Android to GNU/Linux

First written on 2018-08-04

Last updated on 2022-10-08

Overview #

Backups of mobile phone content have been a problem I wanted to solve for years but unfortunately the tools to do a clean job have always been inadeguate. Also, I certainly didn’t want to rely on third party services such as Dropbox, Google, etc…

The first solution was to use bluetooth with an APP I can’t even remember the name of. The problem was its slow speed and non-secureness.

So I started using micro SD cards and manual copying… with the result of losing several times all media on the card because its filesystem got corrupted.

After searching intensively I finally found a reliable and automatic way for mobile phone backups. There is this neat APP called Primitive FTPD that includes an SFTP server. The advantages of SFTP instead of using other systems are:

simpleness
encryption and authentication by default
no need to open yet another port
possibility to use a variety of copying tools such as Rsync.

Steps #

Set a fixed IP address for the phone.
Download the APP from Fdroid.
Open the APP and choose a username and password in the settings menu.
Start the server and tap yes when asked for key generation.
Install SSHFS and Rsync on the target computer.
Adapt the following script I have written for the purpose:

#!/usr/bin/env bash

# The full path of the configuration file
CONFIGURATION="${1}"

{ command -V rsync && command -V bash && command -V mail \
    && command -V printf && command -V sync; command -V fusermount3; } \
    || exit 1
{ [ ! -f "${CONFIGURATION}" ]; } \
    && printf "%s\n" 'check configuration file' 2>&1 && exit 1

. "${CONFIGURATION}"

# See https://gist.github.com/mfellner/5743990
RSYNC_MAIN_OPTS='-rv --size-only --no-group --no-owner --no-perms --no-times --whole-file --inplace'

START=$(date +%s)

# Mount only if not already mounted.
exists_mount="$(mount | grep ""${USERNAME}"@"${ADDRESS}":")"
if [ -z "${exists_mount}" ]; then
        echo "${PASSWORD}" | sshfs -o password_stdin -p "${PORT}" "${USERNAME}"@"${ADDRESS}": "${MOUNT_DIR}"
fi

# Might not be mounted if device not in range or if its server is offline.
exists_mount="$(mount | grep ""${USERNAME}"@"${ADDRESS}":")"
if [ -z "${exists_mount}" ]; then
        echo ""${ALIAS}" not connected"
        exit 1
fi

rsync ${RSYNC_MAIN_OPTS} "${MOUNT_DIR}"/* "${DST_DIR}"
ret_rsync=${?}
sync
fusermount3 -u "${MOUNT_DIR}"

FINISH=$(date +%s)
if [ ${ret_rsync} -eq 0 ]; then
    if [ "${LOG_TO_FILE}" = 'true' ]; then
        printf "%s\n" "[$(date '+%Y-%m-%d, %T') OK backup_mobile_phone] "${ALIAS}" ; \
$((${FINISH}-${START}))s" >> "${LOG_FILE}"
    fi
    if [ "${LOG_TO_EMAIL}" = 'true' ]; then
        printf "%s\n" "Backup on "${ALIAS}" took $((${FINISH}-${START}))s" \
            | mail -r "${EMAIL_SENDER}" -s 'Backup complete' "${EMAIL_RECIPIENT}"
    fi
else
    if [ "${LOG_TO_FILE}" = 'true' ]; then
        printf "%s\n" "[$(date '+%Y-%m-%d, %T') FAILED backup_mobile_phone \
ret_rsync=${ret_rsync}] "${DST}"" \
        >> /var/log/rsync.log
    fi
    if [ "${LOG_TO_EMAIL}" = 'true' ]; then
        printf "%s\n" "Backup error on "${ALIAS}". Error code ${ret_rsync}" \
            | mail -r "${EMAIL_SENDER}" -s 'Backup error' "${EMAIL_RECIPIENT}"
    fi
fi

Edit the following configuration file:

USERNAME='username0'
PASSWORD='password0'
ADDRESS='1.2.3.4'
PORT='65432'
MOUNT_DIR='/home/user/mountpoints/user0'
DST_DIR='/home/user/backup/user0'

# Friendlier name.
ALIAS='user 0'

LOG_TO_EMAIL='true'
EMAIL_SENDER='Computer <your.email@address.com>'
# You can use aliases if your mailing system is set up correctly.
EMAIL_RECIPIENT='all'

LOG_TO_FILE='true'
LOG_FILE='/var/log/rsync.log'

Finally run it. If it works you can add it to crontab for automatic backups. To run the script you simply need to pass the configuration file as argument:

$ ${PATH_TO_SCRIPT} ${PATH_TO_CONFIGURATION_FILE}

Problems and explanation #

(Rsync over SFTP) or (Rsync over directory over SSHFS) #

Since Primitive FTPD is an SFTP server, we can’t use rsync directly on SFTP because this is not supported by rsync (see: $ man 1 rsync). For this reason we must first mount the directory using SSHFS and then rsyncit.

(Public key) or (password authentication) #

I was not able to use public key authentication because of Pftpd reported some obscure error with it. Initially I wasn’t able to find Pftpd’s log file in the phone’s flash memory but I noticed it after several sync operations. Going through the file didn’t clear the situation at all. Even the project issues didn’t solve the problem. So, the only remaining solution was to use password authentication. Since all the process needs to be automatic, the password needs to be stored plaintext and passed through the password_stdin argument. This is of course not the best setup.

Storage type #

Depending on the Android version and device, the type of storage selected from Pftpd might not work and behave differently than expected, so it is kind of a trial and error operation. In my case the option Plain old filesystem worked best, although the location that will be synced will usually be /storage/emulated/0 and cannot be changed to use, for example, the external SD card.

Alternatives #

There aren’t many free software alternatives that work like this for Android. What I found either didn’t involve SSH or lack some functionalities. Syncopoli, for example, was another possible option. The readme file states that it is an Rsync-based application optionally working via SSH and public key authentication, however it is a client, not a server.

Using a client avoids the computer polling for the connection, so it’s a plus. The only problem is that I need to do extra operations like maling and writing to log files after the syncronization finishes. This is not possible with Syncopoli.

Have fun!

Tags