linux

Cloud backup

Introduction

In the past I have used Ubuntu One until it went out of business, then ownCloud, until it went out of business. I have been looking for a method to perform bidirectional backup to a directory on my Hostgator shared server. Osync initially looked promising, but SSHs into the server too many times, exceeding the 12 allowable attempts per IP per 90 seconds imposed by Hostgator. I fell back to rsync, and had to hack the “bidirectional” feature in an unsatisfactory way.

Rsync

See here for setting up SSH keys. The command to upload from a local directory ~/syncd/ to Hostgator, preserving directory structure:

1
2
3
4
5
cd ~/syncd/
rsync -avzP --delete --rsh='ssh -p2222' /home/mbc/syncd mbcladwell@123.123.123.123:/home2/mbcladwell/public_html/

;;for test
rsync -avzP --delete --rsh='ssh -p2222' mbcladwell@123.123.123.123:/home2/mbcladwell/public_html/lnsDFoKytr /home/mbc/

for cron edit the file using the command crontab:

1
2
3
#crontab -e

0,15,30,45 * * * * rsync -azp --delete --rsh='ssh -p2222' /home/mbc/syncd mbcladwell@123.123.123.123:/home2/mbcladwell/public_html/

\$ps aux to list PIDs
\$kill

crontab -l to see crons

flags

-q quiet - use with cron
-v verbose
-r recursive
-R relative
-t times must be used to transfer only modified files in future backups
-a equivalent to -rlptgoD
-o owner
-g group
-p permissions
-D devices - transfer char and block device info
-z compress
-P same as -partial -progress
-partial retain partially transferred files
-progress show progress (should also use -v )

Create 2 batch files, one for upload and one for download.
I will use the upload syntax to register a cron job. The job will run every 15 minutes and back up to the cloud.

rsync-upload.sh
1
2
3

#!/bin/bash
rsync -avzP --delete --rsh='ssh -p2222' /home/mbc/syncd mbcladwell@123.123.123.123:/home2/mbcladwell/public_html/
rsync-download.sh
1
2
3
#!/bin/bash
rsync -avzP --delete --rsh='ssh -p2222' mbcladwell@123.123.123.123:/home2/mbcladwell/public_html/syncd /home/mbc/

If I use a second computer, manually run rsync-download.sh upon login. When returning to the primary computer, again manually run rsync-download.sh to capture any changes made on the auxilliary machine. As a reminder, I created a batch file that will prompt me on login. Add a new autostart with the Applications / Settings / Settings manager / Session and startup / Application autostart dialog with the command: xfce4-terminal -e “bash /home/pl/check-for-download.sh” –hold

check-for-download.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#!/bin/bash

##  xfce4-terminal -e "bash /home/pl/check-for-download.sh" --hold
## by default input assigned to the variable REPLY
echo "Run download script? [Y|n]:"
read 

if [[ ("$REPLY" == "Y") || ( "$REPLY" == "" ) ]];
then
    echo "starting download!"
   ./rsync-download.sh
else
    echo "Download canceled."
fi

Backup to USB

I don’t want private keys in the cloud so I backup personal directories and files to a local USB.

keys-backup.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

#!/bin/bash
rm -r /media/mbc/vol1/bin
rm -r /media/mbc/vol1/.electrum
rm -r /media/mbc/vol1/.electrum-ltc
rm -r /media/mbc/vol1/.gnupg
rm -r /media/mbc/vol1/.keystorefiles
rm -r /media/mbc/vol1/.password-store
rm -r /media/mbc/vol1/.ssh

cp -r /home/mbc/bin /media/mbc/vol1
cp -r /home/mbc/.electrum /media/mbc/vol1
cp -r /home/mbc/.electrum-ltc /media/mbc/vol1
cp -r /home/mbc/.gnupg /media/mbc/vol1
cp -r /home/mbc/.keystorefiles /media/mbc/vol1
cp -r /home/mbc/.password-store /media/mbc/vol1
cp -r /home/mbc/.ssh /media/mbc/vol1

lsyncd

http://blog.acipo.com/using-rsync-to-transfer-files-to-your-hostgator-account/

username: syncd@granitemtn.net
password: EW27&v&cd)z+

We can now restart the lsyncd service by issuing the following command:
sudo service lsyncd restart

sudo su
ssh remote_mirror_ip_address
ls /remotesync

http://www.cerebralmastication.com/2011/04/fast-two-way-sync-in-ubuntu/

Osync

https://github.com/deajan/osync

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ git clone https://github.com/deajan/osync
$ cd osync
# sh install.sh
[root@xps osync]# ./install.sh
Created directory [/etc/osync].
Copied osync.sh to [/usr/local/bin].
Copied osync-batch.sh to [/usr/local/bin].
Copied ssh_filter.sh to [/usr/local/bin].
Created osync-srv service in [/lib/systemd/system] and [/etc/systemd/user].
Can be activated with [systemctl start osync-srv@instance.conf] where instance.conf is the name of the config file in /etc/osync.
Can be enabled on boot with [systemctl enable osync-srv@instance.conf].
In userland, active with [systemctl --user start osync-srv@instance.conf].
osync installed. Use with /usr/local/bin/osync
In order to make install statistics, the script would like to connect to http://instcount.netpower.fr?program=osync&version=1.2-beta3&os=Linux%20unknown%20unknown%20GNU%2FLinux
No data except those in the url will be send. Allow [Y/n]

Test that I can SSH into my server:

1
mbc@dc7700s:~$ ssh -p 2222 mbcladwell@123.123.123.123

Now perform a test transfer before using the configuration file:

1
./osync/osync.sh --initiator="/home/mbc/syncd/" --target="ssh://123.123.123.123:2222//public_html" --rsakey=/home/mbc/.ssh/id_rsa

Set up the configuration file sync.conf with the following variables:

INITIATOR_SYNC_DIR=”/home/mbc/lnsDFoKytr”
TARGET_SYNC_DIR=”ssh://mbcladwell@123.123.123.123:2222//home2/mbcladwell/public_html”
SSH_RSA_PRIVATE_KEY=”/home/mbc/.ssh/id_rsa”

As mentioned above, osync repeatedly confirms SSH accessibility, exceeding the login limits allowed by my provider. Insert a sleep statement in the
CheckConnectivityRemoteHost function to slow the script down. Hostgator allows 12 SSH logins per 90 seconds, with a fresh allocation of logins every 90 seconds.

osync.sh
1
2
3
4
5
6
7

function CheckConnectivityRemoteHost {
        sleep 10
    local retval

.....etc.

Share