How to backup from 1 Linux to another Linux with rsync

Posted by paul on 2014.04.29

How to backup from 1 Linux to another Linux with rsync

2014.04.29 Tue

I wanted an automated backup solution to backup data from multiple Linux computer to one central Linux server. Obviously there are many ways to do it but I came up with a rsync based solution that I'm happy with. Please be careful when following the direction if you are not very familiar with linux commands. With one missing / or a minor typo, you can easily destroy a Linux computer.

Assumptions

  1. server01.home.loc - Main file server running Linux.
  2. web01.home.loc - A Linux computer which is a backup client

Prep server01

  1. On server01, create both acct and group for backup
    2. [[email protected] ~]# useradd backupuser
[[email protected] ~]# passwd backupuser
...
[[email protected] ~]# groupadd backuplocalteam
[[email protected] ~]# usermod -a -G backuplocalteam backupuser
  2. On server01, reate acct and group for backup on
    3. [[email protected] ~]# mkdir /backupinbox /backup
[[email protected] ~]# chown -R backupuser:backuplocalteam /backupinbox
[[email protected] ~]# chmod -R 750 /backupinbox

web01: create ssh keys

Create the a pair of SSH keys which will be used only for rsyncing files from web01 to server01:/backupinbox/ for backup purpose.

  1. On server01, create both acct and group for backup
  2. Change [email protected] to match your environment. I recommend using root.bak...
  3. Also you must specify the name of the private key file by using -f /root/.ssh/id_rsa_bak. Default file name is usually id_rsa but for this ssh being created for dedicated backup, the file name is id_rsa_bak.
    4. [[email protected] ~]# ssh-keygen -t rsa -C "[email protected]"\
-f /root/.ssh/id_rsa_bak
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa_bak.
Your public key has been saved in /root/.ssh/id_rsa_bak.pub.
The key fingerprint is:
23:be:a7:b4:dc:a4:be:98:32:fe:09:9c:19:94:de:39 [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|   .             |
|  o              |
| o . .           |
|  o E . S        |
| . + o . .       |
|  =   o .        |
|  o. * *.        |
| ..+=.O+.        |
+-----------------+
[[email protected] ~]#

Setup ssh key

  1. On web01, copy id_rsa_bak.pub to server01.
  2. I recommend adding $HOSTNAME at end of the destination file, especially when you may be setting up multiple Linux computers to send backup data to server01.
    3. [[email protected] ~]# scp ~/.ssh/id_rsa_bak.pub [email protected]:~/id_rsa_bak.pub.$HOSTNAME
[email protected]'s password:
id_rsa_bak.pub                               100%  401     0.4KB/s   00:00
  3. On server01 as backupuser, run following commands to allow passwordless login from web01. You need to execute line 1, 2, & 5 only once when setting up the first backup client. In line 4, it's very important you use >>, NOT >.
    4. [[email protected] ~]$ mkdir .ssh
[[email protected] ~]$ chmod 755 .ssh/
[[email protected] ~]$ cd
[[email protected] ~]$ cat id_rsa_bak.pub.web01 >> ~/.ssh/authorized_keys
[[email protected] ~]$ chmod 600 ~/.ssh/authorized_keys
  4. At this point, you can login from [email protected] into [email protected] using the ssh key.
    5. [[email protected] ~]# ssh -i ~/.ssh/id_rsa_bak [email protected]
Last login: Thu Apr 17 13:00:45 2014 from 192.168.11.9
#################
Welcome to server01, backup storage server.
#################
[[email protected] ~]$
  5. Log out from server01.

Lock down backupuser on server01

  1. On server01 as backupuser create /home/backupuser/validate-rsync with following content:
    2. #!/bin/sh
case "$SSH_ORIGINAL_COMMAND" in
*\&*)
echo "Rejected"
;;
*\(*)
echo "Rejected"
;;
*\{*)
echo "Rejected"
;;
*\;*)
echo "Rejected"
;;
*\<*)
echo "Rejected"
;;
*\`*)
echo "Rejected"
;;
*\|*)
echo "Rejected"
;;
rsync\ --server*)
$SSH_ORIGINAL_COMMAND
;;
*)
echo "Rejected"
;;
esac
  2. On server01 as backupuser, edit /home/backupuser/.ssh/authorized_keys. Specifically you want to add this: command="/home/backupuser/validate-rsync" to the beginning of the line that ends with [email protected]
  3. Before authorized_keys is edited **all the text is 1 line.
    4. ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6ctzGjCJKxw9NYfDQHI2...NhVWRhCDSd5p0DmIljRgwiN2d+BulMAnjSu3JV1N+29QQ== [email protected]
  4. After authorized_keys is edited **all the text is 1 line.
    5. command="/home/backupuser/validate-rsync" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6ctzGjCJKxw9NYfDQHI2...NhVWRhCDSd5p0DmIljRgwiN2d+BulMAnjSu3JV1N+29QQ== [email protected]
  5. On server01, make /home/backupuser/validate-rsync executable.
    6. [[email protected] ~]$ ls -la validate-rsync
-rw-rw-r-- 1 backupuser backupuser 318 Apr 28 10:47 validate-rsync
[[email protected] ~]$ chmod 700 validate-rsync
[[email protected] ~]$ ls -la validate-rsync
-rwx------ 1 backupuser backupuser 318 Apr 28 10:47 validate-rsync
[[email protected] ~]$

backup-rsync script on web01

  1. Create following script on web01, /root/bin/backup-rsync.sh. Following script allows you to keep different sets of backup. At this point, it can only backup folders in root level ( aka / ). I will update it eventually to allow backing up subfolders.
    2. #!/bin/bash

###
# variables
###
NOW=$(date +"%Y-%m-%d_%H")
NOWSTAT=$NOW
LOGIN=backupuser

###
# variables that  M U S T  be VERIFIED
###
DESTHOST=server01
DESTDIR="backupinbox"
SRC[1]="etc"
SRC[2]="data"


###
# script
###
#create time stamped dir and copy it to the backup server, server01
/bin/mkdir -p /tmp/$HOSTNAME/$NOWSTAT
/bin/chmod -R 770 /tmp/$HOSTNAME/$NOWSTAT
/usr/bin/rsync -ave "ssh -i /root/.ssh/id_rsa_bak" /tmp/$HOSTNAME/$NOWSTAT [email protected]$DESTHOST:/$DESTDIR/$HOSTNAME

#rsync over files
for i in "${SRC[@]}"
do
    [ "$(ls -A /$i)" ] && /bin/tar -czf /tmp/$i--$NOWSTAT.tar.gz /$i 2>/dev/null;\
    /usr/bin/rsync -ave "ssh -i /root/.ssh/id_rsa_bak" --delete /tmp/$i--$NOWSTAT.tar.gz [email protected]$DESTHOST:/$DESTDIR/$HOSTNAME/$NOWSTAT/ 2>/dev/null;\
    rm /tmp/$i--$NOWSTAT.tar.gz 2>/dev/null
done

#rsync files that do not need datestamp
#/usr/bin/rsync -ave "ssh -i /root/.ssh/id_rsa_bak" --delete /data/www/yumrepo [email protected]$DESTHOST:/$DESTDIR/$HOSTNAME/

#clean up
rm -rf /tmp/$HOSTNAME
exit 0
  2. line 6-7: picks current yyyy-mm-dd_hr. Doing this make sure you are dealing with only 1 date-time-stamp within the duration the script is running, no matter now long it runs. Also, limiting the time stamp to the level of hour helps control the amount of data you would have to copy over to server01 if you end up running backups multiples times in a day.
  3. lines 15 -16: list of folders on web01 you want to backup. Note it doesn't have the leading '/'. Here, I'm backing up /etc/ & /data/ from web01.
  4. lines 22-24: creates folder named with data-time-stamp on web01 and copies it to server01:/backupinbox/web01/_data-time-stamp_/.
  5. lines 27 - 32: rsyncs files from web01 --> server01:/backupinbox/web01/_data-time-stamp_/
  6. line 36: This is commented out for now but if you are backing up particularly big data folder (such as /data/www/yumrepo/ on Yumrepo server with 30GB of data in it), perhaps you don't want to backup the entire folder every day. And you don't care about recovering files from older backup set. In that case, you can just backup using line 36. The backup data goes into server01:/backupinbox/web01/.
  7. On web01, make backup-rsync.sh executable and execute it.
    8. [[email protected] ~]# chmod 750 /root/bin/backup-rsync.sh
[[email protected] ~]# /root/bin/backup-rsync.sh
  8. On server01, you should see the backup data being copied over.
    9. [[email protected] ~]# ls /backupinbox/web01/ | sort
2014-04-14_12
  9. On web01, edit /etc/crontab to set backup-rsync.sh to run daily. Add following at end of /etc/crontab.
    10. 55 3 * * * root /root/bin/backup-rsync.sh

Purge old backups

In order to prevent backups from filling up the HD on server01, you should purge old backups regularly by running a script via crontab.

  1. On server01, create script /root/bin/backup-purge-old-backup.sh.
    2. #!/bin/bash
#This script purges old backup sets from /backupinbox/.
#This script should be set to run everynight via crontab.

###
# Variables
###
BAK_DEST=backupinbox

#Array to hold list of folders in $BAK_DEST.
FOLDERS=(/$BAK_DEST/*)

###
# Script
###
cd /$BAK_DEST
for i in ${FOLDERS[@]}
do
    i1=`basename $i`    # Convert devsite/host01 --> host01
    /bin/ls -dt /$BAK_DEST/$i1/* | /usr/bin/tail -n +6 | /usr/bin/xargs /bin/rm -rf  #keep only latest 5 sets and delete older ones
    echo "purged old backups in folder: $i1"
done
exit 0
  2. Make /root/bin/backup-purge-old-backup.sh executable.
  3. Since server01:/backupinbox/web01/ has only 1 backup set from web01, create six more folders to use as dummy backup sets.
    4. [ba[email protected] ~]# cd /backupinbox/web01/
[[email protected] web01]# mkdir 2014-04-14_15 2014-04-14_16 2014-04-14_17 2014-04-14_18 2014-04-14_19 2014-04-14_20
[[email protected] web01]# ls | sort
2014-04-14_12
2014-04-14_15
2014-04-14_16
2014-04-14_17
2014-04-14_18
2014-04-14_19
2014-04-14_20
  4. On server01 as root, execute /root/bin/backup-purge-old-backup.sh. You many need to log into server01 with a new Terminal or switch from backupuser to root.
  5. Check /backupinbox/web01/ again and you should see only the latest 5 folders. Folders with earlier modify time were deleted when backup-purge-old-backup.sh was executed.
    6. [[email protected] web01]# ls | sort
2014-04-14_16
2014-04-14_17
2014-04-14_18
2014-04-14_19
2014-04-14_20
  6. To verify a backup was successfully executed by web01, you can run following command as root on server01 and you should see something like below. It shows there are 5 subfolders under /backupinbox/web01/, and only the latest folder '/backupinbox/web01/2014-04-29_21' actually has a backup set.
    7. [[email protected] ~]# find /backupinbox/web01/ -maxdepth 2 | sort
/backupinbox/web01/
/backupinbox/web01/2014-04-14_15
/backupinbox/web01/2014-04-14_16
/backupinbox/web01/2014-04-14_17
/backupinbox/web01/2014-04-14_18
/backupinbox/web01/2014-04-29_21
/backupinbox/web01/2014-04-29_21/etc--2014-04-29_21.tar.gz
  7. Edit /etc/crontab and add following line at the end to schedule backup-purge-old-backup.sh to run daily
    8. 55 5 * * * root /root/bin/backup-purge-old-backup.sh

Restoring from backup

If you are unfortunate enough to have to restore data from the backup, make sure you tar unzip the .tar.gz file as root. This is required for the restored data to have the original owner/permission. Check below. Restored 'etc' has root:root as owner:group. 

[[email protected] 2014-04-29_21]# ls
etc--2014-04-29_21.tar.gz
[[email protected] 2014-04-29_21]# tar xzf etc--2014-04-29_21.tar.gz
[[email protected] 2014-04-29_21]# ls -lr
total 8592
-rw-r--r--   1 backupuser backupuser 8783695 Apr 29 21:58 etc--2014-04-29_21.tar.gz
drwxr-xr-x 100 root       root         12288 Apr 29 12:10 etc
[[email protected] 2014-04-29_21]#

That's it!