setup GlusterFS untuk NFS sharing backup email account zimbra di Ubuntu 12.04

Mail server Zimbra (mail.abc.co.id) telah menggunakan RAID 1+0 . lihat hasil config status dibawah

/opt/hp/hpssacli/bin # /opt/hp/hpssacli/bin/hpssacli ctrl all show config
Dynamic Smart Array B120i RAID in Slot 0 (Embedded)
Smart Array P222 in Slot 1                (sn: PDSXH0BRH5P0P4)
   array A (SATA, Unused Space: 0  MB)
      logicaldrive 1 (5.5 TB, RAID 1+0, OK)
      physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SATA, 3 TB, OK)
      physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SATA, 3 TB, OK)
      physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SATA, 3 TB, OK)
      physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SATA, 3 TB, OK)
   SEP (Vendor ID PMCSIERA, Model SRCv8x6G) 380 (WWID: 500143802894C36F)
/opt/hp/hpssacli/bin #

RAID is not a backup solution. It is used to improve disk I/O (performance) and reliability of your server or workstation. Can RAID Array Fail ? Yes. The entire RAID array can fail taking down all your data (yes hardware RAID card do dies) link

GlusterFS is used in environments where high performance, redundancy and reliability are of a premium. GlusterFS is a file system that is designed to provide network storage that can be made redundant, fault-tolerant and scalable. It’s particularly well suited to applications that require high-performance access to large files. With GlusterFS, you can have enterprise- or scientific-research-grade storage up and running in minutes. link

Setup GlusterFS ini sebagai langkah akhir atau solusi terakhir (backup), jika raid disk sudah tidak bisa diselamatkan. Dengan menggunakan space ~400 GB, backup harian dari email zimbra yang kurang lebih 1GB/hari, diharapkan dapat menampung selama 365 hari atau satu tahun. setelah satu tahun akan dirotate.

Prepare:

Linux server (menggunakan ubuntu server 12.04). Karena mail server zimbra telah menggunakan ubuntu server 12.04, maka untuk compability glusterfs, lebih baik menggunakan Os yang sama. Pastikan telah terinstall dengan baik.
Install GlusterFS Untuk menginstall glusterfs ketik perintah:

apt-get install glusterfs-server

Setting brick:

brick adalah storage pool yang akan digunakan oleh gluster fs untuk menampung datanya. berikut disk space free dari ubuntu server yang telah kita install:

root@glusterfs:/data1# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       2.9G  1.1G  1.8G  38% /
udev            2.0G  4.0K  2.0G   1% /dev
tmpfs           406M  240K  405M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm
/dev/sdb1       148G  592M  140G   1% /data1
/dev/sdc1       252G  592M  239G   1% /data2

directory /data1 dan directory /data2 memiliki free space yang dapat kita gunakan, jika ditotal maka kurang lebih menjadi 400GB . kita buat directory zimbrabackup misalnya di setiap directory tersebut.

root@glusterfs:~# mkdir /data1/zimbrabackup
root@glusterfs:~# mkdir /data2/zimbrabackup

dari 2 buah directory partisi disk tersebut akan kita gabung menjadi satu dengan perintah: pastikan glusterfs (hostname) sudah didefine di /etc/hosts

root@glusterfs:~# gluster volume create ZimbraBackup transport tcp glusterfs:/data1/zimbrabackup glusterfs:/data2/zimbrabackup
Creation of volume ZimbraBackup has been successful. Please start the volume to access data.
root@glusterfs:~# gluster volume info
Volume Name: ZimbraBackup
Type: Distribute
Status: Created
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: glusterfs:/data1/zimbrabackup
Brick2: glusterfs:/data2/zimbrabackup
root@glusterfs:~#

lalu kita allow host yang dapat mengakses distributed volume tersebut:

root@glusterfs:~# gluster volume set ZimbraBackup auth.allow 192.168.2.2
Set volume successful

lalu cek dengan perintah:

root@glusterfs:~# gluster volume info
Volume Name: ZimbraBackup
Type: Distribute
Status: Created
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: glusterfs:/data1/zimbrabackup
Brick2: glusterfs:/data2/zimbrabackup
Options Reconfigured:
auth.allow: 192.168.2.2
root@glusterfs:~#

Jika sudah lalu start volume dengan perintah

root@glusterfs:~# gluster volume start ZimbraBackup
Starting volume ZimbraBackup has been successful
root@glusterfs:~#

cek status volume

root@glusterfs:~# gluster volume info
Volume Name: ZimbraBackup
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: glusterfs:/data1/zimbrabackup
Brick2: glusterfs:/data2/zimbrabackup
Options Reconfigured:
auth.allow: 192.168.2.2
root@glusterfs:~#

ok. storage pool sudah ready.

Mount dari client

Mount gluster volume dari mail server zimbra untuk mounting NFS dari GlusterFS ketik:

root@mail:~# mount.glusterfs 192.168.1.51:/ZimbraBackup /ZimbraBackup/

lalu cek:

root@mail:~# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/zimbra--vg-root  5.4T  816G  4.4T  16% /
udev                         3.9G  4.0K  3.9G   1% /dev
tmpfs                        783M  248K  782M   1% /run
none                         5.0M     0  5.0M   0% /run/lock
none                         3.9G     0  3.9G   0% /run/shm
/dev/sda2                    237M   32M  193M  14% /boot
192.168.1.51:/ZimbraBackup   400G  119M  379G   1% /ZimbraBackup

terlihat size directory /ZimbraBackup sebesar 400GB . lalu coba transfer file ke host glusterFS:

root@mail:/ZimbraBackup# rsync -avh --progress /backup/zimbra-backup/20150512.tar.bz2 /ZimbraBackup/20150512.tar.bz2
sending incremental file list
20150512.tar.bz2
       1.12G 100%   60.27MB/s    0:00:17 (xfer#1, to-check=0/1)
sent 1.12G bytes  received 31 bytes  60.34M bytes/sec
total size is 1.12G  speedup is 1.00
root@mail:/ZimbraBackup#

cek di host glusterfs:

root@glusterfs:~# ls /data*/zimbrabackup/
/data1/zimbrabackup/:
20150512.tar.bz2
/data2/zimbrabackup/:
root@glusterfs:~#

file 20150512.tar.bz2 sudah tersimpan di /data1/zimbrabackup/

ref:

Final shell script untuk backup:

#!/bin/bash
### START CONFIGURATION ###
DIR="/backup/zimbra-backup";
#BASEDIR="/home/user/backup";
### END OF CONFIGURATION ###

mount | grep Zimbra

if [ "$?" == "0" ];then
    USERS=`su - zimbra -c 'zmprov -l gaa'`;
    ### The above command work on Zimbra 5.x. If you use Zimbra 6.x, use
    ### USERS=`su - zimbra -c 'zmprov -l gaa'`; instead
    DATE=`date +%Y%m%d`;

    ### backup list account
    echo "Processing list account backup..."
    touch $DIR/listaccount-"$DATE".csv
    su - zimbra -c "zmprov -l gaa" > $DIR/listaccount-"$DATE".csv
    echo "backup list account done..."
    HARI=`date --date='1 days ago' +"%m/%d/%Y"`

    #HARI=`date +%m/%d/%Y`
    #query="&query=after:$HARI"
    query="&query=date:$HARI"

    if [ ! -d $DIR ]; then mkdir $DIR; chown zimbra:zimbra $DIR; fi

    ### backup mailbox
    for ACCOUNT in $USERS; do
            NAME=`echo $ACCOUNT`;
            echo "Processing mailbox $NAME backup..."
            #su - zimbra -c "zmmailbox -z -m $ACCOUNT getRestURL '//?fmt=zip$query' > $DIR/$NAME.zip";
            su - zimbra -c "zmmailbox -z -m $ACCOUNT getRestURL '//?fmt=tgz$query' > $DIR/$NAME.tgz";
    done

    echo "Compressing mailbox backup, please wait..."
    #cd $DIR; mkdir $DATE; mv *.zip $DATE
    cd $DIR; mkdir $DATE; mv *.tgz $DATE

    tar cjvf $DIR/$DATE.tar.bz2 $DATE;
    rsync -avh --progress $DIR/$DATE.tar.bz2 /ZimbraBackup/$DATE.tar.bz2
    rsync -avh --progress $DIR/listaccount-"$DATE".csv /ZimbraBackup/listaccount-"$DATE".csv

    logfile="/tmp/zbackup-`date +%F`.log"
    echo "Zimbra mailbox backup has been completed successfully..." > $logfile
    echo "" >> $logfile

    sizeremote=`du -sh /ZimbraBackup/$DATE.tar.bz2`
    sizelocal=`du -sh $DIR/$DATE.tar.bz2`
    checksumlocal=`/usr/bin/md5sum $DIR/$DATE.tar.bz2`
    checksumremote=`/usr/bin/md5sum /ZimbraBackup/$DATE.tar.bz2`

    echo "Size File di NFS: $sizeremote" >> $logfile
    echo "Size File di local: $sizelocal" >> $logfile
    echo "Checksum File di NFS: $checksumremote" >> $logfile
    echo "Checksum File di local: $checksumlocal" >> $logfile

    cd $DIR;rm -rf $DATE;rm -rf $DATE.tar.bz2;
    if [ -f "$DIR/$DATE.tar.bz2" ];then
        echo "WARNING!: File $DIR/$DATE.tar.bz2 still exist" >> $logfile
    else
        echo "File $DIR/$DATE.tar.bz2 was deleted..." >> $logfile
    fi

    echo "" >> $logfile
    echo "GlusterFS Space:" >> $logfile
    /bin/df /ZimbraBackup >> $logfile

    /usr/bin/sendemail -f ngadimin@abc.co.id -u "Zimbra daily account Backup (OK)" -t ngadimin@abc.co.id -o message-content-type=text -o message-file=$logfile -xu user -xp pass -s 192.168.2.2:587 -o tls=no

else
    /usr/bin/sendemail -f ngadimin@abc.co.id -u "Zimbra daily account Backup (Warning)" -t ngadimin@abc.co.id -o message-content-type=text -o -m "Zimbra mailbox backup failed. /ZimbraBackup mount not found" -xu user -xp pass -s 192.168.2.2:587 -o tls=no

fi