INSTALLING AND CONFIGURING DRBD 8.3 AND HEARTBEAT 3.0 ON CENTOS 6 TO BUILD A HIGHLY AVAILABLE FILE SERVER.

DRBD & HEARTBEAT

*********************************************************

INTRODUCTION:DRBD It refers to Distributed Replicated Block Device. It is distributed storage system for the GNU/Linux platform. It consists of a kernel module, several user space management applications and some shell scripts and is normally used on high availability (HA) clusters. DRBD bears similarities to RAID 1, except that it runs over a network. DRBD refers to both the software (kernel module and associated user space tools), and also to specific logical block devices managed by the software. Heartbeat It is a basic heartbeat subsystem for Linux-HA. It will run scripts at initialization, and when machines go up or down. This version will also perform IP address takeover using gratuitous ARPs. It works correctly for a 2-node configuration, and is extensible to larger configurations so in simple language it can be understood as a failover mechanism , whenever your primary host goes down it instantly makes the next secondary host as primary and saves the loss of data.

PREPARING THE SERVERS:You have to take 2 servers and install centos 6 on the both leaving the amount of free space which you wish to use as high available file server. Here leaving free space means just leaving some amount of space in your hard disk, not trying to make a partition because it is a rule that if you mount your partition the DRBD will not consider it and it will generate errors. After we have the OS installed we will configure the machine so that it should be network available and then disable selinux, and iptables and get CentOS up to date. We can activate IPTABLES and selinux back again but for now we disabled it to make things easier to troubleshoot in case we need. Please note that all the below configuration will be done on both the servers until it is directed. Very first I suggest that you should install the NANO editor if its not already with the OS pack because it is much easier to work on NANO(In my view) -Installing NANO #yum install nano - Disabling selinux # nano /etc/selinux/config SELINUX=disabled - Turn Off iptable from system start up # chkconfig iptables off - Update the system # yum update Now we will install some packages that will be necessary for DRBD installation and configuration. # yum install wget ntp #service ntpd start

*********************************************************

Here ntp is network time protocol it is necessary because the both servers will interact and synchronize with each other so if there is time difference it can generate errors. We need to make sure that ntpd is always running to keep both servers time synchronized. # chkconfig ntpd on

HOSTNAME RESOLVE
Now lets suppose we have created two servers :Host name Server1 Server2 test1 test2 ipaddress 10.2.1.230 10.2.1.232

Now we need to make sure that these two servers ping each also other with their hostname and for this we have to open #nano /etc/hosts And write these two lines in the file 10.2.1.230 10.2.1.232 test1 test2

Next try to ping both the servers from each other if they are able to ping then its fine otherwise you need to recheck the /etc/hosts file and also check /etc/sysconfig/network for your hostname and /etc/sysconfig/network-scripts/name of-your-interface for checking your network configuration including IP address, gateway, DNS etc.

CREATING PARTITION
Now we need to create a partition which will be used by DRBD. Please be careful following these steps because the numbers and name could change according machine to machine #fdisk /dev/sda WARNING: DOS-compatible mode is deprecated. It's strongly recommended to Switch off the mode (command 'c') and change display units to sectors (command 'u'). Command (m for help):n (to create new partition) Command action e extended p primary partition (1-4):p (for primary partition)

*********************************************************

select partition (1-4):3 First cylinder (18165-20023, default 18165): Using default value 18165 Last cylinder, +cylinders or +size{K,M,G} (18165-20023, default 20023):+20G In the above line we specified the size of the partition in this case we took it as 20gb Command (m for help):w (to write changes to disk) The changes will be made to disk when you reboot the machine #reboot Now if till now every step is fine we should move further to build and install drbd.

*********************************************************

INSTALLING DRBD
Please note that below anywhere x86_64 is according to the machine architecture which is 64 here and if you are using machine of 32-bit architecture you should replace x86_64 with i386 #yum -y install gcc make automake autoconf flex rpm-build kernel-devel #mkdir -p rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS} #wget http://oss.linbit.com/drbd/8.3/drbd-8.3.11.tar.gz #tar zxvf drbd-8.3.11.tar.gz #cd drbd-8.3.11 drbd-8.3.11]#./configure drbd-8.3.11]#make rpm drbd-8.3.11]#make km-rpm drbd-8.3.11]#cd /root/rpmbuild/RPMS/x86_64 x86_64]#rpm -Uvh drbd-utils-8.3.11-1.el6.x86_64.rpm drbd-km-2.6.32_71.29.1.el6.x86_64-8.3.111.el6.x86_64.rpm Preparing... ########################################### [100%] 1:drbd-utils########################################### [ 50%] 2:drbd-km-2.6.32_71.29.1.################################### [100%] Next, create volumes like follows. #pvcreate /dev/sda3 Physical volume "/dev/sdb1" successfully created #vgcreate vg_r0 /dev/sda3 Volume group "vg_r0" successfully created #lvcreate -L 20G -n lv_r0 vg_r0 Logical volume "lv_r0" created

*********************************************************

Configure DRBD.
Set them on both hosts. #nano /etc/drbd.d/global_common.conf global { #at second line write or modify if previously written as usage-count no; disk { # line 27: add ( detach disks when IO-error ) on-io-error detach; syncer { # line 38: add ( band width for synchronization ) rate 300M; Save the file and exit. Now create a file name main.res #nano /etc/drbd.d/main.res # create new resource main { # DRBD device device /dev/drbd0; # phisical device disk /dev/vg_r0/lv_r0; meta-disk internal; on test1 { # IP address:port address 10.2.1.230:7788; } on test2 {

*********************************************************

address 10.2.1.232:7788; } } Save the file and exit #modprobe drbd #lsmod | grep drbd #drbdadm create-md main --== Thank you for participating in the global usage survey ==-The server's response is: you are the 20298th user to install this version Writing meta data... initializing activity log NOT initialized bitmap New drbd meta data block successfully created. success Now we should start drbd service on both the servers. May I remind one more time till now what have done should be same on both the servers and will be same in both in further configuration until it is directed. #service drbd start Starting DRBD resources: [ d(main) s(main) n(main) ]. #chkconfig drbd on # echo "/sbin/modprobe drbd" >> /etc/rc.local Make a Host be primary, then synchronization starts. #cat /proc/drbd version: 8.3.11 (api:88/proto:86-96) GIT-hash: 0de839cee13a4160eed6037c4bddd066645e23c5 build by root@www01.server.world, 2011-07-15 15:26:20 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:20970844

*********************************************************

run the below command on any one server which you wish to make primary. In this case lets make test1 a primary server so the commands will be run on test1 server. #drbdsetup /dev/drbd0 primary o # set primary #cat /proc/drbd version: 8.3.11 (api:88/proto:86-96) GIT-hash: 0de839cee13a4160eed6037c4bddd066645e23c5 build by root@www01.server.world, 2011-07-15 15:26:20 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----ns:940444 nr:0 dw:0 dr:948480 al:0 bm:56 lo:136 pe:3130 ua:1979 ap:0 ep:1 wo:b oos:20042916 [>....................] sync'ed: 4.5% (19572/20476)M finish: 0:03:57 speed: 84,356 (84,356) K/sec after few minutes later, synchronization completes and the status turns like follows. #cat /proc/drbd version: 8.3.11 (api:88/proto:86-96) GIT-hash: 0de839cee13a4160eed6037c4bddd066645e23c5 build by root@www01.server.world, 2011-07-15 15:26:20 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----ns:20970844 nr:0 dw:0 dr:20971508 al:0 bm:1277 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 Configuration completes. It's OK to create a file system on DRBD device and mount it on a primary host to use it. #mkfs -t ext4 /dev/drbd0 #mkdir /drbd0 #mount /dev/drbd0 /drbd0 #touch /drbd0/test.txt # create a test file

If you'd like to mount DRBD device on secondary Host, unmount DRBD device on primary Host first and make primary Host be secondary. Next make secondary Host be primary and mount DRBD device. In the way we can also check the replication of data as the test.txt file which we created will show on secondary host when we will mount the DRBD device on the secondary host.

*********************************************************

########### on Primary Host ########### #umount /drbd0 #drbdadm secondary main # set secondary

########### on Secondary Host ########### #drbdadm primary main #mount /dev/drbd0 /drbd0 Now when we will list the directories and files of /drbd it will show test.txt. # set primary

*********************************************************

HEARTBEAT INSTALLATION AND CONFIGURATION

# rpm -Uvh http://download.fedora.redhat.com/pub/epel/6/x86_64/epel-release-65.noarch.rpm # nano /etc/yum.repos.d/epel.repo enabled=0 # yum --enablerepo=epel install heartbeat Create the following file on both servers with the exact same content: # nano /etc/ha.d/ha.cf logfile /var/log/ha-log logfacility local0 keepalive 2 deadtime 30 initdead 120 bcast Auto_eth0 udpport 694 auto_failback on node test1 node test2 The next step is to create the resource file for heartbeat on both servers with exact same content again: # nano /etc/ha.d/haresources test1 IPaddr::10.2.1.200/24/Auto_eth0 first word is the hostname of the primary server then the IP 10.2.1.200 is the one I choose to be the virtual IP to be moved to the slave in case of a failure. The last thing is to create the authentication file on both servers again with the same content:

*********************************************************

# nano /etc/ha.d/authkeys auth 2 2 sha1 my-password This password file should only be readable by the root user: # chmod 600 /etc/ha.d/authkeys Ok now we should be ready to go Lets test it!! On both servers start heartbeat service: # service heartbeat start on server1 # ifconfig Auto_eth0 Link encap:Ethernet HWaddr 08:00:27:95:AB:B1

inet addr:10.2.1.230 Bcast:10.2.1.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe95:abb1/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2189638 errors:0 dropped:0 overruns:0 frame:0 TX packets:30442386 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:188528923 (179.7 MiB) TX bytes:45853044392 (42.7 GiB) Auto_eth0:0 Link encap:Ethernet HWaddr 08:00:27:95:AB:B1 inet addr:10.2.1.200 Bcast:10.2.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:305 errors:0 dropped:0 overruns:0 frame:0 TX packets:305 errors:0 dropped:0 overruns:0 carrier:0

*********************************************************

collisions:0 txqueuelen:0 RX bytes:35012 (34.1 KiB) TX bytes:35012 (34.1 KiB)

Testing DRBD fail over

You can shut down server1 or simply stop heartbeat service: On server1: # service heartbeat stop On server2: # ifconfig Auto_eth0 Link encap:Ethernet HWaddr 08:00:27:8F:3B:50

inet addr:10.2.1.232 Bcast:10.2.1.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe8f:3b50/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:30447253 errors:0 dropped:0 overruns:0 frame:0 TX packets:2138369 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:45799991579 (42.6 GiB) TX bytes:173195698 (165.1 MiB) Auto_eth0:0 Link encap:Ethernet HWaddr 08:00:27:8F:3B:50 inet addr:10.2.1.200 Bcast:10.0.4.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:208 errors:0 dropped:0 overruns:0 frame:0 TX packets:208 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0

*********************************************************

RX bytes:23480 (22.9 KiB) TX bytes:23480 (22.9 KiB) Till this point if your machine responds in the same manner , congratulations you have now installed and configured DRBD with Heartbeat. If you get any error refer to /var/log/messages to get more detail.