The recent release of DRBD 8.3 now includes The Third Node feature as a freely available component. This document will cover the basics of setting up a third node on a standard Debian Etch installation. At the end of this tutorial you will have a DRBD device that can be utilized as a SAN, an iSCSI target, a file server, or a database server.
Note: LINBIT support customers can skip Section 1 and utilize the package repositories.
LINBIT has hosted third node solutions available, please contact them at sales_us@linbit.com for more information.
Preface:
The setup is as follows:
- Three servers: alpha, bravo, foxtrot
- alpha and bravo are the primary and secondary local nodes
- foxtrot is the third node which is on a remote network
- Both alpha and bravo have interfaces on the 192.168.1.x network (eth0) for external connectivity.
- A crossover link exists on alpha and bravos (eth1) for replication using 172.16.6.10 and .20
- Heartbeat provides a virtual IP of 192.168.5.2 to communicate with the disaster recovery node located in a geographically diverse location
Section 1: Installing The Source
These steps need to be done on each of the 3 nodes.
Prerequisites:
- make
- gcc
- glibc development libraries
- flex scanner generator
- headers for the current kernel
Enter the following at the command line as a privileged user to satisfy these dependencies:
apt-get install make gcc libc6 flex linux-headers-`uname -r` libc6-dev linux-kernel-headers
Once the dependencies are installed, download DRBD. The latest version can always be obtained at http://oss.linbit.com/drbd/. Currently, it is 8.3.
cd /usr/src/
wget http://oss.linbit.com/drbd/8.3/drbd-8.3.0.tar.gz
After the download is complete:
- Uncompress DRBD
- Enter the source directory
- Compile the source
- Install DRBD
tar -xzvf drbd-8.3.0.tar.gz
cd /usr/src/drbd-8.3.0/
make clean all
make install
Now load and verify the module:
modprobe drbd
cat /proc/drbd
version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@alpha, 2009-02-05 10:36:11
Once this has been completed on each of the three nodes, continue to next section.
Section 2: Heartbeat Configuration
Setting up a third node entails stacking DRBD on top of DRBD. A virtual IP is needed for the third node to connect to, for this we will setup a simple Heartbeat v1 configuration. This section will only be done on alpha and bravo.
Install Heartbeat:
apt-get install heartbeat
Edit the authkeys file :
vi /etc/ha.d/authkeys
auth 1
1 sha1 yoursupersecretpasswordhere
Once the file has been created, change the permissions on the file. Heartbeat will not start if this step is not followed.
chmod 600 /etc/ha.d/authkeys
Copy the authkeys file to bravo:
scp /etc/ha.d/authkeys bravo:/etc/ha.d/
Edit the ha.cf file:
vi /etc/ha.d/ha.cf
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 1
deadtime 10
warntime 5
initdead 60
udpport 694
ucast eth0 192.168.1.10
ucast eth0 192.168.1.20
auto_failback off
node alpha
node bravo
Copy the ha.cf file to bravo:
scp /etc/ha.d/ha.cf bravo:/etc/ha.d/
Edit the haresources file, the IP created here will be the IP that our third node refers to.
vi /etc/ha.d/haresources
alpha IPaddr::192.168.5.2/24/eth0
Copy the haresources file to bravo:
scp /etc/ha.d/haresources bravo:/etc/ha.d/
Start the heartbeat service on both servers to bring up the virtual IP
alpha:/# /etc/init.d/heartbeat start
bravo:/# /etc/init.d/heartbeat start
Heartbeat will bring up the new interface (eth0:0).
Note: It may take heartbeat up to one minute to bring the interface up.
alpha:/# ifconfig eth0:0
eth0:0 Link encap:Ethernet HWaddr 00:08:C7:DB:01:CC
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Section 3: DRBD Configuration
Configuration for DRBD is done via the drbd.conf file. This needs to be the same on all nodes (alpha, bravo, foxtrot). Please note that the usage-count is set to yes, which means it will notify Linbit that you have installed DRBD. No personal information is collected. Please see this page for more information :
global { usage-count yes; }
resource data-lower {
protocol C;
net {
shared-secret "LINBIT";
}
syncer {
rate 12M;
}
on alpha {
device /dev/drbd1;
disk /dev/hdb1;
address 172.16.6.10:7788;
meta-disk internal;
}
on bravo {
device /dev/drbd1;
disk /dev/hdd1;
address 172.16.6.20:7788;
meta-disk internal;
}
}
resource data-upper {
protocol A;
syncer {
after data-lower;
rate 12M;
al-extents 513;
}
net {
shared-secret "LINBIT";
}
stacked-on-top-of data-lower {
device /dev/drbd3;
address 192.168.5.2:7788; # IP provided by Heartbeat
}
on foxtrot {
device /dev/drbd3;
disk /dev/sdb1;
address 192.168.5.3:7788; # Public IP of the backup node
meta-disk internal;
}
}
Section 4: Preparing The DRBD Devices
Now that the configuration is in place, create the metadata on alpha and bravo.
alpha:/usr/src/drbd-8.3.0# drbdadm create-md data-lower
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
bravo:/usr/src/drbd-8.3.0# drbdadm create-md data-lower
Writing meta data...
initialising activity log
NOT initialized bitmap
New drbd meta data block successfully created.
Now start DRBD on alpha and bravo:
alpha:/usr/src/drbd-8.3.0# /etc/init.d/drbd start
bravo:/usr/src/drbd-8.3.0# /etc/init.d/drbd start
Verify that the lower level DRBD devices are connected:
cat /proc/drbd
version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@alpha, 2009-02-05 10:36:11
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:19530844
Tell alpha to become the primary node:
NOTE: As the command states, this is going to overwrite any data on bravo: Now is a good time to go and grab your favorite drink.
alpha:/# drbdadm -- --overwrite-data-of-peer primary data-lower
alpha:/# cat /proc/drbd
version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@alpha, 2009-02-05 10:36:11
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---
ns:3088464 nr:0 dw:0 dr:3089408 al:0 bm:188 lo:23 pe:6 ua:53 ap:0 ep:1 wo:b oos:16442556
[==>.................] sync'ed: 15.9% (16057/19073)M
finish: 0:16:30 speed: 16,512 (8,276) K/sec
After the data sync has finished, create the meta-data on data-upper on alpha, followed foxtrot.
Note the resource is data-upper and the --stacked option is on alpha only.
alpha:~# drbdadm --stacked create-md data-upper
Writing meta data...
initialising activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success
foxtrot:/usr/src/drbd-8.3.0# drbdadm create-md data-upper
Writing meta data...
initialising activity log
NOT initialized bitmap
New drbd meta data block sucessfully created.
Bring up the stacked resource, then make alpha the primary of data-upper
alpha:/# drbdadm --stacked adjust data-upper
foxtrot:~# drbdadm adjust data-upper
foxtrot:~# cat /proc/drbd
version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@foxtrot, 2009-02-02 10:28:37
1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent A r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:19530208
alpha:~# drbdadm --stacked -- --overwrite-data-of-peer primary data-upper
alpha:~# cat /proc/drbd
version: 8.3.0 (api:88/proto:86-89)
GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@alpha, 2009-02-05 10:36:11
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r---
ns:19532532 nr:0 dw:1688 dr:34046020 al:1 bm:1196 lo:156 pe:0 ua:0 ap:156 ep:1 wo:b oos:0
1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent A r---
ns:14512132 nr:0 dw:0 dr:14512676 al:0 bm:885 lo:156 pe:32 ua:292 ap:0 ep:1 wo:b oos:5018200
[=============>......] sync'ed: 74.4% (4900/19072)M
finish: 0:07:06 speed: 11,776 (10,992) K/sec
Drink time again!
After the sync is complete, access your DRBD block device via /dev/drbd3. This will write to both local nodes and the remote third node. In your Heartbeat configuration you will use the "drbdupper" script to bring up your /dev/drbd3 device. Have fun!
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
