{"id":353,"date":"2016-11-29T10:35:49","date_gmt":"2016-11-29T10:35:49","guid":{"rendered":"http:\/\/shijuvarghese.com\/?p=353"},"modified":"2024-06-28T07:52:44","modified_gmt":"2024-06-28T07:52:44","slug":"installing-distributed-replicated-block-device-drbd-in-centos-7","status":"publish","type":"post","link":"http:\/\/shijuvarghese.com\/?p=353","title":{"rendered":"Installing Distributed Replicated Block Device &#8211; DRBD in CentOS 7"},"content":{"rendered":"<p>Distributed Replicated Block Device commonly called as DRBD is a distributed replicated storage system. This is used in Linux platform. It is a distributed replication (RAID 1) via network.<\/p>\n<p>Listed below are the steps involved in creating system in which a partition in two hosts are replicated<\/p>\n<p><strong>Lab setup:<\/strong><\/p>\n<ul>\n<li>Two CentOS linux hosts (hostnames to be DRBD1 and DRBD2 for example). The result of hostname command in both the hosts should return the same<\/li>\n<li>Both systems have a partition \/dev\/sda6 that will store data that will be replicated. This partition <em>should not be formatted or mounted<\/em>.<\/li>\n<li>Hosts has two NICs. One connected to local network, and other to a separate network (drbd vlan)<\/li>\n<li>Both the hosts has hostnames of both nodes in \/etc\/hosts file to connect to each other.<\/li>\n<li>IPs of NICs connected to local lan are 20.10.0.29 and 20.10.0.30<\/li>\n<li>IPs of NICs connected to DRBD vlans are 192.168.0.1 and 192.168.0.2<\/li>\n<li>Ensure SeLinux is <strong>disabled<\/strong><\/li>\n<\/ul>\n<p><strong>Steps:<\/strong><\/p>\n<p>Install elrepo in both the nodes in the system as DRBD packages does not come in<br \/>\n]#\u00a0<strong><em>rpm -ivh http:\/\/www.elrepo.org\/elrepo-release-7.0-2.el7.elrepo.noarch.rpm<\/em><\/strong><\/p>\n<p>Ensure the kernel is up-to-date in both the hosts<br \/>\n]# <strong><em>yum update kernel -y<\/em><\/strong><\/p>\n<p>&nbsp;<\/p>\n<p><em><strong>Note:<\/strong><\/em><\/p>\n<p>Install the below packages in both the nodes<br \/>\n]# <strong><em>yum -y install drbd84-utils kmod-drbd84<\/em><\/strong><br \/>\nNote: Some version of the drbd will not work well with certain new kernel. So we need to find the correct match and get the correct version of kmod-drbd84. For example &#8220;kmod-drbd84-8.4.7&#8221; instead of &#8220;kmod-drbd84&#8221;<\/p>\n<p>Load the module in the kernel in both the nodes<br \/>\n]# <em><strong>\/sbin\/modprobe drbd<\/strong><\/em><\/p>\n<p>Verify if the module is loaded in both the hosts:<br \/>\n]# <em><strong>lsmod | grep drbd<\/strong><\/em><br \/>\ndrbd\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 405309\u00a0 0<br \/>\nlibcrc32c\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 12644\u00a0 2 xfs,drbd<\/p>\n<p>Create a resource file for the clustered device in the first hosts:<br \/>\nDRBD1]# <strong><em>vi \/etc\/drbd.d\/data1.res<\/em><\/strong><\/p>\n<p><em>resource data1 {<\/em><br \/>\n<em>protocol C;<\/em><br \/>\n<em>on DRBD1 {<\/em><br \/>\n<em>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 device \/dev\/drbd0;<\/em><br \/>\n<em>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 disk \/dev\/sda6;<\/em><br \/>\n<em>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 address 192.168.0.1:7788;<\/em><br \/>\n<em>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 meta-disk internal;<\/em><br \/>\n<em>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }<\/em><br \/>\n<em>on DRBD2 {<\/em><br \/>\n<em>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 device \/dev\/drbd0;<\/em><br \/>\n<em>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 disk \/dev\/sda6;<\/em><br \/>\n<em>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 address 192.168.0.2:7788;<\/em><br \/>\n<em>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 meta-disk internal;<\/em><br \/>\n<em>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }<\/em><br \/>\n<em>}<\/em><\/p>\n<p>Copy the same file in the second host too.<br \/>\nDRBD1]# <em><strong>scp \/etc\/drbd.d\/data1.res root@DRBD2:\/etc\/drbd.d\/data1.res<\/strong><\/em><\/p>\n<p>Initialize the disks in both the nodes one by one<br \/>\n]# <em><strong>drbdadm create-md data1<\/strong><\/em><\/p>\n<p>You should see an output as below:<\/p>\n<p>&#8211;==\u00a0 Thank you for participating in the global usage survey\u00a0 ==&#8211;<br \/>\nThe server&#8217;s response is:<br \/>\nyou are the 10680th user to install this version<br \/>\ninitializing activity log<br \/>\nNOT initializing bitmap<br \/>\nWriting meta data&#8230;<br \/>\nNew drbd meta data block successfully created.<br \/>\nsuccess<\/p>\n<p>Start DRBD in both the hosts:<br \/>\n]# <strong><em>systemctl start drbd<\/em><\/strong><br \/>\n]#<strong><em> systemctl enable drbd<\/em><\/strong><\/p>\n<p>Both the nodes will be configured as secondary by default. Check this in both the hosts:<br \/>\n]# <strong><em>cat \/proc\/drbd<\/em><\/strong><br \/>\n]# <strong><em>drbd-overview<\/em><\/strong><\/p>\n<p>Note: Do not panic if you see the message &#8220;cs:Connected ro:Secondary\/Secondary ds:Inconsistent\/Inconsistent C r&#8212;&#8211;&#8221;<\/p>\n<p>Now you should make one node the primary. Let us do this in node 1:<br \/>\nDRBD1]# <em><strong>drbdadm primary data1<\/strong><\/em><br \/>\nIf the above does shows error, you may want to try:<br \/>\n<em><strong>drbdadm primary data1 &#8211;force<\/strong><\/em><\/p>\n<p>Now check the status in both the host:<br \/>\n]# <em><strong>cat \/proc\/drbd<\/strong><\/em><br \/>\nIdentify which is Primary and Sencondary<\/p>\n<p>Format the drbd partition in the PRIMARY node.<br \/>\nDRBD1]# <em><strong>mkfs.xfs \/dev\/drbd0<\/strong><\/em><\/p>\n<p>Create a folder in DRBD1 and mount the partition to it<br \/>\nDRBD1]# <em><strong>mkdir \/data<\/strong><\/em><br \/>\nDRBD1]# <em><strong>mount \/dev\/drbd0 \/data<\/strong><\/em><br \/>\nDRBD1]# <em><strong>touch \/data\/test1.txt<\/strong><\/em><\/p>\n<p>Now let us check if the file got replicated to DRBD2 node.<br \/>\nDRBD1]# <em><strong>umount \/data<\/strong><\/em><br \/>\nDRBD1]# <em><strong>drbdadm secondary data1<\/strong><\/em><\/p>\n<p>Perform the below in 2nd node<br \/>\nDRBD2]# <em><strong>drbdadm primary data1<\/strong><\/em><br \/>\nDRBD2]# <em><strong>mkdir \/data<\/strong><\/em><br \/>\nDRBD2]# <em><strong>mount \/dev\/drbd0 \/data<\/strong><\/em><br \/>\nDRBD2]# <em><strong>ls -l \/data<\/strong><\/em><\/p>\n<p>The file created in 1st node should be present here.<\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: center;\"><strong>Manual split brain recovery<\/strong><\/p>\n<p style=\"text-align: center;\">==================<\/p>\n<p style=\"text-align: left;\">At times, if there is a bad disturbance in the network and if both the node thinks they have the latest data, it gets into a split brain state.<\/p>\n<p>This can be identified if the &#8220;cat \/proc\/drbd&#8221; gives a response with &#8220;Secondary\/Unknown&#8221; or &#8220;Unknown\/Primary&#8221; in the nodes.<\/p>\n<p>Here we have to identify whch is the host that has the latest data and decide that as the primary. Let us say in this example DRBD2 has the latest data.Then we should do the following in the hosts:<\/p>\n<p>DRBD1]# drbdadm secondary data1<br \/>\nDRBD1]# drbdadm &#8212; &#8211;discard-my-data connect data1<\/p>\n<p>DRBD2]# drbdadm connect resource<\/p>\n<p>==== ====== ====== === OR =========<\/p>\n<p>In the one decided to have older data, run the following commands after umnounting the drbd0 device:<\/p>\n<pre class=\"bash\">drbdadm secondary all\r\ndrbdadm disconnect all\r\ndrbdadm -- --discard-my-data connect all<\/pre>\n<p>In the one to be primary, run the following:<\/p>\n<pre class=\"bash\">drbdadm primary all\r\ndrbdadm disconnect all\r\ndrbdadm connect all<\/pre>\n<p>======== ============ ======<\/p>\n<p>There could be situation where DRBD is not enabled to start during boot-up, and have shutdown both hosts. When trying to boot up, if one node does not come up, we will not be able to start DRBD in the other node. In this case we have to do the following:<\/p>\n<p>DRBD2]#\u00a0modprobe drbd<br \/>\nDRBD2]#\u00a0drbdadm create-md\u00a0data1<br \/>\nDRBD2]#\u00a0drbdadm up\u00a0data1<br \/>\nDRBD2]#\u00a0drbdadm primary\u00a0data1\u00a0<em><strong>(or\u00a0drbdadm primary\u00a0data1\u00a0&#8211;force)<\/strong><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"mh-excerpt\"><p>Distributed Replicated Block Device commonly called as DRBD is a distributed replicated storage system. This is used in Linux platform. It is a distributed replication <a class=\"mh-excerpt-more\" href=\"http:\/\/shijuvarghese.com\/?p=353\" title=\"Installing Distributed Replicated Block Device &#8211; DRBD in CentOS 7\">[&#8230;]<\/a><\/p>\n<\/div>","protected":false},"author":1,"featured_media":355,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17,3],"tags":[],"class_list":["post-353","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cluster","category-linux"],"_links":{"self":[{"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/posts\/353","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=353"}],"version-history":[{"count":19,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/posts\/353\/revisions"}],"predecessor-version":[{"id":1525,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/posts\/353\/revisions\/1525"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/media\/355"}],"wp:attachment":[{"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=353"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=353"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=353"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}