{"id":919,"date":"2020-10-05T06:11:24","date_gmt":"2020-10-05T06:11:24","guid":{"rendered":"http:\/\/shijuvarghese.com\/?p=919"},"modified":"2024-07-01T17:22:23","modified_gmt":"2024-07-01T17:22:23","slug":"linux-pacemaker-the-new-ha-high-availability","status":"publish","type":"post","link":"http:\/\/shijuvarghese.com\/?p=919","title":{"rendered":"Linux Pacemaker &#8211;  the new HA (High Availability)"},"content":{"rendered":"<p>In the world of running critical application, where we cannot afford downtime caused by hardware failures, it becomes necessary to run cluster of hardware, where the application can be automatically migrated to another running hardware, in case the initial hardware fails due to whatever reason.<\/p>\n<ul>\n<li>Pacemaker is a high-availability cluster resource manager, that enables the following:<br \/>\nConstantly check the status of all the hardware systems running on the cluster.<\/li>\n<li>Start configured application (eg: httpd) in one of the running hardware based on the preference mentioned in the cluster configuration file.<\/li>\n<li>Move application to another hardware in case the present hardware becomes unavailable due to reasons such as hardware failure, network failure, unavailability of the node due to resource (memory, CPU, etc) over-consumption, etc.<\/li>\n<li>Enable administrator to move application to a different hardware in a seamless manner<\/li>\n<\/ul>\n<p>Pacemaker can be considered as new version on old <strong>HA<\/strong> application that was available in older Linux versions.<\/p>\n<p><span style=\"text-decoration: underline;\"><strong>Pacemaker &#8211; Installation procedure to create a\u00a0TWO\u00a0node cluster<\/strong><\/span><\/p>\n<p>Install the application in both nodes<br \/>\n<strong>[root@star ~]#<\/strong> yum install pacemaker pcs fence-agents -y<\/p>\n<p>&nbsp;<\/p>\n<p>Verify the <strong>hacluster<\/strong> user is created in both nodes<\/p>\n<p><strong>[root@star ~]#\u00a0<\/strong>cat \/etc\/passwd\u00a0| grep\u00a0 hacluster<\/p>\n<p>&nbsp;<\/p>\n<p>Set a password for the\u00a0hacluster used (eg: mypass) in both nodes<\/p>\n<p><strong>[root@star ~]#\u00a0<\/strong>passwd hacluster<\/p>\n<p>&nbsp;<\/p>\n<p>Start the pacemaker applications in both nodes<br \/>\n<strong>[root@star ~]#\u00a0<\/strong> systemctl start pcsd<br \/>\n<strong>[root@star ~]#\u00a0<\/strong> systemctl enable pcsd<\/p>\n<p>&nbsp;<\/p>\n<p>As we will have two nodes in this cluster, and will be addressed as\u00a0ha_node_1 and\u00a0ha_node_2 in this example, we need to add the DNS entry in the local DNS file.<\/p>\n<p><strong>[root@star ~]#\u00a0<\/strong>\u00a0vi \/etc\/hosts<br \/>\n192.168.1.22 ha_node_1<br \/>\n192.168.1.23 ha_node_2<\/p>\n<p>&nbsp;<\/p>\n<p>Disable\u00a0NetworkManager during startup in both the nodes<\/p>\n<p><strong>[root@star ~]#\u00a0<\/strong>\u00a0systemctl disable NetworkManager<\/p>\n<p>&nbsp;<\/p>\n<p>Run the following commands in\u00a0<strong>ha_node_1<\/strong><\/p>\n<p><strong>[root@star ~]#\u00a0<\/strong>pcs cluster auth ha_node_1 ha_node_2 -u hacluster<\/p>\n<p><strong>[root@star ~]#\u00a0<\/strong>pcs cluster setup &#8211;name Cluster ha_node_1 ha_node_2<\/p>\n<p>================<br \/>\nDestroying cluster on nodes: ha_node_1, ha_node_2&#8230;<br \/>\nha_node_1: Stopping Cluster (pacemaker)&#8230;<br \/>\nha_node_2: Stopping Cluster (pacemaker)&#8230;<br \/>\nha_node_2: Successfully destroyed cluster<br \/>\nha_node_1: Successfully destroyed cluster<\/p>\n<p>Sending &#8216;pacemaker_remote authkey&#8217; to &#8216;ha_node_1&#8217;, &#8216;ha_node_2&#8217;<br \/>\nha_node_1: successful distribution of the file &#8216;pacemaker_remote authkey&#8217;<br \/>\nha_node_2: successful distribution of the file &#8216;pacemaker_remote authkey&#8217;<br \/>\nSending cluster config files to the nodes&#8230;<br \/>\nha_node_1: Succeeded<br \/>\nha_node_2: Succeeded<\/p>\n<p>Synchronizing pcsd certificates on nodes ha_node_1, ha_node_2&#8230;<br \/>\nha_node_1: Success<br \/>\nha_node_2: Success<br \/>\nRestarting pcsd on the nodes in order to reload the certificates&#8230;<br \/>\nha_node_1: Success<br \/>\nha_node_2: Success<br \/>\n[root@localhost ~]#<\/p>\n<p>=================================<br \/>\nVerify in both nodes:<br \/>\n<strong>[root@star ~]#\u00a0<\/strong>cat \/etc\/corosync\/corosync.conf<\/p>\n<p>&nbsp;<\/p>\n<p>Run the command in\u00a0<strong>ha_node_1<\/strong><\/p>\n<p><strong>[root@star ~]#\u00a0<\/strong>pcs cluster start &#8211;all<br \/>\nha_node_1: Starting Cluster (corosync)&#8230;<br \/>\nha_node_2: Starting Cluster (corosync)&#8230;<br \/>\nha_node_1: Starting Cluster (pacemaker)&#8230;<br \/>\nha_node_2: Starting Cluster (pacemaker)&#8230;<\/p>\n<p>&nbsp;<\/p>\n<p>Check the status in both nodes<br \/>\n<strong>[root@star ~]#\u00a0<\/strong>pcs status<\/p>\n<p>====== =======<br \/>\nCluster name: Cluster<\/p>\n<p>WARNINGS:<br \/>\nNo stonith devices and stonith-enabled is not false<\/p>\n<p>Stack: corosync<br \/>\nCurrent DC: ha_node_2 (version 1.1.21-4.el7-f14e36fd43) &#8211; partition with quorum<br \/>\nLast updated: Mon Aug 24 13:37:59 2020<br \/>\nLast change: Mon Aug 24 13:37:48 2020 by hacluster via crmd on ha_node_2<\/p>\n<p>2 nodes configured<br \/>\n0 resources configured<\/p>\n<p>Online: [ ha_node_1 ha_node_2 ]<\/p>\n<p>No resources<br \/>\nDaemon Status:<br \/>\ncorosync: active\/disabled<br \/>\npacemaker: active\/disabled<br \/>\npcsd: active\/enabled<br \/>\n[root@localhost ~]#<br \/>\npcs cluster enable -all<\/p>\n<p>====== =======<br \/>\nNow configure the cluster with apache and virtual IP. Run command in 1st node<\/p>\n<p><strong>[root@star ~]#\u00a0<\/strong>pcs resource create VirtIP IPAddr ip=192.168.1.25 cidr_netmask=24 op monitor interval=30<br \/>\nAssumed agent name &#8216;ocf:heartbeat:IPaddr&#8217; (deduced from &#8216;IPAddr&#8217;)<\/p>\n<p>In case you want to delete the VirtIP cluster, execute the following command<\/p>\n<p><strong>[root@star ~]# &lt;&lt;&lt;&lt;\u00a0 \u00a0<\/strong>pcs resource delete VirtIP &gt;&gt;&gt;<\/p>\n<p><strong>[root@star ~]#\u00a0<\/strong>pcs resource create Httpd apache configuration=&#8221;\/etc\/httpd\/conf\/httpd.conf&#8221; op monitor interval=30<\/p>\n<p>If the above does not work we can try this command<br \/>\n<em><strong>[root@star ~]#\u00a0<\/strong> pcs resource create Httpd apache configuration=&#8221;\/etc\/httpd\/conf\/httpd.conf&#8221; op monitor interval=30 &#8211;force<\/em><\/p>\n<p>If apache httpd is already installed you may try the below<\/p>\n<p><em><strong>[root@star ~]#\u00a0<\/strong>\u00a0pcs resource update Httpd apache configfile=&#8221;\/etc\/httpd\/conf\/httpd.conf&#8221; op monitor interval=30 &#8211;force<\/em><\/p>\n<p>&nbsp;<\/p>\n<p>Verify the status of the Virtual IP and httpd<\/p>\n<p><strong>[root@star ~]#\u00a0<\/strong>\u00a0pcs status resources<\/p>\n<p>Now try accessing the web server by accessing the URL with the virtual IP. You may want to try switching off each node one at a time while the other node is running. The website should be accessible immaterial of the node that is dowm<\/p>\n<p style=\"text-align: center;\"><strong><em>Troubleshooting<\/em><\/strong><br \/>\n<strong> <em>= = = = =****= = = = =<\/em><\/strong><\/p>\n<p>In case the below command shows the Stopped result, you may want to start the cluster in one of the node:<\/p>\n<p><strong>[root@drbd1 ~]#<\/strong> pcs status resources<\/p>\n<p>======<br \/>\nHttpd (ocf::heartbeat:apache): Stopped<br \/>\nVirtIP (ocf::heartbeat:IPaddr): Stopped<\/p>\n<p>===========<\/p>\n<p><strong>[root@drbd1 ~]#<\/strong> pcs cluster start &#8211;all<\/p>\n<p><em>======<\/em><br \/>\n<em>ha_node_1: Starting Cluster (corosync)&#8230;<\/em><br \/>\n<em>ha_node_2: Starting Cluster (corosync)&#8230;<\/em><br \/>\n<em>ha_node_1: Starting Cluster (pacemaker)&#8230;<\/em><br \/>\n<em>ha_node_2: Starting Cluster (pacemaker)&#8230;<\/em><br \/>\n<em>=========<\/em><\/p>\n<p><strong>[root@drbd1 ~]#<\/strong> pcs status resources<\/p>\n<p><em>======<\/em><br \/>\n<em> Httpd (ocf::heartbeat:apache): Started ha_node_1<\/em><br \/>\n<em> VirtIP (ocf::heartbeat:IPaddr): Started ha_node_2<\/em><br \/>\n<em>=======<\/em><\/p>\n<p>At times you may see the below issue. In this case verify if the nodes can <em>ping each other<\/em> using the IP configured in <em>\/etc\/hosts<\/em> file<\/p>\n<p><strong>[root@drbd1 ~]#<\/strong> pcs status resources<\/p>\n<p><em>====<\/em><br \/>\n<em>Error: unable to get cluster status from crm_mon<\/em><br \/>\n<em>Error: cluster is not available on this node<\/em><br \/>\n<em>======<\/em><\/p>\n<p>The reason for the network issue could be due to a restart of the node, and the NetworkManager being disabled.<\/p>\n<p>Once the above are taken care we need to run the following command<\/p>\n<p><strong>[root@drbd1 ~]#<\/strong> pcs cluster start ha_node_1<br \/>\nha_node_1: Starting Cluster (corosync)&#8230;<br \/>\nha_node_2: Starting Cluster (pacemaker)&#8230;<\/p>\n<p>The below command moved cluster from one node to another<\/p>\n<p>[root@drbd1 ~]# pcs resource move Httpdha_node_2<br \/>\n[root@drbd1 ~]#<\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"mh-excerpt\"><p>In the world of running critical application, where we cannot afford downtime caused by hardware failures, it becomes necessary to run cluster of hardware, where <a class=\"mh-excerpt-more\" href=\"http:\/\/shijuvarghese.com\/?p=919\" title=\"Linux Pacemaker &#8211;  the new HA (High Availability)\">[&#8230;]<\/a><\/p>\n<\/div>","protected":false},"author":1,"featured_media":252,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17,3],"tags":[],"class_list":["post-919","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cluster","category-linux"],"_links":{"self":[{"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/posts\/919","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=919"}],"version-history":[{"count":7,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/posts\/919\/revisions"}],"predecessor-version":[{"id":1541,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/posts\/919\/revisions\/1541"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/media\/252"}],"wp:attachment":[{"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=919"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=919"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=919"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}