Pacemaker 1.

1-pcs Pacemaker Remote

Pacemaker Remote
Extending High Availablity into Virtual Nodes Edition 1

David Vossel
Primary autho r Red Hat dvossel@redhat.com

Legal Notice
Co pyright 2009-2013 David Vo ssel. The text o f and illustratio ns in this do cument are licensed under a Creative Co mmo ns Attributio nShare Alike 3.0 Unpo rted license ("CC-BY-SA")[1]. In acco rdance with CC-BY-SA, if yo u distribute this do cument o r an adaptatio n o f it, yo u must pro vide the URL fo r the o riginal versio n. In additio n to the requirements o f this license, the fo llo wing activities are lo o ked upo n favo rably: 1. If yo u are distributing Open Publicatio n wo rks o n hardco py o r CD-ROM, yo u pro vide email no tificatio n to the autho rs o f yo ur intent to redistribute at least thirty days befo re yo ur manuscript o r media freeze, to give the autho rs time to pro vide updated do cuments. This no tificatio n sho uld describe mo dificatio ns, if any, made to the do cument. 2. All substantive mo dificatio ns (including deletio ns) be either clearly marked up in the do cument o r else described in an attachment to the do cument. 3. Finally, while it is no t mandato ry under this license, it is co nsidered go o d fo rm to o ffer a free co py o f any hardco py o r CD-ROM expressio n o f the autho r(s) wo rk.

Abstract
The do cument exists as bo th a reference and deplo yment guide fo r the Pacemaker Remo te service. The KVM and Linux Co ntainer walk-thro ugh tuto rials will use: 1. Fedo ra 18 as the ho st o perating system 2. Pacemaker Remo te to perfo rm reso urce management within virtual no des 3. libvirt to manage KVM and LXC virtual no des 4. Co ro sync to pro vide messaging and membership services o n the ho st no des 5. Pacemaker to perfo rm reso urce management o n ho st no des

Pacemaker Remote

5. Pacemaker to perfo rm reso urce management o n ho st no des

Table o f Co ntents 1. Extending High Availability Cluster into Virtual No des 1.1. Overview 1.2. Terms 1.3. Virtual Machine Use Case 1.4. Linux Co ntainer Use Case 1.5. Expanding the Cluster Stack 1.5.1. Traditio nal HA Stack 1.5.2. Remo te-No de Enabled HA Stack 2. Quick Example 2.1. Mile High View o f Co nfiguratio n Steps 2.2. What tho se steps just did 2.3. Accessing Cluster fro m Remo te-no de 3. Co nfiguratio n Explained 3.1. Reso urce Optio ns 3.2. Ho st and Guest Authenticatio n 3.3. Pacemaker and pacemaker_remo te Optio ns 4. KVM Walk-thro ugh 4.1. Step 1: Setup the Ho st 4.1.1. SElinux and Firewall 4.1.2. Install Cluster So ftware 4.1.3. Setup Co ro sync 4.1.4. Verify Cluster So ftware 4.1.5. Install Virtualizatio n So ftware 4.2. Step2: Create the KVM guest 4.2.1. Setup Guest Netwo rk 4.2.2. Setup Pacemaker Remo te 4.2.3. Verify Ho st Co nnectio n to Guest 4.3. Step3: Integrate KVM guest into Cluster. 4.3.1. Start the Cluster 4.3.2. Integrate KVM Guest as remo te-no de 4.3.3. Starting Reso urces o n KVM Guest 4.3.4. Testing Remo te-no de Reco very and Fencing 4.3.5. Accessing Cluster To o ls fro m Remo te-no de 5. Linux Co ntainer (LXC) Walk-thro ugh 5.1. Step 1: Setup LXC Ho st 5.1.1. SElinux and Firewall Rules 5.1.2. Install Cluster So ftware o n Ho st 5.1.3. Co nfigure Co ro sync 5.1.4. Verify Cluster 5.2. Step 2: Setup LXC Enviro nment 5.2.1. Install Libvirt LXC so ftware 5.2.2. Generate Libvirt LXC do mains 5.2.3. Generate the Authkey 5.3. Step 3: Integrate LXC guests into Cluster.

Pacemaker 1.1-pcs Pacemaker Remote

5.3.1. Start Cluster 5.3.2. Integrate LXC Guests as remo te-no des 5.3.3. Starting Reso urces o n LXC Guests 5.3.4. Testing LXC Guest Failure A. Revisio n Histo ry Index List o f T able s 3.1. Me t adat a Opt io ns fo r co nfigurint KVM/LXC re so urce s as re m o t e -no de s

Pacemaker Remote

Chapter 1. Extending High Availability Cluster into Virtual Nodes

Table o f Co ntents 1.1. Overview 1.2. Terms 1.3. Virtual Machine Use Case 1.4. Linux Co ntainer Use Case 1.5. Expanding the Cluster Stack 1.5.1. Traditio nal HA Stack 1.5.2. Remo te-No de Enabled HA Stack

1.1. Overview
The recent additio n o f the pacemaker_remote service suppo rted by Pacemaker version 1.1.10 and greater allo ws no des no t running the cluster stack (pacemaker+co ro sync) to integrate into the cluster and have the cluster manage their reso urces just as if they were a real cluster no de. This means that pacemaker clusters are no w capable o f managing bo th launching virtual enviro nments (KVM/LXC) as well as launching the reso urces that live withing tho se virtual enviro nments witho ut requiring the virtual enviro nments to run pacemaker o r co ro sync.

1.2. Terms
cluster-node - A baremetal hardware no de running the High Availability stack (pacemaker + co ro sync) remote-node - A virtual guest no de running the pacemaker_remo te service. pacemaker_remote - A service daemo n capable o f perfo rming remo te applicatio n management within virtual guests (kvm and lxc) in bo th pacemaker cluster enviro nments and standalo ne (no n-cluster) enviro nments. This service is an enhanced versio n o f pacemakers lo cal reso urce manage daemo n (LRMD) that is capable o f managing and mo nito ring LSB, OCF, upstart, and systemd reso urces o n a guest remo tely. It also allo ws fo r mo st o f pacemakers cli to o ls (crm_mo n, crm_reso urce, crm_master, crm_attribute, ect..) to wo rk natively o n remo te-no des. LXC - A Linux Co ntainer defined by the libvirt-lxc Linux co ntainer driver. http://libvirt.o rg/drvlxc.html

1.3. Virtual Machine Use Case

The use o f pacemaker_remo te in virtual machines so lves a deplo yment scenario that has traditio nally been difficult to so lve. "I want a pacemaker cluster to manage virtual machine resources, but I also want pacemaker to be able to manage the resources that live within those virtual machines." In the past, users desiring this deplo yment had to make a decisio n. They wo uld either have to sacrifice the ability o f mo nito ring reso urces residing within virtual guests by running the cluster stack o n the baremetal no des, o r run ano ther cluster instance o n the virtual guests where they po tentially run into co ro sync scalability issues. There is a third scenario where the virtual guests run the cluster stack and jo in the same netwo rk as the baremetal no des, but that can quickly hit issues with scalability as well. With the pacemaker_remo te service we have a new o ptio n. The baremetal cluster-no des run the cluster stack (paceamaker+co ro sync). The virtual remo te-no des run the pacemaker_remo te service (nearly zero co nfiguratio n required o n the virtual machine side) The cluster stack o n the cluster-no des launch the virtual machines and immediately co nnect to the pacemaker_remo te service, allo wing the virtual machines to integrate into the cluster just as if they were a real cluster-no de. The key difference here between the virtual machine remo te-no des and the cluster-no des is that the remo te-no des are no t running the cluster stack. This means the remo te no des will never beco me the DC, and they do no t take place in quo rum. On the hand this also means that the remo te-no des are no t bo und to the scalability limits asso ciated with the cluster stack either. No 16 node corosync member limits to deal with. That isnt to say remo te-no des can scale indefinitely, but the expectatio n is that remo te-no des scale ho rizo ntally much further than cluster-no des. Other than the quo rum limitatio n, these remo te-no des behave just like cluster no des in respects to reso urce management. The

Pacemaker 1.1-pcs Pacemaker Remote

cluster is fully capable o f managing and mo nito ring reso urces o n each remo te-no de. Yo u can build co nstraints against remo te-no des, put them in standby, o r whatever else yo ud expect to be able to do with no rmal cluster-no des. They even sho w up in the crm_mo n o utput as yo u wo uld expect cluster-no des to . To so lidify the co ncept, an example cluster deplo yment integrating remo te-no des co uld lo o k like this. 16 cluster-no des running co ro sync+pacemaker stack. 64 pacemaker managed virtual machine reso urces running pacemaker_remo te co nfigured as remo te-no des. 64 pacemaker managed webserver and database reso urces co nfigured to run o n the 64 remo te-no des. With this deplo yment yo u wo uld have 64 webservers and databases running o n 64 virtual machines o n 16 hardware no des all o f which are managed and mo nito red by the same pacemaker deplo yment.

1.4. Linux Container Use Case

I want to isolate and limit the system resources (cpu, memory, filesystem) a cluster resource can consume without using virtual machines. Using pacemaker_remo te with Linux co ntainers (libvirt-lxc) o pens up so me interesting po ssibilities fo r iso lating reso urces in the cluster witho ut the use o f a hyperviso r. We no w have the ability to bo th define a co ntained enviro nment with cpu and memo ry utilizatio n limits and then assign reso urces to that co ntained enviro nment all managed fro m within pacemaker. The LXC Walk-thro ugh sectio n o f this do cument o utlines ho w pacemaker_remo te can be used to bring Linux co ntainers into the cluster as remo te-no des capable o f executing reso urces.

1.5. Expanding the Cluster Stack

1.5.1. Tradit ional HA St ack

1.5.2. Remot e-Node Enabled HA St ack

The stack gro ws o ne additio nal layer vertical so we can go further ho rizo ntal.

Pacemaker Remote

Pacemaker 1.1-pcs Pacemaker Remote

Chapter 2. Quick Example

Table o f Co ntents 2.1. Mile High View o f Co nfiguratio n Steps 2.2. What tho se steps just did 2.3. Accessing Cluster fro m Remo te-no de If yo u already kno w ho w to use pacemaker, yo ull likely be able to grasp this new co ncept o f remo te-no des by reading thro ugh this quick example witho ut having to so rt thro ugh all the detailed walk-thro ugh steps. Here are the key co nfiguratio n ingredients that make this po ssible using libvirt and KVM virtual guests. These steps strip everything do wn to the very basics.

2.1. Mile High View of Configuration Steps

Put an authkey with this path, /etc/pacemaker/authkey, on every cluster-node and virtual machine. This secures remo te co mmunicatio n and authenticatio n. Run this co mmand if yo u want to make a so mewhat rando m authkey.
dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4096 count=1

Install pacemaker_remote packages every virtual machine, enable pacemaker_remote on startup, and poke hole in firewall for tcp port 3121.
yum install pacemaker-remote resource-agents systemctl enable pacemaker_remote # If you just want to see this work, disable iptables and ip6tables on most distros. # You may have to put selinux in permissive mode as well for the time being. firewall-cmd --add-port 3121/tcp --permanent

Give each virtual machine a static network address and unique hostname Tell pacemaker to launch a virtual machine and that the virtual machine is a remote-node capable of running resources by using the "remote-node" meta-attribute. with pcs
# pcs resource create vm-guest1 VirtualDomain hypervisor="qemu:///system" config="vmguest1.xml" meta +remote-node=guest1+

raw xml
<primitive class="ocf" id="vm-guest1" provider="heartbeat" type="VirtualDomain"> <instance_attributes id="vm-guest-instance_attributes"> <nvpair id="vm-guest1-instance_attributes-hypervisor" name="hypervisor" value="qemu:///system"/> <nvpair id="vm-guest1-instance_attributes-config" name="config" value="guest1.xml"/> </instance_attributes> <operations> <op id="vm-guest1-interval-30s" interval="30s" name="monitor"/> </operations> <meta_attributes id="vm-guest1-meta_attributes"> <nvpair id="vm-guest1-meta_attributes-remote-node" name="remote-node" value="guest1"/> </meta_attributes> </primitive>

In the example abo ve the meta-attribute remote-node=guest1 tells pacemaker that this reso urce is a remo te-no de with the ho stname guest1 that is capable o f being integrated into the cluster. The cluster will attempt to co ntact the virtual machines pacemaker_remo te service at the ho stname guest1 after it launches.

2.2. What those steps just did

Tho se steps just to ld pacemaker to launch a virtual machine called vm-guest1 and integrate that virtual machine as a remo te-no de called guest1 . Example crm_mo n o utput after guest1 is integrated into cluster.

Pacemaker Remote

Last updated: Wed Mar 13 13:52:39 2013 Last change: Wed Mar 13 13:25:17 2013 via crmd on node1 Stack: corosync Current DC: node1 (24815808) - partition with quorum Version: 1.1.10 2 Nodes configured, unknown expected votes 2 Resources configured. Online: [ node1 guest1] vm-guest1 (ocf::heartbeat:VirtualDomain): Started node1

No w, yo u co uld place a reso urce, such as a webserver o n guest1.

# pcs resource create webserver apache params configfile=/etc/httpd/conf/httpd.conf op monitor interval=30s # pcs constraint webserver prefers guest1

No w the crm_mo n o utput wo uld sho w a webserver launched o n the guest1 remo te-no de.
Last updated: Wed Mar 13 13:52:39 2013 Last change: Wed Mar 13 13:25:17 2013 via crmd on node1 Stack: corosync Current DC: node1 (24815808) - partition with quorum Version: 1.1.10 2 Nodes configured, unknown expected votes 2 Resources configured. Online: [ node1 guest1] vm-guest1 webserver (ocf::heartbeat:VirtualDomain): Started node1 (ocf::heartbeat::apache): Started guest1

2.3. Accessing Cluster from Remote-node

It is wo rth no ting that after guest1 is integrated into the cluster, all the pacemaker cli to o ls immediately beco me available to the remo te no de. This means things like crm_mo n, crm_reso urce, and crm_attribute will wo rk natively o n the remo te-no de as lo ng as the co nnectio n between the remo te-no de and cluster-no de exists. This is particularly impo rtant fo r any master/slave reso urces executing o n the remo te-no de that need access to crm_master to set the no des transient attributes.

Pacemaker 1.1-pcs Pacemaker Remote

Chapter 3. Configuration Explained

Table o f Co ntents 3.1. Reso urce Optio ns 3.2. Ho st and Guest Authenticatio n 3.3. Pacemaker and pacemaker_remo te Optio ns The walk-thro ugh examples use so me o f these o ptio ns, but do nt explain exactly what they mean o r do . This sectio n is meant to be the go -to reso urce fo r all the o ptio ns available fo r co nfiguring remo te-no des.

3.1. Resource Options

When co nfiguring a virtual machine o r lxc reso urce to act as a remo te-no de, these are the metadata o ptio ns available to bo th enable the reso urce as a remo te-no de and define the co nnectio n parameters. T able 3.1. Me t adat a Opt io ns fo r co nfigurint KVM/LXC re so urce s as re m o t e -no de s Opt io n remote-node De fault <no ne> De script io n The name o f the remo te-no de this reso urce defines. This bo th enables the reso urce as a remo te-no de and defines the unique name used to identify the remo te-no de. If no o ther parameters are set, this value will also be assumed as the ho stname to co nnect to at po rt 3121. WARNING This value canno t o verlap with any reso urce o r no de IDs. Co nfigure a custo m po rt to use fo r the guest co nnectio n to pacemaker_remo te. The ip address o r ho stname to co nnect to if remo te-no des name is no t the ho stname o f the guest. Ho w lo ng befo re a pending guest co nnectio n will time o ut.

remote-port remote-addr

3121 remote-node value used as ho stname 60s

remoteconnecttimeout

3.2. Host and Guest Authentication

Authenticatio n and encryptio n o f the co nnectio n between cluster-no des (pacemaker) to remo te-no des (pacemaker_remo te) is achieved using TLS with PSK encryptio n/authenticatio n o n tcp port 3121. This means bo th the cluster-no de and remo te-no de must share the same private key. By default this key must be placed at "/etc/pacemaker/authkey" on both cluster-nodes and remote-nodes.

3.3. Pacemaker and pacemaker_remote Options

If yo u need to change the default po rt o r authkey lo catio n fo r either pacemaker o r pacemaker_remo te, there are enviro nment variables yo u can set that affect bo th o f tho se daemo ns. These enviro nment variables can be enabled by placing them in the /etc/sysco nfig/pacemaker file.
#==#==# Pacemaker Remote # Use a custom directory for finding the authkey. PCMK_authkey_location=/etc/pacemaker/authkey # # Specify a custom port for Pacemaker Remote connections PCMK_remote_port=3121

10

Pacemaker Remote

Chapter 4. KVM Walk-through

Table o f Co ntents 4.1. Step 1: Setup the Ho st 4.1.1. SElinux and Firewall 4.1.2. Install Cluster So ftware 4.1.3. Setup Co ro sync 4.1.4. Verify Cluster So ftware 4.1.5. Install Virtualizatio n So ftware 4.2. Step2: Create the KVM guest 4.2.1. Setup Guest Netwo rk 4.2.2. Setup Pacemaker Remo te 4.2.3. Verify Ho st Co nnectio n to Guest 4.3. Step3: Integrate KVM guest into Cluster. 4.3.1. Start the Cluster 4.3.2. Integrate KVM Guest as remo te-no de 4.3.3. Starting Reso urces o n KVM Guest 4.3.4. Testing Remo te-no de Reco very and Fencing 4.3.5. Accessing Cluster To o ls fro m Remo te-no de What this tutorial is: This tuto rial is an in-depth walk-thro ugh o f ho w to get pacemaker to manage a KVM guest instance and integrate that guest into the cluster as a remo te-no de. What this tutorial is not: This tuto rial is no t a realistic deplo yment scenario . The steps sho wn here are meant to get users familiar with the co ncept o f remo te-no des as quickly as po ssible.

4.1. Step 1: Setup the Host

This tuto rial was created using Fedo ra 18 o n the ho st and guest no des. Anything that is capable o f running libvirt and pacemaker v1.1.10 o r greater will do tho ugh. An installatio n guide fo r installing Fedo ra 18 can be fo und here, http://do cs.fedo rapro ject.o rg/en-US/Fedo ra/18/html/Installatio n_Guide/. Fedo ra 18 (o r similar distro ) ho st preparatio n steps.

4.1.1. SElinux and Firewall

In o rder to simply this tuto rial we will disable the selinux and the firewall o n the ho st. WARNING: These actio ns will o pen a significant security threat to machines expo sed to the o utside wo rld.
# # # # # # # # setenforce 0 sed -i.bak "s/SELINUX=enforcing/SELINUX=permissive/g" /etc/selinux/config systemctl disable iptables.service systemctl disable ip6tables.service rm '/etc/systemd/system/basic.target.wants/iptables.service' rm '/etc/systemd/system/basic.target.wants/ip6tables.service' systemctl stop iptables.service systemctl stop ip6tables.service

4.1.2. Inst all Clust er Soft ware

# yum install -y pacemaker corosync pcs resource-agents

4.1.3. Set up Corosync

Running the co mmand belo w will attempt to detect the netwo rk address co ro sync sho uld bind to .
# export corosync_addr=`ip addr | grep "inet " | tail -n 1 | awk '{print $4}' | sed s/255/0/g`

Display and verify that address is co rrect

Pacemaker 1.1-pcs Pacemaker Remote

11

# echo $corosync_addr

In many cases the address will be 192.168.1.0 if yo u are behind a standard ho me ro uter. No w co py o ver the example co ro sync.co nf. This co de will inject yo ur bindaddress and enable the vo te quo rum api which is required by pacemaker.
# cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf # sed -i.bak "s/.*\tbindnetaddr:.*/bindnetaddr:\ $corosync_addr/g" /etc/corosync/corosync.conf # cat << END >> /etc/corosync/corosync.conf quorum { provider: corosync_votequorum expected_votes: 2 } END

4.1.4. Verify Clust er Soft ware

Start the cluster
# pcs cluster start

Verify co ro sync membership

# pcs status corosync Membership information Nodeid Votes Name 1795270848 1 example-host (local)

Verify pacemaker status. At first the pcs cluster status o utput will lo o k like this.
# pcs status Last updated: Thu Mar 14 12:26:00 2013 Last change: Thu Mar 14 12:25:55 2013 via crmd on example-host Stack: corosync Current DC: Version: 1.1.10 1 Nodes configured, unknown expected votes 0 Resources configured.

After abo ut a minute yo u sho uld see yo ur ho st as a single no de in the cluster.

# pcs status Last updated: Thu Mar 14 12:28:23 2013 Last change: Thu Mar 14 12:25:55 2013 via crmd on example-host Stack: corosync Current DC: example-host (1795270848) - partition WITHOUT quorum Version: 1.1.8-9b13ea1 1 Nodes configured, unknown expected votes 0 Resources configured. Online: [ example-host ]

Go ahead and sto p the cluster fo r no w after verifying everything is in o rder.

# pcs cluster stop

4.1.5. Inst all Virt ualizat ion Soft ware

# yum install -y kvm libvirt qemu-system qemu-kvm bridge-utils virt-manager # systemctl enable libvirtd.service

rebo o t the ho st

4.2. Step2: Create the KVM guest

I am no t go ing to o utline the installatio n steps required to create a kvm guest. There are plenty o f tuto rials available elsewhere that do that. I reco mmend using a Fedo ra 18 o r greater distro as yo ur guest as that is what I am testing this

12

Pacemaker Remote

tuto rial with.

4.2.1. Set up Guest Net work

Run the co mmands belo w to set up a static ip address (192.168.122.10) and ho stname (guest1).
export remote_hostname=guest1 export remote_ip=192.168.122.10 export remote_gateway=192.168.122.1 yum remove -y NetworkManager rm -f /etc/hostname cat << END >> /etc/hostname $remote_hostname END hostname $remote_hostname cat << END >> /etc/sysconfig/network HOSTNAME=$remote_hostname GATEWAY=$remote_gateway END sed -i.bak "s/.*BOOTPROTO=.*/BOOTPROTO=none/g" /etc/sysconfig/network-scripts/ifcfg-eth0 cat << END >> /etc/sysconfig/network-scripts/ifcfg-eth0 IPADDR0=$remote_ip PREFIX0=24 GATEWAY0=$remote_gateway DNS1=$remote_gateway END systemctl systemctl systemctl systemctl restart network enable network.service enable sshd start sshd

echo "checking connectivity" ping www.google.com

To simplify the tuto rial well go ahead and disable selinux o n the guest. Well also need to po ke a ho le thro ugh the firewall o n po rt 3121 (the default po rt fo r pacemaker_remo te) so the ho st can co ntact the guest.
# setenforce 0 # sed -i.bak "s/SELINUX=enforcing/SELINUX=permissive/g" /etc/selinux/config # firewall-cmd --add-port 3121/tcp --permanent

If yo u still enco unter co nnectio n issues just disable iptables and ipv6tables o n the guest like we did o n the ho st to guarantee yo ull be able to co ntact the guest fro m the ho st. At this po int yo u sho uld be able to ssh into the guest fro m the ho st.

4.2.2. Set up Pacemaker Remot e

On the HOST machine run these co mmands to generate an authkey and co py it to the /etc/pacemaker fo lder o n bo th the ho st and guest.
# mkdir /etc/pacemaker # dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4096 count=1 # scp -r /etc/pacemaker root@192.168.122.10:/etc/

No w o n the GUEST install pacemaker-remo te package and enable the daemo n to run at startup. In the co mmands belo w yo u will no tice the pacemaker and pacemaker_remote packages are being installed. The pacemaker package is no t required. The o nly reaso n it is being installed fo r this tuto rial is because it co ntains the a Dummy reso urce agent we will be using later o n to test the remo te-no de.
# yum install -y pacemaker paceamaker-remote resource-agents # systemctl enable pacemaker_remote.service

No w start pacemaker_remo te o n the guest and verify the start was successful.

Pacemaker 1.1-pcs Pacemaker Remote

13

# systemctl start pacemaker_remote.service # systemctl status pacemaker_remote pacemaker_remote.service - Pacemaker Remote Service Loaded: loaded (/usr/lib/systemd/system/pacemaker_remote.service; enabled) Active: active (running) since Thu 2013-03-14 18:24:04 EDT; 2min 8s ago Main PID: 1233 (pacemaker_remot) CGroup: name=systemd:/system/pacemaker_remote.service 1233 /usr/sbin/pacemaker_remoted Mar 14 Mar 14 Mar 14 Starting 18:24:04 guest1 systemd[1]: Starting Pacemaker Remote Service... 18:24:04 guest1 systemd[1]: Started Pacemaker Remote Service. 18:24:04 guest1 pacemaker_remoted[1233]: notice: lrmd_init_remote_tls_server: a tls listener on port 3121.

4.2.3. Verify Host Connect ion t o Guest

Befo re mo ving fo rward its wo rth go ing ahead and verifying the ho st can co ntact the guest o n po rt 3121. Heres a trick yo u can use. Co nnect using telnet fro m the ho st. The co nnectio n will get destro yed, but ho w it is destro yed tells yo u whether it wo rked o r no t. First add guest1 to the ho st machines /etc/ho sts file if yo u havent already. This is required unless yo u have dns setup in a way where guest1s address can be disco vered.
# cat << END >> /etc/hosts 192.168.122.10 guest1 END

If running the telnet co mmand o n the ho st results in this o utput befo re disco nnecting, the co nnectio n wo rks.
# telnet guest1 3121 Trying 192.168.122.10... Connected to guest1. Escape character is '^]'. Connection closed by foreign host.

If yo u see this, the co nnectio n is no t wo rking.

# telnet guest1 3121 Trying 192.168.122.10... telnet: connect to address 192.168.122.10: No route to host

Once yo u can successfully co nnect to the guest fro m the ho st, shutdo wn the guest. Pacemaker will be managing the virtual machine fro m this po int fo rward.

4.3. Step3: Integrate KVM guest into Cluster.

No w the fun part, integrating the virtual machine yo uve just created into the cluster. It is incredibly simple.

4.3.1. St art t he Clust er

On the ho st, start pacemaker.
# pcs cluster start

Wait fo r the ho st to beco me the DC. The o utput o f pcs status sho uld lo o k similar to this after abo ut a minute.
Last updated: Thu Mar 14 16:41:22 2013 Last change: Thu Mar 14 16:41:08 2013 via crmd on example-host Stack: corosync Current DC: example-host (1795270848) - partition WITHOUT quorum Version: 1.1.10 1 Nodes configured, unknown expected votes 0 Resources configured. Online: [ example-host ]

No w enable the cluster to wo rk witho ut quo rum o r sto nith. This is required just fo r the sake o f getting this tuto rial to wo rk with a single cluster-no de.

14

Pacemaker Remote

# pcs property set stonith-enabled=false # pcs property set no-quorum-policy=ignore

4.3.2. Int egrat e KVM Guest as remot e-node

If yo u didnt already do this earlier in the verify ho st to guest co nnectio n sectio n, add the KVM guests ip to the ho sts /etc/ho sts file so we can co nnect by ho stname. The co mmand belo w will do that if yo u used the same ip address I used earlier.
# cat << END >> /etc/hosts 192.168.122.10 guest1 END

We will use the VirtualDomain reso urce agent fo r the management o f the virtual machine. This agent requires the virtual machines xml co nfig to be dumped to a file o n disk. To do this pick o ut the name o f the virtual machine yo u just created fro m the o utput o f this list.
# virsh list --all Id Name State ______________________________________________ guest1 shut off

In my case I named it guest1. Dump the xml to a file so mewhere o n the ho st using the fo llo wing co mmand.
# virsh dumpxml guest1 > /root/guest1.xml

No w just register the reso urce with pacemaker and yo ure set!
# pcs resource create vm-guest1 VirtualDomain hypervisor="qemu:///system" config="/root/guest1.xml" meta remote-node=guest1

Once the vm-guest1 reso urce is started yo u will see guest1 appear in the pcs status o utput as a no de. The final pcs status o utput sho uld lo o k so mething like this.
Last updated: Fri Mar 15 09:30:30 2013 Last change: Thu Mar 14 17:21:35 2013 via cibadmin on example-host Stack: corosync Current DC: example-host (1795270848) - partition WITHOUT quorum Version: 1.1.10 2 Nodes configured, unknown expected votes 2 Resources configured. Online: [ example-host guest1 ] Full list of resources: vm-guest1 (ocf::heartbeat:VirtualDomain): Started example-host

4.3.3. St art ing Resources on KVM Guest

The co mmands belo w demo nstrate ho w reso urces can be executed o n bo th the remo te-no de and the cluster-no de. Create a few Dummy reso urces. Dummy reso urces are real reso urce agents used just fo r testing purpo ses. They actually execute o n the ho st they are assigned to just like an apache server o r database wo uld, except their executio n just means a file was created. When the reso urce is sto pped, that the file it created is remo ved.
# # # # # pcs pcs pcs pcs pcs resource resource resource resource resource create create create create create FAKE1 FAKE2 FAKE3 FAKE4 FAKE5 ocf:pacemaker:Dummy ocf:pacemaker:Dummy ocf:pacemaker:Dummy ocf:pacemaker:Dummy ocf:pacemaker:Dummy

No w check yo ur pcs status o utput. In the reso urce sectio n yo u sho uld see so mething like the fo llo wing, where so me o f the reso urces go t started o n the cluster-no de, and so me started o n the remo te-no de.

Pacemaker 1.1-pcs Pacemaker Remote

15

Full list of resources: vm-guest1 (ocf::heartbeat:VirtualDomain): Started example-host FAKE1 (ocf::pacemaker:Dummy): Started guest1 FAKE2 (ocf::pacemaker:Dummy): Started guest1 FAKE3 (ocf::pacemaker:Dummy): Started example-host FAKE4 (ocf::pacemaker:Dummy): Started guest1 FAKE5 (ocf::pacemaker:Dummy): Started example-host

The remo te-no de, guest1 , reacts just like any o ther no de in the cluster. Fo r example, pick o ut a reso urce that is running o n yo ur cluster-no de. Fo r my purpo ses I am picking FAKE3 fro m the o utput abo ve. We can fo rce FAKE3 to run o n guest1 in the exact same way we wo uld any o ther no de.
# pcs constraint FAKE3 prefers guest1

No w lo o king at the bo tto m o f the pcs status o utput yo ull see FAKE3 is o n guest1 .
Full list of resources: vm-guest1 (ocf::heartbeat:VirtualDomain): Started example-host FAKE1 (ocf::pacemaker:Dummy): Started guest1 FAKE2 (ocf::pacemaker:Dummy): Started guest1 FAKE3 (ocf::pacemaker:Dummy): Started guest1 FAKE4 (ocf::pacemaker:Dummy): Started example-host FAKE5 (ocf::pacemaker:Dummy): Started example-host

4.3.4. Test ing Remot e-node Recovery and Fencing

Pacemakers po licy engine is smart eno ugh to kno w fencing remo te-no des asso ciated with a virtual machine means shutting o ff/rebo o ting the virtual machine. No special co nfiguratio n is necessary to make this happen. If yo u are interested in testing this functio nality o ut, trying sto pping the guests pacemaker_remo te daemo n. This wo uld be equivalent o f abruptly terminating a cluster-no des co ro sync membership witho ut pro perly shutting it do wn. ssh into the guest and run this co mmand.
# kill -9 `pidof pacemaker_remoted`

After a few seco nds o r so yo ull see this in yo ur pcs status o utput. The guest1 no de will be sho w as o ffline as it is being reco vered.
Last updated: Fri Mar 15 11:00:31 2013 Last change: Fri Mar 15 09:54:16 2013 via cibadmin on example-host Stack: corosync Current DC: example-host (1795270848) - partition WITHOUT quorum Version: 1.1.10 2 Nodes configured, unknown expected votes 7 Resources configured. Online: [ example-host ] OFFLINE: [ guest1 ] Full list of resources: vm-guest1 (ocf::heartbeat:VirtualDomain): Started example-host FAKE1 (ocf::pacemaker:Dummy): Stopped FAKE2 (ocf::pacemaker:Dummy): Stopped FAKE3 (ocf::pacemaker:Dummy): Stopped FAKE4 (ocf::pacemaker:Dummy): Started example-host FAKE5 (ocf::pacemaker:Dummy): Started example-host Failed actions: guest1_monitor_30000 (node=example-host, call=3, rc=7, status=complete): not running

Once reco very o f the guest is co mplete, yo ull see it auto matically get re-integrated into the cluster. The final pcs status o utput sho uld lo o k so mething like this.

16

Pacemaker Remote

Last updated: Fri Mar 15 11:03:17 2013 Last change: Fri Mar 15 09:54:16 2013 via cibadmin on example-host Stack: corosync Current DC: example-host (1795270848) - partition WITHOUT quorum Version: 1.1.10 2 Nodes configured, unknown expected votes 7 Resources configured. Online: [ example-host guest1 ] Full list of resources: vm-guest1 (ocf::heartbeat:VirtualDomain): Started example-host FAKE1 (ocf::pacemaker:Dummy): Started guest1 FAKE2 (ocf::pacemaker:Dummy): Started guest1 FAKE3 (ocf::pacemaker:Dummy): Started guest1 FAKE4 (ocf::pacemaker:Dummy): Started example-host FAKE5 (ocf::pacemaker:Dummy): Started example-host Failed actions: guest1_monitor_30000 (node=example-host, call=3, rc=7, status=complete): not running

4.3.5. Accessing Clust er Tools from Remot e-node

Besides just allo wing the cluster to manage reso urces o n a remo te-no de, pacemaker_remo te has o ne o ther trick. The pacemaker_remote daemon allows nearly all the pacemaker tools (crm_resource, crm_mon, crm_attribute, crm_master) to work on remote nodes natively. Try it, run crm_mon o r pcs status o n the guest after pacemaker has integrated the remo te-no de into the cluster. These to o ls just wo rk. These means reso urce agents such as master/slave reso urces which need access to to o ls like crm_master wo rk seamlessly o n the remo te-no des.

Pacemaker 1.1-pcs Pacemaker Remote

17

Chapter 5. Linux Container (LXC) Walk-through

Table o f Co ntents 5.1. Step 1: Setup LXC Ho st 5.1.1. SElinux and Firewall Rules 5.1.2. Install Cluster So ftware o n Ho st 5.1.3. Co nfigure Co ro sync 5.1.4. Verify Cluster 5.2. Step 2: Setup LXC Enviro nment 5.2.1. Install Libvirt LXC so ftware 5.2.2. Generate Libvirt LXC do mains 5.2.3. Generate the Authkey 5.3. Step 3: Integrate LXC guests into Cluster. 5.3.1. Start Cluster 5.3.2. Integrate LXC Guests as remo te-no des 5.3.3. Starting Reso urces o n LXC Guests 5.3.4. Testing LXC Guest Failure What this tutorial is: This tuto rial demo nstrates ho w pacemaker_remo te can be used with Linux co ntainers (managed by libvirt-lxc) to run cluster reso urces in an iso lated enviro nment. What this tutorial is not: This tuto rial is no t a realistic deplo yment scenario . The steps sho wn here are meant to intro duce users to the co ncept o f managing Linux co ntainer enviro nments with Pacemaker.

5.1. Step 1: Setup LXC Host

This tuto rial was tested with Fedo ra 18. Anything that is capable o f running libvirt and pacemaker v1.1.10 o r greater will do tho ugh. An installatio n guide fo r installing Fedo ra 18 can be fo und here, http://do cs.fedo rapro ject.o rg/enUS/Fedo ra/18/html/Installatio n_Guide/. Fedo ra 18 (o r similar distro ) ho st preparatio n steps.

5.1.1. SElinux and Firewall Rules

In o rder to simply this tuto rial we will disable the selinux and the firewall o n the ho st. WARNING: These actio ns po se a significant security issues to machines expo sed to the o utside wo rld. Basically, just do nt do this o n yo ur pro ductio n system.
# setenforce 0 # sed -i.bak "s/SELINUX=enforcing/SELINUX=permissive/g" /etc/selinux/config # firewall-cmd --add-port 3121/tcp --permanent # # # # # # systemctl disable iptables.service systemctl disable ip6tables.service rm '/etc/systemd/system/basic.target.wants/iptables.service' rm '/etc/systemd/system/basic.target.wants/ip6tables.service' systemctl stop iptables.service systemctl stop ip6tables.service

5.1.2. Inst all Clust er Soft ware on Host

# yum install -y pacemaker pacemaker-remote corosync pcs resource-agents

5.1.3. Configure Corosync

Running the co mmand belo w will attempt to detect the netwo rk address co ro sync sho uld bind to .
# export corosync_addr=`ip addr | grep "inet " | tail -n 1 | awk '{print $4}' | sed s/255/0/g`

Display and verify the address is co rrect

18

Pacemaker Remote

# echo $corosync_addr

In mo st cases the address will be 192.168.1.0 if yo u are behind a standard ho me ro uter. No w co py o ver the example co ro sync.co nf. This co de will inject yo ur bindaddress and enable the vo te quo rum api which is required by pacemaker.
# cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf # sed -i.bak "s/.*\tbindnetaddr:.*/bindnetaddr:\ $corosync_addr/g" /etc/corosync/corosync.conf # cat << END >> /etc/corosync/corosync.conf quorum { provider: corosync_votequorum expected_votes: 2 } END

5.1.4. Verify Clust er

Start the cluster
# pcs cluster start

Verify co ro sync membership

# pcs status corosync Membership information Nodeid Votes Name 1795270848 1 example-host (local)

Verify pacemaker status. At first the pcs cluster status o utput will lo o k like this.
# pcs status Last updated: Thu Mar 14 12:26:00 2013 Last change: Thu Mar 14 12:25:55 2013 via crmd on example-host Stack: corosync Current DC: Version: 1.1.10 1 Nodes configured, unknown expected votes 0 Resources configured.

After abo ut a minute yo u sho uld see yo ur ho st as a single no de in the cluster.

# pcs status Last updated: Thu Mar 14 12:28:23 2013 Last change: Thu Mar 14 12:25:55 2013 via crmd on example-host Stack: corosync Current DC: example-host (1795270848) - partition WITHOUT quorum Version: 1.1.8-9b13ea1 1 Nodes configured, unknown expected votes 0 Resources configured. Online: [ example-host ]

Go ahead and sto p the cluster fo r no w after verifying everything is in o rder.

# pcs cluster stop

5.2. Step 2: Setup LXC Environment

5.2.1. Inst all Libvirt LXC soft ware
# yum install -y libvirt libvirt-daemon-lxc wget # systemctl enable libvirtd

At this po int, restart the ho st.

Pacemaker 1.1-pcs Pacemaker Remote

19

5.2.2. Generat e Libvirt LXC domains

Ive attempted to simply this tuto rial by creating a script to auto generate the libvirt-lxc xml do main definitio ns. Do wnlo ad the script to whatever directo ry yo u want the co ntainers to live in. In this example I am using the /ro o t/lxc/ directo ry.
# # # # mkdir /root/lxc/ cd /root/lxc/ wget https://raw.github.com/davidvossel/pcmk-lxc-autogen/master/lxc-autogen chmod 755 lxc-autogen

No w execute the script.

# ./lxc-autogen

After executing the script yo u will see a bunch o f directo ries and xml files are generated. Tho se xml files are the libvirtlxc do main definitio ns, and the directo ries are used as so me special mo unt po ints fo r each co ntainer. If yo u o pen up o ne o f the xml files yo ull be able to see ho w the cpu, memo ry, and filesystem reso urces fo r the co ntainer are defined. Yo u can use the libvirt-lxc drivers do cumentatio n fo und here, http://libvirt.o rg/drvlxc.html, as a reference to help understand all the parts o f the xml file. The lxc-auto gen script is no t co mplicated and is wo rth explo ring in o rder to grasp ho w the enviro nment is generated. It is wo rth no ting that this enviro nment is dependent o n use o f libvirts default netwo rk interface. Verify the co mmands belo w lo o k the same as yo ur enviro nment. The default netwo rk address 192.168.122.1 sho uld have been generated by auto matically when yo u installed the virtualizatio n so ftware.
# virsh net-list Name State Autostart Persistent ________________________________________________________ default active yes yes # virsh net-dumpxml default | grep -e "ip address=" <ip address='192.168.122.1' netmask='255.255.255.0'>

5.2.3. Generat e t he Aut hkey

Generate the authkey used to secure co nnectio ns between the ho st and the lxc guest pacemaker_remo te instances. This is so rt o f a funny case because the lxc guests and the ho st will share the same key file in the /etc/pacemaker/ directo ry. If in a different deplo yment where the lxc guests do no t share the ho sts /etc/pacemaker directo ry, this key will have to be co pied into each lxc guest.
# dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4096 count=1

5.3. Step 3: Integrate LXC guests into Cluster.

5.3.1. St art Clust er
On the ho st, start pacemaker.
# pcs cluster start

Wait fo r the ho st to beco me the DC. The o utput o f pcs status sho uld lo o k similar to this after abo ut a minute.
Last updated: Thu Mar 14 16:41:22 2013 Last change: Thu Mar 14 16:41:08 2013 via crmd on example-host Stack: corosync Current DC: example-host (1795270848) - partition WITHOUT quorum Version: 1.1.10 1 Nodes configured, unknown expected votes 0 Resources configured. Online: [ example-host ]

No w enable the cluster to wo rk witho ut quo rum o r sto nith. This is required just fo r the sake o f getting this tuto rial to wo rk with a single cluster-no de.
# pcs property set stonith-enabled=false # pcs property set no-quorum-policy=ignore

20

Pacemaker Remote

5.3.2. Int egrat e LXC Guest s as remot e-nodes

If yo u ran the lxc-autogen script with default parameters, 3 lxc do main definitio ns were created as .xml files. If yo u used the same directo ry I used fo r the lxc enviro nment, the co nfig files will be lo cated in /ro o t/lxc. Replace the config parameters in the fo llo wing pcs co mmands if yo urs sho uld be different. The pcs co mmands belo w each co nfigure a lxc guest as a remo te-no de in pacemaker. Behind the scenes each lxc guest is launching an instance o f pacemaker_remo te allo wing pacemaker to integrate the lxc guests as remo te-no des. The meta-attribute remote-node=<node-name> used in each co mmand is what tells pacemaker that the lxc guest is bo th a reso urce and a remo te-no de capable o f running reso urces. In this case, the remote-node attribute also indicates to pacemaker that it can co ntact each lxcs pacemaker_remo te service by using the remo te-no de name as the ho stname. If yo u lo o k in the /etc/ho sts/ file yo u will see entries fo r each lxc guest. These entries were auto -generated earlier by the lxc-autogen script.
# pcs resource create container1 config="/root/lxc/lxc1.xml" meta # pcs resource create container2 config="/root/lxc/lxc2.xml" meta # pcs resource create container3 config="/root/lxc/lxc3.xml" meta VirtualDomain force_stop="true" hypervisor="lxc:///" remote-node=lxc1 VirtualDomain force_stop="true" hypervisor="lxc:///" remote-node=lxc2 VirtualDomain force_stop="true" hypervisor="lxc:///" remote-node=lxc3

After creating the co ntainer reso urces yo u pcs status sho uld lo o k like this.
Last updated: Mon Mar 18 17:15:46 2013 Last change: Mon Mar 18 17:15:26 2013 via cibadmin on guest1 Stack: corosync Current DC: example-host (175810752) - partition WITHOUT quorum Version: 1.1.10 4 Nodes configured, unknown expected votes 6 Resources configured. Online: [ example-host lxc1 lxc2 lxc3 ] Full list of resources: container3 container1 container2 (ocf::heartbeat:VirtualDomain): Started example-host (ocf::heartbeat:VirtualDomain): Started example-host (ocf::heartbeat:VirtualDomain): Started example-host

5.3.3. St art ing Resources on LXC Guest s

No w that the lxc guests are integrated into the cluster, lets generate so me Dummy reso urces to run o n them. Dummy reso urces are real reso urce agents used just fo r testing purpo ses. They actually execute o n the no de they are assigned to just like an apache server o r database wo uld, except their executio n just means a file was created. When the reso urce is sto pped, that the file it created is remo ved.
# # # # # pcs pcs pcs pcs pcs resource resource resource resource resource create create create create create FAKE1 FAKE2 FAKE3 FAKE4 FAKE5 ocf:pacemaker:Dummy ocf:pacemaker:Dummy ocf:pacemaker:Dummy ocf:pacemaker:Dummy ocf:pacemaker:Dummy

After creating the Dummy reso urces yo u will see that the reso urce go t distributed amo ng all the no des. The pcs status o utput sho uld lo o k similar to this.

Pacemaker 1.1-pcs Pacemaker Remote

21

Last updated: Mon Mar 18 17:31:54 2013 Last change: Mon Mar 18 17:31:05 2013 via cibadmin on example-host Stack: corosync Current DC: example=host (175810752) - partition WITHOUT quorum Version: 1.1.10 4 Nodes configured, unknown expected votes 11 Resources configured. Online: [ example-host lxc1 lxc2 lxc3 ] Full list of resources: container3 (ocf::heartbeat:VirtualDomain): Started example-host container1 (ocf::heartbeat:VirtualDomain): Started example-host container2 (ocf::heartbeat:VirtualDomain): Started example-host FAKE1 (ocf::pacemaker:Dummy): Started lxc1 FAKE2 (ocf::pacemaker:Dummy): Started lxc2 FAKE3 (ocf::pacemaker:Dummy): Started lxc3 FAKE4 (ocf::pacemaker:Dummy): Started lxc1 FAKE5 (ocf::pacemaker:Dummy): Started lxc2

To witness that Dummy agents are running within the lxc guests bro wse o ne o f the lxc do mains filesystem fo lders. Each lxc guest has a custo m mo unt po int fo r the '/var/run/'directo ry, which is the lo catio n the Dummy reso urces write their state files to .
# ls lxc1-filesystem/var/run/ Dummy-FAKE4.state Dummy-FAKE.state

If yo u are curio us, take a lo o k at lxc1.xml to see ho w the filesystem is mo unted.

5.3.4. Test ing LXC Guest Failure

Yo u will be able to see each pacemaker_remo ted pro cess running in each lxc guest fro m the ho st machine.
# ps -A | grep 9142 pts/2 10148 pts/4 10942 pts/6 -e pacemaker_remote* 00:00:00 pacemaker_remot 00:00:00 pacemaker_remot 00:00:00 pacemaker_remot

In o rder to see ho w the cluster reacts to a failed lxc guest. Try killing o ne o f the pacemaker_remo te instances.
# kill -9 9142

After a few mo ments the lxc guest that was running that instance o f pacemaker_remo te will be reco vered alo ng with all the reso urces running within that co ntainer.

Revision History
Re visio n 1 Impo rt fro m Pages.app T ue Mar 19 2013 David Vo sse l

Index