Setting up a High Availability NAS/SAN using Openfiler

Installing and Configuring Openfiler with DRBD and Heartbeat

Table of contents
Table of contents
l Introduction
l Requirements
Hardware
Software
l Network
l Hostname setup
l SSH Shared keys
l Configure DRBD
l Configure LVM Partition
l Configure Heartbeat
l Openfiler data configuration
Heartbeat Cluster configuration
Samba and NFS Support
iSCSI Support
FTP Support
l Configure Volume Group
l Starting Heartbeat and first-time configuration
l Note:
l Trouble shooting
l Software updates
l Credits



Introduction

This HowTo? focuses on setting up the OpenSource NAS/SAN Openfiler as a High Availability storage. There currently does not appear to be a lot of documentation
available on setting it up in version 2.3, so we thought we would document it here.

We will follow the version 2.2 HowTo? from http://wiki.hyber.dk/doku.php with a few modifications and comments on the setup process.

Requirements

Hardware

l 2 x boxes that meet the minimum spec of Openfiler's hardware specifications.
l 2 x ethernet interfaces in each box
l Openfiler 2.3 installation media.
l Both boxes should have the same size drives in each to avoid any replication inconsistencies.

Software
l Install Openfiler 2.3 on both boxes utilising a disk setup such as the following:
3gig /
512meg swap
300meg partition for cluster_metadata DRBD
Data partition configured as an unmounted LVM (this will be DRBD vg0_drbd)


Network

As you are using two network interfaces on each box, you need to select a range of IP addresses you want to use. The first interface is used to talk to your network... this
is what your computers will see when they talk to your NAS/SAN.

The second interface is used for the DRBD data replication (dedicated interface) between the two filers. Ideally this interface has at least a gigabit of bandwidth, but it does
work on 100mbit (depending on how much data you are putting through your NAS/SAN).

5
6
3 4
[root@filer1 ~]# fdisk -l

Disk /dev/hdb: 40.0 GB, 40020664320 bytes
255 heads, 63 sectors/track, 4865 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hdb1 * 1 382 3068383+ 83 Linux
/dev/hdb2 383 4206 30716280 8e Linux LVM
/dev/hdb3 4207 4239 265072+ 82 Linux swap / S
/dev/hdb4 4240 4276 297202+ 83 Linux
http://www.the-mesh.org/tiki-print.php?page=OpenFilerHaSetup Page 1 / 9
One thing to note in High Availability is you will need three IP addresses allocated to your cluster. The first IP address is the address that all of the computers on your
network will talk to. Depending on which of your filers is Primary at the time, it will be configured to use this IP address. In an event of a fail over, the Secondary filer will
become Primary and take over this IP address.

The other two IP addresses are used purely for administration, so you can directly communicate with each filer.

So, as an example, we will use a 192.168.8.0/24 subnet on our first interfaces, and 192.168.9.0/29 on the second interface.

So we can allocate the following:
l filer1
LAN Interface (eth0) 192.168.8.11
Replication Interface (eth1) 192.168.9.11

l filer2
LAN Interface (eth0) 192.168.8.12
Replication Interface (eth1) 192.168.9.12

l HA NAS/SAN Address (eth0) 192.168.8.10

Note: The Heartbeat service controls the HA Address, so DO NOT declare this address in your network config scripts... it will be declared when we configure Heartbeat.

Hostname setup

We want both filers to be able to recognise each other. As Openfiler works off it's FQDN (Fully Qualified Domain Name), we will use that.

On filer1:
root@filer1 ~# hostname --fqdn
filer1


Modify our /etc/hosts (on filer1):


On filer2:
root@filer2 ~# hostname --fqdn
filer2


Modify our /etc/hosts (on filer2):


SSH Shared keys

We wan the two boxes to talk to each other without having to use a password. So, we will use SSH shared keys as mentioned in the Openfiler 2.2 Howto:

On filer1:
root@filer1 ~# ssh-keygen -t dsa

Hit enter at the prompts (we don't want to set a password on the key).

On filer2:
root@filer2 ~# ssh-keygen -t dsa

Hit enter at the prompts (we don't want to set a password on the key).

The above command will generate a file called "id_dsa.pub" in ~/.ssh/, which is the public key we need to copy over to our other filer:

root@filer1 ~# scp ~/.ssh/id_dsa.pub root@filer2:~/.ssh/authorized_keys2

root@filer2 ~# scp ~/.ssh/id_dsa.pub root@filer1:~/.ssh/authorized_keys2


Configure DRBD

DRBD is what we will use to keep the data between the two filers consistent. DRBD allows us to run a RAID-1 (mirror) over the network, so if one host goes down, the
other can come up in it's place without having inconsistent data (that's the theory anyway).

On filer1:
root@filer1 ~# mv /etc/drbd.conf /etc/drbd.conf.org


# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 filer1 localhost.localdomain localhost
192.168.8.12 filer2
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 filer2 localhost.localdomain localhost
192.168.8.11 filer1
http://www.the-mesh.org/tiki-print.php?page=OpenFilerHaSetup Page 2 / 9

Then we make our drbd.conf (version 8) like the following:
global {
# minor-count 64;
# dialog-refresh 5; # 5 seconds
# disable-ip-verification;
usage-count ask;
}

common {
syncer { rate 100M; }
}

resource cluster_metadata {

protocol C;

handlers {
pri-on-incon-degr "echo O > /proc/sysrq-trigger ; halt -f";
pri-lost-after-sb "echo O > /proc/sysrq-trigger ; halt -f";
local-io-error "echo O > /proc/sysrq-trigger ; halt -f";
# outdate-peer "/usr/sbin/drbd-peer-outdater";
}

startup {
# wfc-timeout 0;
degr-wfc-timeout 120; # 2 minutes.
}

disk {
on-io-error detach;

}

net {
after-sb-0pri disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
rr-conflict disconnect;
}

syncer {
# rate 10M;
# after "r2";
al-extents 257;
}

on filer1 {
device /dev/drbd0;
disk /dev/sda3;
address 192.168.9.11:7788;
meta-disk internal;
}

on filer2 {
device /dev/drbd0;
disk /dev/sda3;
address 192.168.9.12:7788;
meta-disk internal;
}
}

resource vg0_drbd {

protocol C;
startup {
wfc-timeout 0; ## Infinite!
degr-wfc-timeout 120; ## 2 minutes.
}

disk {
on-io-error detach;
}

net {
# timeout 60;
# connect-int 10;
# ping-int 10;
# max-buffers 2048;
# max-epoch-size 2048;
}

syncer {
after "cluster_metadata";
}

on filer1 {
device /dev/drbd1;
disk /dev/sda5;
address 192.168.9.11:7789;
meta-disk internal;
}
http://www.the-mesh.org/tiki-print.php?page=OpenFilerHaSetup Page 3 / 9


Take note of the "disk" variable, as these partitions may be different on your filers. Basically just need to make sure the 300meg partitions are used for cluster_metadata
and the LVM partition is used for our vg0_drbd.

We get to use /dev/drbd1 as a PV (Physical Volume) for our LVM, which may not sound that exiting if you're not used to using LVMs, but it allows us to have a dynamic
volume control of our data volumes, on top of a network RAID1 device (/dev/drbd1).

l More information on LVM here.

Both hosts need the SAME drbd.conf, so we overwrite filer2's config with our modified filer1 configuration:
root@filer1 ~# scp /etc/drbd.conf root@filer2:/etc/drbd.conf


Initialise metadata on /dev/drbd0 (cluster_metadata) and /dev/drbd1 (vg0_drbd) on both filers:
root@filer1 ~# drbdadm create-md cluster_metadata

root@filer1 ~# drbdadm create-md vg0_drbd

root@filer2 ~# drbdadm create-md cluster_metadata

root@filer2 ~# drbdadm create-md vg0_drbd


Note: if you get errors about needing to zero out the file system you can use the following command:
root@filer1 ~# dd if=/dev/zero of=/dev/hdb2
Please be careful with this command and make sure its on the correct drive


We can now start DRBD on both hosts:
root@filer1 ~# service drbd start

root@filer2 ~# service drbd start


If all goes well, they should connect and you should be able to run "service drbd status" and get a result similar to the following:
root@filer1 /# service drbd status
drbd driver loaded OK; device status:
version: 8.0.12 (api:86/proto:86)
GIT-hash: 5c9f89594553e32adb87d9638dce591782f947e3 build by phil@mescal, 2008-04-24 13:29:44
m:res cs st ds p mounted fstype
0:cluster_metadata Connected Secondary/Secondary Inconsistent/Inconsistent C
1:vg0_drbd Connected Secondary/Secondary Inconsistent/Inconsistent C


So, once both drbd resources are connected and both filers are in Secondary state (as above), we need to set a Primary filer... which we will make filer1:
root@filer1 ~# drbdsetup /dev/drbd0 primary -o
root@filer1 ~# drbdsetup /dev/drbd1 primary -o


This should give you a status result of something like the following:
root@filer1 /# service drbd status
drbd driver loaded OK; device status:
version: 8.0.12 (api:86/proto:86)
GIT-hash: 5c9f89594553e32adb87d9638dce591782f947e3 build by phil@mescal, 2008-04-24 13:29:44
m:res cs st ds p mounted fstype
... sync'ed: 17.9% (247232/297152)K
0:cluster_metadata SyncSource? Primary/Secondary UpToDate/Inconsistent C
1:vg0_drbd PausedSyncS? Primary/Secondary UpToDate/Inconsistent C


Enable DRBD to startup at boot:
root@filer1 ~# chkconfig --level 2345 drbd on

root@filer2 ~# chkconfig --level 2345 drbd on


We now create our cluster_metadata filesystem. We use this 300meg partition to keep all of the Openfiler configuration data and the data for the services we want to be
available in HA (eg. NFS, iSCSI, SMB).

root@filer1 ~# mkfs.ext3 /dev/drbd0


We don't add this partition to an /etc/fstab, as like our HA Interface, this is managed by Heartbeat... we will configure this shortly.


on filer2 {
device /dev/drbd1;
disk /dev/sda5;
address 192.168.9.12:7789;
meta-disk internal;
}
}
http://www.the-mesh.org/tiki-print.php?page=OpenFilerHaSetup Page 4 / 9
Configure LVM Partition

As mentioned before, we create our /dev/drbd1 as a PV (Physical Volume) for our data volume group, which we then use to create Logical Volumes for our data.

First we need to edit /etc/lvm/lvm.conf and modify the filter line:
From:

To:


Note: Change /dev/sda5 to reflect the partition of your LVM, as it may be different. Also remember to apply these changes on both filer1 and filer2.

My understanding of this is so when the host scans for LVM partitions, it excludes our LVM partition, as this seems to cause problems with drbd consistency when this filter
is not enabled.

Create our LVM Physical Volume (we only do this on our Primary filer, as it will replicate to the Second filer via drbd):
root@filer1 /# pvcreate /dev/drbd1
Physical volume "/dev/drbd1" successfully created


Configure Heartbeat

As mentioned before, Heartbeat controls our fail_over between hosts. Basically the two filers run the Heartbeat service, that sends out a heartbeat pulse on our secondary
interface (eth1). If one of the filers dies, then Heartbeat detects this and roles over the surviving filer to Primary (if it wasn't already) using startup scripts available
in /etc/ha.d/resources.d.

We need to make modifications to /etc/ha.d/ha.cf and /etc/ha.d/authkeys. Make sure you make these changes on both filers.

In /etc/ha.d/authkeys, add:
auth 2
2 crc


You may have to actually create the /etc/ha.d/authkeys file, as it does not appear to exist in Openfiler 2.3.

We also need restrict permissions to authkeys to just "root":
root@filer1 ~# chmod 600 /etc/ha.d/authkeys

root@filer2 ~# chmod 600 /etc/ha.d/authkeys


Now we create a /etc/ha.d/ha.cf on both filers (needs to be identical on both, just like /etc/drbd.conf.

debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
bcast eth1
keepalive 5
warntime 10
deadtime 120
initdead 120
udpport 694
auto_failback off
node filer1
node filer2


Enable Heartbeat to startup at boot:
root@filer1 ~# chkconfig --level 2345 heartbeat on

root@filer2 ~# chkconfig --level 2345 heartbeat on


Openfiler data configuration

As mentioned above, we have created a 300meg partition to keep our configuration and HA services available when we do a fail_over. To get this working, we copy our
services and Openfiler config data over to our new partition - symbolically linking it back to it's original location. If this does not make much sense now, it should be a little
clearer when we start testing the fail_over of services.

filer1:

root@filer1 ~# mkdir /cluster_metadata
root@filer1 ~# mount /dev/drbd0 /cluster_metadata
root@filer1 ~# mv /opt/openfiler/ /opt/openfiler.local
root@filer1 ~# mkdir /cluster_metadata/opt
root@filer1 ~# cp -a /opt/openfiler.local /cluster_metadata/opt/openfiler
root@filer1 ~# ln -s /cluster_metadata/opt/openfiler /opt/openfiler
root@filer1 ~# rm /cluster_metadata/opt/openfiler/sbin/openfiler
filter = [ "a/.*/" ]
filter = [ "r|/dev/sda5|" ]
http://www.the-mesh.org/tiki-print.php?page=OpenFilerHaSetup Page 5 / 9
root@filer1 ~# ln -s /usr/sbin/httpd /cluster_metadata/opt/openfiler/sbin/openfiler
root@filer1 ~# rm /cluster_metadata/opt/openfiler/etc/rsync.xml
root@filer1 ~# ln -s /opt/openfiler.local/etc/rsync.xml /cluster_metadata/opt/openfiler/etc/


We then edit our /opt/openfiler.local/etc/rsync.xml file:
<?xml version="1.0" ?>
<rsync>
<remote hostname="192.168.9.12"/> ## IP address of peer filer.
<item path="/etc/ha.d/haresources"/>
<item path="/etc/ha.d/ha.cf"/>
<item path="/etc/ldap.conf"/>
<item path="/etc/openldap/ldap.conf"/>
<item path="/etc/ldap.secret"/>
<item path="/etc/nsswitch.conf"/>
<item path="/etc/krb5.conf"/>
</rsync>


root@filer1 ~# mkdir -p /cluster_metadata/etc/httpd/conf.d


filer2:

root@filer2 ~# mkdir /cluster_metadata
root@filer2 ~# mv /opt/openfiler/ /opt/openfiler.local
root@filer2 ~# ln -s /cluster_metadata/opt/openfiler /opt/openfiler

Change the /opt/openfiler.local/etc/rsync.xml to reflect below:
<?xml version="1.0" ?>
<rsync>
<remote hostname="192.168.9.11"/> ## IP address of peer filer.
<item path="/etc/ha.d/haresources"/>
<item path="/etc/ha.d/ha.cf"/>
<item path="/etc/ldap.conf"/>
<item path="/etc/openldap/ldap.conf"/>
<item path="/etc/ldap.secret"/>
<item path="/etc/nsswitch.conf"/>
<item path="/etc/krb5.conf"/>
</rsync>


Heartbeat Cluster configuration

We then modify the /cluster_metadata/opt/openfiler/etc/cluster.xml config file. This config file generates our /etc/ha.d/haresources file, which tells Heartbeat what we want
it to do in an event of a fail_over.

filer1 Only:



Note how we have declared the HA IP Address here (192.168.8.10). As mentioned before, Heartbeat controls both the setup of the network interface and the mounting of
our LVM VG and /cluster_metadata.

Samba and NFS Support

Modify Samba and NFS so it's available on our /cluster_metadata drbd resource.

filer1:

root@filer1 ~# mkdir /cluster_metadata/etc
root@filer1 ~# mv /etc/samba/ /cluster_metadata/etc/
root@filer1 ~# ln -s /cluster_metadata/etc/samba/ /etc/samba
root@filer1 ~# mkdir -p /cluster_metadata/var/spool
root@filer1 ~# mv /var/spool/samba/ /cluster_metadata/var/spool/
root@filer1 ~# ln -s /cluster_metadata/var/spool/samba/ /var/spool/samba
root@filer1 ~# mkdir -p /cluster_metadata/var/lib
root@filer1 ~# mv /var/lib/nfs/ /cluster_metadata/var/lib/
root@filer1 ~# ln -s /cluster_metadata/var/lib/nfs/ /var/lib/nfs
root@filer1 ~# mv /etc/exports /cluster_metadata/etc/
root@filer1 ~# ln -s /cluster_metadata/etc/exports /etc/exports


Note: We have moved /var/spool/samba into /cluster_metadata, which is only a 300meg partition. So, if you put some large print jobs through Samba, you will find the
5
6
3 4
<?xml version="1.0" ?>
<cluster>
<clustering state="on" />
<nodename value="filer1" />
<resource value="MailTo::root@localhost::ClusterFailover"/>
<resource value="IPaddr::192.168.8.10/24" />
<resource value="drbddisk::" />
<resource value="LVM::vg0_drbd" />
<resource value="Filesystem::/dev/drbd0::/cluster_metadata::ext3::d
<resource value="MakeMounts" />
</cluster>
http://www.the-mesh.org/tiki-print.php?page=OpenFilerHaSetup Page 6 / 9
free space on this volume will get eaten up pretty quickly. So, if this is the case, you may want to create a separate DRBD resource for your /var directory. Or you may
want to reconsider hosting your print services on a server that isn't your SAN/NAS.

filer2:

root@filer2 ~# rm -rf /etc/samba/
root@filer2 ~# ln -s /cluster_metadata/etc/samba/ /etc/samba
root@filer2 ~# rm -rf /var/spool/samba/
root@filer2 ~# ln -s /cluster_metadata/var/spool/samba/ /var/spool/samba
root@filer2 ~# rm -rf /var/lib/nfs/
root@filer2 ~# ln -s /cluster_metadata/var/lib/nfs/ /var/lib/nfs
root@filer2 ~# rm -rf /etc/exports
root@filer2 ~# ln -s /cluster_metadata/etc/exports /etc/exports


iSCSI Support

filer1:

root@filer1 ~# mv /etc/ietd.conf /cluster_metadata/etc/
root@filer1 ~# ln -s /cluster_metadata/etc/ietd.conf /etc/ietd.conf
root@filer1 ~# mv /etc/initiators.allow /cluster_metadata/etc/
root@filer1 ~# ln -s /cluster_metadata/etc/initiators.allow /etc/initiators.allow
root@filer1 ~# mv /etc/initiators.deny /cluster_metadata/etc/
root@filer1 ~# ln -s /cluster_metadata/etc/initiators.deny /etc/initiators.deny


filer2:

root@filer2 ~# rm /etc/ietd.conf

root@filer2 ~# ln -s /cluster_metadata/etc/ietd.conf /etc/ietd.conf
root@filer2 ~# rm /etc/initiators.allow
root@filer2 ~# ln -s /cluster_metadata/etc/initiators.allow /etc/initiators.allow
root@filer2 ~# rm /etc/initiators.deny
root@filer2 ~# ln -s /cluster_metadata/etc/initiators.deny /etc/initiators.deny


FTP Support

filer1:

root@filer1 ~# mv /etc/proftpd /cluster_metadata/etc/
root@filer1 ~# ln -s /cluster_metadata/etc/proftpd/ /etc/proftpd


filer2:

root@filer2 ~# rm -rf /etc/proftpd
root@filer2 ~# ln -s /cluster_metadata/etc/proftpd/ /etc/proftpd


Configure Volume Group

filer1:

Create a Volume group from our /dev/drbd1:
root@filer1 etc# vgcreate vg0_drbd /dev/drbd1
Volume group "vg0_drbd" successfully created


Note: If you plan on using Windows to connect your iSCSI targets, do not use the "_" character or any other special characters when creating your Volume group.

Once the Heartbeat service has been configured and started (see below), you should be able to use the Openfiler web administration on https://192.168.8.10:446 to create
LVM volumes and to export them via iSCSI etc.

Starting Heartbeat and first-time configuration

In order to get Openfiler to write the /etc/ha.d/haresources file based on the cluster.xml config file, we need to restart the Openfiler service and then log onto the web
interface (using the Primary filer's direct IP) and click on System -> HA Cluster Setup.

Make sure you do this on the Primary filer (assumably filer1).

root@filer1 ~# rm /opt/openfiler/etc/httpd/modules

root@filer1 ~# ln -s /usr/lib/httpd/modules /opt/openfiler/etc/httpd/modules

root@filer1 ~# service openfiler restart


Note
http://www.the-mesh.org/tiki-print.php?page=OpenFilerHaSetup Page 7 / 9
On a 64bit platform the lib folder is called lib64;
root@filer1 ~# ln -s /usr/lib64/httpd/modules /opt/openfiler/etc/httpd/modules


After Openfiler restarts and you log onto the web interface, it should have a whole heap of values that reflect the clustes.xml file. With any luck, Openfiler has written this
file out to /etc/ha.d/haresources. If haresources exist, you should be right to copy it over to filer2.

Before we can start Heartbeat we must create a volume for our heartbeat to start (Otherwise HA will fail to start).

[root@filer1 ~# lvcreate -L 50M -n StartVolume? vg0_drbd


When you log onto the web interface and activate a service such as NFS or iSCSI. this will force Openfiler to rewrite the /etc/ha.d/haresources file. You will then need to
scp this file over to the second filer as follows:

root@filer1 ~# scp /etc/ha.d/haresources root@filer2:/etc/ha.d/haresources


Since we have added heartbeat to our start up scripts earlier, we should be right to reboot filer1 then after that to reboot filer2.

If all goes to plan you should be able to access the primary filer with your web browser via https://192.168.8.10:446 (the High Available ip address). You can then remove
the 'StartVolume' you created above and create any other volumes you need via the web interface (Make sure you add one in before restarting otherwise HA will fail to
start)

If the webserver does not work for any reason a good place to look for errors is in /var/log/ha-log this should allow you to work out where the problem may be.



Note:
The rsync configuration is meant to synchronise changes between the two filers, but from observation this does not appear to happen. So, if you update your clusters.xml
file, make sure you put the re-generated haresources file on both filers. Do not modify /etc/ha.d/haresources directly, as it will get overwritten. Always
use /cluster_metadata/opt/openfiler/etc/cluster.xml.

Also it is a good idea to copy over the /etc/ha.d/haresources file everytime you change your lvm volumes as this can alter the haresources file and prevent the second
filer from failing over.

We have filed a bug report which has since been fixed, I am yet to try just updating this file to see if it works or not.

https://project.openfiler.com/tracker/ticket/791


Trouble shooting

If you get an error during the creation of a LVM volume, similar to to:
Found duplicate PV m8M0tNOCx83QS53D07kybsEDl8oNt4X6: using /dev/sdb1 not /dev/drbd0

It is using the raw device instead of the DRBD'd device, to fix this edit /etc/lvm/lvm.conf and change the filter line to:


Software updates

Note from the 2.2 HA Cluster Guide:
When updating Openfiler, you need to run conary updateall replace-fileson the primary active filer in the cluster (the one with /cluster_metadata currently mounted.)
Running the update on the secondary filer in the cluster will fail because /cluster_metadata (which /opt/openfiler is symlinked to) does not exist.


Not sure if the second filer is meant to be updated by becoming Primary and updating as the note above suggests for the first filer. Assumably this would not cause any
problems, but at this stage it is something we have not tested.

Credits

We would like to acknowledge that this HowTo? is heavily based on the Openfiler 2.2 HA Cluster Guide. Please make sure this is signified in all reproductions of this
documentation. We have copied information from the 2.2 HA Cluster Guide with an understanding it is a public domain document. If this is not the case, please notify us
and we will correct this.

Installing and Configuring Openfiler with DRBD and Heartbeat

Copyright (c) 2005 The Mesh.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.2
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover? Texts, and no Back-Cover?
Texts. A copy of the license is included in the section entitled
"GNU Free Documentation License".


Created by: Ji nx. Last Modification: Friday 05 of February, 2010 00:25:37 CST by WickedWesley.
filter = [ "a|/dev/drbd*|", "r/.*/" ]
http://www.the-mesh.org/tiki-print.php?page=OpenFilerHaSetup Page 8 / 9
The content on this page is licensed under the terms of the GNU FDL.
The original document is available at http://the-mesh.org/tiki-index.php?page=OpenFilerHaSetup

http://www.the-mesh.org/tiki-print.php?page=OpenFilerHaSetup Page 9 / 9