You are on page 1of 38
High Availability of IBM Security Directory Server using Heartbeat A highly available authentication system Prabir Meher
High Availability of IBM Security Directory
Server using Heartbeat
A highly available authentication system
Prabir Meher
IBM India Software Lab, Pune
2014/02/20, 1.0
Abstract:
The purpose of this article is to demonstrate the use of ISDS (IBM Security Directory Server), formerly known
as ITDS (IBM Tivoli Directory Server), using Heartbeat for creating a highly available authentication system
with fail-over mechanism. High availability is very critical for enterprise authentication services since
consolidating any service on a particular server is not at all reliable. Depending on a single server eventually
creates a "single point of failure" (SPOF), which can break the entire organization's authentication system.
You will see here one method of creating a reliable authentication server using IBM Security Directory Server,
which can be adapted by many different applications. We will use the Heartbeat package from the Linux HA
project (see the "Resources" section) to design a highly available authentication system using IBM Security
Directory Server.

Table of Contents

1Introduction .............................................................................................................................................4 2Environment setup...................................................................................................................................5 3About IBM Security Directory Server (ISDS).........................................................................................5 4About Heartbeat.......................................................................................................................................5

5Prerequisite...............................................................................................................................................6

6IBM Security Directory Server replication..............................................................................................7 6.1Log in to Web Administration Tool..................................................................................................8 6.2WAT introduction page.....................................................................................................................8 6.3Adding servers to WAT....................................................................................................................9 6.4Log in to the ISDS instance to configure replication.....................................................................11 6.5Setting up the replication................................................................................................................11 6.5.1Check replication-related entries in ISDS using command line.............................................23 6.6Synchronize data between two replication servers.........................................................................23 6.6.1Exporting data.........................................................................................................................24 6.6.2Exporting data using salt and seed of the destination server..................................................24 How to obtain salt and seed of an ISDS server..........................................................................25 6.6.3Importing data.........................................................................................................................25 6.7Resume replication queue..............................................................................................................26 7Heartbeat setup.......................................................................................................................................28 7.1Installing Heartbeat........................................................................................................................28 7.2Configuring Heartbeat....................................................................................................................28 7.2.1The authkeys file.....................................................................................................................28 7.2.2The ha.cf file...........................................................................................................................29 7.2.3The haresources file................................................................................................................30 7.3Propagating configuration files to peer servers..............................................................................30 7.4Starting Heartbeat...........................................................................................................................31 7.4.1Network configuration before starting heartbeat....................................................................31 7.4.2Network configuration after starting heartbeat.......................................................................34 7.4.3Checking Heartbeat status......................................................................................................34 8Testing high-availability scenario..........................................................................................................35 8.1.1Checking the network configuration on ldaphost2.................................................................35 9DNS hack...............................................................................................................................................37 10Query ISDS server using virtual IP......................................................................................................37

11Resources.............................................................................................................................................38

Table of Figures

Figure 1: Directory operations under normal condition.............................................................................4 Figure 2: Directory operations under failover condition............................................................................5 Figure 3: WAT Console administration login.............................................................................................8 Figure 4: WAT console introduction page..................................................................................................8 Figure 5: Adding servers to WAT...............................................................................................................9 Figure 6: Adding servers to WAT(2)..........................................................................................................9 Figure 7: WAT servers list........................................................................................................................10 Figure 8: WAT Directory server login......................................................................................................11 Figure 9: WAT Introduction page.............................................................................................................12 Figure 10: Add replication subtree...........................................................................................................12 Figure 11: Browse subtree(1)...................................................................................................................13

Figure 12: Select replication suffix..........................................................................................................13 Figure 13: Browse subtree (2)..................................................................................................................13 Figure 14: Show topology........................................................................................................................14 Figure 15: Show topology for o=ibm,c=in...............................................................................................15 Figure 16: Replication pop-up menu.......................................................................................................15 Figure 17: Add peer server to replication topology.................................................................................16 Figure 18: Add credentials (1).................................................................................................................16 Figure 19: Add credentials (2).................................................................................................................17 Figure 20: Add credentials (3).................................................................................................................17 Figure 21: Select credential......................................................................................................................18 Figure 22: Add master server...................................................................................................................18 Figure 23: Add credential on peer server.................................................................................................19 Figure 24: Collect peer agreement information.......................................................................................19 Figure 25: Enter ldaphost1 credential information..................................................................................20 Figure 26: Replication setup success information...................................................................................23 Figure 27: Resume replication queue from ldaphost1 to ldaphost2.........................................................27 Figure 28: Resume replication queue from ldaphost2 to ldaphost1.........................................................27

Table of Listings

Listing 1: "idsilist" output from ldaphost1.................................................................................................6 Listing 2: "idsilist" output from ldaphost2.................................................................................................7 Listing 3: ISDS and admin server running on ldaphost1...........................................................................7 Listing 4: ISDS and admin server running on ldaphost2...........................................................................7 Listing 5: ISDS instance list.....................................................................................................................10 Listing 6: Information added to DIT after adding replication suffix.......................................................14 Listing 7: Bind credential.........................................................................................................................18 Listing 8: Supplier bind credential information.......................................................................................19 Listing 9: Information added to server ldaphost1....................................................................................21 Listing 10: Information added to server ldaphost2..................................................................................22 Listing 11: Exporting data with seed and salt..........................................................................................24 Listing 12: ISDS server's crypto salt........................................................................................................25 Listing 13: Copy the exported LDIF file to the target server...................................................................25 Listing 14: Importing data into the server................................................................................................26 Listing 15: View Heartbeat's document...................................................................................................28 Listing 16: The /etc/ha.d/authkeys file.....................................................................................................29 Listing 17: authkeys permission..............................................................................................................29 Listing 18: Sample ha.cf file....................................................................................................................29 Listing 19: Sample haresources file.........................................................................................................30 Listing 20: ha_propagate script location..................................................................................................30 Listing 21: ha_propagate..........................................................................................................................31 Listing 22: ifconfig output before starting heartbeat................................................................................32 Listing 23: Starting heartbeat...................................................................................................................32 Listing 24: log snippet from /var/log/ha-log on ldaphost1......................................................................33 Listing 25: ifconfig output after starting heartbeat..................................................................................34 Listing 26: Stopping heartbeat.................................................................................................................35 Listing 27: ifconfig output from ldaphost2 after stopping heartbeat on ldaphost1..................................35 Listing 28: log snippet from /var/log/ha-log on ldaphost2 after stopping heartbeat on ldaphost1..........36 Listing 29: /etc/hosts file from server ldaphost1 and ldaphost2..............................................................37

Listing 30: idsldapsearch using virtual IP/hostname...............................................................................38

1

Introduction

As organizations add applications and services, centralized authentication services can increase security and decrease administrative tasks. However, being dependent on a single server eventually creates a "single point of failure" (SPOF), which can break the entire organization's authentication system. To overcome this SPOF, we will be discussing here how to configure IBM Security Directory Server (ISDS) with Heartbeat to deliver a highly available authentication system.

In this article, we will demonstrate one method to create a reliable authentication server using ISDS, which can be adapted by many organization-wide applications. We will use the Heartbeat package from the Linux HA project.

Starting with two identical ISDS servers (peer-to-peer replication), several configurations can be used. First, we could do a "cold standby" where the master ISDS server has a virtual IP and a running ISDS instance. The secondary ISDS server sits idle. When the master server fails, the ISDS instance and virtual IP move to the cold node (secondary server). This is a very simple setup to implement. See Figure 1: Directory operations under normal condition. However, the data synchronization between the master and secondary servers could be a problem. To solve that, we will configure the servers with live ISDS instances running on both the servers. In this way, updates to the master server are immediately replicated to the secondary server.

Listing 30: idsldapsearch using virtual IP/hostname...............................................................................38 1 Introduction As organizations add applications and services, centralized authentication

Figure 1: Directory operations under normal condition

Failure of the master ISDS server leaves our secondary ISDS server available to respond to client queries. See Figure 2: Directory operations under failover condition

Figure 2: Directory operations under failover condition

Figure 2: Directory operations under failover condition

  • 2 Environment setup

The example described in this article is based on a setup that requires:

Two ISDS v6.3.1 Servers, installed on RHEL 6.5 64-bit, configured in a peer-to-peer or master-master replication.

Server1 hostname – ldaphost1.in.ibm.com - 192.168.56.20

Server2 hostname – ldaphost2.in.ibm.com - 192.168.56.21

Heartbeat version 3.0.4 installed on both servers (i.e., ldaphost1 and ldaphost2)

Note: See the "Resources" section to download ISDS and Heartbeat packages.

  • 3 About IBM Security Directory Server (ISDS)

IBM® Security Directory Server is an enterprise identity management software that uses the Lightweight Directory Access Protocol (LDAP) and provides a trusted identity data infrastructure for authentication. See the "Resources" section to find out more about IBM Security Directory Server.

  • 4 About Heartbeat

In an enterprise environment, certain servers (such as an authentication server) must always be up and running for the business to keep functioning smoothly. These servers provide services that need to be always available.

A cornerstone of any mission-critical service that always needs to be up with no downtime is being able to transfer the service from one system to another gracefully. The magic that makes this possible is a service called Heartbeat. Heartbeat is the main product of the High-Availability Linux project.

In this article, we have tested only an active/passive method with two ISDS peer servers, where the active server provides the services and the passive server waits to take over in case the active server goes down.

The best part of this method is that you do not need any hardware devices, which tend to be expensive, to build a highly available authentication system.

5

Prerequisite

In this section, I assume that you have already installed ISDS on both systems (i.e., ldaphost1 and

ldaphost2).

I also assume that you have created an ISDS instance dsrdbm01 on both the servers, which runs on port 389. If you have set up the instances properly, then you will see output from both servers similar to the following two listings.

[root@ldaphost1 ~]# idsilist -a Directory server instance(s):

--------------------------------------

Instance 1:

Name: dsrdbm01 Version: 6.3.1 Location: /home/dsrdbm01 Description: IBM Security Directory Server Instance V6.3.1 IP Addresses: All available Port: 389 Secure Port: 636 Admin Server Port: 3538 Admin Server Secure Port: 3539 Type: Directory Server

Listing 1: "idsilist" output from ldaphost1

Listing 1 shows sample output from ldaphost1.in.ibm.com. Listing 2 shows sample output from

ldaphost2.in.ibm.com.

[root@ldaphost2 ~]# idsilist -a Directory server instance(s):

--------------------------------------

Instance 1:

Name: dsrdbm01 Version: 6.3.1 Location: /home/dsrdbm01 Description: IBM Security Directory Server Instance V6.3.1 IP Addresses: All available Port: 389 Secure Port: 636 Admin Server Port: 3538 Admin Server Secure Port: 3539 Type: Directory Server

Listing 2: "idsilist" output from ldaphost2

To configure the replication, both the instances on server ldaphost1 and ldaphost2 must be up and running. To confirm that they are, type the following command on both servers:

[root@ldaphost1 ~]# netstat -tnlp | grep ibm

Listing 3shows sample output from ldaphost1:

[root@ldaphost1 ~]# netstat -tnlp | grep ibm

 

tcp

0

0 :::389

:::*

LISTEN

1755/ibmslapd

tcp

0

0 :::3538

:::*

LISTEN

29489/ibmdiradm

Listing 3: ISDS and admin server running on ldaphost1

 

Listing 4 shows sample output from ldaphost2:

 

[root@ldaphost2 ~]# netstat -tnlp | grep ibm

 

tcp

0

0 :::389

:::*

LISTEN

14158/ibmslapd

tcp

0

0 :::3538

:::*

LISTEN

4581/ibmdiradm

Listing 4: ISDS and admin server running on ldaphost2

 
  • 6 IBM Security Directory Server replication

Replication is a technique used by directory servers to improve performance, availability, and reliability. The replication process keeps the data in multiple directory servers synchronized.

We will set up a peer-to-peer replication using the Web Administration Tool (WAT), which uses a graphical user interface to administer ISDS servers.

If you do not have WAT installed on your machine, see "Setting up Tivoli Directory Server replication using the command line," an excellent article on developerWorks that presents a step-by-step method of configuring replication in an ISDS server using command-line tools. It can be accessed through the link

provided below in "Resources." For the current article, I have set up WAT on machine ldaphost1.in.ibm.com.

  • 6.1 Log in to Web Administration Tool

You can log in to the Web Administration Tool (WAT) using the following url:

http://ldaphost1.in.ibm.com:12100/IDSWebApp/. You will need to change the hostname of the server

where you have deployed the WAT. If the above url doesn't work, try the direct login page:

http://ldaphost1.in.ibm.com:12100/IDSWebApp/IDSjsp/Login.jsp

provided below in "Resources." For the current article, I have set up WAT on machine ldaphost1.in.ibm.comhttp://ldaphost1.in.ibm.com:12100/IDSWebApp/ . You will need to change the hostname of the server where you have deployed the WAT. If the above url doesn't work, try the direct login page: http://ldaphost1.in.ibm.com:12100/IDSWebApp/IDSjsp/Login.jsp Figure 3: WAT Console administration login Use the default username superadmin and password secret to log in to the WAT. 6.2 WAT introduction page After you have logged in to the WAT for the first time, click on Manage Console Servers to begin adding the servers. Figure 4: WAT console introduction page " id="pdf-obj-7-16" src="pdf-obj-7-16.jpg">

Figure 3: WAT Console administration login

Use the default username superadmin and password secret to log in to the WAT.

  • 6.2 WAT introduction page

After you have logged in to the WAT for the first time, click on Manage Console Servers to begin adding the servers.

provided below in "Resources." For the current article, I have set up WAT on machine ldaphost1.in.ibm.comhttp://ldaphost1.in.ibm.com:12100/IDSWebApp/ . You will need to change the hostname of the server where you have deployed the WAT. If the above url doesn't work, try the direct login page: http://ldaphost1.in.ibm.com:12100/IDSWebApp/IDSjsp/Login.jsp Figure 3: WAT Console administration login Use the default username superadmin and password secret to log in to the WAT. 6.2 WAT introduction page After you have logged in to the WAT for the first time, click on Manage Console Servers to begin adding the servers. Figure 4: WAT console introduction page " id="pdf-obj-7-32" src="pdf-obj-7-32.jpg">

Figure 4: WAT console introduction page

6.3

Adding servers to WAT

Click on Add or servers. ... to add your servers to the Web Administration Tool to
Click on Add
or servers.
...
to add your servers to the Web Administration Tool to manage from a remote location
Figure 5: Adding servers to WAT

In the Server name field, enter a descriptive name for your ISDS instance. The best practice in creating a server name is to use the hostname followed by the instance name (i.e., hostname- instance_name). To use only the hostname, leave the field blank.

Hostname is the real server's name on which TDS instance is running, and the hostname must be resolvable by your DNS, otherwise WAT will fail to contact the server. On the other hand, Server name is for end user reference to give a meaningful name to the instance. Server name is confined to the WAT portal only, whereas Hostname must be resolvable to a specific IP address.

Figure 6: Adding servers to WAT(2)

Figure 6: Adding servers to WAT(2)

The panel in Figure 6 also prompts you for a Port number and an Administration port number. Use the command line tool idsilist -a to find out these values. See, for example, the output in Listing

5.

[root@ldaphost1 ~]# idsilist -a Directory server instance(s):

--------------------------------------

Instance 1:

Name: dsrdbm01 Version: 6.3.1 Location: /home/dsrdbm01 Description: IBM Security Directory Server Instance V6.3.1 IP Addresses: All available

Port: 389

Secure Port: 636

Admin Server Port: 3538

Admin Server Secure Port: 3539 Type: Directory Server

Listing 5: ISDS instance list

After you have added each server, press OK. Add the rest of the servers that you want to administer remotely using WAT.

After all the servers have been added, you can display the list in the Manage console servers page.

Figure 7: WAT servers list

Figure 7: WAT servers list

Now click Logout to log out of the WAT.

6.4

Log in to the ISDS instance to configure replication

You can now log in to any of the ISDS servers that you added in the "Adding servers to WAT" section. Here we will log in to the instance dsrdbm01 on server ldaphost1.

The LDAP Server Name drop-down field will pop out a list of all the servers that you have added. See Figure 8: WAT Directory server login.

Select ldaphost1-dsrdbm01 as the LDAP Server Name and enter the ISDS administrative user ID (in most cases, it is cn=root) and password, which was configured when the ISDS instance dsrdbm01 was created on hostldaphost1.

Note: If you do not know the administrative user ID and password, you cannot log in to WAT as a privileged user and cannot configure the replication. Please refer to the command line tool idsdnpw for more information on how to configure the administrative DN and password for an ISDS instance.

6.4 Log in to the ISDS instance to configure replication You can now log in to

Figure 8: WAT Directory server login

  • 6.5 Setting up the replication

When you click Login—see Figure 8—you are presented with the introduction page of the Web Administration Tool, where you can administer your ISDS instance from a local or remote system. See Figure 9: WAT Introduction page.

Figure 9: WAT Introduction page

Figure 9: WAT Introduction page

Near the top of Figure 9, two values have been outlined in red. The left box signifies the physical server name and the port on which the ISDS instance is running along with the server name in brackets, which you configured in the "Adding servers to WAT" section. The right box indicates the current logged-in user, which in this case is cn=root (ISDS root administrator). Click on Manage Replication to configure the replication for a particular suffix or subtree. See Figure 10: Add replication subtree

Figure 10: Add replication subtree

Figure 10: Add replication subtree

Click on Add subtree

...

to add the suffix or subtree that you want to replicate. You are presented with

a screen similar to Figure 11: Browse subtree(1) to add your subtree.

Now click on Browse

...

to navigate through the

Directory Information Tree (DIT) of your ISDS

server.

Accept the default value in the Master server referral LDAP URL field.

Figure 11: Browse subtree(1)

Figure 11: Browse subtree(1)

After clicking on OK, as shown in Figure 11: Browse subtree(1), you are presented with a screen similar to Figure 12: Select replication suffix, on which to choose the suffix or subtree to replicate.

Figure 12: Select replication suffix

Figure 12: Select replication suffix

Click the radio button on the left to choose the suffix that you want to replicate. Click on Select. You are redirected to a page similar to Figure 13: Browse subtree (2). This screen resembles the one displayed in Figure 11, except that the Subtree DN value is filled in with the selected subtree name: in this case, o=ibm,c=in. Next, click on the OK button to continue. WAT adds the following three replication stanzas (shown in Listing 6: Information added to DIT after adding replication suffix) to the DIT of instance dsrdbm01, which is

running on host ldaphost1.in.ibm.com.

Figure 13: Browse subtree (2)

Figure 13: Browse subtree

(2)

o=ibm,c=in objectclass=organization objectclass=top objectclass=ibm-replicationcontext o=ibm

ibm-replicareferralurl=ldap://ldaphost1.in.ibm.com:389

ibm-replicaGroup=default,o=ibm,c=in

ibm-replicagroup=default

objectclass=ibm-replicagroup

objectclass=top

cn=ldaphost1.in.ibm.com:389,ibm-replicaGroup=default,o=ibm,c=in

objectclass=ibm-replicasubentry objectclass=top

ibm-replicaserverid=de8e2ec0-bfa2-1032-950c-8d69c0d06df0

ibm-replicationserverismaster=TRUE

cn=ldaphost1.in.ibm.com:389

Listing 6: Information added to DIT after adding replication suffix

Refer to "Check replication-related entries in ISDS using command line" to find out the Listing 6 information for your server.

After clicking on OK in Figure 13, you see the following screen:

Figure 14: Show topology

Figure 14: Show topology

Notice that o=ibm,c=in has been added successfully as a replication subtree. The ISDS instance dsrdbm01, which is running on port 389 on server ldaphost1.in.ibm.com, will act as a master server for this subtree in the replication topology. Click on Show topology to see a screen similar to that in Figure 15: Show topology for o=ibm,c=in:

Figure 15: Show topology for o=ibm,c=in
Figure 15: Show topology for o=ibm,c=in

Click on the Topology for selected subtree button to get a pop-up menu, as shown in Figure 16: Replication pop-up menu.

This menu allows you to add a peer master or a replica or even manage your gateway servers.

Since we are setting up a peer-to-peer replication topology,we need to add a master server under the server ldaphost1.in.ibm.com which is also a master server for the suffix o=ibm, c=in. Click on Add master to add another peer server to

Figure 16: Replication pop-up menu

Figure 16: Replication pop-up menu

ldaphost1.in.ibm.com. After clicking on Add master, you are presented with a screen similar to that in Figure 17: Add peer server to replication topology:

Figure 17: Add peer server to replication topology

Figure 17: Add peer server to replication topology

Click on the drop-down list for the field Server hostname:port to select your peer master server. Click on Get server ID to get the server ID of ldaphost2.in.ibm.com. You will now need bind credentials which your server ldaphost1.in.ibm.com will use to bind to the server ldaphost2.in.ibm.com and make the necessary changes during replication. If any entry information on server ldaphost1.in.ibm.com changes by any client, then the same operation will be replicated to server ldaphost2.in.ibm.com using these bind credentials.

Click Select as shown Figure 17. You will see a screen similar to Figure 18: Add credentials (1). No credentials have yet been created; therefore, the Select credential field is empty. Select the replication suffix (o=ibm,c=in) radio button under Location for credentials and click on Add credentials.

Figure 18: Add credentials (1)

Figure 18: Add credentials (1)

You now see a screen similar to Figure 19: Add credentials (2), where you need to enter a CN (Common Name) under which your bind credentials will be stored. Click on Next to continue.

Figure 19: Add credentials (2)

Figure 19: Add credentials (2)

You now see a screen similar to Figure 20: Add credentials (3), where you need to enter the bind DN and a password. The DN will be used to authenticate the replication changes from server ldaphost1.in.ibm.com to ldaphost2.in.ibm.com. Optionally, you can also enter a description for the bind DN.

Figure 20: Add credentials (3)

Figure 20: Add credentials (3)

Click Finish on Figure 20: Add credentials (3). You now see a screen similar to Figure 21: Select credential.

Figure 21: Select credential

Figure 21: Select credential

A stanza has been added to your DIT under DN ibm-replicaGroup=default,O=IBM,C=IN as shown in Listing 7: Bind credential.

cn=bindcreds,ibm-replicaGroup=default,O=IBM,C=IN replicacredentials=manager description=Bind to replication server ldaphost2 objectclass=ibm-replicationcredentials objectclass=ibm-replicationcredentialssimple objectclass=top replicabinddn=cn=manager cn=bindcreds

Listing 7: Bind credential

Refer to "Check replication-related entries in ISDS using command line" to find out the Listing 6 information for your server.

Click OK on Figure 21. You see a screen similar to that shown in Figure 22: Add master server.

Figure 21: Select credential A stanza has been added to your DIT under DN ibm-replicaGroup=default,O=IBM,C=IN as

Figure 22: Add master server

Click on Additional to configure the peer replication server (ldaphost2.in.ibm.com). You are presented with a new screen. Scroll down to the Consumer section and add the details as shown in Figure 23:

Add credential on peer server.

Figure 23: Add credential on peer server

Figure 23: Add credential on peer server

Click the Add credential information on consumer check box and type in the admin DN for the peer replication server. In this case, the admin DN is cn=root. Enter the password root .

Click OK. The following supplier bind credential information, as shown in Listing 8: Supplier bind credential information, for server ldaphost1.in.ibm.com will be added to server ldaphost2.in.ibm.com.

cn=Supplier1397407251698, cn=configuration

cn=Supplier1397407251698

ibm-slapdmasterdn=cn=manager

ibm-slapdmasterpw={AES256}eKP6VmxELqJHSNAlGYWUkw==

ibm-slapdreplicasubtree=O=IBM, C=IN objectclass=ibm-slapdconfigentry objectclass=ibm-slapdsupplier objectclass=top

Listing 8: Supplier bind credential information

You now see a screen similar to Figure 24: Collect peer agreement information.

Figure 24: Collect peer agreement information

Figure 24: Collect peer agreement information

Click OK.You are asked to enter the credential information to be added to server ldaphost1.in.ibm.com. See Figure 25: Enter ldaphost1 credential information.

Figure 25: Enter ldaphost1 credential information

Figure 25: Enter ldaphost1 credential information

Enter all the information requested in Figure 25. Click OK.

The following stanzas have now been added to server ldaphost1.in.ibm.com (see Listing 9: Information added to server ldaphost1):

cn=Supplier1397416499356, cn=configuration

cn=Supplier1397416499356

ibm-slapdmasterdn=cn=manager

ibm-slapdmasterpw={AES256}mU4i4ChMyT7OytFazN+hIA==

ibm-slapdreplicasubtree=O=IBM, C=IN objectclass=ibm-slapdconfigentry objectclass=ibm-slapdsupplier objectclass=top

cn=ldaphost2.in.ibm.com:389,ibm-replicaGroup=default,O=IBM,C=IN

objectclass=ibm-replicasubentry objectclass=top

ibm-replicaserverid=7cf882c0-46f0-1033-8038-b508ef5ab1d7

ibm-replicationserverismaster=TRUE

cn=ldaphost2.in.ibm.com:389

cn=ldaphost1.in.ibm.com:389,cn=ldaphost2.in.ibm.com:389,ibm-

replicaGroup=default,O=IBM,C=IN

ibm-replicamethod=1

ibm-replicaconsumerid=de8e2ec0-bfa2-1032-950c-8d69c0d06df0

ibm-replicationonhold=TRUE ibm-replicacredentialsdn=cn=bindcreds,ibm-replicaGroup=default,O=IBM,C=IN

ibm-replicaurl=ldap://ldaphost1.in.ibm.com:389

objectclass=ibm-replicationagreement objectclass=top

cn=ldaphost1.in.ibm.com:389

cn=ldaphost2.in.ibm.com:389,cn=ldaphost1.in.ibm.com:389,ibm-

replicaGroup=default,O=IBM,C=IN

ibm-replicamethod=1

ibm-replicaconsumerid=7cf882c0-46f0-1033-8038-b508ef5ab1d7

ibm-replicationonhold=TRUE ibm-replicacredentialsdn=cn=bindcreds,ibm-replicaGroup=default,O=IBM,C=IN

ibm-replicaurl=ldap://ldaphost2.in.ibm.com:389

objectclass=ibm-replicationagreement

objectclass=top

cn=ldaphost2.in.ibm.com:389

Listing 9: Information added to server ldaphost1

In addition, the following information has been added to server ldaphost2.in.ibm.com (Listing 10:

Information added to server ldaphost2):

o=ibm,c=in objectclass=organization objectclass=top objectclass=ibm-replicationcontext o=ibm

ibm-replicareferralurl=ldap://ldaphost1.in.ibm.com:389

ibm-replicaGroup=default,o=ibm,c=in

ibm-replicagroup=default

objectclass=ibm-replicagroup

objectclass=top

cn=ldaphost1.in.ibm.com:389,ibm-replicaGroup=default,o=ibm,c=in

objectclass=ibm-replicasubentry objectclass=top

ibm-replicaserverid=de8e2ec0-bfa2-1032-950c-8d69c0d06df0

ibm-replicationserverismaster=TRUE

cn=ldaphost1.in.ibm.com:389

cn=ldaphost2.in.ibm.com:389,ibm-replicaGroup=default,O=IBM,C=IN

objectclass=ibm-replicasubentry objectclass=top

ibm-replicaserverid=7cf882c0-46f0-1033-8038-b508ef5ab1d7

ibm-replicationserverismaster=TRUE

cn=ldaphost2.in.ibm.com:389

cn=ldaphost1.in.ibm.com:389,cn=ldaphost2.in.ibm.com:389,ibm-

replicaGroup=default,O=IBM,C=IN

ibm-replicamethod=1

ibm-replicaconsumerid=de8e2ec0-bfa2-1032-950c-8d69c0d06df0

ibm-replicationonhold=TRUE ibm-replicacredentialsdn=cn=bindcreds,ibm-replicaGroup=default,O=IBM,C=IN

ibm-replicaurl=ldap://ldaphost1.in.ibm.com:389

objectclass=ibm-replicationagreement objectclass=top

cn=ldaphost1.in.ibm.com:389

cn=ldaphost2.in.ibm.com:389,cn=ldaphost1.in.ibm.com:389,ibm-replicaGroup=default

ibm-replicamethod=1

ibm-replicaconsumerid=7cf882c0-46f0-1033-8038-b508ef5ab1d7

ibm-replicationonhold=TRUE ibm-replicacredentialsdn=cn=bindcreds,ibm-replicaGroup=default,O=IBM,C=IN

ibm-replicaurl=ldap://ldaphost2.in.ibm.com:389

objectclass=ibm-replicationagreement objectclass=top

cn=ldaphost2.in.ibm.com:389

cn=bindcreds,ibm-replicaGroup=default,O=IBM,C=IN replicacredentials=manager description=Bind to replication server ldaphost2 objectclass=ibm-replicationcredentials objectclass=ibm-replicationcredentialssimple objectclass=top replicabinddn=cn=manager cn=bindcreds

Listing 10: Information added to server ldaphost2

Next, you are presented with a screen similar to that in Figure 26: Replication setup success information.

Figure 26: Replication setup success information

Figure 26: Replication setup success information

At this point, you have successfully configured the peer-to-peer replication between host ldaphost1.in.ibm.com and ldaphost2.in.ibm.com.

Note: If you already have data under suffix o=ibm,c=in, then you need to synchronize both servers manually. When WAT is used to set up the replication, WAT doesn't synchronize the data between the hosts but replicates only the configuration information essential for replication to work. By default, the replication queue is in suspended mode. You need to resume the queue manually to allow the changes made to both the servers to be replicated. Please refer to the section "Resume replication queue" to find out how to resume the queue using WAT.

6.5.1 Check replication-related entries in ISDS using command line

Sometimes it is useful to check the replication-related entries in ISDS to find out the basic reason why the replication is not working: whether an replication entry is missing or whether you have input the replication hostname incorrectly.

Use the following command to find out all the replication related entries in ISDS.

idsldapsearch -h <hostname/IP> -p <port> -D <adminDN> -w <adminDN_PW> -s sub -b "" objectclass=ibm-repl*

For example,

idsldapsearch -h ldaphost1.in.ibm.com -p 389 -D cn=root -w root -s sub -b "" objectclass=ibm-repl*

Note: You don't have to specify the port if it's 389, because 389 is the default port for LDAP. If the port number is other than 389, then you must specify the port or else you'll receive an error message.

Use the following command to find out the replication-related entries within a subtree. For example, to find out the replication-related entries under suffix "O=IBM,C=IN"

idsldapsearch -h ldaphost1.in.ibm.com -D cn=root -w root -s sub -b "O=IBM,C=IN" objectclass=ibm-repl*

  • 6.6 Synchronize data between two replication servers

Suppose you already have data under the suffix o=ibm,c=in. After OK on Figure 26: Replication setup success information is clicked, any changes made to or under suffix o=ibm, c=in will be replicated to server ldaphost2.in.ibm.com. But what about the data that was there under o=ibm, c=in on server ldaphost1.in.ibm.com before the replication was set up? No need to worry. All you have to do is to export the data under o=ibm,c=in from server ldaphost1.in.ibm.com into the server

ldaphost2.in.ibm.com.

Note: For better performance, synchronize the directory servers that are taking part in the replication cryptographically. See "Resources" for a link to Appendix J in the IBM Security Directory Server Administration Guide.

  • 6.6.1 Exporting data

You can export the data from a particular suffix using the command line tool idsdb2ldif. This will create an LDIF file that will contain all the data under a particular suffix.

Syntax:

idsdb2ldif -I <instance_name> -o <outfile.ldif>

Note: If your server instances are not synchronized cryptographically, then exported data (in this case <outfile.ldif> from server ldaphost1.in.ibm.com) cannot be imported into another server ldaphost2.in.ibm.com. In that case, you need to export the data from server ldaphost1.in.ibm.com using the salt and seed of the destination server ldaphost2.in.ibm.com. See "How to obtain salt and seed of an ISDS server" to get the salt value (-t); the seed value should be known to the person who originally created the ISDS instance.

  • 6.6.2 Exporting data using salt and seed of the destination server

After you have obtained the seed and salt value of your destination server—in this case, ldaphost2.in.ibm.com—perform the following command to export the data from server

ldapohost1.in.ibm.com.

Syntax:

idsdb2ldif -I <instance_name> -k <seed_value> -t <salt_value> -o <outfile.ldif>

For example, see Listing 11: Exporting data with seed and salt.

[root@ldaphost1 sbin]# pwd

/opt/ibm/ldap/V6.3.1/sbin

[root@ldaphost1 sbin]# ./idsdb2ldif -I dsrdbm01 -k abc0123456789 -t 'umU"mYJ4Oct0' -o dsrdbm01_fullbackup.ldif GLPCTL113I Largest core file size creation limit for the process (in bytes):

'0'(Soft limit) and '-1'(Hard limit).

GLPCTL119I Maximum Data Segment(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is 262144. GLPCTL119I Maximum File Size(512 bytes block) soft ulimit for the process is -1 and the prescribed minimum is 2097152. GLPCTL122I Maximum Open Files soft ulimit for the process is 1024 and the prescribed minimum is 500. GLPCTL122I Maximum Stack Size(Kbytes) soft ulimit for the process is 10240 and the prescribed minimum is 10240. GLPCTL119I Maximum Virtual Memory(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is 1048576. GLPSRV221I Replication of security attributes feature is disabled. GLPSRV200I Initializing primary database and its connections.

GLPD2L011I 61 entries have been successfully exported from the directory.

Listing 11: Exporting data with seed and salt

How to obtain salt and seed of an ISDS server Obtain the encryption salt value by performing the following search:

ldapsearch -D <adminDN> -w <adminPw> -b "cn=crypto,cn=localhost" objectclass=* ibm-slapdCryptoSalt For example, to obtain the salt value of ldaphost2.in.ibm.com, execute the following command:

[root@ldaphost1 ~]# idsldapsearch -h ldaphost2.in.ibm.com -L -D cn=root -w root -s base -b cn=crypto,cn=localhost objectclass=* ibm-slapdCryptoSalt dn: cn=crypto,cn=localhost ibm-slapdCryptoSalt: umU"mYJ4Oct0

Listing 12: ISDS server's crypto salt

Note: For security purposes, the encryption seed value is not stored anywhere in the server. While creating the ISDS instance, be sure to make note of it so that you can remember it later.

6.6.3 Importing data

You can import an LDIF file into an ISDS server using a command line tool called idsldif2db. Now you can copy the newly generated LDIF file to the destination server ldaphost2.in.ibm.com so that you can import it into the ISDS instance.

Remote copy the LDIF file as shown in Listing 13: Copy the exported LDIF file to the target server (on a Linux system):

[root@ldaphost1 ~]# scp dsrdbm01_fullbackup.ldif ldaphost2.in.ibm.com:~ The authenticity of host 'ldaphost2.in.ibm.com (192.168.56.21)' can't be established. RSA key fingerprint is ab:20:99:59:74:25:dc:96:20:47:e5:ae:7e:9c:a5:90. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ldaphost2.in.ibm.com' (RSA) to the list of known hosts. root@ldaphost2.in.ibm.com's password:

dsrdbm01_fullbackup.ldif

100% 44KB 43.7KB/s 00:00 [root@ldaphost1 ~]#

Listing 13: Copy the exported LDIF file to the target server

On the destination server, ldaphost2.in.ibm.com, execute the command shown in Listing 14: Importing data into the server to import the LDIF file.

[root@ldaphost2 sbin]# ./idsldif2db -I dsrdbm01 -i ~/dsrdbm01_fullbackup.ldif GLPCTL113I Largest core file size creation limit for the process (in bytes):

'0'(Soft limit) and '-1'(Hard limit). GLPCTL119I Maximum Data Segment(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is 262144. GLPCTL119I Maximum File Size(512 bytes block) soft ulimit for the process is -1 and the prescribed minimum is 2097152. GLPCTL122I Maximum Open Files soft ulimit for the process is 1024 and the prescribed minimum is 500. GLPCTL122I Maximum Stack Size(Kbytes) soft ulimit for the process is 10240 and the prescribed minimum is 10240. GLPCTL119I Maximum Virtual Memory(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is 1048576. GLPCOM022I The database plugin is successfully loaded from libback-config.so. GLPSRV221I Replication of security attributes feature is disabled. GLPSRV200I Initializing primary database and its connections. GLPRPL137I Restricted Access to the replication topology is set to false. GLPRDB052E Entry CN=IBMPOLICIES already exists. GLPRDB052E Entry IBM-REPLICAGROUP=DEFAULT,CN=IBMPOLICIES already exists. GLPRDB052E Entry globalGroupName=GlobalAdminGroup,cn=ibmpolicies already exists. GLPRDB052E Entry cn=pwdpolicy,cn=ibmpolicies already exists. GLPRDB052E Entry CN=REPLICATION,CN=IBMPOLICIES already exists. GLPRDB052E Entry o=ibm,c=in already exists. GLPRDB052E Entry ibm-replicaGroup=default,o=ibm,c=in already exists. GLPRDB052E Entry cn=ldaphost1.in.ibm.com:389,ibm-replicaGroup=default,o=ibm,c=in already exists. GLPRDB052E Entry cn=bindcreds,ibm-replicaGroup=default,O=IBM,C=IN already exists. GLPRDB052E Entry cn=ldaphost2.in.ibm.com:389,ibm-replicaGroup=default,O=IBM,C=IN already exists. GLPRDB052E Entry cn=ldaphost1.in.ibm.com:389,cn=ldaphost2.in.ibm.com:389,ibm- replicaGroup=default,O=IBM,C=IN already exists. GLPRDB052E Entry cn=ldaphost2.in.ibm.com:389,cn=ldaphost1.in.ibm.com:389,ibm- replicaGroup=default,O=IBM,C=IN already exists.

GLPRDB002W ldif2db: 49 entries have been successfully added out of 61 attempted.

Listing 14: Importing data into the server

Note: The ISDS instance must be stopped before importing the LDIF file; otherwise it will not allow you to import the data into it. Also, you can ignore the error message GLPRDB052E shown in Listing 14, as the entries were already present on the destination server, ldaphost2.in.ibm.com. The idsldif2db utility will not change these entries; it will simply skip them.

At this point, both the source server (ldaphost1.in.ibm.com) and the destination server (ldaphost2.in.ibm.com) are perfectly in sync. Now you can go ahead and resume the replication queue to allow both the servers to replicate data to each other.

  • 6.7 Resume replication queue

Log in to the instance that is running on server ldapohost1.in.ibm.com and click on Replication management on the left pane. Under it, click on Manage queues.

Figure 27: Resume replication queue from ldaphost1 to ldaphost2

Figure 27: Resume replication queue from ldaphost1 to ldaphost2

Click on Suspend/resume to resume the replication queue. If ldaphost2.in.ibm.com is displayed in the Replica column, the replication queue flow is from server ldaphost1.in.ibm.com to

ldaphost2.in.ibm.com.

Next, log in to the instance that is running on server ldaphost2.in.ibm.com and click on Replication management on the left pane. Under it, click on Manage queues.

Figure 28: Resume replication queue from ldaphost2 to ldaphost1

Figure 28: Resume replication queue from ldaphost2 to ldaphost1

Click on Suspend/resume to resume the replication queue. If ldaphost1.in.ibm.com is displayed in the Replica column, the replication queue flow is from server ldaphost2.in.ibm.com to

ldaphost1.in.ibm.com.

Now both the servers are perfectly in sync and the replication queue is in Ready state. We will now

configure high availability (HA) for these servers.

  • 7 Heartbeat setup

Heartbeat is very flexible and powerful. In this article, I have touched on only the basic active/passive mode with two servers, where the active server (ldaphost1.in.ibm.com) is providing the service and the passive server (ldaphost2.in.ibm.com) is waiting to take over if necessary.

  • 7.1 Installing Heartbeat

Installing the Heartbeat package from the yum repository is always easier than building it from the source. The Heartbeat package is available in the EPEL (Extra Package for Enterprise Linux) repository. See "Resources" for download information.

Execute the following command to install Heartbeat:

yum install heartbeat --enablerepo=epel

  • 7.2 Configuring Heartbeat

You need three configuration files to work with Heartbeat. Each goes into the directory /etc/ha.d/

authkeys

File containing keys for mutual node authentication

ha.cf

Main Heartbeat configuration file

haresources

Resource configuration file

The ha.cf and hareources files may be readable by everyone, but the authkeys file must not be.

The good news is that sample versions of these files may be found in the documentation directory. If you installed Heartbeat using yum, then the following command will show you where they are on your system:

rpm -q heartbeat -d

For example, see Listing 15: View Heartbeat's document.

[root@ldaphost1 ~]# rpm -q heartbeat -d ...... ......

/usr/share/doc/heartbeat-3.0.4/authkeys

/usr/share/doc/heartbeat-3.0.4/ha.cf

/usr/share/doc/heartbeat-3.0.4/haresources

......

......

Listing 15: View Heartbeat's document

7.2.1 The authkeys file

The authkeys file must be owned by root and be set to chmod 600. The actual format of the authkeys file is very simple; it consists of only two lines. The first line contains an auth directive with an associated method ID number, and the second line identifies the authentication method and the key that go with the auth directive.

Three authentication methods are supported: CRC, MD5, and SHA1. CRC doesn't accept a key. You normally have only one authentication method listed in a CRC file. It adds no security, except from packet corruption, and should be used only on physically secure networks. Of the remaining two, SHA1 is usually considered to be the best, followed by MD5.

Listing 16: The /etc/ha.d/authkeys file shows an example. Make the key long as it will improve security and you will not have to type it again.

auth 1 1 sha a9e2189843e9c3fb7c5a1bf81587fc11

Listing 16: The /etc/ha.d/authkeys file

Check the permission on the file /etc/ha.d/authkeys. If the permission is not 600, then change it to 600 as shown in Listing 17: authkeys permission:

[root@ldaphost1 ~]# ls -lh /etc/ha.d/authkeys -rw-r--r-- 1 root root 46 Apr 19 01:51 /etc/ha.d/authkeys [root@ldaphost1 ~]# chmod 600 /etc/ha.d/authkeys [root@ldaphost1 ~]# ls -lh /etc/ha.d/authkeys -rw------- 1 root root 46 Apr 19 01:51 /etc/ha.d/authkeys

Listing 17: authkeys permission

7.2.2 The ha.cf file

This is the main Heartbeat configuration file. The content of this file should be same on both the servers with a couple of exceptions.

Heartbeat ships with an example file in the documentation directory that is well worth a look. Also, when creating your ha.cf file, the order in which things appear matters. Do not move them around!

debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 bcast eth0 warntime 5 deadtime 15 initdead 60 keepalive 2 auto_failback on node ldaphost1.in.ibm.com node ldaphost2.in.ibm.com

Listing 18: Sample ha.cf file

The auto_failback on command tells Heartbeat to prefer ldaphost1.in.ibm.com. If ldaphost1.in.ibm.com should go down, ldaphost2.in.ibm.com will take over the Virtual IP Address and start the highly available services. When ldaphost1.in.ibm.com comes back online, ldaphost2.in.ibm.com will transfer (automatically failback) the Virtual IP and continue serving the

clients without any downtime.

You should familiarize yourself with the sample ha.cf file that comes with the Heartbeat package. Refer to the Listing 15: View Heartbeat's document to find out more about ha.cf options. Also see the "Resources" section for a link to the official Heartbeat documentation.

7.2.3 The haresources file

This file should be exactly the same on both the servers.

ldaphost1.in.ibm.com IPaddr::192.168.56.252/24/eth0:1/192.168.56.255

Listing 19: Sample haresources file

ldaphost1.in.ibm.com is the hostname of our first or primary node, and IPaddr::192.168.56.252/24/eth0:1/192.168.56.255 is the virtual IP address with a subnet mask 24. It will be attached to an Ethernet interface eth0:1 with a broadcast address of 192.168.56.255. See the official Heartbeat documentation to learn more about the options available.

  • 7.3 Propagating configuration files to peer servers

The Heartbeat package provides a script ha_propagate to copy the configuration files over to peer servers that are listed in the ha.cf file with a node directive. By default, the script ha_propagate copies only the two files authkeys and ha.cf. However, I have tweaked my version of ha_propagate to copy all the three files to the peer servers. See the "Resources" section to download the tweaked version of the script.

If you want to find out where the default ha_propagate script is located, then execute the following command:

# rpm -ql heartbeat | grep -i ha_propagate /usr/share/heartbeat/ha_propagate

Listing 20: ha_propagate script location

Since we have created all the three files (authkeys, ha.cf, and haresources) in ldaphost1.in.ibm.com, we need to copy them over to ldaphost2.in.ibm.com. Execute the downloaded script to copy the three configuration files. See Listing 21: ha_propagate

[root@ldaphost1 heartbeat]# pwd /usr/share/heartbeat [root@ldaphost1 heartbeat]# ./ha_propagate Propagating HA configuration files to node ldaphost2.in.ibm.com. ha.cf

100%

259

0.3KB/s

00:00

authkeys

 

100%

47

0.1KB/s

00:00

haresources

 

100%

69

0.1KB/s

00:00

Setting HA startup configuration on node ldaphost2.in.ibm.com. ..... .....

heartbeat

0:off

1:off

2:on

3:on

4:on

5:on

6:off

httpd

0:off

1:off

2:off

3:off

4:off

5:off

6:off

ip6tables

0:off

1:off

2:off

3:off

4:off

5:off

6:off

.....

.....

Listing 21: ha_propagate

When you execute the ha_propagate script, it may ask you for the password to the remote system to which the configuration files are being copied. Provide the password to continue. In the above case, since both the servers are configured for passwordless ssh, ha_propagate didn't prompt for any password.

Note: ha_propagate uses scp to copy the files to the remote server. If you have set up passwordless ssh, it will not ask you for the remote system's password.

  • 7.4 Starting Heartbeat

Now that both the servers (ldaphost1 and ldaphost2) have the same Heartbeat configuration files, you are almost ready to start the Heartbeat service on both.

7.4.1 Network configuration before starting heartbeat

Before starting Heartbeat, check your network configuration with the command line tool ifconfig. See Listing 22: ifconfig output before starting heartbeat

[root@ldaphost1 ~]# ifconfig

eth0

Link encap:Ethernet HWaddr 08:00:27:63:97:31 inet addr:192.168.56.20 Bcast:192.168.56.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe63:9731/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:27950 errors:0 dropped:0 overruns:0 frame:0 TX packets:23545 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5230588 (4.9 MiB) TX bytes:5613926 (5.3 MiB)

lo

Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:239275 errors:0 dropped:0 overruns:0 frame:0 TX packets:239275 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:34065653 (32.4 MiB) TX bytes:34065653 (32.4 MiB)

Listing 22: ifconfig output before starting heartbeat

Now start the Heartbeat service on server ldaphost1.in.ibm.com and ldaphost2.in.ibm.com.

[root@ldaphost1 ~]# service heartbeat start Starting High-Availability services: INFO: Resource is stopped Done.

[root@ldaphost2 ~]# service heartbeat start Starting High-Availability services: INFO: Resource is stopped Done.

Listing 23: Starting heartbeat

Next, check the log file /var/log/ha-log on the primary server, which is ldaphost1.in.ibm.com in this case. Refer to Listing 18: Sample ha.cf file.

Apr 20 00:26:47 ldaphost1.in.ibm.com heartbeat: [30705]: info: Pacemaker support:

false ........... Apr 20 00:26:47 ldaphost1.in.ibm.com heartbeat: [30705]: info: Configuration

validated. Starting heartbeat 3.0.4 Apr 20 00:26:47 ldaphost1.in.ibm.com heartbeat: [30706]: info: heartbeat: version

3.0.4

...........

output

omitted

...........

.................................... Apr 20 00:26:49 ldaphost1.in.ibm.com heartbeat: [30706]: info: Local status now set to: 'up' Apr 20 00:26:49 ldaphost1.in.ibm.com heartbeat: [30706]: info: Link ldaphost1.in.ibm.com:eth0 up. Apr 20 00:26:57 ldaphost1.in.ibm.com heartbeat: [30706]: info: Link ldaphost2.in.ibm.com:eth0 up. Apr 20 00:26:57 ldaphost1.in.ibm.com heartbeat: [30706]: info: Status update for node ldaphost2.in.ibm.com: status up

...........

output

omitted

...........

....................................

ResourceManager(default)[30916]:

resource group: ldaphost1.in.ibm.com

ResourceManager(default)[30916]:

2014/04/20_00:27:17 info: Acquiring

2014/04/20_00:27:21 info: Running

/etc/ha.d/resource.d/IPaddr 192.168.56.252/24/eth0:1/192.168.56.255 start

IPaddr(IPaddr_192.168.56.252)[31072]:

2014/04/20_00:27:24 INFO: Adding inet

address 192.168.56.252/24 with broadcast address 192.168.56.255 to device eth0 (with label eth0:1)

IPaddr(IPaddr_192.168.56.252)[31072]:

eth0 up

...........

output

omitted

...........

....................................

2014/04/20_00:27:24 INFO: Bringing device

/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.56.252)[31046]:

2014/04/20_00:27:25 INFO: Success Apr 20 11:06:01 ldaphost1.in.ibm.com heartbeat: [30706]: info: No pkts missing from

ldaphost2.in.ibm.com!

Listing 24: log snippet from /var/log/ha-log on ldaphost1

7.4.2

Network configuration after starting heartbeat

[root@ldaphost1 ~]# ifconfig

eth0

Link encap:Ethernet HWaddr 08:00:27:63:97:31 inet addr:192.168.56.20 Bcast:192.168.56.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe63:9731/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:34202 errors:0 dropped:0 overruns:0 frame:0 TX packets:29170 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:6692040 (6.3 MiB) TX bytes:7078873 (6.7 MiB)

eth0:1

Link encap:Ethernet HWaddr 08:00:27:63:97:31 inet addr:192.168.56.252 Bcast:192.168.56.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

lo

Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:239275 errors:0 dropped:0 overruns:0 frame:0 TX packets:239275 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:34065653 (32.4 MiB) TX bytes:34065653 (32.4 MiB)

Listing 25: ifconfig output after starting heartbeat

If you see the above ifconfig output, a new Ethernet alias eth0:1 has been added with an IP address of 192.168.56.252 as specified in the file /etc/ha.d/haresources.

  • 7.4.3 Checking Heartbeat status

You can check the status of the Heartbeat service from any server listed in the node directive as specified in the ha.cf file. Use the command line tool /usr/bin/cl_status,which is provided by the Heartbeat package. Refer to Listing 18: Sample ha.cf file.

Check the list of the servers that are configured to run heartbeat

[root@ldaphost1 ~]# cl_status listnodes

ldaphost2.in.ibm.com

ldaphost1.in.ibm.com

Check the heartbeat status on a particular server

[root@ldaphost1 ~]# cl_status hbstatus Heartbeat is running on this machine. [root@ldaphost2 ~]# cl_status hbstatus Heartbeat is stopped on this machine. [root@ldaphost1 ~]# cl_status nodestatus ldaphost1.in.ibm.com active [root@ldaphost1 ~]# cl_status nodestatus ldaphost2.in.ibm.com dead

If you see the above output, since Heartbeat is running on ldaphost1, it is shown as being "active." Server ldaphost2 is shown as being "dead."

Now start Heartbeat on server ldaphost2. See Listing 23.

Check the Heartbeat status on server ldaphost2:

[root@ldaphost2 ~]# cl_status hbstatus Heartbeat is running on this machine. [root@ldaphost1 ~]# cl_status nodestatus ldaphost2.in.ibm.com active

  • 8 Testing high-availability scenario

At this point, Heartbeat is configured on both the servers and is running successfully as we saw in the section "Checking Heartbeat status."

In order to test the high-availability scenario, we can simulate a system failure by stopping Heartbeat on one server (ldaphost1.in.ibm.com) so that the peer server (ldaphost2.in.ibm.com) will not receive any Heartbeat response and will therefore assume that the system is dead.

Stop the Heartbeat service on server ldaphost1.in.ibm.com.

[root@ldaphost1 ~]# service heartbeat stop Stopping High-Availability services: Done.

Listing 26: Stopping heartbeat

8.1.1 Checking the network configuration on ldaphost2

[root@ldaphost2 ~]# ifconfig

eth0

Link encap:Ethernet HWaddr 08:00:27:03:6B:CD inet addr:192.168.56.21 Bcast:192.168.56.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe03:6bcd/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:28666 errors:0 dropped:0 overruns:0 frame:0 TX packets:24660 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:6159029 (5.8 MiB) TX bytes:6471877 (6.1 MiB)

eth0:1

Link encap:Ethernet HWaddr 08:00:27:03:6B:CD inet addr:192.168.56.252 Bcast:192.168.56.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

lo

Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:155 errors:0 dropped:0 overruns:0 frame:0 TX packets:155 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0

RX bytes:21318 (20.8 KiB)

TX bytes:21318 (20.8 KiB)

Listing 27: ifconfig output from ldaphost2 after stopping heartbeat on ldaphost1

Check the log file /var/log/ha-log on server ldaphost2.in.ibm.com to see how the resources were acquired. In this case, the IP alias eth0:1 will be destroyed on server ldaphost1.in.ibm.com and will be created on the server ldaphost2.in.ibm.com.

Apr 20 13:02:18 ldaphost2.in.ibm.com heartbeat: [26755]: info: Received shutdown notice from 'ldaphost1.in.ibm.com'. Apr 20 13:02:18 ldaphost2.in.ibm.com heartbeat: [26755]: info: Resources being acquired from ldaphost1.in.ibm.com.

output

omitted

........ Apr 20 13:02:18 ldaphost2.in.ibm.com heartbeat: [27067]: info: local HA resource

........

acquisition completed (standby). Apr 20 13:02:18 ldaphost2.in.ibm.com heartbeat: [26755]: info: Standby resource acquisition done [foreign].

harc(default)[27093]:

status

2014/04/20_13:02:18 info: Running /etc/ha.d//rc.d/status

mach_down(default)[27110]:

2014/04/20_13:02:18 info: Taking over resource

group IPaddr::192.168.56.252/24/eth0:1/192.168.56.255

ResourceManager(default)[27137]:

resource group: ldaphost1.in.ibm.com

2014/04/20_13:02:18 info: Acquiring

IPaddr::192.168.56.252/24/eth0:1/192.168.56.255

/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.56.252)[27165]:

2014/04/20_13:02:19 INFO: Resource is stopped

ResourceManager(default)[27137]:

2014/04/20_13:02:19 info: Running

/etc/ha.d/resource.d/IPaddr 192.168.56.252/24/eth0:1/192.168.56.255 start

IPaddr(IPaddr_192.168.56.252)[27293]:

2014/04/20_13:02:19 INFO: Adding inet

address 192.168.56.252/24 with broadcast address 192.168.56.255 to device eth0

(with label eth0:1)

IPaddr(IPaddr_192.168.56.252)[27293]:

eth0 up

2014/04/20_13:02:19 INFO: Bringing device

output

omitted

........ Apr 20 13:02:19 ldaphost2.in.ibm.com heartbeat: [26755]: info: mach_down takeover complete. Apr 20 13:02:34 ldaphost2.in.ibm.com heartbeat: [26755]: WARN: node ldaphost1.in.ibm.com: is dead Apr 20 13:02:34 ldaphost2.in.ibm.com heartbeat: [26755]: info: Dead node ldaphost1.in.ibm.com gave up resources.

........

Listing 28: log snippet from /var/log/ha-log on ldaphost2 after stopping heartbeat on ldaphost1

If you see output such as that in Listing 27, a new IP alias eth0:1 has been created and the same alias has been destroyed on the server ldaphost1.in.ibm.com. Because of this floating IP address, your services will never be down as long as one of the server nodes is running. Whenever the server ldaphost1.in.ibm.com is up, resources from the secondary server ldaphost2.in.ibm.com will be acquired again (i.e., IP alias eth0:1 will be destroyed on server ldaphost2.in.ibm.com and will be created on primary server ldaphost1.in.ibm.com; the primary server will continue serving its clients).

Note: You need to remember that the ISDS client must be pointed to at its virtual IP address (i.e., 192.168.56.252 here and not the physical IP address of the system), because this IP address will be available at all times in case any of the physical servers goes down. Distributing the new IP address (192.168.56.252) will not be a good idea, as it is very difficult to remember the IP addresses. Instead, you should distribute a new hostname. At this point, however, you don't have a hostname for IP address 192.168.56.252. Read on in the next section ("DNS hack") to solve this problem.

9

DNS hack

When you start Heartbeat, it creates a new IP alias with a new virtual IP address as specified in the file /etc/ha.d/haresources. Refer to Listing 19: Sample haresources file.

Previously, we had two IP addresses: 192.168.56.20 with hostname ldaphost1.in.ibm.com and 192.168.56.21 with hostname ldaphost2.in.ibm.com. Now we have another IP address:

192.168.56.252, for which there is no DNS entry or any entry in file /etc/hosts (for Linux system).

You have two options:

You can tell your network administrator to add a new DNS entry for the new IP address (in this case 192.168.56.252) with a new hostname, or

If the servers are serving a limited number of clients, then you can make use of the local hosts file (/etc/hosts for Linux systems and %SystemRoot %\System32\drivers\etc\hosts file for Windows systems) to resolve the IP address- to-hostname issue.

The first option is recommended for organizations having a large-scale deployment. However, for this article, we are choosing the option 2 to simulate the behavior. See Listing 29: /etc/hosts file from server ldaphost1 and ldaphost2

[root@ldaphost1 ~]# cat /etc/hosts

  • 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 localhost localhost.localdomain localhost6 localhost6.localdomain6

::1

  • 192.168.56.20 ldaphost1.in.ibm.com ldaphost1

  • 192.168.56.21 ldaphost2.in.ibm.com ldaphost2

    • 192.168.56.252 ldapserver.in.ibm.com ldapserver

[root@ldaphost1 ~]# ssh ldaphost2 'cat /etc/hosts'

  • 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 localhost localhost.localdomain localhost6 localhost6.localdomain6

::1

  • 192.168.56.21 ldaphost2.in.ibm.com ldaphost2

  • 192.168.56.20 ldaphost1.in.ibm.com ldaphost1

    • 192.168.56.252 ldapserver.in.ibm.com ldapserver

Listing 29: /etc/hosts file from server ldaphost1 and ldaphost2

10 Query ISDS server using virtual IP

Now all the clients should point to the ISDS server using a virtual IP 192.168.56.252 with a hostname of ldapserver.in.ibm.com. You can perform the search operation in Listing 30: idsldapsearch using virtual IP/hostnameto check whether everything is working as it should.

[root@ldaphost2 ~]# idsldapsearch -h ldapserver.in.ibm.com -s base objectclass=* namingcontexts

namingcontexts=CN=SCHEMA

namingcontexts=CN=LOCALHOST

namingcontexts=CN=IBMPOLICIES

namingcontexts=O=SAMPLE

namingcontexts=O=IBM,C=IN

Listing 30: idsldapsearch using virtual IP/hostname

11 Resources

Learn

For assistance in installing IBM Security Directory Server, read the IBM Security Directory Server Installation and Configuration Guide.

Visit the IBM Security Directory Server Version 6.3.1 information center,where you can find information about installing, configuring, administering, and using the Directory Server.

Read "Setting up Tivoli Directory Server replication using the command line" for help with configuring replication on an IBM Security Directory Server.

Get product and technologies Get the IBM Security Directory Server Version 6.3.1 here. Download the EPEL yum repository rpm package. You can also download the source code for heartbeat at http://www.linux-ha.org/wiki/Downloads Download the tweaked version of ha_propagate script.