unixwerk: how to add a node to an hacmp cluster http://www.unixwerk.eu/aix/addnode.

html

unixwerk > AIX > add a cluster node January 3, 2013

unixwerk How to Add a No de to an HAC MP C luster

Contents
AIX
1. Introduction
BSD 2. STEP 1: Additional Boot Addresses
3. STEP 2: Import LUNs of shared volume groups into new node
HP-UX 4. STEP 3: Add the new node to the cluster
5. STEP 4: Restart cluster services on new node
Linux 6. STEP 5: Add new IP addresses to the cluster configuration
7. STEP 6: Start HACMP on new node
Solaris
8. STEP 7: Remove Dependencies between Resource Groups
9. STEP 8: Add new node to all resource groups
Others
10. STEP 9: Restore dependencies between Resource Groups
Un*x 11. STEP 10: Final synchronization of the cluster

A. Related Information
Topics

Introduction

Guestbook This article describes how to add a third node hoppy to an HACMP cluster. We find the following situation:

Contact barney# clRGinfo

-----------------------------------------------------------------------------
Archive Group Name Group State Node
-----------------------------------------------------------------------------
RES_GRP_01 ONLINE barney
OFFLINE betty

RES_GRP_02 ONLINE barney

OFFLINE betty

Conclusion: barney and betty are members of two resource groups, but all resources are currently active on
barney.

STEP 1: Additional Boot Addresses

Before we can start we need new boot IP addresses for all HACMP networks. The boot addresses have to
be assigned to the network interfaces of our new node hoppy in AIX:

hoppy# smitty chinet

[Entry Fields]
Network Interface Name en0
INTERNET ADDRESS (dotted decimal) [<boot ip address>]
Network MASK (hexadecimal or dotted decimal) [<netmask>>]
Current STATE up +
Use Address Resolution Protocol (ARP)? yes +

Note: Usually the boot addresses are not equal to the host specific ip addresses of the interfaces (such as
the hostname's ip address). They become persistent addresses instead. However, in a virtualized
environment we could use the host specific IP addresses as boot addresses and could avoid the need for

1 of 6 Saturday 06 July 2013 12:33 PM

unixwerk: how to add a node to an hacmp cluster http://www.unixwerk.eu/aix/addnode.html

persistent addresses at all.

We add the new node's boot and persistent addresses to /etc/hosts on all nodes. The old node's IP
addresses as well as all cluster service addresses have to be added to the new node's /etc/hosts.

Finally we copy /usr/es/sbin/cluster/netmon.cf to the new node:

hoppy# scp barney:/usr/es/sbin/cluster/netmon.cf /usr/es/sbin/cluster

STEP 2: Import LUNs of Shared Volume Groups into New Node

All storage assigned to the resource groups RES_GRP_01 and RES_GRP_02 has to be visible to the new
node. Be sure to set the reserve_policy to no_reserve for all shared storage on the new node:

hoppy# cfgmgr
hoppy# chdev -a reserve_policy=no_reserve hdiskN1
hoppy# chdev -a reserve_policy=no_reserve hdiskN2
hoppy# chdev -a reserve_policy=no_reserve hdiskN3
.
.
.

Now we are ready to import all shared Volume Groups into the new node.

hoppy# importvg -n -V <Major Number> -y <Volume Group> <PVID>

Get information about major number, Volume Group, and PVID from existing nodes.

STEP 3: Add the New Node to the Cluster

Now we are ready to add hoppy as new node to the cluster. We do this on one of the existing nodes.

barney# smitty hacmp

-> Extended Configuration
-> Extended Topology Configuration
-> Configure HACMP Nodes
-> Add a Node to the HACMP Cluster

[Entry Fields]
* Node Name [hoppy]
Communication Path to Node [<boot address of new node

STEP 4: Restart Cluster Services on new Node

We need to restart the cluster services on the new node. First we verify the configuration and synchronize it
to the other nodes:

barney# smitty hacmp

-> Extended Configuration
-> Extended Verification and Synchronization

clstat shows new node as down.

2 of 6 Saturday 06 July 2013 12:33 PM

unixwerk: how to add a node to an hacmp cluster http://www.unixwerk.eu/aix/addnode.html

To be sure we check /usr/es/sbin/cluster/etc/rhosts on all nodes -> all cluster addresses should be in it!

At this point the cluster services on the new node can be restarted:

hoppy# stopsrc -g cluster

hoppy# stopsrc -s clcomdES
hoppy# startsrc -s clcomdES
hoppy# startsrc -g cluster

STEP 5: Add new IP Addresses to the Cluster Configuration

First we let HACMP discover what we have...

barney# smitty hacmp

-> Extended Configuration
-> Discover HACMP-related Information from Configured Nodes

Then we add the new ip addresses to the cluster

barney# smitty hacmp

-> Extended Configuration
-> Extended Topology Configuration
-> Configure HACMP Communication Interfaces/Devices
-> Add Communication Interfaces/Devices
-> Add Pre-defined Communication Interfaces and Devices
-> Communication Interfaces
-> Select Network

[Entry Fields]
* IP Label/Address [<new boot address
* Network Type ether
* Network Name (selected network)
* Node Name []
Network Interface []

When we are done with the boot addresses we can put the new node's persistent addresses to the
interfaces:

barney# smitty hacmp

-> Extended Configuration
-> Extended Topology Configuration
-> Configure HACMP Persistent Node IP Labels/Addresses
-> Add a Persistent Node IP Label

[Entry Fields]
* Node Name hoppy
* Network Name [(selected network)]
* Node IP Label/Address [<persistent IP address
Netmask(IPv4)/Prefix Length(IPv6) []

If our persistent address is in a different network than the boot address and there are routes through the
interface (what is the classical setup) they have to be defined in the ODM on hoppy. For a default route to a
gateway 111.111.111.1 the command would be

hoppy# chdev -l inet0 -a route=net,-hopcount,0,,0,111.111.111.1 -P

3 of 6 Saturday 06 July 2013 12:33 PM

unixwerk: how to add a node to an hacmp cluster http://www.unixwerk.eu/aix/addnode.html

Again we need to synchronize the cluster

barney# smitty hacmp

-> Extended Configuration
-> Extended Verification and Synchronization

STEP 6: Start HACMP on New Node

Finally we are ready to start the cluster on the new node.

hoppy# smitty clstart

hoppy# mkdev -l inet0

this command activates the route set with STEP 5

STEP 7: Remove Dependencies between Resource Groups

Any online on same node dependencies have to be removed before we are able to add the new node to
resource groups:

barney# smitty hacmp

-> Extended Resource Configuration
-> Configure Resource Group Run-Time Policies
-> Configure Dependencies between Resource Groups
-> Configure Online on the Same Node Dependency
-> Remove Online on the Same Node Dependency Between Resource Groups

STEP 8: Add New Node to All Resource Groups

First we need to copy the application start/stop scripts to the new node (hoppy):

hoppy# scp barney:/path/to/application_servers/* /path/to/application_servers

Then we are ready to add new node to the HACMP configuration:

barney# smitty hacmp

-> Extended Configuration
-> Extended Resource Configuration
-> HACMP Extended Resource Group Configuration
-> Change/Show a Resource Group

Select resource groups one by one and add the new node to the end of the list of Participating Nodes:

[Entry Fields]
Resource Group Name (Resource Group)
New Resource Group Name []
Participating Nodes (Default Node Priority) [barney betty hoppy

Note: Really add the new node to the end of the list. Don't use the F4 selection, because the order might be

4 of 6 Saturday 06 July 2013 12:33 PM

unixwerk: how to add a node to an hacmp cluster http://www.unixwerk.eu/aix/addnode.html

wrong! To be sure we verify the node order in the resource group with

barney# clshowres | egrep 'Resource Group Name|Participating Node Name'

Resource Group Name RES_GRP_01
Participating Node Name(s) barney betty hoppy
Resource Group Name RES_GRP_02
Participating Node Name(s) barney betty hoppy

Finally we synchronize the cluster:

barney# smitty hacmp

-> Extended Configuration
-> Extended Verification and Synchronization

STEP 9: Restore Dependencies between Resource Groups

If you had to remove any dependencies between resource groups in STEP 7 now it's time to add the
dependencies again:

barney# smitty hacmp

-> Extended Resource Configuration
-> Configure Resource Group Run-Time Policies
-> Configure Dependencies between Resource Groups
-> Configure Online on the Same Node Dependency
-> Add Online on the Same Node Dependency Between Resource Groups

Use <F4> to get a list of known resource groups.

STEP 10: Final Synchronization of the Cluster

Now synchronize the cluster...

barney# smitty hacmp

-> Extended Configuration
-> Extended Verification and Synchronization

and clRGinfo shows three nodes:

barney# clRGinfo
-----------------------------------------------------------------------------
Group Name Group State Node
-----------------------------------------------------------------------------
RES_GRP_01 ONLINE barney
OFFLINE betty
OFFLINE hoppy

RES_GRP_02 ONLINE barney

OFFLINE betty
OFFLINE hoppy

A. Related Information

5 of 6 Saturday 06 July 2013 12:33 PM

unixwerk: how to add a node to an hacmp cluster http://www.unixwerk.eu/aix/addnode.html

AIX 7.1 Information Center > PowerHA SystemMirror > HACMP Version 6.1 > Adding a cluster node
to the HACMP configuration

Also on unixwerk
HACMP: Cluster Commandline
How to Add a New VG to an Active HACMP Resource Group
How to Add a New Resource Group to an Active Cluster
How to Remove a Node from an HACMP Cluster
Setup a Two-Node Cluster with HACMP

(c) unixwerk Content is available under the BSD Documentation License.

6 of 6 Saturday 06 July 2013 12:33 PM