IP Multipathing..................................................................................1-1 Objectives ........................................................................................... 1-1 Introducing IPMP .............................................................................. 1-2 Self-Test ............................................................................................... 1-3 Hardware and Software Requirements...........................................

2-1 Objectives ........................................................................................... 2-1 Supported Hardware......................................................................... 2-2 Supported Software........................................................................... 2-3 Self-Test ............................................................................................... 2-4 Data Path...........................................................................................3-1 Objectives ........................................................................................... 3-1 The ifconfig Output ....................................................................... 3-2 Standby Configuration...................................................................... 3-3 Example Failure ....................................................................... 3-5 Multipath Configuration................................................................... 3-7 Self-Test ............................................................................................. 3-10 Control Path ......................................................................................4-1 Objectives ........................................................................................... 4-1 The ifconfig Subcommands.......................................................... 4-2 The if_mpadm Command ................................................................. 4-3 Alternate Pathing and Dynamic Reconfiguration Interaction With IPMP .......................................................... 4-3 The in.mpathd Daemon................................................................... 4-4 Multipathing Configuration File ............................................ 4-5 Self-Test ............................................................................................... 4-7 Installation.........................................................................................5-1 Objectives ........................................................................................... 5-1 Checking the IPMP Package............................................................. 5-2 Configuration ................................................................................... 6-1 Objectives ........................................................................................... 6-1 System Preparation............................................................................ 6-2 Standby Configuration...................................................................... 6-3 Performing a Manual Configuration With IPv4................... 6-3 Performing an Automatic Configuration With IPv4 ........... 6-4 Performing a Manual Configuration With IPv6................... 6-6 Performing an Automatic Configuration With IPv6 ........... 6-7 Multipath Configuration................................................................... 6-8 Performing a Manual Multipath Configuration With IPv4................................................................................. 6-8 Performing an Automatic Multipath Configuration With IPv4................................................................................. 6-9 Performing a Manual Multipath Configuration With IPv6.............................................................................. 6-12 Performing an Automatic Multipath Configuration With IPv6............................................................................... 6-12 Systems Administration Tasks....................................................... 7-1 Objectives ........................................................................................... 7-1 Stopping and Starting the in.mpathd Daemon........................... 7-2 Automatically Starting the in.mpathd Daemon.......................... 7-2 Self-Test ............................................................................................... 7-3 Message Logs and Errors ............................................................... 8-1

IPMP HOWTO

Objectives ........................................................................................... 8-1 Messages.............................................................................................. 8-2 The in.mpathd Diagnostics ............................................................. 8-3 Self-Test ............................................................................................... 8-5 Troubleshooting and Tuning .......................................................... 9-1 Objectives ........................................................................................... 9-1 Debug Mode for the in.mpathd Daemon ....................................... 9-2 The snoop IP Packets......................................................................... 9-4 Tracing Routing With the netstat Command................................. 9-7 Self-Test ............................................................................................... 9-9 IPMP Lab......................................................................................... 10-1 Objectives: ........................................................................................ 10-1 Exercise: Configuring IPMP........................................................... 10-2 Preparation.............................................................................. 10-3 Task 1 Create an IPMP Standby IPMP Configuration....... 10-3 Preparation.............................................................................. 10-4 Task 2 Create a Multipath IPMP Configuration................. 10-4 Task 3 Verify Load Spreading.............................................. 10-5 Task 4: Verify an IPMP Failover and Failback Operation .............................................................................. 10-5 Exercise Summary............................................................................ 10-6

IP Multipathing
Objectives
Upon completion of this module, you should be able to: Describe IP multipathing (IPMP) Define load spreading Introducing IPMP

Introducing IPMP
IPMP provides the Solaris operating environment (Solaris OE) with the ability to recover from single point failures with a network adapter, and also increased throughput when multiple network adapters are connected to the same IP link (for example, the same ethernet switch). If a failure occurs in the network adapter, and if an alternate adapter has been configured, the system fails over, with all the network access changing automatically from the failed adapter to the alternate adapter allowing uninterrupted access to the network. Also, when there are multiple network adapters connected to the IP link, increased throughput will be achieved by spreading the load across multiple network adapters. Following are the summary of features available from IP multipathing: Outbound load spreading Outbound network packets are spread across multiple network adapters without affecting the ordering of packets to achieve higher throughput. The load spreading is done only when the server is talking to more than one client. Failure detection Ability to detect when a network adapter has failed, and automatically failing over the network access to an alternate network adapter. This assumes that an alternate network adapter was configured. Repair detection Ability to detect when a network adapter that failed previously has been repaired and automatically failing back the network access to an alternate network adapter. This process assumes that failbacks are enabled.

Hardware and Software Requirements

Objectives
Upon completion of this module, you should be able to: List supported hardware List software requirements List limitations of the Solaris operating environment (Solaris OE) implementation Supported Hardware

Supported Hardware

IP Multipathing (IPMP) is supported on both the SPARC and Intel Architecture (IA). Only similar interfaces are supported in a multipathing group. For example, you cannot mix Token Ring and Ethernet in the same multipath group. You can mix different Ethernets, such as le, hme, ge, and qfe, in one multipath group. Asynchronous Transfer Mode (ATM) is only supported in LAN Emulation (LANE) mode. On SPARC, the OpenBoot PROM (OBP) local-mac-address? variable must be set to true if multipathing is being used.

Supported Software

IPMP is supported on the Solaris 8 OE Update 2 (10/00). Currently there is no backport to previous versions of the Solaris OE. The /sbin/in.mpathd daemon detects failed interfaces. The daemon is started with the ifconfig utility when the subcommand group is used. IPMP is configured with the ifconfig utility. There are a few new sub-commands in the ifconfig utility to configure multipathing. Configuration with the ifconfig utility and in.mpathd daemon is described in detail in Unit 6, Configuration.

Self-Test

1. Which platform type is supported by IPMP? A. SPARC only B. Intel only C. Sun4u D. SPARC and Intel architecture Answer: D. SPARC and Intel architecture

Objectives
Upon completion of this module, you should be able to: Identify test address interfaces Identify multipath configuration Identify standby configuration Identify possible paths for multipath Internet Protocol (IP) datagrams between client and server. Diagram IP failover data flow between client and server

The ifconfig output has new flags and information to identify if

multipathing is being used. The flags are: DEPRECATED, NOFAILOVER, STANDBY, INACTIVE, and FAILED. Other information given by this output is groupname, which should be a non-null name to identify the physical group name to which the interface belongs. A description of the flags for ifconfig output follows: DEPRECATED The interface address should not be preferred as the source address for outbound packets. If any non-deprecated addresses are present, they are preferred over deprecated. The interface still responds to pings and can make a connection. NOFAILOVER The interface address does not fail over when a failed interface is detected. STANDBY The interface is marked as a standby interface for the physical interface group. INACTIVE Use this flag with STANDBY, and it indicates that the interface is not currently being used. FAILED This flag marks the interface as failed, and normal traffic is re-routed to another interface (if possible).

Standby Configuration

The following is an example of partial ifconfig -a output produced when a system is set up with a standby interface: hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255 groupname standby-test ether 8:0:20:b3:22:f2 hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4, NOFAILOVER> mtu 1500 index 2 inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255 qfe0: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4, NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index 3 inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255 groupname standby-test ether 8:0:20:b7:41:d8 The hme0 interface has the primary IP address for this system, and it is designated 172.20.4.181. The hme0 interface is a part of the physical interface group called standby-test. The hme0:1 flags DEPRECATED and NOFAILOVER indicate that this interface is not to be used by a normal application and the IP address will not be moved if the interface is detected as having failed. Standby Configuration qfe0 interface is the standby interface and is only used when there is a detected failure on the hme0 interface72.20.4.183
hme0 IP hme0:1 qfe0 172.20.4.183 group standby-test deprecated -failover standby # cat /etc/hostname.qfe0 172.20.4.181 group standby-test up \ addif 172.20.4.182 -failover deprecated up # cat /etc/hostname.hme0 lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1

inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255 groupname standby-test ether 8:0:20:b3:22:f2 hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2 inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255 qfe0: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER, STANDBY,INACTIVE> mtu 1500 index 3 inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255 # ifconfig -a

Standby Configuration This is an example of a failure on the hme0 interface, which is a partial listing of the output from the ifconfig -a command: hme0: flags=19000842<BROADCAST,RUNNING,MULTICAST,IPv4, NOFAILOVER,FAILED> mtu 0 index 2 inet 0.0.0.0 netmask 0 groupname standby-test ether 8:0:20:b3:22:f2 hme0:1: flags=19040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4, NOFAILOVER,FAILED> mtu 1500 index 2 inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255 qfe0: flags=29040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4, NOFAILOVER,STANDBY> mtu 1500 index 3 inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255 groupname standby-test ether 8:0:20:b7:41:d8 qfe0:1: flags=21000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,STANDBY> mtu 1500 index 3 inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.25 The preceding example shows how the hme0 interface has failed and now has 0.0.0.0 as its Internet address. There is an new logical interface on the qfe0 interface. It is qfe0:1 and now has the internet address of hme0 (127.20.4.181). All connections that were on hme0 transferred to qfe0:1. Notice the flags on hme0 are FAILED and NOFAILOVER and the flag on hme0:1 is FAILED. Note The reason that hme0 has 0.0.0.0 as the address is because hme0 is the name of an interface and also represents the zeroth address (for example, the hme0 interface is actually the hme0:0 interface). To address the interface as hme0 (for example in ioctls), even after a failover, you need to have a zeroth address. Standby Configuration When the in.mpathd daemon detects that the hme0 interface has been repaired, failback occurs and the routes and interfaces return to a normal configuration. The messages in the following output show that link down message (Line 1) of the original failure, to the failback (Line 9), to the original interface: 1 Aug 2 11:54:44 proto181 hme: SUNW,hme0 : No response from Ethernet network : Link down -- cable problem? 2 Aug 2 11:54:52 proto181 in.mpathd[31]: NIC failure detected on hme0 3 Aug 2 11:54:52 proto181 in.mpathd[31]: Successfully failed over from NIC hme0 to NIC qfe0

4 Aug 2 11:54:55 proto181 hme: SUNW,hme0 : No response from Ethernet network : Link down -- cable problem? 5 Aug 2 11:56:35 proto181 last message repeated 9 times 6 Aug 2 11:56:36 proto181 hme: SUNW,hme0 : Internal Transceiver Selected. 7 Aug 2 11:56:36 proto181 hme: SUNW,hme0 : Auto-Negotiated 100 Mbps Half-Duplex Link Up 8 Aug 2 11:56:51 proto181 in.mpathd[31]: NIC repair detected on hme0 9 Aug 2 11:56:51 proto181 in.mpathd[31]: Successfully failed back to NIC hme0

Multipath Configuration
The following code example, which is a partial listing of the ifconfig -a command, shows a system that is set up with a multipath interface: hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.25 groupname multipath-test ether 8:0:20:b3:22:f2 hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4, NOFAILOVER> mtu 1500 index 2 inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255 qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255 groupname multipath-test ether 8:0:20:b7:41:d8 qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4, NOFAILOVER> mtu 1500 index 3 inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.25 The hme0 interface, which is normally running a 172.20.4.181 IP address, has a physical interface group name multipath-test. The hme0:1 interface is the test address for the hme0 interface with an 172.20.4.182 IP address. Its flags are DEPRECATED and NOFAILOVER and are special for multipathing. No application should select this as a source address, and the address will not be failed over. Caution This does not prevent someone from connecting to the test address from another host. It is still a legal address that can be used but the address will not failover. The qfe0 interface, which is normally running a 172.20.4.183 IP address, has a physical interface group name multipath-test. The qfe0:1 interface is the test address for the hme0 interface with an 172.20.4.184 IP address. Its flags are also DEPRECATED and NOFAILOVER so no application should select this as a source address, and the address will not fail over. The hme0 interface address can be relocated to the qfe0 interface if there is a failed interface detected. The reverse is true if the qfe0 interface fails then, the hme0 interface takes over. Multipath Configuration

You can use the qfe0 (172.20.4.183) and the hme0 (172.20.4.181)

addresses as a usable source addresses for applications from this host. For instance, a telnet session can have hme0 address as its source address to one host, and during the next telnet session, it might have the address of the qfe0 as the source address to another host. Figure 3-2 shows the network when it is configured for IP Multipathing (IPMP). There are four possible paths that IPMP can use with IP traffic to and from this host with two interfaces. If there are more interfaces, then more possible paths are added: In/out on hme0 In/out on qfe0 In on hme0/out on qfe0 In on qfe0/out on hme0
IP 172.20.4.181 group multipath-test up \ addif 172.20.4.182 -failover deprecated up 172.20.4.183 group multipath-test up \ addif 172.20.4.184 -failover deprecated up # cat /etc/hostname.hme0 lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255 groupname multipath-test ether 8:0:20:b3:22:f2 hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2 inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255 qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255 groupname multipath-test ether 8:0:20:b7:41:d8 qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3 inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.255 # ifconfig -a

Multipath Configuration Logical interfaces are also supported on IPMP. All connections work as normal. There is a change in source address selection in Solaris 8 OE Update 2. Now, the Solaris OE uses the inbound packet destination address as the source address of the outbound packet. This means that packets that come in on a certain IP address now leave with that IP address as its source address and not the primary physical address interface as in previous releases of the Solaris OE. 1. Which flag is not new to the ifconfig utility in the Solaris 8 OE Update 2? A. DEPRECATED B. NOFAILOVER C. STANDBY D. INACTIVE E. FAILED F. IPv4 Answer: F. IPv4

Control Path
Objectives
Upon completion of this module, you should be able to: Locate the configuration file Identify parameters in the configuration file Look up possible values and actions for each parameter The ifconfig Subcommands

The ifconfig Subcommands

The ifconfig(1M) tool configures Internet Protocol (IP) Multipathing (IPMP). Several new sub-commands were added for use with IPMP: deprecated Do not use the interface with normal IP traffic. failover The interface address does not fail over when a failed interface is detected. group Use this sub-command to create an physical interface group. standby - Use this sub-command to mark an interface as a standby for the physical interface group. addif - Use this sub-command to create the next unused logical interface on the specified physical interface. This is a sub-command new to Solaris 8 Operating Environment (Solaris 8 OE) FCS. The if_mpadm Command .

Alternate Pathing and Dynamic Reconfiguration Interaction With IPMP

Currently, there is no support for using IPMP concurrently with Alternate Pathing (AP) on the same network interfaces. You can use AP for supported storage arrays and use IPMP for your network interfaces. Currently, IPMP does not automatically fail over an active network interface when a Dynamic Reconfiguration operation is initiated. The system administrator must manually issue the following command: /usr/sbin/if_mpadm -d network_interface_name /usr/sbin/if_mpadm -r network_interface_name where: The -d option detaches the network interface or to take it offline The -r option reattaches a previously detached network interface. network_interface_name is the network interface to be managed. The if_mpadm command allows administrators to manually force IPMP to stop using a redundant network interface and then make it possible to use Dynamic Reconfiguration to remove the network interface card. This feature of IPMP is in the Solaris 8 OE Update 3 (1/01). The in.mpathd Daemon

The in.mpathd Daemon

The in.mpathd multipathing daemon detects failures and repairs by sending out probes on all the interfaces that are part of a group. When an interface is part of a group and has a test address, the daemon starts sending out probes for determining failures on that interface. If there are no replies to five consecutive probes, the interface is considered to have failed. The probing rate depends on the failure detection time. By default,

failure detection time is 10 seconds. Thus, the probing rate is one probe every two seconds. To avoid synchronization in the network, probing is not periodic. If five consecutive probes fail, the in.mpathd daemon considers the interface as failed and performs a failover of the network access from the failed interface to another interface in the group that is functioning properly. If a standby interface is configured, it is chosen for failover of the IP addresses, and broadcasts and multicasts memberships. If no standby interface exists, the interface with the least number of IP addresses is chosen. Because the in.mpathd daemon determines what targets to probe dynamically, you cannot configure the targets. Routers connected to the link are chosen as targets for probing. If no routers exist on the link, arbitrary hosts on the link are chosen. A multicast packet sent to the all hosts multicast address (224.0.0.1 in Internet Protocol version 4 [IPv4] and ff02::1 in Internet Protocol version 6 [IPv6]) determines the arbitrary hosts. The first few hosts that respond to the echo packets are chosen as targets for probing. If the in.mpathd daemon cannot find routers or hosts that responded to ICMP echo packets, the in.mpathd daemon cannot detect failures. The in.mpathd daemon detects an NIC failure and repair by sending and receiving ICMP echo requests and replies on each NIC. The in.mpathd daemon sends the probes to on-link routers. If no routers are available, it sends the probes to neighboring hosts. Thus, for network failure detection and repair, there must be at least one neighbor on each link that responds to ICMP echo request probes. The in.mpathd daemon works on both IPv4 and IPv6. If IPv4 is plumbed on a NIC, an IPv4 test address is configured on the NIC, and the NIC is configured as part of a network multipathing group, then the in.mpathd daemon starts sending ICMP probes on the NIC using IPv4. In the case of IPv6, you must configure the link-local address as the test address. The in.mpathd daemon does not accept a non-link-local address as a test address. If the NIC is part of a multipathing group and the test address has been configured, then the in.mpathd daemon probes the NIC for failures using IPv6. Even if both the IPv4 and IPv6 protocol streams are plumbed, it is sufficient to configure only one of the two, that is, either an IPv4 test address or an IPv6 test address on a NIC. If only an IPv4 test address is configured, it probes using only ICMPv4. If only an IPv6 test address is configured, it probes using only ICMPv6. If both type test addresses are configured, it probes using both ICMPv4 and ICMPv6.

Multipathing Configuration File

The in.mpathd daemon uses the settings in the /etc/default/mpathd configuration file to invoke multipathing. Changes to this file are read by the in.mpathd daemon at startup and on a SIGHUP signal. This file

contains the following default settings and information: # # Time taken by mpathd to detect a NIC failure in ms. The minimum time # that can be specified is 100 ms. # FAILURE_DETECTION_TIME=10000 # # Failback is enabled by default. To disable failback turn off this option # FAILBACK=yes # # By default only interfaces configured as part of multipathing groups # are tracked. Turn off this option to track all network interfaces # on the system # TRACK_INTERFACES_ONLY_WITH_GROUPS=yes The in.mpathd Daemon For more information on configuring the /etc/default/mpathd configuration file, refer to the IP Multipathing Administration Guide for Solaris 8 OE and for Solaris 9 see Solaris 9 Systems Administration Guide: IP Services.

Failure Detection Time

You can set a lower value of failure detection time. Sometimes these values might not be achieved if the load on the network is too high. Then, the in.mpathd daemon prints a message on the console, indicating that the time cannot be met. The daemon also prints the time that it can meet currently. If the response comes back correctly, the in.mpathd daemon meets the failure detection time provided in this file.

Failback
After a failover, failbacks take place when the failed interface is repaired. However, the in.mpathd daemon does not fail back the interface if the FAILBACK flag is set to no.

Tracking Interfaces Only With the groups Option

By turning off the groups option, the in.mpathd daemon tracks all interfaces in the system. When a failure is detected, an appropriate message is logged on the console. For this option to function properly, Ethernet addresses on all the interfaces must be unique.

Installing IPMP
Checking the IPMP Package
Internet Protocol Multipathing (IPMP) does not require special installation. It is installed with the Solaris OE core installation. To check for the package, use the pkginfo command: % pkginfo | grep -i csr system SUNWcsr Core Solaris, (Root)

System Preparation
These conditions must exist before Internet Protocol Multipathing (IPMP)

can be used: The set of STREAMS modules on all network adapters in the multipathing group must be the same. If you have plumbed Internet Protocol version 4 (IPv4) on one network adaptor, then you must plumb IPv4 on all network adaptors in the multipathing group. When you plumb a device, you open the device associated with the physical interface name and set up the streams needed for IP to use the device. When used with a logical interface name, this command creates a a specific named logical interface. An interface must be separately plumbed for use by IPv4 and IPv6. The address_family parameter controls whether the ifconfig command applies to IPv4 or IPv6. If you have plumbed Internet Protocol version 6 (IPv6) on one network adapter, then you must plumb IPv6 on all network adaptors in the multipathing group. All Ethernet network adapters in the system should have unique Media Access Control (MAC) addresses. This is achieved by setting the local-mac-address? parameter to true in the OpenBoot PROM (OBP) for SPARC platforms. Nothing needs to be done for Intel architecture (IA) platforms. All network adapters of the multipathing group must be connected to the same IP link. The multipathing group should not contain dissimilar interfaces. The interfaces that are grouped together should be of the same interface type that is defined in the /usr/include/net/if_types.h file. For example, you cannot combine Ethernet with Token Ring, and you cannot combine a Token bus with asynchronous transfer mode (ATM). When you use IP network multipathing with ATM, you must configure the ATM for LAN emulation (multipathing over classical IP instances is not currently supported).

Performing a Manual Configuration With IPv4

To perform a manual configuration with IPv4, follow these steps: 1. Assume that the primary interface (hme0, 172.20.4.181) is running. 2. Make sure that the local-mac-address? parameter is set to true. # eeprom local-mac-address? local-mac-address?=true 3. Join an interface group called standby-test. # ifconfig hme0 group standby-test 4. Create a test address (172.20.4.182) for hme0. # ifconfig hme0 addif 172.20.4.182 deprecated -failover netmask + broadcast + up 5. Use the plumb command on the second interface. # ifconfig qfe0 plumb 6. Join interface group standby-test. # ifconfig qfe0 group standby-test 7. Configure qfe0 as a standby with test address of 172.20.4.183. # ifconfig qfe0 addif 172.20.4.183 deprecated -failover standby netmask + broadcast + up 8. Check your work.

# ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255 groupname standby-test ether 8:0:20:b3:22:f2 hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4, NOFAILOVER> mtu 1500 index 2 inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255 qfe0: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4, NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index 3 inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255 groupname standby-test

Performing an Automatic Configuration With IPv4

To make the configuration a part of the boot process, configure the /etc/hostname.hme0 and /etc/hostname.qfe0 files similar to the following example: # cat /etc/hostname.hme0 172.20.4.181 group standby-test up addif 172.20.4.182 -failover deprecated up # cat /etc/hostname.qfe0 172.20.4.183 group standby-test deprecated -failover standby The system takes care of the netmask and broadcast as a part of startup. Standby Configuration

Performing a Manual Configuration With IPv6

To perform a manual configuration with IPv6, follow these steps: 1. Make sure that the MAC addresses on your interfaces are unique. If the interfaces are not already members of an interface group, they need to join one. Again, assuming that IPv6 addresses are already configured, issue the following commands: # ifconfig hme0 group standby-test # ifconfig qfe0 group standby-test172.20.4.184
172.20.4.254 cat # /etc/hostname.qfe0 172.20.4.181 group multipath-test up \ addif 172.20.4.182 -failover deprecated up 172.20.4.183 group multipath-test up \ addif 172.20.4.184 -failover deprecated up # cat /etc/hostname.hme0 lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255 groupname multipath-test ether 8:0:20:b3:22:f2 hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2 inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255 qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255 groupname multipath-test ether 8:0:20:b7:41:d8

qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3 inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.255 # ifconfig -a

2. When the interfaces are joined, set up qfe0 as the standby interface: # ifconfig qfe0 inet6 standby -failover Caution You must not mark IPv6 link-local addresses as DEPRECATED .

Performing an Automatic Configuration With IPv6

To make the configuration a part of the boot process, configure the /etc/hostname6.hme0 and /etc/hostname6.qfe0 files similar to the following example: # cat /etc/hostname6.hme0 group standby-test -failover up # cat /etc/hostname6.qfe0 group standby-test -failover standby up

Performing a Manual Multipath Configuration With IPv4

To perform a manual multipath configuration with IPv4, follow these steps: 1. Assume that the primary interface (hme0, 172.20.4.181) is running. Make sure that local-mac-address? is set to true. # eeprom local-mac-address? local-mac-address?=true 2. Join an interface group called multipath-test. # ifconfig hme0 group multipath-test 3. Create a test address (172.20.4.182) for hme0. # ifconfig hme0 addif 172.20.4.182 deprecated -failover netmask + broadcast + up The second interface is already running with the IP address 172.20.4.183. 4. Join interface group multipath-test. # ifconfig qfe0 group multipath-test 5. Configure qfe0 as a standby with test address of 172.20.4.184. # ifconfig qfe0 addif 172.20.4.184 deprecated -failover netmask + broadcast + up 6. Check your work. # ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255 groupname multipath-test ether 8:0:20:b3:22:f2 hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAIL OVER> mtu 1500 index 2 inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255 qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

Multipath Configuration inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255 groupname multipath-test ether 8:0:20:b7:41:d8 qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAIL OVER> mtu 1500 index 3 inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.255

Performing an Automatic Multipath Configuration With IPv4

To make the configuration a part of the boot process, configure the /etc/hostname.hme0 and /etc/hostname.qfe0 files similar to the following example: # cat /etc/hostname.hme0: 172.20.4.181 group multipath-test up addif 172.20.4.182 -failover deprecated up # cat /etc/hostname.qfe0: 172.20.4.183 group multipath-test up addif 172.20.4.184 -failover deprecated up Multipath Configuration

hme0 hme0:1 qfe0 qfe0:# cat /etc/hostname.qfe0 172.20.4.181 group multipath-test up \ addif 172.20.4.182 -failover deprecated up 172.20.4.183 group multipath-test up \ addif 172.20.4.184 -failover deprecated up # cat /etc/hostname.hme0 lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255 groupname multipath-test ether 8:0:20:b3:22:f2 hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2 inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255 qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255 groupname multipath-test ether 8:0:20:b7:41:d8 qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3 inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.255 # ifconfig -a Multipath Configuration172.20.4.254 cat # /etc/hostname.qfe0 172.20.4.181 group multipath-test up \ addif 172.20.4.182 -failover deprecated up 172.20.4.183 group multipath-test up \ addif 172.20.4.184 -failover deprecated up # cat /etc/hostname.hme0 lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255 groupname multipath-test ether 8:0:20:b3:22:f2

hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2 inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255 qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255 groupname multipath-test ether 8:0:20:b7:41:d8 qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3 inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.255 # ifconfig -a

Performing a Manual Multipath Configuration With IPv6

To perform a manual multipath configuration with IPv6, follow these steps: 1. Make sure the MAC addresses are unique. If the interfaces are not already members of an interface group, they need to join one. Again assuming that IPv6 addresses are already configured, issue the following command: # ifconfig hme0 group multipath-test # ifconfig qfe0 group multipath-test 2. When the interfaces are joined, hme0 and qfe0 can be set up for multipath: # ifconfig hme0 inet6 -failover # ifconfig qfe0 inet6 -failover Caution You must not mark IPv6 link-local addresses as DEPRECATED.

Performing an Automatic Multipath Configuration With IPv6

To make the configuration a part of the boot process, configure the /etc/hostname6.hme0 and /etc/hostname6.qfe0 files similar to the following example: # cat /etc/hostname6.hme0 group multipath-test -failover up # cat /etc/hostname6.qfe0 group multipath-test -failover up

Stopping and Starting the in.mpathd Daemon

To start the in.mpathd daemon by manually, issue the following command as root: # /sbin/in.mpathd To start the in.mpathd daemon with debugging turned on: # /sbin/in.mpathd -d Note The -d flag is an undocumented command-line option.

Automatically Starting the in.mpathd Daemon

The multipath daemon does not have a start and stop script in the /etc/init.d directory. The in.mpathd daemon is executed from the ifconfig command when using the group sub-command. The in.mpathd daemon does not create multiple incarnations. Self-Test in.mpathd messages in the /var/adm/messages file, issue the

following command as root: # grep mpathd /var/adm/messages*

This section describes the diagnostics associated with the in.mpathd daemon. IFF_NOFAILOVER address The address is not unique, and failure detection is not possible. Every test address that is configured must be unique on the system. Otherwise, the in.mpathd daemon cannot perform failure detection. Because the IPv6 test address is a link-local address, which in turn is derived from the Ethernet address, each network interface card (NIC) must have a unique MAC address. Also, when multiple NICs are connected to the same subnet, one should have a unique Media Access Control (MAC) address to avoid duplicate packets being picked up by the interface. NIC interface_name of group group_name is not plumbed for IPv[4|6] and may affect failover capability All NICs in a multipathing group must be homogeneously plumbed. For example, if a NIC is plumbed for IPv4, then all NICs in the group must be plumbed for IPv4. The streams modules pushed on all NICs must be identical. Failures cannot be detected on interface_name as no IFF_NOFAILOVER address is available Every NIC that is configured as part of a multipathing group must have a test address, which can be either IPv4, IPv6, or both. Otherwise, the in.mpathd daemon cannot probe the NIC, and the preceding message appears. Invalid failure detection time assuming default 10000 An invalid value was encountered for FAILURE_DETECTION_TIME in the /etc/default/mpathd file. Too small failure detection time of time assuming minimum 100 The minimum value that can be specified for FAILURE_DETECTION_TIME is currently 100 milliseconds. Invalid value for FAILBACK value Valid values for the boolean variable FAILBACK are yes or no. The in.mpathd Diagnostics

in.mpathd Diagnostics

Invalid value for TRACK_INTERFACES_ONLY_WITH_GROUPS value Valid values for the Boolean variable TRACK_INTERFACES_ONLY_WITH_GROUPS are yes or no. Cannot meet requested failure detection time of time ms on (inet[4|6] interface_name) new failure detection is time ms The round trip time for ICMP probes is higher than the specified failure detection time. The network is probably congested or the probe targets are loaded. The in.mpathd daemon automatically increases the failure detection time to whatever it can achieve under these conditions. Improved failure detection time time ms The round trip time for ICMP probes has now decreased and the in.mpathd daemon has lowered the failure detection time correspondingly.

NIC failure detected on interface_name The in.mpathd daemon has detected NIC failure on interface_name, and has set the IFF_FAILED flag on NIC interface_name. Successfully failed over from NIC interface_name1 to NIC interface_name2 The in.mpathd daemon has caused the network traffic to failover from NIC interface_name1 to NIC interface_name2, which is part of the multipathing group. NIC repair detected on interface_name The in.mpathd daemon has detected that NIC interface_name is repaired and operational. If the IFF_FAILED flag on the NIC was previously set, it will be reset. Successfully failed back to NIC interface_name The in.mpathd daemon has restored network traffic back to NIC interface_name, which is now repaired and operational.

Debug Mode for the in.mpathd Daemon

1. Place the in.mpathd daemon into debug mode, type: /sbin/in.mpath -d Caution Debug mode is undocumented. Debugging output is considered unstable and scripts or tools should not depend on it to be the same from release to release. The following is sample output from debug mode: run_timeouts(inet qfe0): next scheduled for this phyint inst 997, next scheduled global 997 ms phyint_inst_timer(inet hme0) cur_time 159084781 snxt_time 159085886 snxt_basetime 159086211 interval 1105 probe(inet hme0 4660 159084781) run_timeouts(inet hme0): next scheduled for this phyint inst 1105, next scheduled global 997 ms run_timeouts: 997 ms timer_schedule(997) timer_schedule(997): sec 0 usec 997000 in_data(inet hme0) incoming_echo_reply: inet hme0 172.20.4.254 seq 2 pi_set_crtt: target - m 2 in_signal() got 14 in_signal(SIGALRM) delta 1 run_timeouts() phyint_inst_timer(inet qfe0) cur_time 159085780 snxt_time 159086917 snxt_basetime 159087423 interval 1137 probe(inet qfe0 4660 159085780) run_timeouts(inet qfe0): next scheduled for this phyint inst 1137, next scheduled global 1137 ms 2. Print statistics either in normal daemonize or debug modes. Send a USR1 signal to the daemon, type: # pkill -USR1 mpath Debug Mode for the in.mpathd Daemon The following is sample output with an explanation of the following terms:

Probe stats on (inet qfe0) Number of probes sent 4142 Number of probe acks received 4142 Number of probes/acks lost 0 Number of valid unacknowled probes 0 Number of ambiguous probe acks received 0 Probe stats on (inet hme0) Number of probes sent 4142 Number of probe acks received 4142 Number of probes/acks lost 0 Number of valid unacknowled probes 0 Number of ambiguous probe acks received 0 Note These statistics are on a per interface, per protocol basis. Number of probes The total number of ICMPv4 echo requests sent out on interface after the in.mpathd daemon was started until now. Number of probe acks received The total number of ICMPv4 echo replies received in response to the preceding echo requests. Number of probes/acks lost The number of echo requests that did not get a reply. Number of valid unacked probes The number of echo requests to which you have not received replies and which have not yet timed out. The probe is considered timed out if more than one RTT has elapsed since it was sent. Number of ambiguous probe acks If an echo reply is received long after it is sent, that is after the circular buffer has wrapped around, and you cannot determine anything about the packet, it is considered ambiguous. The same applies to unsolicited echo replies; for example, a duplicate. The snoop IP Packets

The snoop IP Packets

The hme0 and qfe0 interfaces are configured to use multipathing. # ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255 groupname multipath-test ether 8:0:20:b3:22:f2 hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAIL OVER> mtu 1500 index 2 inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255 qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255 groupname multipath-test ether 8:0:20:b7:41:d8 qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4, NOFAILOVER> mtu 1500 index 3 inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.255 The test address is 172.20.4.182 on hme0:1. The snoop command was

taken on the hme0 interface and filtering on ICMP packets. The source Ethernet address matches hme0, and source IP address is 172.20.4.182. This ICMP is directed at a router. For the test address, 172.20.4.184 on qfe0:1, you should see a similar packet with the ethernet source address of the qfe0 interface and source IP address of 172.20.4.184: 10 ETHER: ----- Ether Header ----11 ETHER: 12 ETHER: Packet 1 arrived at 15:28:27.19 13 ETHER: Packet size = 50 bytes 14 ETHER: Destination = 0:d0:d3:35:46:d0, 15 ETHER: Source = 8:0:20:b3:22:f2, Sun 16 ETHER: Ethertype = 0800 (IP) 17 ETHER: 18 IP: ----- IP Header ----19 IP: 20 IP: Version = 4 21 IP: Header length = 20 bytes 22 IP: Type of service = 0x00 The snoop IP Packets 23 IP: xxx. .... = 0 (precedence) 24 IP: ...0 .... = normal delay 25 IP: .... 0... = normal throughput 26 IP: .... .0.. = normal reliability 27 IP: Total length = 36 bytes 28 IP: Identification = 40936 29 IP: Flags = 0x4 30 IP: .1.. .... = do not fragment 31 IP: ..0. .... = last fragment 32 IP: Fragment offset = 0 bytes 33 IP: Time to live = 1 seconds/hops 34 IP: Protocol = 1 (ICMP) 35 IP: Header checksum = 7814 36 IP: Source address = 172.20.4.182, proto182.Central.Sun.COM 37 IP: Destination address = 172.20.4.254, protorouter.Central.Sun.COM 38 IP: No options 39 IP: 40 ICMP: ----- ICMP Header ----41 ICMP: 42 ICMP: Type = 8 (Echo request) 43 ICMP: Code = 0 (ID: 7170 Sequence number: 60558) 44 ICMP: Checksum = 5860 45 ICMP: 46 47 48 ETHER: ----- Ether Header ----49 ETHER: 50 ETHER: Packet 2 arrived at 15:28:27.19 51 ETHER: Packet size = 60 bytes 52 ETHER: Destination = 8:0:20:b3:22:f2, Sun 53 ETHER: Source = 0:d0:d3:35:46:d0, 54 ETHER: Ethertype = 0800 (IP) 55 ETHER:

56 IP: ----- IP Header ----57 IP: 58 IP: Version = 4 59 IP: Header length = 20 bytes 60 IP: Type of service = 0x00 61 IP: xxx. .... = 0 (precedence) 62 IP: ...0 .... = normal delay 63 IP: .... 0... = normal throughput 64 IP: .... .0.. = normal reliability 65 IP: Total length = 36 bytes 66 IP: Identification = 40936 67 IP: Flags = 0x4 68 IP: .1.. .... = do not fragment The snoop IP Packets 69 IP: ..0. .... = last fragment 70 IP: Fragment offset = 0 bytes 71 IP: Time to live = 255 seconds/hops 72 IP: Protocol = 1 (ICMP) 73 IP: Header checksum = 7a13 74 IP: Source address = 172.20.4.254, protorouter.Central.Sun.COM 75 IP: Destination address = 172.20.4.182, proto182.Central.Sun.COM 76 IP: No options 77 IP: 78 ICMP: ----- ICMP Header ----79 ICMP: 80 ICMP: Type = 0 (Echo reply) 81 ICMP: Code = 0 (ID: 7170 Sequence number: 60558) 82 ICMP: Checksum = 6060 83 ICMP: 84 Note Currently, the snoop command cannot watch multiple interfaces at once. To make sure that you capture all packets destined for an IP address, you must have multiple snoop commands running. Tracing Routing With the netstat Command You can use netstat and route to verify load spreading in the Solaris OE. 1. First, verify the Transmission Control Protocol (TCP) connections into the system that you are examining. In this case, examine the telnet connections into the host: 1 # netstat -n 2 3 TCP: IPv4 4 Local Address Remote Address Swind Send-Q Rwind Recv-Q State 5 -------------------- -------------------- ----- ------ ----- ------ -----6 172.20.4.181.513 129.147.48.184.532 24820 0 24820 0 ESTABLISHED 7 172.20.4.181.23 129.147.28.54.35430 24820 0 24820 0 ESTABLISHED 8 172.20.4.181.23 172.20.4.170.33065 24820 0 24820 0 ESTABLISHED 9 172.20.4.181.23 172.20.13.122.32863 8760 1 24820 0

ESTABLISHED Lines 7, 8, and 9 are the telnet connections into the host. You can see this because it is on port number 23 and the fact that it is on the local address. 2. Examine the routing tables using the following command: 10 # netstat -ran | grep UHA 11 129.147.62.15 172.20.4.254 UHA 1 18 qfe0 12 172.20.4.254 -- UHA 1 788 qfe0 13 172.20.4.254 -- UHA 1 788 hme0 14 129.147.48.184 172.20.4.254 UHA 1 1 qfe0 15 172.20.13.122 172.20.4.254 UHA 2 100 qfe0 Tracing Routing With the netstat Command The 172.20.13.122 remote host is using the qfe0 interface, which indicates that multipathing is working. Another way to verify that multipathing is working is to use the following command: 16 # route -n get 172.20.13.122 17 route to: 172.20.13.122 18 destination: default 19 mask: default 20 gateway: 172.20.4.254 21 interface: qfe0 22 flags: <UP,GATEWAY,DONE,STATIC> 23 recvpipe sendpipe ssthresh rtt,ms rttvar,ms hopcount mtu expire 24 0 0 0 0 0 0 1500 0 Line 21 show IP is using the qfe0 interface to communicate with the 172.20.13.122 remote host. Self-Test