Link Aggregation Control Protocol (LACP): The 802.

3ad IEEE standard presents the means for the |forming of a single Ethernet link automatically from two or more Ethernet |links using LACP. This protocol provides the means of assuring that |both ends of the Ethernet link are functional and agree to be members |of the aggregation group before the link is added to the group. LACP |must be enabled at both ends of the link to be operational. |Should LACP not be available, 802.3ad provides a manual |aggregation that only requires both ends of the link be functional. |LACP provides for the controlled addition and removal of physical |links for the aggregation group such that no frames are lost or duplicated.| The |removal of aggregate links is provided by the LACP protocol. The 802.3ad |specification also provides for manual aggregation for the deployment |of multiple links between switches without performing the LACP message |exchanges. Manual aggregation is not as reliable or manageable as |an LACP negotiated link. The virtual switch supports the following LACP modes of operation:

ACTIVE (default) - Full LACP negotiation with the partner switch, including the initiation of negotiations. INACTIVE - No LACP negotiations with the partner switch and any LACP packets received from the partner switch are ignored (PASSIVE mode is not provided).

In order for LACP to determine whether a set of links can aggregate, several identifiers are required as outlined below. These identifiers allow exchanges to occur between systems so that on an ongoing basis, the maximum level of aggregation capability can be achieved. LACP communications occur on all OSA-Express ports that are part of the aggregated group. 1. System ID This identifier uniquely identifies the virtual switch and consists of a MAC address concatenated with a System priority. The system administrator can specify a MACID (to be concatenated with the MACPREFIX) on the virtual switch definition if a predictable MAC address is desired. Otherwise, the MAC address is assigned from within the system range (that is, outside the user range defined on the VMLAN system configuration statement) and using the MACPREFIX. 2. Aggregation ID Each Aggregator (the function comprised of the Frame Collector and the Frame Distributor) requires a unique MAC address. This MAC address is the same address that is used to create the System ID. 3. Port ID and Capabilities This ID consists of a Port Priority concatenated with a Port Number. Port Numbers are unique within the VSWITCH and are assigned by CP. Each port also requires capability identification, known as a Key that is used during negotiation to compare aggregate capabilities. There are two Keys - an operational Key and an administrative Key. These keys will be assigned from CP. CP assigns these keys along with the other IDs because the virtual switch supports only a single aggregation group per instance so there is no requirement for the System Programmer to differentiate between multiple groups. 4. Link Aggregation Group Id (LAG ID) The LAG ID is constructed of: a. The System ID b. The operational Key assigned to all ports in the LAG c. The Port ID

Link Aggregation Port Group: The grouping of links (OSA-Express adapters) provides a single logical link from which the virtual switch will send and receive data from a partner switch device. The Port Group defines the set of OSA ports that will comprise the logical aggregated link. A LAG is the definition of an ETHERNET virtual switch with the GROUP attribute. The construction of a Port Group is a 2-step process: 1. The virtual switch and group name is defined with the DEFINE VSWITCH command/statement. 2. SET PORT GROUP command/statement is used to identify the OSA devices that are to make up the link aggregation group. These commands are entered in either order but the virtual switch connection to the real hardware network typically will not be operational until both are entered. However, if the virtual switch definition is entered first, specifying backup RDEVs, the virtual switch connection will be brought up using one of the backups. The virtual switch will convert to use the port group once it is defined. The SET PORT GROUP command can also be used to make changes to the list of devices associated with the virtual switch. Options are provided to add (JOIN) or remove (LEAVE) a device. The processing to remove a device involves suspending outbound traffic for the conversations associated with the link, sending a marker PDU, and reassigning all conversations when the marker PDU response is received or it times out. When adding, deleting or using an OSA port in an aggregated group, the port will be in one of the following states: Attached OSA port is currently available and in use for data communication within the aggregated group. Waiting OSA port is available but is not currently being used for data communication within the aggregated group. Suspended All new data transmissions are suspended until pending requests complete or they can be moved to another OSA port. Error OSA port is unavailable due to a problem with the QDIO Connection. The TCPIP controller is having a problem or the OSA Port is not compatible with the aggregated group. The current state of the OSA port is reported by the QUERY VSWITCH and QUERY PORT CP commands. In addition to the state, the reason the port is in the specific state is also returned in the response. Link Aggregation without LACP: The means to provide manual link aggregation is also provided by the 802.3ad IEEE standard. This method of aggregating links does not provide the control and reliability mechanisms that are inherent with LACP constructed and managed link aggregation. The functionality and port states on both switched require manual verification and monitoring. Each partner has no knowledge of the others functional state or capabilities. In fact you may find some switch products that do not support link aggregation without the deployment of LACP. The port states available for LACP are also supported in manual mode.

Load Balancing within a Virtual Switch: All active OSA-Express ports within a virtual switch port group are used in the transmission of data between z/VM's virtual switch and the connected physical switch. The virtual switch logic will attempt to distribute the load equally across all the ports within the group. The actual load balancing achieved will depend on the frame rate of the different

conversations taking place across the OSA-Express ports and the load balance interval specified by the SET PORT GROUP INTERVAL command as to how well the load is balanced. |From |the perspective of the virtual switch, a conversation is based on |the source and destination MAC addresses of a frame being transmitted. All frames from a virtual NIC destined to the same destination MAC are considered the same conversation. All the outbound frames associated with this conversation can be routed automatically to any one of the active OSA-Express ports within the group. This is the smallest unit of work, which can be moved from one port to another to balance the workload. The frequency in which a load balance operation will move a conversation is based on the interval specified by the SET PORT GROUP INTERVAL command. The default load balance interval will look at the number of frames received and transmitted across each active port within the last five minutes. If there is more than a ten percent frame rate difference between the most and least utilized OSA-Express port, an attempt will be made to move a conversation from a virtual NIC currently routed to the most utilized OSA-Express port to the least utilized port. The conversation selected will be the one that will make the frame rates between these two ports as equal as possible. Of course this also assumes the frame rate remains relatively the same in the next load balance interval. Lowering the SET PORT GROUP INTERVAL value increases the ability to balance the load across the active OSAExpress ports. However, decreasing the interval time does increase the number of times the network workload must be examined by CP, thus increasing the CPU overhead to manage the virtual switch. Selecting the largest interval value to meet your network load is the best way to go. There is also the ability to turn off load balancing for a specific port group anytime by issuing the SET PORT GROUP INTERVAL OFF command. Likewise, it can also be turned back on to any allowable interval at anytime. The number of frames transmitted for a specific OSA-Express port in both the current and previous load balance interval can be seen at anytime by issuing a QUERY PORT GROUP DETAILS command. The PORT Information" section of the response displays a list of the active ports within the group in addition to the load balance frame counts. Each frame count includes the sum of both inbound and outbound frames transmitted by the OSA-Express port within the load balance interval. In addition, the ROUTING Information" section of the same response displays the current routing of outbound frames for the ports within the group. Each row displayed corresponds to the least three significant bits of the destination MAC for outbound frames. The Device" column indicates the OSA -Express port currently assigned to handle frames with the same last three destination MAC bits and the number of frames sent out across that link in both the previous and current load balance interval.

Virtual Link Aggregation Control Protocol (VLACP): Virtual Link Aggregation Control Protocol (VLACP) is extension of the Link Aggregation Control Protocol (LACP) developed by Nortel to detect end-to-end failure over an Ethernet network. Weve been deploying VLACP within our network for the past year with great success. We were eager to deploy VLACP because the Nortel Ethernet Switch 470 Gigabit Ethernet fiber ports (GBIC) did not support auto-negotiation and are required to be hard set to 1000/Full Duplex when connecting to a Nortel Ethernet Routing Switch 8600. Without auto-negotiation there is no mechanism to provide link failure notification (RFI, FEFI) on the specific interface. The problem can arise if you have a GBIC malfunction or a single fiber strand breaks leaving one side of the link up and the other side down. VLACP mitigates this problem by providing a mechanism to detect the path failure and can be applied to provide end-to-end failure notification over a telco carrier network. Heres what Nortel has to stay in their document, Link Aggregation Control Protocol (LACP) 802.3ad and VLACP Technical Configuration Guide dated August 2007; Virtual LACP (VLACP) is an extension to LACP, used to detect end-to-end failure. VLACP takes the point-to-point hello mechanism of LACP and uses it to periodically send hello packets to ensure end-to-end reachability and provide

failure detection (across any L2 domain). When Hello packets are not received, VLACP transitions to a failure state and the port will be brought down. The benefit of this over LACP is that VLACP timers can be reduced to 400 milliseconds between a pair of ERS8600 switches. This will allow for approximately one-second-failure detection and switchover. Note that the lowest VLACP timer on an ES460/470 is 500ms. VLACP can also be used with Nortels proprietary aggregation mechanism (MLT) to complement its capabilities and provide quick failure detection. VLACP is recommended for all SMLT access links when the links are configured as MLT to ensure both end devices are able to communicate. By using VLACP over Single-Port SMLT, enhanced failure detection is extended beyond the limits of the number of SMLT or LACP instances that can be created on the ERS8600. VLACP can also be used as a loop prevention mechanism in SMLT configurations and should be used when setting up the IST. It also protects against CPU failures by causing traffic to be switched or rerouted to the SMLT peer in the case the CPU fails or gets hung up. Please refer to the Technical Configuration Guide for Switch Clustering using Split-Multilink Trunking (SMLT) with ERS8600 for more details. NOTE: In regards to the ERS8600, although either the CLI or JDM interface allows you to configure the short timers to less than 400ms, Nortel does not support this configuration unless the ERS8600 is equipped with the SuperMezz daughter module for the 8692SF. The SuperMezz allow for very quick sub 100ms failure detection. Although functions such as Remote fault indication (RFI) or Far-end fault indication (FEFI) can be used to indicate link failure, there are some limitations with these mechanisms. The first limitation is that with either of these mechanisms, they terminate at the next Ethernet hop. Hence, failures cannot be detected on an end-to-end basis over multiple hops such as LAN Extension services. The second limitation is both of these mechanisms required AutoNegotiation to be enabled on the Ethernet interface. Hence, if an Ethernet interface does not support Auto-Negotiation; neither of these mechanisms can be used. The third limitation is if an Ethernet interface should fail and still provide a transmit signal, RFI nor FEFI will be able to detect a failure. Hence, the far-end interface will still think the link up and continue to transmit traffic. VLACP will only work for port-to-port applications when there is a guarantee for a logical port-port match. It will not work in a port-to-multi-port scenario where there is no guarantee for a point-point match. NOTE: Please note that VLACP does not perform link aggregation. Is it simply used to detect end-to-end link failures and can be enabled over single links or even MLT trunks. VLACP does not require LACP to be enabled; LACP and VLACP are independent features. NOTE: When configuring VLACP, both ends of the link must be configured with the same Ether Type, Multicast MAC address, and same timers. By default, the VLACP parameters across all ES and ERS switches are the same with the exception of the Fast Periodic Timer, which is set to 200ms on the ERS8600 and 500ms on all other switches. When connecting, for example, an ERS8600 to and ERS5500, the recommendation is to use 500ms Fast Periodic Timers with Short Timeout in order to achieve fast failover. Also, when using the ES460/470 in the 3.6.x software release, the VLACP Ether Type must be configured with a different value on each MLT link. The Ether Type must match the Ether Type value at the far end of the MLT link. NOTE: If VLACP is used with LACP, there is no difference in how VLACP and LACP bring down a port if no LACP or VLACP PDUs are received. VLACP will declare the VLACP status as down and will report the event in the log file whereas LACP will not synchronize, not activate Collecting and Distributing on this port, and not report a message in the log file. The end result is the same where the port will block traffic; the physical layer for this port will remain up. Although you can enable VLACP with LACP, there is no practical reason why you would do so. There was an interim solution before VLACP developed by Nortel called Single Fiber Fault Detection (SFFD) specifically designed to allow remote fault detection on Gigabit Ethernet fiber ports that did not support auto negotiation. Unfortunately we had some issues with SFFD and never really deployed the feature beyond our test lab environment.

Ethernet Routing Switch Heres how you would configure VLACP on the MLT uplinks to an ERS 8600 Switch. Youll need to connect to the 5510 switch and enter the Command Line Interface if you have the menu up. Reference: http://publib.boulder.ibm.com/infocenter/zvm/v5r4/index.jsp?topic=/com.ibm.zvm.v54.hcpa6/hcsc9 b3169.htm