Вы находитесь на странице: 1из 184

Cisco MDS 9000 Family Cookbook for SAN-OS 2.

x
10/4/05

Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100

CCSP, the Cisco Square Bridge logo, Cisco Unity, Follow Me Browsing, FormShare, and StackWise are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, and iQuick Study are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Empowering the Internet Generation, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, GigaDrive, GigaStack, HomeLink, Internet Quotient, IOS, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, LightStream, Linksys, MeetingPlace, MGX, the Networkers logo, Networking Academy, Network Registrar, Packet, PIX, Post-Routing, Pre-Routing, ProConnect, RateMUX, Registrar, ScriptShare, SlideCast, SMARTnet, StrataView Plus, SwitchProbe, TeleRouter, The Fastest Way to Increase Your Internet Quotient, TransPath, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0406R)

Cisco MDS 9000 Family Cookbook Copyright 2004 Cisco Systems, Inc. All rights reserved.

Comments: mds-cookbook@cisco.com

Preface
This document provides a reference guide to the configuration and implementation of fabrics using the MDS-9000 family of switches. The configurations and components used herein have been tested and validated by Ciscos Solution-Interoperability Engineering for supporting risk-free deployment of fabrics using Ciscos MDS 9000 family of Fibre Channel Switch and Director Class products.

Audience
This reference document is designed for use by Cisco TAC, Sales, Support Engineers, Professional Service Partners, Systems Administrators and others, responsible for the design and deployment of Storage Area Networks in the data center environment.

Inquiries
Please send your comments via email to: Email: mds-cookbook@cisco.com State the document name, page number, and details of the request. Thank you. Latest version of this document is available on Ciscos Partner website at: http://www.cisco.com

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT T CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCBs public domain version of the UNIX operating system. All rights reserved. Copyright 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED AS IS WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED ORIMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

iii

Preface

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

iv

78-xxxxx-xx

CONTENTS
1

CHAPTER

Management

13

CFS: Cisco Fabric Services 13 Fabric Manager and CFS 15 Which Switch does FM Distribute from? 15 CFS CLI Commands 16 What Switches are CFS capable? 16 What CFS applications do I have and what is their scope? Locked out of an application by CFS 17 Command Scheduler 17 Regularly Backing up the Switch Configuration Copying files to or from the MDS
19 21 18

16

Managing Files on the Standby Supervisor

Firmware Upgrades 22 Upgrading firmware with the CLI 22 Downgrading firmware with the CLI 24 Upgrading Firmware with Fabric Manager Password Recovery
28

25

Installing a License 30 Using the CLI to Install a License: 31 Using Fabric Manager to Install a License 32 What Feature is Enabling the License Grace Period? Copying core files from the MDS Fixed Switch Config Restoration
35 36

34

Configuring an NTP Server 38 Configuring NTP with CFS 38 Without CFS 40 Steps to Perform Before Calling TAC How to Disable the Webserver
41 42

Saving the configuration across the fabric


42

Device Aliases 43 Device Aliases with the CLI 43 Examples of Devices Aliases Displayed in the CLI: Creating Device Aliases with the CLI 44

44

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

Contents

Converting FC Aliases into Device Aliases 45 Device Aliases with FM 46 Enabling Fabric Manager to use Device Alias 46 Creating a Device Alias for an Existing Device 47 Managing Fabric Manager 48 Operating Fabric Manager Through a Firewall using SNMP Proxy Configuration using a non-NAT Packet Filter 49
2
49

CHAPTER

Account Management

51 52

Creating User Accounts

Creating a MDS User Role 52 Creating a Role with Device Manager Creating Roles with CLI 56

52

Configuring TACACS+ with Cisco SecureACS 57 Authentication and Authorization with TACACS+ Configure SecureACS Server 57 Configure TACACS+ on the MDS 62 Accounting with TACACS+ 63 Configuring the MDS 63 Configuring SecureACS 64 Providing Passwordless Access Leveraging SSH
3
66

57

CHAPTER

Physical Interfaces

69

Configuring FC ports 69 Port Description 69 Port speed 69 Port mode (auto) 70 Port mode (E) 70 Port mode (F) 70 Port mode (FL) 71 Port mode (Fx) 71 Port mode (SD) 71 Port mode (ST) 71 Port mode (TL) 72 Configuring Trunking E ports 72 Trunk port mode 72 Configuring trunk ports to filter specific VSANs Enabling port beaconing 73 Configuring Gigabit Ethernet Ports
Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

72

73

vi

Contents

Configuring VRRP

73

Implementing WWN Based VSANs (DPVM) 75 Adding Existing Devices to DPVM 77 Adding New Devices to DPVM 79 Modify the VSAN Assignment of a DPVM Entry DPVM Conflicting Entries 82 DPVM with the CLI 84 Adding Existing Devices to DPVM 84 Adding New Devices to DPVM 85 Modify the VSAN Assignment of a DPVM Entry
4

81

86

CHAPTER

VSANs

87 87

Creating a VSAN and Adding Interfaces

Modifying VSAN Attributes with Fabric Manager 88 Converting an Existing VSAN to Static DomainID and Enabling Persistent FCID using Fabric Manager. 89 Modifying VSAN Attributes with the CLI 91 Creating a VSAN on a single switch and adding an Interface 91 Setting VSAN Interop Mode 92 Changing Load-balancing scheme 93 Sequence Level load-balancing (Source_ID, Destination_ID) 93 Exchange level load balancing (S_ID, D_ID, OX_ID) 93 Converting an Existing VSAN to Static DomainID and Enabling Persistent FCID using CLI. 93 Restarting a VSAN 94 Assigning a Predetermined FCID to a PWWN 95
5

CHAPTER

Zoning

97

Enhanced Zoning 97 How to Enable Enhanced Zoning 99 Enabling Enhanced Zoning with the CLI 99 Enabling Enhanced Zoning with Fabric Manager 99 How to Display which user has obtained the lock 100 ZoneSets 101 Zoneset Distribution 101 Automatic Zoneset Distribution 101 Manual Zoneset Distribution: 102 Zones 102 Zone Creation and Adding it to a Zoneset with Fabric Manager
103

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

vii

Contents

How to Create Non-PWWN based zones 108 Creating a zone and adding it to a zoneset (CLI standalone method) 109 Creating a zone and adding it to a zoneset (CLI inline method) 110 Creating a FC Alias based zone (CLI) 111 Creating an Interface based zone (CLI method) 113
6

CHAPTER

Inter-VSAN Routing 115 IVR with CFS 115 Configuring a Three Switch, Two Transit VSAN Topology with CFS IVR-1 121 Enable IVR-1 121 To enable IVR-1 via the CLI: 121 To enable IVR-1 via FM: 122 Single Switch, Two VSANs 122 IVR-2: FC-NAT 125 Enabling IVR-2 125 How to Upgrade from IVR-1 to IVR-2: 128 How to Configure Persistent FCIDs in IVR 129 Single Switch: Two VSANs 131 Adding a new IVR Enabled Switch 134

117

CHAPTER

FCIP

137 137 137 141

How to Enable FCIP

Configuring FCIP on MDS

Configuring multiple FCIP tunnels using a single gigE port Configuring FCIP with IPsec using FM
149

Tuning FCIP 158 TCP Tuning: Latency and Available Bandwidth FCIP Write Acceleration 159 FCIP Compression 159 Enabling Tape Acceleration 161 SAN Extention Tuner (SET)
8
164

158

CHAPTER

iSCSI

171 171 171 176

How to Enable iSCSI

Configuring iSCSI on MDS in Transparent Mode Configuring iSCSI on MDS in Proxy initiator mode Configuring iSCSI Client (initiators) on hosts
Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

180

viii

Contents

iSCSI on Windows 180 iSCSI on Linux 183

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

ix

Contents

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

Introduction
While deploying or configuring the MDS platform, a task may come up that the administrator does not know how to perform and requires a simple recipe to complete it. This guide wont replace the MDS 9000 Family Configuration Guides, but compliment them with simple, concise procedures to accomplish a specific task. This is field driven book, meaning that the intended audience, the storage administrators, technical support engineers, SEs and CEs are the source for these procedures. Their requirements for a procedure to perform XYZ are what determine the content. If there are procedures that should be covered in this book, please notify by Email: mds-cookbook@cisco.com. Within this book, there are sections that look like this:

Tip

These tips refer to a MDS best practice towards implementing the MDS. They come from in depth knowledge of the platform as well as years of experience of implementing SANs.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

xi

Introduction

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

xii

78-xxxxx-xx

C H A P T E R

Management
This chapter will provide recipes for the aspects of managing the MDS as an entity. These non-datapath areas include access control, accounting, event resolution and monitoring.

CFS: Cisco Fabric Services


With the introduction of SAN-OS version 2.0, the MDS gained the ability for the same application on multiple switches to have its configuration be synchronized across the fabric. This infrastructure, Cisco Fabric Servers (CFS), provides the underlying transport for such applications as NTP, device-aliases and IVR to have their configuration pushed out or distributed to other MDSs in the fabric. This feature provides the administrator with a central point of management for any of the supported applications. Prior to SAN-OS 2.0, on each switch in the fabric, the administrator had to either perform the configuration manually, leverage host based scripting or run Fabric Manager. With CFS, the administrator can execute the commands from one MDS and have them be distributed to the rest of the switches in the fabric. In addtion, the CFS protocol provides application locking so that two admins could not simultaneously perform configuration changes to the same application. Cisco Fabric Services uses common terminology across its supported applications:

Pending Database: When configuration changes are made to a CFS application, they are first made to the pending database which is always distributed to all switches in the fabric.To activate these changes into the MDSs running-configuration, an explicit commit command must be executed. Alternatively, the user can clear the applications pending database by issuing an explicit abort command. Locking. Prior to modifying the pending database, the application via the CFS transport obtain a lock, thus preventing other users and switches from modifying the pending database. Applications outside the scope of the lock can still be modified.
Upon the start of configuration, the application implicitly attempts to obtain a lock. The CFS

infrastructure knows which switch and user has obtained the lock.

Scope. The scope of an application can be either Physical or Logical. This determines if other users can be modifying the same application simultaneously.
A physical scope encompasses all the switches in the physical fabric such as NTP. While a NTP

lock has been obtained, no other user may modify NTP within the physical fabric.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-13

Chapter 1 CFS: Cisco Fabric Services

Management

A logical scope encompasses only the VSAN that the application is being configured for such

as Port-Security. While a port-security lock is obtained, no other user can modify port-security in the VSAN for which the lock is applied to. However, port-security could be modified for another VSAN as it is outside of the scope of the lock.

Merge Control. Each application is responsible for the merging of its configuration with that of the same application in another physical fabric if two fabrics are merged. The basic rule for merging is that a union of the two configurations will be performed however, conflicting entries will not be merged. Those conflicting entries will have to be manually created in the resulting configuration.

Note

Failure to fully merge a CFS application when two merging fabrics, will NOT isolate the ISL.

Tip

If CFS is to be used with an application, all the MDSs in the fabric should be configured to use CFS for that application. For example, if there are five switches in a fabric, and NTP will be configured leveraging CFS, all five switches should have NTP leverage CFS, not just a subset of them. As illustrated in Figure 1-1, a CFS application would follow the below procedure:

Step 1 Step 2

Prior to the first Administrator enables CFS distribution for the application. Then enters configuration mode for the specified application. Local MDS implicitly attempts to obtain a an application lock from the other switches in the fabric according to the scope of the application (VSAN or physical). If available, other MDSs grant the lock to the local MDS. If the lock is not granted, the admin cannot make changes to the applications pending database. Admin makes configuration changes to the pending database and when that is completed, the changes are explicitly committed or aborted by the admin. Local MDS informs the other MDSs in the scope to commit the changes and the lock is released. Until the lock is released by the application, other users on other switch cannot make changes to the locked application. However, other applications can still be modified.

Step 3 Step 4 Step 5

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-14

78-xxxxx-xx

Chapter 1

Management CFS: Cisco Fabric Services

Figure 1-1

CFS Application Flow

Fabric Manager and CFS


Prior to 2.0, Fabric Manager (FM) had the ability to configure multiple switches simultaneously by sending the configuration commands to all the switches that were selected by the user. FM still has this ability, but now also has the ability to perform the same action leveraging CFS. The user will still need to commit the changes, as committing is an explicit activity.

Tip

If an application is configured to use CFS in the fabric, CFS should be enabled for that application on all switches in the fabric. FM will use either CFS or the legacy method (send multiple requests), not both.

Which Switch does FM Distribute from?


If Fabric Manager is leverages CFS to distribute a configuration, it will rely on one switch to perform the locking and distribution. This is referred to as the Master switch as displayed in Figure 1-2. FM determines the master by selecting the switch with the lowest WWN.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-15

Chapter 1 CFS: Cisco Fabric Services

Management

Figure 1-2

CFS Master in FabricManager

CFS CLI Commands


When interacting with CFS it will be via the application that leverages CFS (ex: NTP, DPVM etc), not with CFS directly. As it is more important to know the status of a NTP merge or commit than how CFS is setup. However, there are a few commands that may need to be invoked:

What Switches are CFS capable?


172.22.36.9# show cfs peers Physical Fabric -------------------------------------------------Switch WWN IP Address -------------------------------------------------20:00:00:05:30:00:86:9e 172.22.36.9 [Local] 20:00:00:05:30:00:68:5e 172.22.36.11 20:00:00:0d:ec:02:1d:40 172.22.36.8 20:00:00:0c:85:e9:d2:c0 172.22.36.142 Total number of entries = 4

What CFS applications do I have and what is their scope?


172.22.36.9# show cfs application ------------------------------------------Application Enabled Scope ------------------------------------------ntp Yes Physical sfm Yes Physical/Logical fscm Yes Physical role No Physical radius No Physical

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-16

78-xxxxx-xx

Chapter 1

Management Command Scheduler

tacacs fctimer syslogd callhome device-alias port-security

No No No No Yes No

Physical Physical Physical Physical Physical Logical

Total number of entries = 11

Note

Remember that a physical scope spans all switches physically connected together, regardless of VSAN configuration. Logical scope applies only to the VSAN that the configuration applies to. SFM - SCSI Flow Manager for use with monitoring scsi flows with the Storage Services Module FSCM - Fabric Startup Configuration Manager enables copy running-config startup-config
fabric

Locked out of an application by CFS


CFS provides locking on a physical or logical scope, and if a configuration change is made by a user that doesnt have the lock the following error might be seen:
172.22.36.9(config)# ntp peer 172.22.36.99 Failed to acquire Lock

To find out which user on which switch has the lock:


172.22.36.9# show cfs lock Application: ntp Scope : Physical -------------------------------------------------------------------Switch WWN IP Address User Name User Type -------------------------------------------------------------------20:00:00:0c:85:e9:d2:c0 172.22.36.142 admin CLI/SNMP v3 Total number of entries = 1

Until that user commits their changes to the database or their lock expires another user cannot modify the pending database. However, if its required, then to break a lock. This will clear the pending-database and all pending changes for the specified application will be lost.
172.22.36.9# clear ntp session

Command Scheduler
The following section provides recipes leveraging the MDSs command scheduler, which provides the user with the ability to schedule tasks for the MDS.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-17

Chapter 1 Command Scheduler

Management

Regularly Backing up the Switch Configuration


Prior to SAN-OS 2.0, the only method for automating the backing up of the switch configuration was to set up a management station to periodically log into the MDS, and issue the appropriate scripting commands to copy off the configuration to a tftp server. However, if the management station went down, then the configuration would nolonger be backed up. This procedure demonstrates how to leverage the Command Scheduler to regularly back up the configuration to a tftp server. For this recipe the following resources will be used: Switch: 172.22.36.142 TFTP Server: 171.71.58.69 Schedule: nightly_10pm Every night at 10PM.
Step 1

Enable the command scheduler:


ca-9506# config terminal Enter configuration commands, one per line. ca-9506(config)# scheduler enable End with CNTL/Z.

Step 2

Define the job to be run. The job will consist of saving the running configuration and then copying it to a tftp server. the {config-job} prompt is the same as the MDS exec-mode prompt. Thus any command on the MDS could be executed.
ca-9506(config)# scheduler job name backup_config ca-9506(config-job)# copy running-config startup-config ca-9506(config-job)# copy startup-config tftp://171.71.58.69/ca-9506_config

Step 3

Display the defined job:


ca-9506# show scheduler job name backup_config Job Name: backup_config ----------------------copy running-config startup-config copy startup-config tftp://171.71.58.69/ca-9506/ca-9506_config ==============================================================================

Step 4

Create the schedule, assign the time (20:00) and the job that will be assigned to it (backup_config).
ca-9506# conf terminal Enter configuration commands, one per line. End with CNTL/Z. ca-9506(config)# scheduler schedule name nightly_10pm ca-9506(config-schedule)# time daily 20:00 ca-9506(config-schedule)# job name backup_config

Step 5

Display the schedule:


ca-9506# show scheduler schedule name nightly_10pm Schedule Name : nightly_10pm ---------------------------------User Name : admin Schedule Type : Run every day at 20 Hrs 0 Mins Last Execution Time : Yet to be executed ----------------------------------------------Job Name Last Execution Status backup_config n/a -----------------------------------------------

Step 6

After the job has run, the status of the job and the details of the execution can be examined:
ca-9506# show scheduler schedule name nightly_10pm

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-18

78-xxxxx-xx

Chapter 1

Management Copying files to or from the MDS

Schedule Name : nightly_10pm ---------------------------------User Name : admin Schedule Type : Run every 0 Days 0 Hrs 1 Mins Start Time : Fri Apr 22 20:00:00 2005 Last Execution Time : Fri Apr 22 20:00:00 2005 Last Completion Time: Fri Apr 22 20:00:15 2005 Execution count : 1 ----------------------------------------------Job Name Last Execution Status ----------------------------------------------backup_config Success (0) ==============================================================================

Detailed Log:
ca-9506# show scheduler logfile ============================================================================== Job Name : backup_config Job Status: Success (0) Schedule Name : nightly_10pm User Name : admin Completion time: Fri Apr 22 20:00:15 2005 --------------------------------- Job Output --------------------------------`copy running-config startup-config ` [#### ] 7% [####### ] 14% [########## ] 23% [############# ] 30% [################ ] 37% [################### ] 46% [###################### ] 53% [######################### ] 60% [############################ ] 69% [############################### ] 76% [################################## ] 84% [##################################### ] 92% [########################################] 100% `copy startup-config tftp://171.71.58.69/ca-9506_config` Trying to connect to tftp server...... TFTP put operation was successful ==============================================================================

Tip

Some tftp servers may have to be configured to allow the overwriting of files. To avoid overwriting the previous night configuration, create multiple jobs. Specify each job using a different destination filename (ca-9506_monday, ca-9506-tuesday... etc).

Copying files to or from the MDS


It may be required to move files to or from an MDS switch. These files may include log, configuration or firmware files. There are two methods for copying files to/from the MDS: CLI and FM. This procedure will cover the CLI:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-19

Chapter 1 Copying files to or from the MDS

Management

The CLI offers a broad range of protocols to use for copying to or from the MDS. Note that the MDS always acts as a client, such that an ftp/scp/tftp session will always originate from the MDS and either push files to an external system or pull files from an external system. File Server: 172.22.36.10 File to be copied to the switch: /etc/hosts The MDSs copy command supports 4 transfer protocols and 12 different sources for files.

ca-9506# copy ? bootflash: core: debug: ftp: licenses log: modflash: nvram: running-config scp: sftp: slot0: startup-config system: tftp: volatile:

Select source filesystem Select source filesystem Select source filesystem Select source filesystem Backup license files Select source filesystem Select source filesystem Select source filesystem Copy running configuration to destination Select source filesystem Select source filesystem Select source filesystem Copy startup configuration to destination Select source filesystem Select source filesystem Select source filesystem

To use scp (Secure copy) as the transfer mechanism, the following syntax would be used: "scp:[//[username@]server][/path]" To copy /etc/hosts from 172.22.36.10 using the user user1, and the destination would be hosts.txt use:
switch# copy scp://user1@172.22.36.10/etc/hosts bootflash:hosts.txt user1@172.22.36.10's password: hosts 100% |*****************************| 2035 00:00

To backup the startup-configuration to a sftp server:


switch# copy startup-config sftp://user1@172.22.36.10/MDS/startup-configuration.bak1 Connecting to 172.22.36.10... User1@172.22.36.10's password: switch#

Tip

Backing up the startup-configuration to a server should be done on a daily basis and prior to any changes. A short script could be written to be run on the MDS to perform a save and then backup of the configuration. The script only needs to contain two commands: copy running-configuration startup-configuration and then copy startup-configuration tftp://<server>/<name>. To execute the script use: run-script <filename>.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-20

78-xxxxx-xx

Chapter 1

Management Managing Files on the Standby Supervisor

Managing Files on the Standby Supervisor


Oftentimes a file may need to be copied to, off or deleted from the supervisor, or even deleted from the standby supervisor. To do this, attach to the standby supervisor and use the normal dir: and delete commands.

Note

This recipe is most often invoked when a firmware upgrade fails because there is not be enough free bootflash: capacity on the standby supervisor for the firmware images. Determine which supervisor is the standby. In this case it is module 6 as displayed below.
switch# show module Mod Ports Module-Type --- ----- ------------------------------1 16 1/2 Gbps FC Module 2 16 1/2 Gbps FC Module 3 8 IP Storage Services Module 4 0 Caching Services Module 5 0 Supervisor/Fabric-1 6 0 Supervisor/Fabric-1

Step 1

Model -----------------DS-X9016 DS-X9016 DS-X9308-SMIP DS-X9560-SMAP DS-X9530-SF1-K9 DS-X9530-SF1-K9

Status -----------ok ok ok ok active * ha-standby

Step 2

Next, connect to the standby supervisor using the attach command. Note how the prompt displays the word standby.
ca-9506# attach module 6 Attaching to module 6 ... To exit type 'exit', to abort type '$.' Cisco Storage Area Networking Operating System (SAN-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2004, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by Andiamo Systems, Inc. and/or other third parties and are used and distributed under license. Some parts of this software are covered under the GNU Public License. A copy of the license is available at http://www.gnu.org/licenses/gpl.html. ca-9506(standby)#

Step 3

List the files on the bootflash to be deleted:


ca-9506(standby)# dir bootflash: 12330496 Jun 30 21:11:33 2004 2035 Jun 17 16:30:18 2004 43705437 Jun 30 21:11:58 2004 12288 Dec 31 17:13:48 1979 12334592 Jun 23 17:02:16 2004 43687917 Jun 23 17:02:42 2004 99 Apr 07 19:28:54 1980 Usage for bootflash://sup-local 126340096 bytes used 59745280 bytes free 186085376 bytes total

boot-1-3-4a hosts.txt isan-1-3-4a lost+found/ m9500-sf1ek9-kickstart-mz.1.3.4b.bin m9500-sf1ek9-mz.1.3.4b.bin security_cnv.log

Step 4

Delete the file with the delete command:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-21

Chapter 1 Firmware Upgrades

Management

ca-9506(standby)# delete bootflash:hosts.txt

Step 5

To return to the active supervisor, type exit, and the prompt will also return to the active supervisors prompt:
ca-9506(standby)# exit rlogin: connection closed. ca-9506#

Firmware Upgrades
Within SAN-OS 2.x, the process of upgrading has not changed from within SAN-OS 1.x. However, if the need to downgrade from SAN-OS 2.x to 1.x is required, special attention needs to be paid to the procedure.

Upgrading firmware with the CLI


The process for upgrading using the 2.x images should leverage either the install all command, or the Firmware Upgrade wizard in Fabric Manager. The example below demonstrates upgradeing from SAN-OS 2.0(2b) to 2.1(1a) using the install all command with the source images located on a scp server.

Tip

Always carefully read the output of install alls compatibility check. This will tell you exactly what needs to be upgraded (BIOS, loader, firmware) and what modules are not hitless. If there are any questions or concerns about the results of the output, select n to not proceed with the installation and contact the next level of support.

ca-9506# install all system scp://setmason@dino/tftpboot/rel/qa/2_1_1a/final/m95 00-sf1ek9-mz.2.1.1a.bin kickstart scp://setmason@dino/tftpboot/rel/qa/2_1_1a/fin al/m9500-sf1ek9-kickstart-mz.2.1.1a.bin For scp://setmason@dino, please enter password: For scp://setmason@dino, please enter password: Copying image from scp://setmason@dino/tftpboot/rel/qa/2_1_1a/final/m9500-sf1ek9 -kickstart-mz.2.1.1a.bin to bootflash:///m9500-sf1ek9-kickstart-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Copying image from scp://setmason@dino/tftpboot/rel/qa/2_1_1a/final/m9500-sf1ek9 -mz.2.1.1a.bin to bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Verifying image bootflash:///m9500-sf1ek9-kickstart-mz.2.1.1a.bin [####################] 100% -- SUCCESS Verifying image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin [####################] 100% -- SUCCESS

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-22

78-xxxxx-xx

Chapter 1

Management Firmware Upgrades

Extracting "slc" version from image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Extracting "ips" version from image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Extracting "svclc" version from image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Extracting "system" version from image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Extracting "kickstart" version from image bootflash:///m9500-sf1ek9-kickstart-mz .2.1.1a.bin. [####################] 100% -- SUCCESS Extracting "loader" version from image bootflash:///m9500-sf1ek9-kickstart-mz.2. 1.1a.bin. [####################] 100% -- SUCCESS

Compatibility check is done: Module bootable Impact ------ -------- -------------1 yes non-disruptive 2 yes non-disruptive 3 yes disruptive 4 yes disruptive 5 yes non-disruptive 6 yes non-disruptive

Install-type -----------rolling rolling rolling rolling reset reset

Reason ------

Hitless upgrade is not supported Hitless upgrade is not supported

Images will be upgraded according to following table: Module Image Running-Version New-Version ------ ---------- -------------------- -------------------1 slc 2.0(2b) 2.1(1a) 1 bios v1.1.0(10/24/03) v1.1.0(10/24/03) 2 slc 2.0(2b) 2.1(1a) 2 bios v1.1.0(10/24/03) v1.1.0(10/24/03) 3 ips 2.0(2b) 2.1(1a) 3 bios v1.1.0(10/24/03) v1.1.0(10/24/03) 4 svclc 2.0(2b) 2.1(1a) 4 svcsb 1.3(5m) 1.3(5m) 4 svcsb 1.3(5m) 1.3(5m) 4 bios v1.1.0(10/24/03) v1.1.0(10/24/03) 5 system 2.0(2b) 2.1(1a) 5 kickstart 2.0(2b) 2.1(1a) 5 bios v1.1.0(10/24/03) v1.1.0(10/24/03) 5 loader 1.2(2) 1.2(2) 6 system 2.0(2b) 2.1(1a) 6 kickstart 2.0(2b) 2.1(1a) 6 bios v1.1.0(10/24/03) v1.1.0(10/24/03) 6 loader 1.2(2) 1.2(2) Do you want to continue with the installation (y/n)? Install is in progress, please wait. [n] y

Upg-Required -----------yes no yes no yes no yes no no no yes yes no no yes yes no no

Syncing image bootflash:///m9500-sf1ek9-kickstart-mz.2.1.1a.bin to standby. [####################] 100% -- SUCCESS Syncing image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin to standby.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-23

Chapter 1 Firmware Upgrades

Management

[####################] 100% -- SUCCESS Setting boot variables. [####################] 100% -- SUCCESS Performing configuration copy. [####################] 100% -- SUCCESS Module 5: Waiting for module online. 2005 May 20 15:46:03 ca-9506 %KERN-2-SYSTEM_MSG: mts: HA communication with standby terminated. Please check the standby supervisor. -- SUCCESS "Switching over onto standby".

The %Kern-2-SYSTEM_MSG displayed at the end is displayed when the standby supervisor is rebooted as part of the rolling upgrade of the supervisor modules. It is expected and upon the next message, the user should reconnect (telnet/ssh) back into the switch on the new active supervisor. The continued progress of the installation can be watched on the new active supervisor using the show install all status command.

ca-9506# show install all status There is an on-going installation... Enter Ctrl-C to go back to the prompt. Continue on installation process, please wait. The login will be disabled until the installation is completed. Trying to start the installer... Module 5: Waiting for module online. -- SUCCESS Module 1: Non-disruptive upgrading. -- SUCCESS

Downgrading firmware with the CLI


When downgrading firmware, it is imparative that any features that are not supported in the new, lower level be turned off or disabled prior to the downgrade. Failure to do so can lead to a disruptive downgrade. This command should always be run prior to a downgrade even if no SAN-OS 2.x features were explicitly enabled.
Step 1

Verify there are no features enabled that are not supported in the lower level which should be disabled.

Warning

Failure to disable a feature listed in the incompatibility check can result in a disruptive firmware downgrade.

If downgrading from 2.x to 1.x the following features might be seen:


ca-9506# show incompatibility system bootflash:m9500-sf1ek9-mz.1.3.5.bin The following configurations on active are incompatible with the system image 1) Service : cfs , Capability : CAP_FEATURE_CFS_ENABLED_DEVICE_ALIAS

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-24

78-xxxxx-xx

Chapter 1

Management Firmware Upgrades

Description : CFS - Distribution is enabled for DEVICE-ALIAS Capability requirement : STRICT

If downgrading from 2.1(x) to 2.0(x) as IVR with FC-Nat was not available in SAN-OS 2.0(x).
ca-9506# show incompatibility system bootflash:m9500-sf1ek9-mz.2.0.2b.bin The following configurations on active are incompatible with the system image 1) Service : ivr , Capability : CAP_FEATURE_IVR_FCID_NAT_ENABLED Description : ivr fcid-nat mode is enabled Capability requirement : STRICT 2) Service : ivr , Capability : CAP_FEATURE_IVR_AUTO_VSAN_TOPOLOGY_ENABLED Description : ivr auto vsan-topology mode is enabled Capability requirement : STRICT

Step 2

Disable the features that are not supported:


ca-9506# ca-9506# conf t Enter configuration commands, one per line. ca-9506(config)# no device-alias distribute

End with CNTL/Z.

Step 3

Rerun the incompatibility check to verify:


ca-9506# show incompatibility system bootflash:m9500-sf1ek9-mz.1.3.5.bin No incompatible configurations

Step 4

Proceed with the downgrade procedure using the install all command as described in Upgrading firmware with the CLI, page 1-22.

Upgrading Firmware with Fabric Manager


To upgrade the firmware of one or more MDS switches Fabric Manager can be leveraged.

Step 1

Select the Software Install Wizard from the toolbar in Fabric Manager.
Figure 1-3 Image Installation with Fabric Mgr

Step 2

Choose the switches to upgrade and select Next:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-25

Chapter 1 Firmware Upgrades

Management

Figure 1-4

Select Switches to Upgrade

Step 3

Specify the location of the firmware images


a. b. c.

Fill in the file information to transfer the file from the server to the switch. If the files are to be downloaded during the install, the path and filename of the images are required to be filled in as well. By selecting Skip Image Download, an upgrade can be performed using images which are already located on the supervisors bootflash.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-26

78-xxxxx-xx

Chapter 1

Management Firmware Upgrades

Figure 1-5

Specify Firmware Images

Step 4

Select Next. Depending on the method for installing (already downloaded to bootflash or download during the install), the wizard may prompt for additional file locations. The fourth and final screen will provide a summary and enable the user to start the install. During the install, a compatibility screen will pop-up displaying the same version compatibility information that was displayed in the CLI upgrade. The user must select Yes to continue with the upgrade.

Note

Unlike the CLI, the FM maintains connectivity to the switch and provides detailed information during the entire upgrade sequence without requiring the user to manually reestablish connectivity to the switch during the supervisor switchover. If there is a failure, the last screen will display any reasons for a failed upgrade.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-27

Chapter 1 Password Recovery

Management

Password Recovery
If there are no accounts accessible on the MDS that have either network-admin or user account creation privileges, due to lost passwords, it may be required to perform a password recovery on the admin account.

Warning

This procedure requires console access to the switch and will require the switch to be rebooted.

Tip

It is possible for another CLI user with network-admin privileges to change the password of the admin user, which can alleviate reloading the switch.

To recover the admin accounts password the following procedure can be used:
Step 1

If possible, save the current configuration:


switch# copy running-config startup-config [########################################] 100%

Step 2

Connect a console cable to the active supervisor of the MDS Switch:


Figure 1-6 Console Connection on 9500 and 9200 series MDS switches.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-28

78-xxxxx-xx

Chapter 1

Management Password Recovery

Step 3 Step 4

Attach the RS-232 end of the console cable to a PC. Configure Hyperterm or similar terminal emulation software for 9600 baud, 8 data bits, no parity, 1 stop bit and no flow control:
Figure 1-7 HyperTerm Terminal Settings.

Step 5 Step 6

Establish a connection to the switch if possible or enough to display the login prompt if no user accounts are available. For a multi supervisor switch, MDS-9509 or MDS-9506, physically remove the standby supervisor. It is not necessary to remove it from the chassis, just enough such that it does not make contact with the backplane.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-29

Chapter 1 Installing a License

Management

Step 7 Step 8 Step 9

Reboot the switch either by cycling the power or issuing the reload command. Press the Ctrl-] key sequence (when the switch begins its SAN-OS software boot sequence) to enter the switch(boot)# prompt. Enter configuration mode:
switchboot# config terminal

Step 10

Issue the command admin-password <new password>


switch(boot-config)# admin-password temppassword switch(boot-config)# exit

Step 11

Load the system image to finish the boot sequence.


switch(boot)# load bootflash: m9500-sf1ek9-mz.2.0.2b.bin

Step 12

Log into the switch using the admin account and the temporary password.
switch login: admin Password: Cisco Storage Area Networking Operating System (SAN-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2004, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by Andiamo Systems, Inc. and/or other third parties and are used and distributed under license. Some parts of this software are covered under the GNU Public License. A copy of the license is available at http://www.gnu.org/licenses/gpl.html. switch#

Step 13

Change the admin password to a new permanent password:


ca-9506# config terminal Enter configuration commands, one per line. End with CNTL/Z. ca-9506(config)# username admin password g05ox

Step 14

Save the configuration which will include the new password.


switch# copy running-config startup-config [########################################] 100%

Installing a License
In order to run the advanced features of the MDS platform, a license key is required for permanent usage. A 120 day grace period allows the user to try out the feature or if to resume operation in the case where a chassis is replaced until a license key is installed.

Warning

If a licenses grace period expires, all of the features that depend on that license will be disabled even if they are currently running or in production.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-30

78-xxxxx-xx

Chapter 1

Management Installing a License

To install a license key there are two methods available, using either the CLI or Fabric Manager.

Using the CLI to Install a License:


Step 1

Copy the license file to the bootflash of the supervisor


switch# copy scp://user1@172.22.36.10/tmp/FM_Server.lic bootflash:FM_Server.lic user1@172.22.36.10's password: FM_Server.lic 100% |*****************************| 2035 00:00

Step 2

Verify the license file:


switch# show license file FM_Server.lic lic.lic: SERVER this_host ANY VENDOR cisco INCREMENT FM_SERVER_PKG cisco 1.0 permanent uncounted \ VENDOR_STRING=MDS HOSTID=VDH=FOX0713037X \ NOTICE="<LicFileID>lic_template</LicFileID><LicLineID>0</LicLineID> \ <PAK>dummyPak</PAK>" SIGN=D8CF07EA26C2

Step 3

Cross reference the switchs host-id (VDH=FOX0713037X) with that which is listed in the license file:
ca-9506# show license host-id License hostid: VDH=FOX0713037X

Step 4

Install the license file:


switch# install license bootflash:FM_Server.lic Installing license ..done

Step 5

Verify license has been installed:


switch# show license lic.lic: SERVER this_host ANY VENDOR cisco INCREMENT FM_SERVER_PKG cisco 1.0 permanent uncounted \ VENDOR_STRING=MDS HOSTID=VDH=FOX0713037X \ NOTICE="<LicFileID>lic_template</LicFileID><LicLineID>0</LicLineID> \ <PAK>dummyPak</PAK>" SIGN=D8CF07EA26C2

To display a summary of the installed licenses use the command show license summary:

switch# show license usage Feature Insta License Status Expiry Date Comments lled Count ----------------------------------------------------------------FM_SERVER_PKG Yes In use never MAINFRAME_PKG No Unused -

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-31

Chapter 1 Installing a License

Management

ENTERPRISE_PKG Yes In use never SAN_EXTN_OVER_IP Yes 2 In use never SAN_EXTN_OVER_IP_IPS4 No 0 Unused -----------------------------------------------------------------

To display which features within a license package are being used, specify the package name. In this case QOS is using the Enterprise package.:
ca-9506# show license usage ENTERPRISE_PKG Application ----------Qos Manager -----------

Using Fabric Manager to Install a License


Step 1

To install the licenses launch the License Installation Wizard:


Figure 1-8 License Installation Wizard

Step 2

This will bring up the following screen which will allow the user to specify how to install the keys based upon whether or not the user has already obtained the license key files or if they only have a Product Authorization Key (PAK). If the user has a PAK then the license file can be downloaded and installed from Ciscos website.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-32

78-xxxxx-xx

Chapter 1

Management Installing a License

Figure 1-9

License Installation Method

a.

If the keys already exist on a server then the following dialog box will be provided to specify the name and location of the license key files:

Figure 1-10 License File Location

b.

If the license files are not already available, and the user only has the PAK numbers, then Fabric Manager can obtain the license files directly from Cisco.com.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-33

Chapter 1 Installing a License

Management

Figure 1-11 Install License using PAK

At this point, the license keys can be installed and the licensable feature can be used.

What Feature is Enabling the License Grace Period?


Often a user may enable a feature that is licensed and being to receive syslog, email or other warnings informing them that a feature, for example Fabric Manager Server, has entered its grace period. To determine what features have enabled a specific license one of the following two methods can be run with either the CLI or Fabric manager.
1.

With Fabric Manager, in the Physical Attributes Pane, select Switches, then Licenses. In the resulting middle pane, select the Usage tab:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-34

78-xxxxx-xx

Chapter 1

Management Copying core files from the MDS

Figure 1-12 License Usage

2.

With the CLI, run the command show license usage <license name>:

switch1# show license usage ENTERPRISE_PKG Application ----------SFM Qos Manager Port Security -----------

At this point the feature has been identified and it can be either disabled or if the feature is needed then a license should be installed.

Copying core files from the MDS


If an MDS process crashes it may create a core file which would then be sent to Cisco TAC for further troubleshooting. This procedure will cover how to copy a core file off of the MDS switch. The resources used in this procedure are: FTP Server: 172.22.36.10

Note

The destination system can any of the possible destinations including tftp, scp, sftp or slot0. Before copying a core file to another server, the pid of the core file must be identified.
switch# show cores Module-num Process-name PID Core-create-time

Step 1

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-35

Chapter 1 Fixed Switch Config Restoration

Management

---------- ------------ --- ---------------5 fspf 1524 Sep 27 03:11

Step 2

Then to copy the core file using, for example, ftp issue the following command syntax:
"core://<module-number>/<process-id>" switch# copy core://5/1524 ftp://172.22.36.10/tmp/fspfcore

The file can now be sent to Cisco TAC according to the TAC engineers directions.

Fixed Switch Config Restoration


This procedure will cover the process for backing up and restoring a switch configuration for one of the MDS 9000 Familyfixed configuration switches such as the 9100 or 9200 series switches. Parts of this procedure are disruptive and should only be done in the event of an emergency, including if the chassis or fixed supervisor needs to be replaced. This procedure will leverage the following resources:
Old Switch: switch1: (172.22.36.8) New Switch: switch2 File Server: host1

Note

Only restore a switch configuration to a switch which has the exact same firmware version on it as which was used to create the switch configuration. If an upgrade is required, restore the configuration then upgrade the firmware. Save the running configuration:
switch1# copy running-config startup-config [########################################] 100%

Step 1

Step 2

Copy the startup-configuration to the file server using any of the available methods on the MDS (ftp, tftp, sftp, scp):
switch1# copy startup-config scp://user@host1/switch1.config user@switch1's password: sysmgr_system.cfg 100% |*****************************| 10938 switch1#

00:00

Step 3

Capture the port assignments using the FLOGI database. This will be used to verify that all of the cables are placed in the correct locations.
switch1# show flogi database --------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME --------------------------------------------------------------------------fc1/8 600 0x7c0007 50:05:07:63:00:ce:a2:27 50:05:07:63:00:c0:a2:27 fc1/13 1001 0xef0001 50:06:0e:80:03:4e:95:13 50:06:0e:80:03:4e:95:13 fc1/15 600 0x7c0004 50:06:0b:00:00:13:37:ae 50:06:0b:00:00:13:37:af

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-36

78-xxxxx-xx

Chapter 1

Management Fixed Switch Config Restoration

Note

At this point the old switch is no longer needed and its mgmt0 port should be disconnected from the LAN.

Step 4

Log on to the new switch using the console connection and clear the switchs configuration. Do not run the setup script, if prompted. The write erase command will erase the switchs configuration.
switch2# write erase Warning: This command will erase the startup-configuration. Do you wish to proceed anyway? (y/n) [n] y

Step 5

Reload the switch.


switch2# reload This command will reboot the system. (y/n)? [n] y

Step 6

The switch will come up in its factory default mode and will prompt for the Basic System Configuration Dialog, skip it, as all the configuration options are contained in the old switchs startup-config. Manually configure the IP address.
switch2# config terminal Enter configuration commands, one per line. End with CNTL/Z. switch2(config)# int mgmt 0 switch2(config-if)# ip address 172.22.36.8 255.255.254.0 switch2(config-if)# no shut

Step 7

If interface (fcX/Y) based zoning was done, obtain the new switchs wwn. Otherwise, this step can be skipped.
switch2# show wwn switch Switch WWN is 20:00:00:0d:ec:02:1d:40

Step 8

On the file server, make a copy of the configuration file and then open it in a text editor (notepad, vi).
a.

Remove the lines that contain the snmp user accounts, as the encrypted passwords are tied to the MAC address of the chassis.:
$ cp switch1.config switch1.config.orig $ vi switch1.config

The user accounts are all grouped together and begin with the command snmp-server user
snmp-server user admin network-admin auth md5 0x46694cac2585d39d3bc00c8a4c7d48a6 localizedkey snmp-server user guestadmin network-admin auth md5 0xcae40d254218747bc57ee1df348 26b51 localizedkey

b.

If interface (fcX/Y) zoning was not done skip this step. Otherwise, replace the old switchs wwn in the zone member commands with the the new switchs wwn:
zone name Z_1 vsan 9 member interface fc1/9 swwn 20:00:00:0d:ec:02:1d:40

c. Step 9

Save and exit the config file.

From the new switch, copy the modified config file from the fileserver onto the new switchs running configuration. As the file is copied it will be executed on the switch as the configuration is applied. The commands being applied with be contained in single-quotes. Any errors due to applying the commands will be displayed immediately after the error causing command is executed. The prompt will change to reflect the new switchname.
switch2# copy scp://user@host1/switch1.config running-config user@host1's password: switch1.config 100% |*****************************| 10938

00:00

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-37

Chapter 1 Configuring an NTP Server

Management

Step 10

Save the configuration, by copy startup-config to running-config


switch1# copy running-config startup-config [########################################] 100%:

Step 11

At this point, thecan be accessed via the CLI and the following remaining items will need to be performed:
a. b. c.

SNMP user accounts will need to be recreated. Also, If the switch is accessed via ssh, the hosts known_hosts file will need to have the MDS switchs entry removed from it, because the switchs public key is different. License keys, if required, will need to be installed.

Step 12 Step 13

Move the cables from the old switch to the new switch, using the old switchs show flogi database as a reference to verify that each cable is in the correct location. Verify that all devices have logged in and all features are running is as they are supposed to be, save the running-configuration to the startup-config with the command copy running-config startup-config. Reload the switch to verify that it boots correctly with the configuration.

Step 14

Configuring an NTP Server


Network Time Protocol (NTP) is a protocol used by devices to synchronize their internal clocks with other devices. The MDS can only be used as an NTP client and would talk to other NTP systems which are considered to have a higher stratum (or authority). NTP is hierarchical in nature such that the lower stratum numbers are closer to the source of the time authority. Devices that are at the same stratum can be configured as peers so that they will work together to determine the correct time by making minute adjustments. Normally, MDSs would be configured as peers while a router or other dedicated machine would be used as a NTP server.

Configuring NTP with CFS


Leveraging the CFS facility as explored in CFS: Cisco Fabric Services, page 1-13, allows the administrator to perform a single configuration for NTP and have it be propagated to other switches. Also, if a new switch comes online, it can be set to inherit the NTP configuration from the existing switches, the new switch would merge its configuration (no configuration) with the existing NTP CFS configuration and the result would be that the new switch would have an NTP configuration.

Note

NTP will not set the timezone (or offset from UTC) for the switch, it must be set manually using for example for Eastern Standard Time and Eastern Daylight-Savings Time: clock timezone EST -5.0 clock summer-time EDT 1 Sunday Apr 02:00 5 Sunday Oct 02:00 60 For this example:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-38

78-xxxxx-xx

Chapter 1

Management Configuring an NTP Server

Switch #1 IP Address: 172.22.36.142 Switch #2 IP Address: 172.22.36.8 NTP Server: 171.69.16.26 To configure ntp for switch1, the following procedure would be used:
Step 1

Enter configuration mode and enable CFS distribution for NTP


a.

switch1:
switch1# conf t Enter configuration commands, one per line. switch1(config)# ntp distribute End with CNTL/Z.

b.

switch2
switch2# conf t Enter configuration commands, one per line. switch2(config)# ntp distribute End with CNTL/Z.

Step 2

Add the ntp server (this and the following steps can be done from either switch1 or switch2).
switch1# conf t switch1(config)# ntp server 171.69.16.26

Step 3

Add the ntp peer switches


switch1(config)# ntp peer 172.22.36.8 switch1(config)# ntp peer 172.22.36.142

Step 4

Commit the ntp configuration via CFS


switch1(config)# ntp commit switch1(config)# end

At this point, NTP is configured and the switch will slowly adjust to the new time. To view the NTP configuration on the local switch:
switch1# show ntp peers -------------------------------------------------Peer IP Address Serv/Peer -------------------------------------------------172.22.36.142 Peer 171.69.16.26 Server 172.22.36.8 Peer To view the NTP configuration on the remote switch: switch2# show ntp peers -------------------------------------------------Peer IP Address Serv/Peer -------------------------------------------------172.22.36.142 Peer 171.69.16.26 Server 172.22.36.8 Peer

Step 5

Save the configuration fabric wide leveraging the fabric keyword. Which instructs all CFS capable switches to perform copy running-config startup-config. See Saving the configuration across the fabric, page 1-42

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-39

Chapter 1 Configuring an NTP Server

Management

switch1# copy running-config startup-config fabric [########################################] 100%

Without CFS
Configuring NTP without CFS will require the user to log onto every switch in the fabric and configure NTP. This is the same procedure that would be used in a SAN-OS 1.x environment.

Note

NTP will not set the timezone (or offset from UTC) for the switch, it must be set manually using for example for Eastern Standard Time and Eastern Daylight-Savings Time: clock timezone EST -5.0 clock summer-time EDT 1 Sunday Apr 02:00 5 Sunday Oct 02:00 60 For this example: Switch #1 IP Address: 172.22.36.142 Switch #2 IP Address: 172.22.36.9 NTP Server: 171.69.16.26 To configure ntp for switch1, the following procedure would be used:

Step 1

Enter configuration mode and add the ntp server


switch1# conf t Enter configuration commands, one per line. switch1(config)# ntp server 171.69.16.26

End with CNTL/Z.

Step 2

Add the ntp peer switch


switch1(config)# ntp peer 172.22.36.9 switch1(config)# end

At this point, NTP is configured and the switch will slowly adjust to the new time.

To view the NTP configuration:


switch1# show ntp peers -------------------------------------------------Peer IP Address Serv/Peer -------------------------------------------------171.69.16.26 Server 172.22.36.9 Peer

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-40

78-xxxxx-xx

Chapter 1

Management Steps to Perform Before Calling TAC

Steps to Perform Before Calling TAC


At some point, the administrator may need to contact the Cisco TAC or their OSM for some additional assistance. This section outlines the steps that the admin should perform prior to contacting their next level of support, as this will reduce the amount of time to resolve the issue.

Step 1 Step 2

Do not reload the line card or the switch until atleast Step 2has been done. Some logs and counters are kept in volatile storage and will not survive a reload. Collect switch information and configuration. This should be done before and after the issue has been resolved. The three methods below each provide the same information.
a. b.

CLI: Configure the telnet/ssh application to log the screen output to a text file and issue the command show tech-support details. CLI: Issue the command tac-pac <filename> (ex: tac-pac bootflash://showtech.switch1). The tac-pac command will redirect the output of show tech-support details to a file then gzip the file. If no filename is specified, the file created is volatile:show_tech_out.gz. The file should then be copied off the MDS using the procedure outlined inCopying files to or from the MDS on page 19. Fabric Manager: Select Tools, then select Show tech support. Fabric manager can capture switch configuration information from multiple switches simultaneously. The file can be saved on the local PC. For IVR related issues capture the output of show ivr tech-support If the error occurs in Fabric Manager, take a screen shot of the error. In windows, to capture the active window use ALT+PrintScreen, and to capture the entire desktop hit just PrintScreen. Then paste this into a new MSpaint.exe (or similiar program) session. Copy the error from the message log which can be displayed using either show logging log or to view the last X lines of the log show logging last #

c.

d. Step 3

Capture the exact error codes:


a.

b.

The following questions should be answered prior to placing the call to TAC:
1.

Which switch, hba, or storage port is the problem occurring?


a. List MDS firmware, driver versions, operating systems versions and storage device firmware.

2. 3. 4. 5. 6. 7. 8. 9.

What is the network topology? (FM/tools ->show tech & save map) Were any changes being made to the environment (zoning, adding linecards, upgrades) prior to or at the time of this event? Are there other similarly configured devices that could have this problem but do not have? Where this problematic device connected to (MDS switch Z, interface x/y)? When did this problem first occur? When did this problem last occur? How often does this problem occur? How many devices have this problem? have already been done? Using such tools as:

10. Were any traces or debug outputs captured during the problem time? What troubleshooting steps

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-41

Chapter 1 Saving the configuration across the fabric

Management

a. Fcanalyzer, PAA-2, Ethereal, local or remote SPAN b. CLI debug commands c. FC traceroute, FC ping d. FM/DM

Saving the configuration across the fabric


In large installations a change may have been propagated either via a host based script or CFS and then the the configuration needs to be saved. Rather than logging into every switch using a script or launching Fabric Manager. The copy running-config startup-config can be propagated using CFS to all the switches. Once this has finished all the switches will have saved their configuration:
switch# copy running-config startup-config fabric [########################################] 100%

This command will take slightly longer than the local-only copy running-config startup-config would.

How to Disable the Webserver


On the MDS platform, a built in webserver is used to provide users with a method of downloading and installing Device and Fabric Manager. After installation of these tools, the webserver is nolonger needed, in fact it is possible to download a specific version of Fabric Manager and Device Manager directly from CCO (http://www.cisco.com). However, to increase the security of the MDS platform and IP ACL can be leveraged to block all access to TCP port 80 of the MDS. The IP ACL that will be set up will deny access to TCP port 80 on the MDS but allow access to all other TCP ports.

Note

IP ACLs have an implicity DENY ALL added to the end of the users ACL, so atleast one permit statement must be used otherwise all traffic will be denied.

Tip

Whenever using ACLs on mgmt0, make sure that it is tested out on a switch that has console access in case a typo is made. When deploying multiple MDSs, the commands in this recipe can be integrated into a script, by taking the bolded commands and pasting them into a CLI session.

The following CLI statements will create an IP ACL filter and then apply it to the mgmt0 interface:
Step 1

First create the access list called disable_webserver. Notice that the second entry for the access list is a permit any statement.
switch1#conf t

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-42

78-xxxxx-xx

Chapter 1

Management Device Aliases

switch1(config)#ip access-list disable_webserver deny tcp any any eq port www switch1(config)#ip access-list disable_webserver permit ip any any

Step 2

Then apply it to mgmt0


switch1(config)#interface mgmt0 switch1(config-if)#ip access-group disable_webserver in

Device Aliases
Device aliases provide a plain text name to pwwn mapping. This one to one mapping was brought upon by the need to be able to identify devices within an MDS environment by easier to understand labels rather than pwwns. The device alias can be used in areas such as, zoning, QOS, IVR, and throughout Fabric Manager and the Performance Manager utilities. This CFS aware application extends to the various CLI output by inserting the Device Alias where appropriate.

Note

The rules for Device Alias are the following:


It is a one to one mapping of pwwn to plain text name. The CFS scope is physical. A Device Alias database is not tied to a VSAN so if you move an end device (ex: hba) from one VSAN to another the device alias still applies. The merging of two device alias databases produces a union of the two databases. Conflicts are not imported into the resulting database and must be manually resolved. This will not keep the non-conflicting entries from being merged into the new database. The merge failure will not isolate an ISL. The CLI manages Device Aliases in a central database while Fabric Manager manages them under their respective type (Hosts and Storage). For services that leverage Device Alias (zoning, IVR, QOS etc), updating the Device Alias with a new pwwn, will not automatically propagate to the services. Those services should have the device removed, then update the device alias, and then re-add the device alias to the service.

Tip

Use the plain text name to describe the hostname and hba instance of the device. For example, host123_lpfc0 or SYMM7890_FA14ab is preferrable to just host123 and SYMM7890. The device alias can then become the basis for a Fabric Manager Enclosure. Device Aliases SYMM7890_FA14ab and SYMM7890_FA14ba should be part of the enclosure SYMM7890.

Device Aliases with the CLI


Device Aliases can be leveraged with the CLI the below recipes demonstrate how.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-43

Chapter 1 Device Aliases

Management

Examples of Devices Aliases Displayed in the CLI:


The CLI can display Device Aliases in show commands, below are a few examples:
1.

Displaying the Nameserver:

ca-9506# show fcns database VSAN 1: -------------------------------------------------------------------------FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------0x620000 N 10:00:00:00:c9:32:8b:a8 (Emulex) scsi-fcp:init [ca-sun1_lpfc0] 0x65000a N 10:00:00:00:c9:34:a6:3e (Emulex) [ca-aix1_fcs0]

2.

Displaying the active zoneset containing IVR and regular zones:

zoneset name nozoneset vsan 501 zone name IVRZ_IvrZone1 vsan 501 pwwn 50:06:0e:80:03:4e:95:23 [HDS20117-c20-9] pwwn 10:00:00:00:c9:32:8b:a8 [ca-sun1_lpfc0] zone name ca_aix2_HDS vsan 600 pwwn 10:00:00:00:c9:34:a5:94 [ca-aix2_fcs1] pwwn 50:06:0e:80:03:4e:95:23 [HDS20117-c20-8]

3.

Displaying the flogi database:

ca-9506# show flogi database --------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME --------------------------------------------------------------------------fc2/5 1000 0xef0008 50:06:0e:80:03:4e:95:23 50:06:0e:80:03:4e:95:23 [HDS20117-c20-9]

Creating Device Aliases with the CLI


To create a Device Alias in the CLI the following recipe should be followed using the below resources as an example: Host: ca-aix1 HBA Instance: fcs0 PWWN: 10:00:00:00:c9:34:a6:3e
Step 1

Device Alias is enabled by default, so first enter the Device Alias database:
ca-9506# conf terminal Enter configuration commands, one per line. ca-9506(config)# device-alias database ca-9506(config-device-alias-db)# End with CNTL/Z.

Step 2

Create the entry for the Device Alias using a meaingful name.
ca-9506(config-device-alias-db)# device-alias name ca-aix1_fcs0 pwwn 10:00:00:00 :c9:34:a6:3e

Step 3

Display the pending changes. A + means that the entry is being added to the database, while a - means that it will be removed from the database, upon performing a CFS commit.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-44

78-xxxxx-xx

Chapter 1

Management Device Aliases

ca-9506(config-device-alias-db)# do show device-alias database pending-diff + device-alias name ca-aix1_fcs0 pwwn 10:00:00:00:c9:34:a6:3e

Step 4

CFS Commit the changes:


ca-9506(config-device-alias-db)# device-alias commit ca-9506(config)#

Converting FC Aliases into Device Aliases


An existing MDS environment that uses FC Aliases can be converted to use Device Aliases by importing the FC Aliases using the CLI. Only those FC Aliases which are also valid Device Aliases will be imported. A FC Alias is eligible to be imported if:
The FC Alias represents a pwwn. The FC Aliases represents exactly one device and not a group of devices. A Device Alias does not already exist with the same name. A Device Alias does not already exist with the same pwwn.

Note

While the Device Alias database is maintained on all switches as it has a physical scope, FC Aliases have a vsan scope and may not be present or replicated to all switches if Full Zoneset Distribution is not enabled. It may be required to log onto multiple switches to import all of the FC Aliases. Importing FC Aliases does not automatically update any zones that are based upon FC Aliases. The zones will need to manually converted to Device Alias based zones. Import FC Aliases does not automatically delete the FC Aliases. The FC Aliases will need to be manually deleted later.

In this example the following FC Aliases will be imported and converted into Device Aliases:
fcalias name alias123 vsan 1 fcalias name temphost vsan 1 pwwn 11:11:11:11:11:11:11:11 fcalias name temphost vsan 2 pwwn 11:11:11:11:11:11:99:99 fcalias name temphost2 vsan 2 pwwn 11:11:11:11:11:22:22:22

Ineligible FC Aliases are listed in vsan 1 and 2 to illustrate how to determine which FC Aliases failed to be imported.
Step 1

Enter config mode:


switch# conf terminal Enter configuration commands, one per line. switch(config)# End with CNTL/Z.

Step 2

Import the FC Aliases


switch(config)# device-alias import fcalias vsan 1-2 WARNING: Some fcaliases from the specified VSAN range could not be imported due to conflicts.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-45

Chapter 1 Device Aliases

Management

a.

If a Warning is displayed then examine the logs to determine which FC Aliases did not import:
switch(config)# do show device-alias internal errors 1) Event:E_DEBUG, length:111, at 608209 usecs after Wed Sep 14 16:25:13 2005 [109] ddas_import_fcalias(703): CONFLICT: fcalias temphost on vsan 2 has same name as fcalias temphost on vsan 1 2) Event:E_DEBUG, length:127, at 607826 usecs after Wed Sep 14 16:25:13 2005 [109] ddas_import_getnext_alias_resp_handler(1119): not importing alias alias123 from vsan 1 as the number of members are not 1.

Examining the contents of the FC Alias alias123 shows that it is not a valid Device Alias because it doesnt have any pwwn members, and that vsan 2s FC Alias temphost will not be imported as the same name was imported from vsan 1.
b.

At this point the conflicts should be resolved, by deleting the FC Alias (as in the case of alias123) or renaming one of the conflicting aliases (as in case of temphost).

Note

FC Aliases might be part of an existing zone and so the appropriate zone(s) should be updated accordingly. Display the pending Device Alias Database to display the newly imported Device Aliases
switch(config)# do show device-alias database pending-diff + device-alias name temphost2 pwwn 11:11:11:11:11:22:22:22

Step 3

Step 4

CFS Commit the pending changes:


switch(config)# device-alias commit

Device Aliases with FM


Enabling Fabric Manager to use Device Alias
Fabric Manager can leverage Device Aliases to provide plain text names in many locations including the map, zoning, and QOS. However, FM needs to be configured to use Device Aliases instead of FC Aliases. It can be enabled either during installation or afterwards. For example, in the figure below, DPVM is using Device Aliases to represent the pWWNs in its configuration. HDS20117-c20-8 in VSAN 1000 is plugged into MDS 172.22.36.11 port fc1/1, is easier to understand than the same description but using a wwn. In this naming scheme the model (HDS), serial number (20117) cluster (20) and port (8) were all used in the name to accurately describe the device.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-46

78-xxxxx-xx

Chapter 1

Management Device Aliases

Figure 1-13 DPVM Leveraging Device Aliases

To enable during installation check the box marked Use Global Device Aliases in place of FC Aliases, at the initial installation screen. For SAN-OS 2.0 through 2.1(1a):
Step 1

In MDS SAN-OS 2.1(1a) and below, to enable after installation, modify the server.properties file (default location: C:\Program Files\Cisco Systems\MDS 9000\server.properties) entry to true:
# Specify whether to discover device aliases from a global alias server. # Default is false if unspecifed. fabric.globalAlias = true

Step 2 Step 3

Restart the Fabric Manager service. Note: Logging out and back into FM will not reread the file. Log back into FM.

For SAN-OS 2.1(2) and later:


Step 1

FM can be configured to use Device Aliases after installation by selecting the Device Alias checkbox under the Admin -> Server menus.

Creating a Device Alias for an Existing Device


To create a Device Alias in Fabric Manager for a device already logged into the fabric, the following recipe should be followed using the below resources as an example: Host: ca-aix1

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-47

Chapter 1 Managing Fabric Manager

Management

HBA Instance: fcs0 PWWN: 10:00:00:00:c9:34:a6:3e


Step 1

Prior to performing any Device Alias work in Fabric Manager, FM should be configured to leverage Device Aliases as in Enabling Fabric Manager to use Device Alias, page 1-46. In Fabric Managers Physical Attributes pane, expand End Devices then since the the wwn corresponds to a host, select the Hosts menu item. For a storage device select the Storage menu item. Find the pwwn and in the Device Alias column enter in the device alias for the referenced pwwn. Press the Apply changes button.
Figure 1-14 Create Device Alias with Fabric Manager

Step 2 Step 3

Managing Fabric Manager


The recipes contained within the following sections pertain to configuring Fabric Manager Client and Server applications rather than objects that reside stricly on the MDS.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-48

78-xxxxx-xx

Chapter 1

Management Managing Fabric Manager

Operating Fabric Manager Through a Firewall using SNMP Proxy


Leveraging the Fabric Manager Server application a storage administrator can connect to an MDS switch that is behind a firewall. The Fabric Manager Client application will encapsulate the SNMP protocol in TCP which will traverse the firewall and connect to the Fabric Manager Server, which in turn will forward the SNMP packets to the MDS. For the Fabric Manager Client to connect to the Fabric Manager Server the following TCP ports will need to be opened on the firewall:
1. 2. 3.

TCP port 9198.. The FM Client uses this TCP port to send the encapsulated SNMP packets to the FM Server. TCP ports 9099-9200. TCP port 9099 is used for java.rmi.registry.port while the other ports are used for java.rmi.server.remoteObjectPort. TCP port 80. Used to view the Performance Monitors web based statistics.

Additionally, for some features the Fabric Manager Client uses the CLI to obtain information on the switch directly. Therefore one of the two options should be used to allow the Fabric Manager Client to access the switch.
1. 2.

TCP port 23. When the Fabric Manager Client has been configured to use telnet for CLI access into the Cisco MDS. This is the default setting for the Fabric Manager Client. TCP port 22. When the Fabric Manager Client has been configured to use SSH for CLI access into the Cisco MDS. This is the preferred setting for this environment, and the MDS will need to have SSH enabled.

Note

All of the ports that Fabric Manager Client and Server use and which if need be modified are contained in the server.properties file, which may be found the following directory C:\Program Files\Cisco Systems\MDS 9000.

Configuration using a non-NAT Packet Filter


In this recipe, the Fabric Manager Client is separated from the Fabric Manager Server and the Cisco MDS by a packet filtering devices which does not perform network address translation (NAT). Additionally, the packet filter allows any internal device unrestricted access to the external network. Lastly, SSH has been configured on the MDS and telnet has been disabled.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

1-49

Chapter 1 Managing Fabric Manager

Management

Figure 1-15 SNMP Proxy Topology

Step 1

Configure the firewall to allow the following TCP connections through:


1. 2. 3.

TCP port 9198. The FM Client uses this TCP port to send the encapsulated SNMP packets to the FM Server. TCP ports 9099-9200. TCP port 9099 is used for java.rmi.registry.port while the other ports are used for java.rmi.server.remoteObjectPort. TCP port 23. Some features of the FabricManager client require telnet access into the MDS. However, if SSH is enabled on the MDS, the FM Client can leverage SSH instead. If SSH is used, then this port is not required. TCP port 80. Used to view the Performance Monitors web based statistics. TCP port 22. Should be enabled if telnet on the MDS is disabled and SSH is enabled.

4. 5. Step 2

Configure the host running the FM Client how to reach the network behind the firewall where the FM Server resides. The 10.0.0.0 network is behind the firewall while 157.11.22.99 is the external IP address of the firewall. This is not the mgmt0 address of the firewall. On a windows host this may be done by:
C:\>route add 10.0.0.0 mask 255.255.255.0 157.11.22.99

Step 3

Configure the MDS to be able to reach the Fabric Manager Client (157.11.1.50) which is outside of the firewall. The internal IP address (the side which faces the MDS) assigned to the firewall is 10.0.0.44.
switch(config)# ip route 157.11.1.50 255.255.255.0 10.0.0.44

Step 4

Launch the Fabric Manager Client, and in the login screen, select the Use SNMP Proxy checkbox and enter the IP addresses of the Fabric Manager Server and the MDS Switch.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

1-50

78-xxxxx-xx

C H A P T E R

Account Management
This chapter provides recipes for managing users and their accounts. In MDS firmware versions prior to 2.0, a separate account was required for both SNMP and CLI access. After 2.0, a single username grants access to both CLI and SNMP.

Note

A new MDS with SAN-OS 2.x preinstalled will not have a existing admin password. The user will need to choose one the first time they run the setup script. Existings accounts will not be forced to change their password. Upgrading from 1.x to 2.x will cause the 1.x CLI password to be used for both accounts where there are different passwords for the same user with regard to CLI and SNMP access. SAN-OS 2.x enforces strong passwords for all accounts created after the installation/upgrading to 2.x. A strong password must have the following characteristics
Must be atleast 8 characters long. Should not have too many consecutive characters. Should not have too many repeated characters. Should not contain easily guessable dictionary words.

Tip

The admin account should only be used during initial setup. Afterwards, other user accounts should be created. Each administrator should have their own individual account. Always change the admin password from the factory default value. Users should be granted the minimum amount of rights or abilities required to perform their job function. Implementing TACACS+, Configuring TACACS+ with Cisco SecureACS, page 2-57 eliminates the need to perform password recovery on the MDS.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

2-51

Chapter 2 Creating User Accounts

Account Management

Creating User Accounts


In order to access the MDS switch a username must be created. To create such a user, the following procedure can be used:
Step 1

Enter the configuration submode:


switch# config terminal

Step 2

Create the CLI user with using the following syntax: username <username> password <password> role <role>
switch(config)# username user1 password sox2004ch@mps role network-admin

At this point the user user1 can access the switch using the password sox2004ch@mps via console, ssh, telnet or snmp.

Creating a MDS User Role


The MDS switch has two default roles, network-admin and network-operator. Network-admin has write privileges to all parts of the MDS while network-operator has read-only access to the switch. It may be required that a user with write access to specific areas of the MDS is required while eliminating write access to other areas. The MDS switch comes with two predefined roles:

Network-admin is the role assigned to the predefined user admin. It has the ability to perform any modification to the MDS platform. There are no restrictions on this user. Network-operator is a predefined read-only role. It does not have the ability to make modifications to the MDS. There are no predefined users assigned to this role.

Tip

Provide each user with a role that provides them with the minimum amount of abilities to perform their job function. Leverage the read only role network-operator for those users who do not require the ability to modify the MDS. VSAN based role based access can allow administrators to have complete control over their VSANs while having read only or no access to other VSANs.

Creating a Role with Device Manager


In this example, a role will be created providing the ability to only modify the zoning configuration on the switch.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

2-52

78-xxxxx-xx

Chapter 2

Account Management Creating a MDS User Role

Step 1

In Fabric Manager, enable role distribution for all switches in the fabric. In the Physical Attributes pane, select Switches, Security then SNMP. Under the Enable Admin column, change the cells to enable. Then hit the Apply button. For this procedure, Fabric Manager is nolonger needed.
Figure 2-1 Enabling CFS distribution for Roles

Step 2

Open up Device Manager (DM) from any one of the switches that were just enabled for CFS distribution of roles, select Security and then Roles. DM will inform you that CFS is enabled, click continue.
Figure 2-2 Initial Roles Screen

Step 3

Select Create

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

2-53

Chapter 2 Creating a MDS User Role

Account Management

Figure 2-3

Create Common Roles

Fill in a name and description (do not use spaces) for the role. Optionally, a VSAN scope can be specified limiting this specific role to a subset of VSANs. Such that a zoning admin role could be created for zone admins that could only modify VSANs 1-10. This example will not specify a VSAN scope.
Step 4

To define what this role can and cannot do within the optional VSAN scope, select Rules
Figure 2-4 Create Rules

Click the CLI Command column to sort the table by CLI command. Select the show checkbox at the top and all of the zone, zone-attribute-group and zoneset commands. Additionally the copy Exec command should be checked so that the zoning admin can save the configuration.
Step 5

Select Apply. This will save the role configuration to CFSs pending database. Not until the CFS commit command has been executed will this role be part of the running-configuration of the switches in the fabric. This will bring up the previous dialog box.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

2-54

78-xxxxx-xx

Chapter 2

Account Management Creating a MDS User Role

Figure 2-5

Create Common Roles

Step 6

Select Create to add the role to the CFS pending database for Roles and then Close this will bring up the original Roles screen. The ZoningAdmin role only exists in the pending-database.
Figure 2-6 Display Role

Step 7

To commit the changes select the CFS button and then commit. To abort the changes and flush the pending-database, choose abort. To see the status of the CFS operation, from the main window, select Admin then CFS.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

2-55

Chapter 2 Creating a MDS User Role

Account Management

Figure 2-7

Create Role CFS Success

Creating Roles with CLI


In order to provide access to one or more switches, the CLI leveraging CFS can distribute the roles throughout the physical fabric.
Step 1

Enable CFS distribution for roles on all switches that should receive this role. This step should be done the first time for all switches. The other steps should be do not need to be repeated on each switch.
switch# conf t Enter configuration commands, one per line. switch(config)# role distribute End with CNTL/Z.

Step 2

Create the role:


switch(config)#role name ZoningAdmin switch(config-role)# description Zoneset_Administrator switch(config-role)# rule 1 permit show switch(config-role)# rule 2 permit config feature zoneset switch(config-role)# rule 3 permit exec feature zoneset switch(config-role)# rule 4 permit clear feature zone switch(config-role)# rule 5 permit config feature zone switch(config-role)# rule 6 permit debug feature zone switch(config-role)# rule 7 permit exec feature zone switch(config-role)# rule 8 permit exec feature copy

Step 3

Commit the role with CFS and distribute it to the other switches.
switch(config)# role commit

Step 4

To create a user, zoning_user, with the new role:

switch# config terminal switch(config)# username zoning_user password g0s0x456 role ZoningAdmin

Step 5

Save the configuration fabric wide

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

2-56

78-xxxxx-xx

Chapter 2

Account Management Configuring TACACS+ with Cisco SecureACS

switch(config)# copy running-config startup-config fabric [########################################] 100%

Configuring TACACS+ with Cisco SecureACS


Ciscos SecureACS product can enhance the MDSs management security, and provide centralized authentication, authorization and accounting of users.

Note

It is recommended that a tacacs+ server be used for both authentication, authorization and accounting. Implementing TACACS+ prevents having to perform password recovery on the MDS.

Authentication and Authorization with TACACS+


Configuring an MDS switch to use tacacs+ will allow centralized account management of the MDS. This centralized management will allow an admin to not have to create and maintain usernames and passwords on individual switches. The SecureACS server will provide the authentication to a switch login as well as assigning the role that the user is a member of. A shared secret key is used to provide encryption and authentication between the tacacs client (MDS-9500) and the tacacs server (Cisco SecureACS). In this procedure: The switchs ip address is 172.22.36.142 The tacacs+ servers ip address is 172.22.36.10 The tacacs+ shared secret key is: WarEagle

Configure SecureACS Server


Step 1

Prior to configuring the MDS, the SecureACS server must be configured. The first step is to configure SecureACS to allow the modification of advanced tacacs settings.
a. b.

On the main screen, on the left hand pane, select Interface Configuration, In the resulting screen select TACACS+ (Cisco IOS).

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

2-57

Chapter 2 Configuring TACACS+ with Cisco SecureACS

Account Management

c. d.

At the bottom of the screen, check Advanced TACACS+ Features and Display a window...attributes. This should result in the following screen. Select the Submit button at the bottom to save the changes.
SecureACS Configure Display

Figure 2-8

Step 2

Next is to define the MDS-9506 to the tacacs+ server, so that the MDS can be authenticated by the tacacs+ server. On the left hand pane, select Network Configuration and then Add Entry.
a. b.

Fill in the MDSs ip address, 172.22.36.142, and shared secret key WarEagle. Select Submit to save the information.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

2-58

78-xxxxx-xx

Chapter 2

Account Management Configuring TACACS+ with Cisco SecureACS

Figure 2-9

SecureACS Client Setup

Step 3

Next is to define a group. These groups should be used to provide an easy way to assign the same role to multiple users without having to modify the attributes of each user individually. On the left hand pane, select Group Setup.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

2-59

Chapter 2 Configuring TACACS+ with Cisco SecureACS

Account Management

Figure 2-10 SecureACS: Group Setup

Step 4

Select an available group and select Rename Group. In the resulting box, choose a new name for this group.

Tip

Use the same SecureACS group name as the MDS role, this will ease creation of tacacs based users. Click Submit to save the name change. On the left hand pane, select Group Setup, then select the newly renamed group and click Edit Settings. Scroll to the section labeled TACACS+ Settings, check Shell and Custom attributes. In the Custom attributes field, input the av-pair string corresponding to the role that is defined on the MDS for users. The syntax is: cisco-av-pair=shell:roles=<role> Select Submit + Restart to save and apply the configuration.

Step 5 Step 6

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

2-60

78-xxxxx-xx

Chapter 2

Account Management Configuring TACACS+ with Cisco SecureACS

Figure 2-11 SecureACS Adding MDS Role

Step 7 Step 8

Next is to define a user. Select User Setup from the left hand pane. Then type in a new or existing username and select Add/Edit. In the resulting window, fill in the fields for Password, Confirm Password and Group to which the user is assigned as illustrated below.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

2-61

Chapter 2 Configuring TACACS+ with Cisco SecureACS

Account Management

Figure 2-12 SecureACS Creating tacacs+ user

Configuring the SecureACS server has been completed and he remaining steps involve configuring the MDS Switch itself. It can be done from the CLI or SNMP.

Configure TACACS+ on the MDS


Step 1

Enable tacacs+

ca-9506# conf t Enter configuration commands, one per line. ca-9506(config)# tacacs+ enable

End with CNTL/Z.

Step 2

Define the tacacs+ server and the shared secret key to use with it.
ca-9506(config)# tacacs-server host 172.22.36.10 key WarEagle

Step 3

Define a group of authentication servers to use and add the tacacs+ server to the group.

ca-9506(config)# aaa group server tacacs+ tacacs-group1 ca-9506(config-tacacs+)# server 172.22.36.10

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

2-62

78-xxxxx-xx

Chapter 2

Account Management Configuring TACACS+ with Cisco SecureACS

Step 4

Define the methods for the switch to perform authentication for the telnet/ssh/snmp access.

ca-9506(config)# aaa authentication login default group tacacs-group1

Below are some show commands to display the configuration:

ca-9506# show tacacs-server timeout value:5 total number of servers:1 following TACACS+ servers are configured: 172.22.36.10: available on port:49 TACACS+ shared secret:******** ca-9506# show aaa authentication default: group tacacs-group1 console: local iscsi: local dhchap: local ca-9506# show user-account user:admin this user account has no expiry date roles:network-admin user:seth expires on Fri Jun 18 23:59:59 2004 roles:network-admin account created through REMOTE authentication Local login not possible

Note

The user seth is not available locally on the switch and yet is a member of the group/role network-admin. This means the user was authenticated by the tacacs+ server and not by the switch.

Accounting with TACACS+


Ciscos SecureACS server can be leveraged to provide a command history of what users performed which actions. This information is similar to the CLI command show accounting log. However, by placing it on a remote system, the logs can be independently examined and are available in case the MDS is inaccessible. This configuration will build upon the configuration defined in Authentication and Authorization with TACACS+, page 2-57.

Configuring the MDS


To configure the MDS to leverage a tacacs+ server for accounting the following procedure should be used:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

2-63

Chapter 2 Configuring TACACS+ with Cisco SecureACS

Account Management

Step 1

Configure the MDS to use the tacacs-group1 server group. The local keyword denotes to log locally on the switch if all servers listed in the server group are unavailable. If the server group is available, commands/events will not be logged locally.
switch# conf t switch(config)# aaa accounting default group tacacs-group1 local

Configuring SecureACS
Since this procedure builds on the configuration defined in Authentication and Authorization with TACACS+, page 2-57, only small modifications need to be made:
Step 1 Step 2 Step 3

Configure the SecureACS server to monitor for Update/Watchdog packets, by modifying the clients configuration. In SecureACS, in the left pane, click on Network Configuration. Select the client to be modified. Check the Log Update/Watchdog Packets from this AAA Client checkbox.
Figure 2-13 Enabling Accounting on the SecureACS server

Step 1 Step 2 Step 3 Step 4

Next is to configure the SecureACS to display the commands. Click on System Configuration in the left pane. Click on Logging. Click on CSV TACACS+ Accounting. Add the column err_msg, and check the Log to CSV TACACS+ Accounting report box.and click submit at the bottom.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

2-64

78-xxxxx-xx

Chapter 2

Account Management Configuring TACACS+ with Cisco SecureACS

Figure 2-14 Add MDS command logging to report.

Step 1 Step 2 Step 3

To view the accounting report, click Reports and Activity in the left pane. Click TACACS+ Accounting. In the right hand pane, select the day to view. The current day is called TACACS+ Accounting active.csv.
Figure 2-15 SecureACS Accounting Log

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

2-65

Chapter 2 Providing Passwordless Access Leveraging SSH

Account Management

Providing Passwordless Access Leveraging SSH


In some instances it may be required to allow access to the MDS without using a password, such as automated scripts or agents. Providing a null password or hard coding the password into the script or agent could be considered a weak security practice. However, leveraging the private/public key infrastructure associated with ssh maintains a solid secure environment. The procedure will include creating the appropriate key on a host and then adding it to a new user. Since SSH leverages a private/public key exchange, the MDS will know only the public key which the host will know both the public and private keys.

Tip

Passwordless logins should be assigned to either a read only role like network-operator or to one with a minimal set of privileges.

Warning

Only having the public key will not cause the MDS to grant access to a user the private key is required to be on the host. The private key should be guarded or treated like a password.

This procedure will cover the process of setting up a read-only (network-operator) based account which will only allow access if the user comes from a host that knows both the public and private keys.

Step 1

On the host, create a ssh rsa1 public/private key:

$ /usr/bin/ssh-keygen -t rsa1 Generating public/private rsa1 key pair. Enter file in which to save the key (/users/testuser/.ssh/identity): /users/setmason/.ssh/identity already exists. Overwrite (y/n)? y Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /users/testuser/.ssh/identity. Your public key has been saved in /users/testuser/.ssh/identity.pub. The key fingerprint is: c2:4d:6d:26:21:9d:79:9b:c3:86:dc:a5:07:d2:62:d4 testuser@host

On the host, the file /users/testuser/.ssh/identity.pub is the ssh public key that is encrypted using the rsa1 algorithm. The contents of this file will be used in the creation of the MDS user. In this example, the file looks like this:
$ cat /users/testuser/.ssh/identity.pub 1024 35 139198677264732164858153476357747926024656548233745027006381178621992083524037906211714241 450436547019604214530354070873624269283640613058470615170649963414635036859628344005142227 886318134122126153182906740418449098047827961768214148936752631482459130056603268404256522 191410368204629699075809390037814979061 testuser@host

Step 2

On the MDS create all of the ssh keys, even though in this case the client is using rsa1.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

2-66

78-xxxxx-xx

Chapter 2

Account Management Providing Passwordless Access Leveraging SSH

172.22.36.11# conf t Enter configuration commands, one per line. 172.22.36.11(config)# ssh key rsa1 generating rsa1 key(1024 bits)..... generated rsa1 key ca-9506(config)# ssh key dsa generating dsa key(1024 bits)..... generated dsa key ca-9506(config)# ssh key rsa generating rsa key(1024 bits)..... generated rsa key

End with CNTL/Z.

Step 3

Enable ssh on the MDS.

172.22.36.11(config)# ssh server enable

Step 4

On the MDS create the user, pasting in the contents of the identity.pub file after the sshkey parameter:

172.22.36.11# conf t Enter configuration commands, one per line. End with CNTL/Z. 172.22.36.11(config)# username testuser role network-operator warning: password for user:setmason not set. S/he cannot login currently 172.22.36.11(config)# username testuser sshkey 1024 35 139198677264732164858153476357747926024656548233745027006381178621992083524037906211714241 450436547019604214530354070873624269283640613058470615170649963414635036859628344005142227 886318134122126153182906740418449098047827961768214148936752631482459130056603268404256522 191410368204629699075809390037814979061 testuser@host 172.22.36.11(config)# end

Step 5

The configuration of the user can be listed with the following:

172.22.36.11# show user-account testuser user: testuser this user account has no expiry date roles:network-operator no password set. Local login not allowed Remote login through RADIUS/TACACS+ is possible ssh public key: 1024 35 139198677264732164858153476357747926024656548233 74502700638117862199208352403790621171424145043654701960421453035407087362426928 36406130584706151706499634146350368596283440051422278863181341221261531829067404 18449098047827961768214148936752631482459130056603268404256522191410368204629699 075809390037814979061 testuser@host

Step 6

Test the log in process from the host:


$ ssh testuser@172.22.36.11 Warning: Remote host denied X11 forwarding. Cisco Storage Area Networking Operating System (SAN-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2004, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license. Some parts of this software are covered under the GNU Public License. A copy of the license is available at

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

2-67

Chapter 2 Providing Passwordless Access Leveraging SSH

Account Management

http://www.gnu.org/licenses/gpl.html. 172.22.36.11#

If the same user tries logging in from another host which does not have both the private key file (/users/testuser/.ssh/identity) and the public key (/users/testuser/.ssh/identity), then it will be denied access to the MDS. The fact that the public key has testuser@host at the end of it, does not tie it to a specific host but allows an admin to determine which host it was generated from.

Tip

One simple method to leverage this feature is to set up a scheduled, for example using cron, to backup the switchs configuration on a nightly basis using ssh. This could be done with the following commands executed on a host, provided the user that is used has the privileges to issue the copy command:
#!/bin/sh ##################################################### # #/usr/local/bin/backup_mds_config.sh # This is used for a cron entry. No arguments are # allowed in cron.Absolute paths to commands must # be specified to ssh for it to work properly # ssh key exchange must be separately configured # for the account "USER" # # Adjust the variables for your host and switch ###################################################### DIR=/mds_config DATE=`date "+%m%d%y_%H%M%S"` SWITCH_NAME=beat_bama FILE=$SWITCH_NAME"_run_cfg_"$DATE USER=cwilliams COMMAND1="copy running-config startup-config COMMAND2="show startup-config" #Copy running to startup-config /usr/local/bin/ssh -l $USER $SWITCH_NAME $COMMAND1 #Backup MDS config to local file /usr/local/bin/ssh -l $USER $SWITCH_NAME $COMMAND2 > $DIR/$FILE

Then setup cron to execute the script, the cron job must be run by the user specified in the script. Configure the crontab for the user. This example runs at 11:00pm every Sunday.
#Backup MDS config: 00 23 * * 0 /usr/local/bin/backup_mds_config.sh > /mds_logs/beat_bama1

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

2-68

78-xxxxx-xx

C H A P T E R

Physical Interfaces
The MDS is a multi protocol switch, and in this section various options will be used to configure the FibreChannel (FC) and Gigabit Ethernet ports. The recipes below show how to configure various parameters and modes for a physical port on the MDS

Configuring FC ports
Port Description
The port description allows a user to provide a description of plain text name for the interface. In the below example the fibre channel interface fc 1/1 is given a description storage array 17 port 1. This is just naming the port on the switch.

switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# interface fc 1/1 switch(config-if)# switchport description storage array 17 port 1 switch(config-if)# end switch#

Port speed
This example show how to the set the port speed for fc 1/1 to 1Gb, 2Gb or auto negotiate speed.

Note

A port can be set to only one speed. The default is auto-negotiate.

switch# conf t Enter configuration commands, one per line. switch(config)# interface fc 1/1

End with CNTL/Z.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

3-69

Chapter 3 Configuring FC ports

Physical Interfaces

switch(config-if)# switch(config-if)# switch(config-if)# switch(config-if)# switch#

switchport speed 1000 switchport speed 2000 switchport speed auto ^Z

<- port speed set to 1 GB <- port speed set to 2 GB <- port speed set to auto negotiate

Port mode (auto)


Note

A FC port can be set to only one port mode at any given time. The default is auto on the 16 port linecards and FX on the 32 port linecards.

Setting the port mode to auto, allows the port to negotiate to either a F, FL or E port. fc 1/1 is set to auto port mode. It can not negotiate to either ST, SD or TL port modes. This is the default setting for all ports on a 16 port linecard.
switch# conf t Enter configuration commands, one per line. switch(config)# interface fc 1/1 switch(config-if)# switchport mode auto switch(config-if)# end switch#

End with CNTL/Z.

Port mode (E)


Setting the port mode to E, restricts the port to coming up as an Eport (can be either trunking or non-trunking depending on the trunking port mode) fc 1/1 is set to E port mode. E port mode is used when the port function as one end of an ISL.
switch# conf t Enter configuration commands, one per line. switch(config)# interface fc 1/1 switch(config-if)# switchport mode E switch(config-if)# end

End with CNTL/Z.

Port mode (F)


Setting the port mode to F, restricts the port to coming up as an F port. fc 1/1 is set to F port mode. F port mode is used for end devices which can only communicate in point to point mode or to a switch.
switch# conf t Enter configuration commands, one per line. switch(config)# interface fc 1/1 switch(config-if)# switchport mode F switch(config-if)# end switch#

End with CNTL/Z.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

3-70

78-xxxxx-xx

Chapter 3

Physical Interfaces Configuring FC ports

Port mode (FL)


Setting the port mode to FL, restricts the port to coming up as an FL port. fc 1/1 is set to FL port mode. FL port mode is used for end devices which can only communicate as a public loop device.
switch# conf t Enter configuration commands, one per line. switch(config)# interface fc 1/1 switch(config-if)# switchport mode FL switch(config-if)# end switch#

End with CNTL/Z.

Port mode (Fx)


Setting the port mode to Fx, restricts the port to coming up as an F or FL port. fc 1/1 is set to Fx port mode. Fx port mode is used exclusively for end devices and prevents a port from autonegotiating to an Eport.
switch# conf t Enter configuration commands, one per line. switch(config)# interface fc 1/1 switch(config-if)# switchport mode Fx switch(config-if)# end switch#

End with CNTL/Z.

Port mode (SD)


Setting the port mode to SD, configures the port for usage with a span session as the span destiation (SD) port. fc 1/1 is set to SD port mode. This is used in conjunction with the PAA to span a port and obtain FC traces without a FC analyzer.
switch# conf t Enter configuration commands, one per line. switch(config)# interface fc 1/1 switch(config-if)# switchport mode SD switch(config-if)# end

End with CNTL/Z.

Port mode (ST)


Setting the port mode to ST, configures the port for usage with a remote span session as the span tunnel (ST) port. fc 1/1 is set to ST port mode. This is used to set up a remote SPAN session to a remote switch in which a PAA or protocol analyzer is connected.
switch# conf t Enter configuration commands, one per line. switch(config)# interface fc 1/1 switch(config-if)# switchport mode ST switch(config-if)# end switch#

End with CNTL/Z.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

3-71

Chapter 3 Configuring FC ports

Physical Interfaces

Port mode (TL)


Setting the port mode to TL, restricts the port to coming up as a TL port. fc 1/1 is set to TLport mode. TL port mode is used exclusively for end devices which can only communicate as a private loop device.
switch# conf t Enter configuration commands, one per line. switch(config)# interface fc 1/1 switch(config-if)# switchport mode TL switch(config-if)# end switch#

End with CNTL/Z.

Configuring Trunking E ports


A Trunking port is used to carry VSAN enabled frames between switches. The section below shows the various configuration options for a trunking port.

Note

These same principals apply to port-channels. Just specify the port channel interface, int port-channel 1, rather than an individual link, interface fc 1/1.

Trunk port mode


This example shows how to the set the trunk port mode for fc 1/1 to auto, on and off. The default mode is auto. One end of an ISL should be set to on when connected between two MDS switches, while the other end can be either on or auto. It needs to be off when talking to non-MDS switches.
switch# conf t Enter configuration commands, one per line. End switch(config)# interface fc 1/1 switch(config-if)# switchport trunk mode auto < switch(config-if)# switchport trunk mode on < switch(config-if)# switchport trunk mode off < switch(config-if)# ^Z switch#

with CNTL/Z. auto negotiates trunk port mode sets trunk port mode to on sets trunk port mode to off

Configuring trunk ports to filter specific VSANs


This example shows how to configure the allowed VSAN traffic through the interface fc 1/1. The all keyword allows all VSAN traffic to go through the port. The add 2 adds VSAN 2 to the list of allowed VSANs through the port. The add 2-4 adds VSANs 2 to 4 to the list of allowed VSANs through the port. The default mode is to allow all the VSAN traffic to pass through the port.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

3-72

78-xxxxx-xx

Chapter 3

Physical Interfaces Configuring Gigabit Ethernet Ports

switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# interface fc 1/1 switch(config-if)# switchport trunk allowed vsan all < all VSAN traffic switch(config-if)# switchport trunk allowed vsan add 2 < only VSAN 2 traffic switch(config-if)# switchport trunk allowed vsan add 2-4 < VSAN 2 to 4 traffic switch(config-if)# ^Z switch#

Enabling port beaconing


This example causes the LEDs below the port fc 1/1 to start flashing. This is useful in identifying a port for physical cabling or trouble shooting.
switch# conf t Enter configuration commands, one per line. switch(config)# interface fc 1/1 switch(config-if)# switchport beacon switch(config-if)# end

End with CNTL/Z.

Configuring Gigabit Ethernet Ports


Configuring VRRP
Virtual Router Redundancy Protocol (VRRP) allows two gigabit Ethernet interfaces to provide failover capability for an IP address. The two interfaces will form an active/passive or master/backup state in which one interface will service requests for the shared IP address while the other will remain in a backup or standby state. It is ideal for providing port level redundancy in iSCSI configurations. A gigabit Ethernet port can still have its own IP address while partaking in a VRRP configuration. A VRRP session has an ID assigned to it for which the two interfaces will communicate to identify its peer. The same ID must be used on both switches. The procedure for having both members of the VRRP pair on the same switch would be the same as if the two members were on different switches.

Note

To have one interface become the master interface whenever it is online, also known as premption, set the gigabit Ethernet interface to have the same IP address as the VRRP IP address.

In this example, the configuration is as follows: VRRP ID: 1 VRRP IP address: 192.168.1.40 Switch 1: Interface gige3/3 (192.168.1.20)

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

3-73

Chapter 3 Configuring Gigabit Ethernet Ports

Physical Interfaces

Switch 2: Interface gige4/1 (192.168.1.30)


Step 1

Configure the IP address on the two Gige interfaces:


Switch1# config terminal Enter configuration commands, one per line. End with CNTL/Z. Switch1(config)# interface gigabitethernet 3/3 Switch1(config-if)# ip address 192.168.1.20 255.255.255.0 Switch1(config-if)# no shut Switch2# config terminal Enter configuration commands, one per line. End with CNTL/Z. Switch2(config)# interface gigabitethernet 4/1 Switch2(config-if)# ip address 192.168.1.30 255.255.255.0 Switch2(config-if)# no shut

At this point it is a good idea to verify that a host on the local subnet can ping both IP addresses (192.168.1.20 and 192.168.1.30). Alternatively, the ips measure-rtt command could be used to ping from one GigE port to the other.
Switch1# ips measure-rtt 192.168.1.30 interface gigabitethernet 3/3 Round trip time is 172 micro seconds (0.17 milli seconds)

Step 2

Configure the VRRP session on both switches using the VRRP id (1)
Switch1# conf t Enter configuration commands, one per line. End with CNTL/Z. Switch1(config)# interface gigabitethernet 3/3 Switch1(config-if)# vrrp 1 Switch1(config-if-vrrp)# address 192.168.1.40 Switch1(config-if-vrrp)# no shut Switch1(config-if-vrrp)# end Switch2# conf t Enter configuration commands, one per line. End with CNTL/Z. Switch2(config)# interface gigabitethernet 4/1 Switch2(config-if)# vrrp 1 Switch2(config-if-vrrp)# address 192.168.1.40 Switch2(config-if-vrrp)# no shut Switch2(config-if-vrrp)# end

Step 3

Verify that the VRRP session is up and which interface has become the master perform the following:
Switch2# show vrrp vr 1 Interface VR Status ------------------------------------------------------GigabitEthernet3/3 1 backup

switch1# show vrrp vr 1 interface gig3/3 status vr id 1 status MAC address 00:00:5e:00:01:01 Operational state: master Up time 8 sec

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

3-74

78-xxxxx-xx

Chapter 3

Physical Interfaces Implementing WWN Based VSANs (DPVM)

To view the configuration:


Switch1# show vrrp vr 1 interface gigabitethernet 3/3 configuration vr id 1 configuration admin state up priority 100 associated ip: 192.168.1.40 no authentication advertisement-interval 1 preempt no protocol IP

Implementing WWN Based VSANs (DPVM)


Dynamic Port VSAN Membership (DPVM) provides the user with the ability for an interfaces VSAN assignment to be determined by the WWN of the device that is logging in not by the configuration of the physical port. The primary advantage is if a device is moved from one port on a switch to another port on the same or different switch, it will end up in the same VSAN, thus preventing further configuration changes or the device ending up in the wrong VSAN. It can also be leveraged if the wwn is known for a device, but the interface that it will be plugged into is not yet known. DPVM can leverage the CFS infrastructure and it is recommended to use it to maintain database synchronization and locking. To populate the database either auto-learning can be enabled which will use the VSAN that each device that is currently logged, or a VSAN can be manually specified. The second method can be used for devices that are not yet in the fabric.

Note

A DPVM configuration will override the VSAN that is assigned to the port. Therefore, changing the VSAN membership of an interface that has a DPVM configured device attached will have no effect on the VSAN of the device. DPVMs CFS scope is physical. DPVM can work in conjunction with Persistent FCIDs, however if the device moves to another switch it will be assigned a different WWN.

After configuring DPVM, if the DPVM assigned VSAN is different from the port assigned VSAN, the operational value for the VSAN will be that of the DPVM assigned value:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

3-75

Chapter 3 Implementing WWN Based VSANs (DPVM)

Physical Interfaces

Figure 3-1

Interface Configuration after DPVM.

Alternatively, in the CLI the operational VSAN assignment will be displayed in the Port vsan field. The configured vsan displays the VSAN that the port would be in if DPVM was not configured.
switch# show int fc2/5 fc2/5 is up Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:45:00:0c:85:e9:d2:c0 Admin port mode is auto, trunk mode is on Port mode is F, FCID is 0xef0008 Configured Port vsan is 1 Port vsan is 1000 Speed is 2 Gbps Transmit B2B Credit is 7 Receive B2B Credit is 16 Receive data field Size is 2112

Prior to any actual configuration activities take place, DPVM must be enabled via CLI or Fabric Manager:
switch# conf terminal Enter configuration commands, one per line. switch(config)# dpvm enable End with CNTL/Z.

Alternatively, in Fabric Manager, under Logical Domains, select All VSANs, then DPVM. Under the Control tab, set the command field to enable and press the Apply Changes button as in Figure 3-2.

Figure 3-2

Enabling DPVM with Fabric Manager

After enabling DPVM, proceed to either Adding Existing Devices to DPVM on page 77 or Adding New Devices to DPVM on page 79 for the rest of the recipe.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

3-76

78-xxxxx-xx

Chapter 3

Physical Interfaces Implementing WWN Based VSANs (DPVM)

Adding Existing Devices to DPVM


If DPVM is being configured for an environment in which end devices are already present and ports hav e been assigned to VSANs, this recipe will describe how to use the DPVM wizard to import the VSAN configuration into DPVM. In this recipe the following resources will be configured: Hosts: 50:06:0e:80:03:4e:95:23 21:00:00:e0:8b:09:78:47 VSAN: 1000
Step 1 Step 2

Enable DPVM as per Implementing WWN Based VSANs (DPVM) on page 75. In Fabric Manager, select Tools, Other, and then DPVM Setup.

Figure 3-3

DPVM Setup Wizard

Select any of the switches that have been enabled for DPVM. DPVM is a CFS aware application and so the DPVM configuration will be propagated to all switches. If a switch is listed as not having DPVM configured then the wizard will automatically enable DPVM on those switches. Press Next.
Step 3 Step 4

Since this is a new DPVM configuration, select the Create Configuration From Currently Logged In End Devices checkbox and press Next. If an existing DPVM configuration exists, do not check this box. At this point, FM will determine the VSAN assignment of all the devices in the fabric and present a table listing the proposed configuration:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

3-77

Chapter 3 Implementing WWN Based VSANs (DPVM)

Physical Interfaces

Figure 3-4 a. b. Step 5 Step 6 Step 7

Proposed DPVM Configuration

If additional entries are desired, select Insert... whereby a dialog box will prompt for a WWN and VSAN. If an above entry should be deleted. Select the row to be removed and select Delete.

When the above changes are completed, select Finish which will CFS commit the action. To view the DPVM configuration in Fabric Manager, in the Logical Domains pane, select the Fabric to view, All VSANs and then DPVM. Click the CFS tab, which will activate the other CFS tabs, and then select the Active Database tab:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

3-78

78-xxxxx-xx

Chapter 3

Physical Interfaces Implementing WWN Based VSANs (DPVM)

Figure 3-5

DPVM Active Database

In the above output, Device Aliases (see Device Aliases, page 1-43) have been enabled therefore the pWWN column displays the Device Alias of the device rather than the WWNs.

Adding New Devices to DPVM


In this recipe the following resources will be configured: Hosts: 50:06:0e:80:03:4e:95:aa VSAN: 1000

Step 1 Step 2 Step 3

Enable DPVM as per Implementing WWN Based VSANs (DPVM) on page 75. To modify the DPVM configuration in Fabric Manager, in the Logical Domains pane, select the Fabric to view, All VSANs and then DPVM. Click the CFS tab, which will activate the other CFS tabs, and then select the Config Database tab:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

3-79

Chapter 3 Implementing WWN Based VSANs (DPVM)

Physical Interfaces

Figure 3-6 Step 4 Step 5

DPVM Config Database

Click the Create Row... button, located above the CFS tab. Either select from the list the device or wwn to be added or manually type in the wwn. Then indicate the VSAN to be assigned. Click Create. When all entries have been created, click Close. In this example, Device Aliases have been configured (see Device Aliases, page 1-43), therefore devices aliases are displayed instead of the WWNs.

Figure 3-7 Step 6

Creating the DPVM entry.

Next, select the Actions tab. Change the Action column to activate and click the green Apply Changes icon. Using the activate action will ensure that a device that is currently logged into the fabric does not get accidentally moved into another VSAN which could potentially disrupt IO. To determine which devices causing the CFS commit to fail see section on DPVM Conflicting Entries, page 3-82. If the commit should ignore this safety check, then use the activate force action. Since DPVM is CFS enabled, a CFS commit is still required. Also, at this point the Config and Active Databases are different.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

3-80

78-xxxxx-xx

Chapter 3

Physical Interfaces Implementing WWN Based VSANs (DPVM)

Step 7 Step 8

To Commit the changes click the CFS button and and select Commit. If the commit succeeds, in the bottom left corner of Fabric Manager will display CFS(dpvm):Committed.. At this point the Active Database will now contain the new entry.

Modify the VSAN Assignment of a DPVM Entry


In this recipe the VSAN assignment of a device will be changed to a new VSAN. PWWN: 50:06:0e:80:03:4e:95:33 Old VSAN: 501 New VSAN: 1000
Step 1 Step 2

To modify the DPVM configuration in Fabric Manager, in the Logical Domains pane, select the Fabric to view, All VSANs and then DPVM. Click the CFS tab, which will activate the other CFS tabs, and then select the Config Database tab:

Figure 3-8 Step 3

DPVM Config Database

Select the cell corresponding to the Login VSAN Id that will be modified, and then enter in the new VSAN ID (1000). Then click the Apply Changes button.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

3-81

Chapter 3 Implementing WWN Based VSANs (DPVM)

Physical Interfaces

Figure 3-9 Step 4

Modify VSAN Id and Apply Changes

Next, select the Actions tab. Change the Action column to activate and click the green Apply Changes icon. Using the activate action will ensure that a device that is currently logged into the fabric does not get accidentally moved into another VSAN which could potentially disrupt IO. To determine which devices causing the CFS commit to fail see section on DPVM Conflicting Entries, page 3-82. If the commit should ignore this safety check, then use the activate force action. Since DPVM is CFS enabled, a CFS commit is still required. Also, at this point the Config and Active Databases are different. To Commit the changes click the CFS button and and select Commit. If the commit succeeds, in the bottom left corner of Fabric Manager will display CFS(dpvm):Committed.. At this point the Active Database will now contain the new entry, and the CFS tab will display a success.

Step 5 Step 6

DPVM Conflicting Entries


Upon committing a DPVM configuration change, a pop-up box may be displayed if a device is already logged into the fabric and performing this DPVM commit would change its VSAN assignment and potentially cause an IO disruption. If one of the switches cannot successfully perform the operation then none of the switches will do it. The error should be resolved and then DPVM should be reactivated and committed.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

3-82

78-xxxxx-xx

Chapter 3

Physical Interfaces Implementing WWN Based VSANs (DPVM)

Figure 3-10 DPVM CFS Commit Error

This procedure follows the procedure outlined in Adding New Devices to DPVM, page 3-79. To determine what entries caused the error perform the following:
Step 1

Access the MDS via the CLI using either telnet or SSH. Connect to the switch listed in the CFS tab that contains the erroneous result:

Figure 3-11 DPVM Conflicting Error: Switch Login. Step 2

Run the following command to see the conflict, the VSAN listed is the new VSAN for the listed device.
172.22.36.11# show dpvm pending-diff Session is on, Lock Taken DPVM Pending Status ------------------Active DB : Activate Auto Learn : None Pending Database Diff --------------------Legend: "+" New Entry, "-" Missing Entry, "*" Possible Conflict Entry --------------------------------------------------------------------* pwwn 50:06:0e:80:03:4e:95:33 vsan 1000

Step 3

To determine the physical location of this device, in Fabric Manager in the Physical Attributes pane, select the End Devices folder. Then sort by pwwn:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

3-83

Chapter 3 Implementing WWN Based VSANs (DPVM)

Physical Interfaces

Figure 3-12 PWWNs location in the fabric

At this point, if its determined that the port(s) can be safely moved to a new VSAN, then the force activate action can be used, or the conflicting entry should be removed from the config database.

DPVM with the CLI


DPVM can be manipulated with the CLI just as easily as it can be done with Fabric Manager. It has the same underlying CFS infrastructure which Fabric Manager leverages. To enable DPVM:
ca-9506# config terminal Enter configuration commands, one per line. ca-9506(config)# dpvm enable End with CNTL/Z.

Adding Existing Devices to DPVM


To add existing devices that are logged into the fabric, use the procedure outlined in section Adding Existing Devices to DPVM, page 3-77.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

3-84

78-xxxxx-xx

Chapter 3

Physical Interfaces Implementing WWN Based VSANs (DPVM)

Adding New Devices to DPVM


In this recipe, a new entry will be entered into the DPVM database and the following resources will be configured: Hosts: 50:06:0e:80:03:4e:95:cc VSAN: 1000
Step 1

After logging into the switch enter config mode and the dpvm database:
ca-9506# config terminal Enter configuration commands, one per line. ca-9506(config)# dpvm database ca-9506(config-dpvm-db)# End with CNTL/Z.

Step 2 Step 3

Enter the pwwn and the VSAN:


ca-9506(config-dpvm-db)# pwwn 50:06:0e:80:03:4e:95:cc vsan 1000

Activate the changes. This can be done from the current CLI prompt or the previous prompt which can be reached by typing exit.:
ca-9506(config-dpvm-db)#dpvm activate

Step 4

Prior to activating the changes, the proposed changes can be seen with the following show command. A conflicting device will cause the CFS commit to fail.

Note

The + represents devices that are being added to the database. The - represents devices that are being removed from the database. The * represents devices that are being modified in the database, including those that are
currently logged into the fabric and are changing VSAN assignment.

The do keyword is required for exec commands while in config mode.


ca-9506(config)# do show dpvm pending-diff Session is on, Lock Taken DPVM Pending Status ------------------Active DB : Activate Auto Learn : None Pending Database Diff --------------------Legend: "+" New Entry, "-" Missing Entry, "*" Possible Conflict Entry --------------------------------------------------------------------+ pwwn 50:06:0e:80:03:4e:95:cc vsan 1000

Step 5

Lastly, CFS commit the changes.


ca-9506(config)#dpvm commit

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

3-85

Chapter 3 Implementing WWN Based VSANs (DPVM)

Physical Interfaces

Modify the VSAN Assignment of a DPVM Entry


This procedure is the same as Adding New Devices to DPVM, page 3-85. However, if the device that is being reassigned to a new VSAN is currently logged into the fabric, the dpvm activate force command should be used.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

3-86

78-xxxxx-xx

C H A P T E R

VSANs
VSANs (Virtual SAN) are a logical fabric. It is a logical grouping of ports in a single switch or across multiple switches that function like a single fabric. Each VSAN has all the required fabric services independent of the other VSANs configured on the same switch or set of switches. VSANs also provide SAN island consolidation on a higher port density physical switch, traffic isolation and increased security. VSANs can be numbered from 1 4094. VSAN 1 and VSAN 4094 are predefined and have very specific roles. VSAN 1 is the default VSAN which holds all the ports by default and the VSAN 4094 is the isolated VSAN into which orphaned ports are assigned.

Creating a VSAN and Adding Interfaces


This recipe will demonstrate using Fabric Manager how to create a VSAN (3005) and add an interface to it.

Note

Moving a port from one VSAN to another will not change its configuratin (F, FL, TL), its speed or its administrative state (shut/noshut). However, any device attached to the port will need to FLOGI back into the switch. In Fabric Manager, on the toolbar select the Create VSAN icon. Select the switch(es) to create the VSAN on. Enter the VSAN ID. Enter in a name for the VSAN. If the VSAN will be attaching to a third party switch select the appropriate Interop mode.

Step 1 Step 2 Step 3 Step 4 Step 5

Note

In SAN-OS 2.1(2), a static domainID can be specified for the VSAN at the time of creation. Otherwise the recipe in Converting an Existing VSAN to Static DomainID and Enabling Persistent FCID using CLI., page 4-93 should be followed to specify the new domainID. Press Create.

Step 6

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

4-87

Chapter 4 Modifying VSAN Attributes with Fabric Manager

VSANs

To add interfaces to the previously created VSAN perform the following:


Step 1 Step 2 Step 3 Step 4 Step 5 Step 6

In Fabric Manager, in the Physical Attributes pane, expand Switches. Expand Interfaces. Select FC Physical. Modify the Port VSAN Config field for the switchs interface that should be moved to the specified VSAN. Optionally, If the port has an Admin Status of down, it can be enabled by changing the value to up. Press the green Apply Changes... button.

Note

A warning will be displayed informing that moving a port between two VSANs can be disruptive to that port as it will have to relogin to the fabric and will nolonger have access to resources in the previous VSAN. Press Yes to move the port to the new VSAN.

Step 7

Modifying VSAN Attributes with Fabric Manager


The following recipes outline how to modify the attributes of a VSAN using Fabric Manager. This includes interop modes, load balancing and setting static domain IDs and persistent FCIDs. To modify:
VSAN Name LoadBalancing (src/dst or src/dst/ox-id) Administrative State (suspended or active) Interoperability Mode to work with third party switches. In Order Delivery Step 1 Step 2 Step 3 Step 4

Access the VSAN attributes pane by selecting the Fabric in the Logical Domains pane. Expand the VSAN to be modified, in the below example, VSAN 3001 was chosen. Select VSAN Attributes. Make the appropriate changes to the desired fields.

Note

Standard editting keyboard shortcuts (ctrl-x to cut; ctrl-c to copy; ctrl-v to paste) can be used to edit the text fields. Press the green Apply Changes... button.

Step 5

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

4-88

78-xxxxx-xx

Chapter 4

VSANs Modifying VSAN Attributes with Fabric Manager

Figure 4-1

Modify VSAN Attributes

Converting an Existing VSAN to Static DomainID and Enabling Persistent FCID using Fabric Manager.
Within a VSAN, the domain manager process on the principal switch in a fabric is responsible for assigning a domain_ID to switch that is joining the fabric. When a switch boots up or joins a new fabric it can request a specific domain_ID or take any available domain_ID. A domain_ID can be configured in one of two ways:
Preferred: The new switch will request a specific domain_ID, however, if it receives a different

domain_ID it will accept it.


Static: The new switch will request a specific domain_ID, however, if it receives a different

domain_ID, it will isolate itself from the fabric. This is the case when the same domain_ID must be maintained under all circumstances. After obtaining the domain_ID from the principal switch in the VSAN, the local switch will assign Fibre Channel Identifiers (FC_IDs) to each end device as they log in to the fabric. A process known as FLOGI (Fabric Login).

Tip

HPUX and AIX are two operating systems that utilize the FC_ID in the device path to the storage. In order to have the switch always assign the same FC_ID to a device across switch reboots, persistent FC_ID and static domain_ID must be configured for the VSAN. If an FC_ID changes for a device accessed by either an AIX or a HPUX host, the host may lose access to the device. By default, the switch will assign the same FC_ID to a device, however, if the switch is rebooted this database of pwwn/FC_ID mapping is not maintained. Enabling persistent FC_IDs will make this database persistent across reboots.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

4-89

Chapter 4 Modifying VSAN Attributes with Fabric Manager

VSANs

A persistent FC_ID can be configured in two ways:


Dynamic (default): The FC_ID is determined and assigned by the switch and if the persistent

FC_ID database is manually purged by the user this entry will be deleted. These entries are persistent across reboots of the switch and are VSAN specific.
Static: The FC_ID is determined by the user prior to attaching the device to the switch. If the

persistent FC_ID database is manually purged by the user these entries will not be removed. These entries are persistent across reboots of the switch and are VSAN specific. When persistent FC_ID is enabled, the MDS will make persistent all of the devices in that VSAN, therefore the admin is not required to manually type in devices entering that VSAN.

In this procedure an existing VSAN (3000) on switch 172.22.36.11 with domain_ID 239 will be statically statically configured and then persistent FCID will be enabled using FM. This recipe does not alter the running domain_ID.

Step 1 Step 2 Step 3 Step 4

In Fabric Manager, in the Logical Domains pane, expand the VSAN to be modified and then select the Domain Manager entry. Select the Configuration tab. Enter the domain_ID that is in the Running domain_ID (239) column into the Config domain_ID column (239). Change the IdType field to static.
Figure 4-2 Enabling Static domain_ID

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

4-90

78-xxxxx-xx

Chapter 4

VSANs Modifying VSAN Attributes with the CLI

Step 5 Step 6 Step 7 Step 8

Press the green Apply Changes... button. Select the Persistent Setup tab. Select the Enable checkbox. Press the green Apply Changes... button. At this point, the domain_ID has been statically set and FC_IDs will remain persistent across reboots for VSAN 3000 on the switch 172.22.36.11. The persistent FC_ID database can be viewed in the Persistent FCIDs tab.
Figure 4-3 Persistent FC_ID database

Modifying VSAN Attributes with the CLI


The following recipes outline how to modify the attributes of a VSAN using the CLI. This includes interop modes, load balancing and setting static domain IDs and persistent FCIDs.

Creating a VSAN on a single switch and adding an Interface


This recipe shows the steps to create and name a VSAN on a single switch. VSAN 200 is created with the name TapeVSAN and fibre channel interface fc 1/1 is added.
switch1# conf t Enter configuration commands, one per line. switch(config)# vsan database

End with CNTL/Z.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

4-91

Chapter 4 Modifying VSAN Attributes with the CLI

VSANs

switch(config-vsan-db)# vsan 200 name TapeVSAN switch(config-vsan-db)# vsan 200 interface fc 1/1 switch(config-vsan-db)# ^Z switch#

Setting VSAN Interop Mode


Interop mode is set for VSANs that need to interact with other 3rd party switches. Interop mode one is required when all vendor switches are set in their respective interop modes. In interop mode 1 only Domain IDs 97 to 127 are allowed. Interop 2 is required when a VSAN has to work with a Brocade 2800/3800 switch in its native corePID 0 mode. Interop mode 3 is required when the VSAN has to work with Brocade switch running in corePID 1 mode. For more information please refer to the interop guide. The MDS Switch to Switch Interoperability Configuration Guide, should be consulted prior to performing any interoperability work, explains the different interop modes and can be found on CCO.

This recipe shows the steps to set the interop modes 1, 2, 3 for a VSAN. Interop mode 1 (ensure that the domain ID of the VSAN is between 97 127 for this mode to work). To change the interop mode of VSAN 200 to interop mode 1:
switch# conf t Enter configuration commands, one per line. switch(config)# vsan database switch(config-vsan-db)# vsan 200 interop 1 switch(config-vsan-db)# ^Z switch#

End with CNTL/Z.

Interop mode 2 (for Brocade switches running corePID mode 0). To change the interop mode of VSAN 200 to interop mode 2.

switch# conf t Enter configuration commands, one per line. switch(config)# vsan database switch(config-vsan-db)# vsan 200 interop 2 switch(config-vsan-db)# ^Z switch#

End with CNTL/Z.

Interop mode 3 (for Brocade switches running corePID mode 1). To change the interop mode of VSAN 200 to interop mode 3.
switch# conf t Enter configuration commands, one per line. switch(config)# vsan database switch(config-vsan-db)# vsan 200 interop 2 switch(config-vsan-db)# ^Z switch#

End with CNTL/Z.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

4-92

78-xxxxx-xx

Chapter 4

VSANs Modifying VSAN Attributes with the CLI

Changing Load-balancing scheme


The load-balancing scheme can be configured per VSAN. S_ID (source id), D_ID (destination id) based load balancing and the exchange level (S_ID, D_ID, OX_ID) load balancing can be configured on the MDS This recipe shows the steps to configure load-balancing schemes for VSAN 200.

Sequence Level load-balancing (Source_ID, Destination_ID)


To change the load-balancing scheme for VSAN 200 to S_ID, D_ID mode.

switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# vsan database switch(config-vsan-db)# vsan 200 loadbalancing src-dst-id switch(config-vsan-db)# ^Z switch#

Exchange level load balancing (S_ID, D_ID, OX_ID)


To change the load-balancing scheme for VSAN 200 to S_ID, D_ID, and OX_ID mode. This is the default load balancing scheme.
switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# vsan database switch(config-vsan-db)# vsan 200 loadbalancing src-dst-ox-id switch(config-vsan-db)# ^Z switch#

Converting an Existing VSAN to Static DomainID and Enabling Persistent FCID using CLI.
This recipe shows the steps to configure static Domain_ID for a VSAN and also how to enable persistent FC_ID for the same VSAN. In this procedure an existing VSAN (3000) on switch 172.22.36.11 with domain_ID 239 will be statically statically configured and then persistent FCID will be enabled using FM. This recipe does not alter the running domain_ID.
Step 1

Display the current domain_ID for VSAN 3000


172.22.36.11# show fcdomain domain-list vsan 3000

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

4-93

Chapter 4 Modifying VSAN Attributes with the CLI

VSANs

Number of domains: 2 Domain ID WWN ------------------------------0xef(239) 2b:b8:00:05:30:00:68:5f [Local] [Principal]

Step 2

Configure the static domain_ID


172.22.36.11# conf t Enter configuration commands, one per line. End with CNTL/Z. 172.22.36.11(config)# fcdomain domain 239 static vsan 3000

Step 3

Configure persistent FC_ID


172.22.36.11(config)# fcdomain fcid persistent vsan 3000 172.22.36.11(config)# end

Step 4

Save the configuration:


172.22.36.11# copy running-config startup-config [########################################] 100%

Note

If the domain ID of VSAN 200 is different than what is currently running (22 in this case) then the VSAN will have to be restarted for the configuration changes to the Domain_ID and FC_ID persistence to take effect. Changing Domain_IDs and hence FC_IDs for a device is disruptive as an end device will have to relogin to the fabric (FLOGI) to obtain a new FCID.

Warning

Changing Domain_IDs and therefore FC_IDs for a device is disruptive, as an end device will have to relogin to the fabric (FLOGI) to obtain a new FCID. However, making a Domain_ID static without changing its value is not disruptive.

Restarting a VSAN
Sometimes the VSAN on a switch will need to be restarted. For example, after changing the Domain_ID of a VSAN, the VSAN should be restarted for the new Domain_ID to take effect. The recipe below shows how a VSAN (200) can be restarted.
switch# conf t Enter configuration commands, one per line. switch(config)# vsan database switch(config-vsan-db)# vsan 200 suspend switch(config-vsan-db)# no vsan 200 suspend switch(config-vsan-db)# end

End with CNTL/Z.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

4-94

78-xxxxx-xx

Chapter 4

VSANs Modifying VSAN Attributes with the CLI

Assigning a Predetermined FCID to a PWWN


When performing a migration or hba replacement, the same FCID as previously used may need to be assigned to the new pwwn. This recipe show steps to assign pre-determined FC_ID to a specific PWWN.

Note

A new FCID cannot be assigned to a pwwn which is already logged into the fabric. Log the device out of the fabric, for example by shutting the FC interface before assigning the new FCID. FC_ID 0x160000 will be assigned to pwwn 50:06:0b:82:bf:d1:db:cd permanently. Therefore when the pwwn logs into the switch (FLOGI) it will get this assigned FC_ID.

Note

The FC_ID to be assigned (0x160000) should contain the same Domain_ID (0x16) as the currently running domain in the VSAN.

switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# fcdomain fcid database switch(config-fcid-db)# vsan 22 wwn 50:06:0b:82:bf:d1:db:cd fcid 0x160000 dynamic switch(config-fcid-db)# ^Z switch#

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

4-95

Chapter 4 Modifying VSAN Attributes with the CLI

VSANs

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

4-96

78-xxxxx-xx

C H A P T E R

Zoning
Zones and zonesets are the basic form of datapath security within a fibre channel environment. A zoneset is a collection of zones which in turn have individual members in them. Only those members within the same zone can communicate with each other. A device can be a member of multiple zones and those devices not in a zone are in the default-zone. The policy for the default-zone can either be permit (devices can see each other) or deny (devices in the default-zone cannot see each other). The basic flow of zoning is to:
1. 2. 3. 4. 5.

Create zones Add hosts and storage to the zones. Create Zonesets. Add the zones to Zonesets Activate the Zoneset.

This chapter will focus on the creation of zones, zonesets and how to manipulate them. In both basic zoning (FC-GS-3) and Enhanced Zoning (FC-GS-4) which was introduced in SAN-OS 2.0, both use the concepts of zones and zonesets.

Enhanced Zoning
Introduced in SAN-OS 2.0, Enhanced Zoning, defined in the FC-GS-4 and FC-SW-3 standards provides significant enhancement to basic zoning.

Has a VSAN scope, so that while VSAN X is using enhanced zoning. Other VSANs can continue to use basic zoning. IVR compatible. Provides session locking so that two SAN administrators cannot modify the zoning database within a VSAN at the same time, Provides implicit full zoneset distribution so that the zoneset database that is local to each switch remain in sync when a zoneset is modified. Full zoneset changes can be distributed without having to activate a zoneset. This can be leveraged if changes want to be readied during the daytime, and then have the zoneset activated during the night.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

5-97

Chapter 5 Enhanced Zoning

Zoning

Modifications are staged and have to be explicity committed or aborted, allowing the SAN admin to review all changes prior to them going into production. Can control how a zone merge should be done. Either by performing a union of the two zonesets according to the same rules as basic zoning, or by a strict policy in which if the two candidate active zonesets are not exactly the same, the two zonesets will not merge. Thereby preventing accidental merging. Enhanced Zoning has a similar behavior to CFS: Cisco Fabric Services, page 1-13.

Enhanced zoning continues to use the same techniques and tools as basic zoning with a few added commands which will be covered in the subsequent recipes. However, the flow of using enhanced zoning is slightly different. First, a VSAN wide lock, if available is implicitly obtained. Next, all zone and zoneset modifications are done including activation. Last, the changes are either committed (put into production) or aborted (pending changes are scrapped). The flow is illustrated below:
Figure 5-1 Enhanced Zoning Flowchart

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

5-98

78-xxxxx-xx

Chapter 5

Zoning Enhanced Zoning

How to Enable Enhanced Zoning


Enhanced zoning, with its VSAN scope, requires that all switches that take part in a VSAN be enhanced zoning capable and will need to have enhanced zoning enabled on. Due to its distributed architecture and abilities, enhanced zoning only needs to be enabled on one switch in the VSAN. Then this command will be propagated to the other switches in the VSAN. The rules for enabling enhanced zoning are:
1. 2. 3.

Enhanced zoning only needs to be enabled on one switch in the VSAN. Attempting to enable it on multiple switches in the same VSAN via Fabric Manager can result in a failure to activate. Enabling enhanced zoning does not cause a zoneset activation. The switch that is chosen to initiate the migration to enhanced zoning will distribute its full zone database to the other switches in the VSAN. Thereby overwriting the destination switches full zoneset database. Therefore it is critical that you select the correct switch to enable enhanced zoning on to prevent deleting your full zoneset database.

Enabling Enhanced Zoning with the CLI


To enable enhanced zoning with the CLI:
Step 1

Enter config term mode and enabling enhanced zoning.


switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# zone mode enhanced vsan 3000 Set zoning mode command initiated. Check zone status switch(config)#

Step 2

Display the zoning status


switch# show zone status vsan 3000 VSAN: 3000 default-zone: deny distribute: active only Interop: default mode: enhanced merge-control: allow session: none hard-zoning: enabled Default zone: qos: low broadcast: disabled ronly: disabled Full Zoning Database : Zonesets:1 Zones:3 Aliases: 1 Attribute-groups: 1 Active Zoning Database : Name: ZoneSet1 Zonesets:1 Zones:2 Status: Set zoning mode complete at 16:22:51 pacific Oct 03 2005

Enabling Enhanced Zoning with Fabric Manager


To enabling Enhanced Zoning in Fabric Manager perform the following:
Step 1 Step 2 Step 3

In the Logical Domains pane select the VSAN to enable VSANs and then the folder corresponding to the name of the active zoneset. If no active zoneset exists choose default zone instead. Select the Enhanced Tab. In the action column for the switch you want to enable enhanced zoning from, change the cell to Enhanced. Remember that for this VSAN, this switch will distribute its full zone database thus overwriting other switches full zone database.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

5-99

Chapter 5 Enhanced Zoning

Zoning

Figure 5-2

Enabling Enhanced Zoning with Fabric Manager

Step 4

Press the Apply Changes button.

How to Display which user has obtained the lock


In enhanced zoning, only one user may make changes to the zone database within a VSAN at any one time due to an implicitly obtained lock. To determine which user currently has this lock run the following commands:
switch# show zone status vsan 3000 VSAN: 3000 default-zone: deny distribute: active only Interop: default mode: enhanced merge-control: allow session: cli [admin] hard-zoning: enabled Default zone: qos: low broadcast: disabled ronly: disabled Full Zoning Database : Zonesets:1 Zones:3 Aliases: 1 Attribute-groups: 1 Active Zoning Database : Name: ZoneSet1 Zonesets:1 Zones:2 Status: Operation failed: [Error: Zoneset already present]: at 17:00:10 pacific Oct 03 2005

Altneratively in Fabric Manager to display the user with the lock, select the Logical Domains Pane, the VSAN to be investigated, the name of the Zoneset (or Default Zone if there is no active zoneset), then the Enhanced Tab. The user is displayed in the Config DB Locked By column:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

5-100

78-xxxxx-xx

Chapter 5

Zoning ZoneSets

Figure 5-3

Displaying Enhanced Zoning Lock

ZoneSets
Zonesets are containers of zones and of which there are two types on the MDS platform:
1.

Active Zoneset: is the rules by which the MDS platform enforces its zoning security policy.It cannot be modified and is distributed to all switches in the VSAN. There are specific rules to merging the active zoneset when two switches are connected by an ISL as set by the FC standards. Local Zoneset: The local zonesets are contained in the full zoneset database on the switch. The zonesets are editted directly and then activated to become the active zoneset. They can optionally be distributed to other switches, either manually or when a zoneset is activated.

2.

Zoneset Distribution
The zoneset in the full zoneset database can be distributed to other switches either during activation or manually when basic zoning is enabled. When enhanced zoning is enabled, the full zoneset is always distributed upon committing any changes to the full zoneset database. It is always enabled for enhanced zoning.

Automatic Zoneset Distribution


To enable the switch to distribute the local zoneset to all other switches in the VSAN when a zoneset is activated, use the following command:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

5-101

Chapter 5 Zones

Zoning

Tip

This feature should be enabled on all switches in the fabric, and can be initially specified in the initial setup script. This is the default behavior when enhanced zoning is enabled.
ca-9506# conf t Enter configuration commands, one per line. End with CNTL/Z. ca-9506(config)# zoneset distribute full vsan 804

Manual Zoneset Distribution:


To distribute the full zoneset database to other switches without activating a zoneset: This can be effective when a new switch is brought into the fabric and the zoneset with its zones and FC aliases need to be distributed. This command will overwrite the exiting zoneset database in the target switch.
ca-9506# zoneset distribute vsan 804 Zoneset distribution initiated. check zone status

Zones
In order for two devices to communicate, they must be in the same zone. Valid members of a zone can be:
Port WWN FC Alias FCID FWWN (WWN of a fc interface) Switch Interface (fc X/Y) Symbolic nodename Distributed Device Alias

The three most common types of zone members are the pwwn, Device Alias, FC alias and the switch interface.

Tip

It is recommended that pwwn or Device Alias be used for zoning as it provides hardware enforced zoning and ties a zone member to a specific hba rather than to the switch port. Also, Device Aliases have the added benefit of being VSAN independent and are based on an easy to understand name rather than a cryptic pwwn. Equally important is the name of the zone. Many environments use different zone names, however, all name formats should provide relevant information as to their contents. Names like Zone1 or TapeZone do not provide sufficient information about their contents.

Tip

A zone name, should contain two members and within the zone name contain identifiers related to the two devices (eg: Z_testhost_fcaw0_symm13FA3aa). It may be longer than Z_testhost_hba0, but provides detailed information about the contents without having to consult further documentation.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

5-102

78-xxxxx-xx

Chapter 5

Zoning Zones

Zone Creation and Adding it to a Zoneset with Fabric Manager


This recipe will demonstrate how to create a zoneset, create zones, add them to a zoneset, and then activate the zoneset. The method is the same when using basic zoning or enhanced zoning. The following topology will be used:
Figure 5-4 Fabric Manager Zoning Topology

Zoneset: ZS_SJC_Bldg6 Solaris Host: ca-sun1,hba lpfc0 HDS Array: HDS20117-c20-8 First a zoneset will need to be created.
Step 1

In the Logical Domains pane, right click on the VSAN, and select Edit Local Full Zone Database.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

5-103

Chapter 5 Zones

Zoning

Figure 5-5

Edit Local Full Zone Database

Note

The VSAN field displays the VSAN to which the zone database will be modified and The Switch field shows which switch is being editted. The Name column represents either FC Aliases or Global Device Aliases, (Device Aliases, page 1-43) if they are used. If Full Zoneset Distribution is enabled, and there are existing zones then the left hand column should contain the existing zonesets and zones. If Active Zoneset Distribution is enabled, then choose the switch that contains the Full Zone Database.

Step 2 Step 3 Step 4

In the left pane, Right click on Zonesets. Select Insert... Enter a Zoneset name such as ZS_SJ_Bldg6 and press OK.

At this point a zoneset has been created. The next phase is to create a zone and add members to it.
Step 1 Step 2 Step 3

Right click on Zones. Select Insert... Enter a meaningful zone name such as ca-sun1_lpfc0_HDS20117-c20-8 to represent both the initiator and the target in the zones name.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

5-104

78-xxxxx-xx

Chapter 5

Zoning Zones

Step 4

Click OK. The resulting dialog should look like:


Figure 5-6 Zone Database after Creating Zoneset and Zones

Step 5

Next, drag the two end devices from the bottom pane into the zone that was just created. This will create a pwwn based zone. If non-pwwn zone members are desired, such as interface, FCID or Global Device Alias see How to Create Non-PWWN based zones, page 5-108 to specify these member types before continuing to the next step.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

5-105

Chapter 5 Zones

Zoning

Figure 5-7

Zone with newly added members

This completes the steps to create the zone. Next, the zone must be added to the zoneset.

Add the Zone to the Zoneset:


Step 1

In the left hand pane, drag the zone (ca-sun1_lpfc0_HDS20117-c20-8) into the zoneset (ZS_SJC_Bldg6). The zonesets icon will change by appending a folder icon and it will expand with the newly added zone underneath it.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

5-106

78-xxxxx-xx

Chapter 5

Zoning Zones

Figure 5-8

Zone Added to Zoneset

The last phase is to Activate the Zoneset. This will instruct the switch program its Access Control Lists, modify the running configuration of the zoneserver to finally allow the two devices to communicate. When enhanced zoning is enabled, the activate button will commit the changes as well.
Step 1 Step 2

In the left pane, right click the Zoneset (ZS_SJC_BLDG6). Choose Activate...
a. b.

If the current Active Zoneset is empty, then press Continue Activation... If the current Active Zoneset is not empty, pressing Proposed Changes will display what is being added or removed from the active zoneset. Then press Continue Activation...
Zoneset Activation Proposed Changes

Figure 5-9

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

5-107

Chapter 5 Zones

Zoning

After activating the zoneset, the two end devices will be able to communicate.

How to Create Non-PWWN based zones


This recipe outlines the procedure necessary to create a zone that is not based upon pwwn. This method is the same if you are performing basic zoning or enhanced zoning.
Step 1 Step 2 Step 3 Step 4

In the Logical Domains pane, right click on the VSAN, and select Edit Local Full Zone Database. In the resulting dialog box, Right click on Zones and select Insert... Specify a zone name and press OK. Right click on the newly created zone and select Insert...
Figure 5-10 Possible zone member types

Step 5

Select the button for the type of zone member required. The labels for the three text boxes will change depending on the selection in the Zone By panel. For example, if Switch & Port is selected, then the text boxes change to Switch Interface (e.g: fc1/1) and Switch Address (e.g. 192.168.1.2). Also the meaning of the ... and pulldown buttons change depending on the selection in the Zone By panel.

Note

Domain & Port zoning should only be done when working in interop mode 2 or 3. See section Setting VSAN Interop Mode, page 4-92 for more information about interop modes.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

5-108

78-xxxxx-xx

Chapter 5

Zoning Zones

Alias refers to both FC Alias and Global Device Alias depending on the mode that Fabric Manager is running in.

Step 6

The resulting zone still needs to be added to a zoneset and then the zoneset needs to be activated, which is outlined in Zone Creation and Adding it to a Zoneset with Fabric Manager, page 5-103.

Creating a zone and adding it to a zoneset (CLI standalone method)


This procedure will demonstrate how to create a single zone with a solaris host and a disk storage port in it, then add it to the zoneset ZS_Engr_primary. This will utilize the standalone method, which will not automatically add the zone to the zoneset upon creation of the zone. This procedure also can be used to to determine how to add an existing zone to a zoneset. This procedure is the same for both basic zoning and enhanced zoning. However, with enhanced zoning the pending database needs to be committed at the end. This example will use pwwns as the zone members, which can be obtained either from the device itself or from the show flogi database vsan 804 command. Zoneset: ZS_Engr_primary Solaris host1, hba instance fcaw0: 22:35:00:0c:85:e9:d2:c2 Symmetrix 78, FA port 03ab: 10:00:00:00:c9:32:8b:a8
Figure 5-11 Standalone Zoning Topology

Step 1

Create the zone, building a zone name that reflects the names of the members.
ca-9506# config terminal Enter configuration commands, one per line. End with CNTL/Z. ca-9506(config)# zone name Z_host1_fcaw0_symm78FA03ab vsan 804 ca-9506(config-zone)# member pwwn 22:35:00:0c:85:e9:d2:c2 ca-9506(config-zone)# member pwwn 10:00:00:00:c9:32:8b:a8

Step 2

Then add the zone to the zoneset


ca-9506(config)# zoneset name ZS_Engr_primary vsan 804 ca-9506(config-zoneset)# member Z_host1_fcaw0_symm78FA03ab

Step 3

Display the zoneset:


ca-9506# show zoneset name ZS_Engr_primary vsan 804 zoneset name ZS_Engr_primary vsan 804

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

5-109

Chapter 5 Zones

Zoning

zone name Z_host1_fcaw0_symm78FA03ab vsan 804 pwwn 22:35:00:0c:85:e9:d2:c2 pwwn 10:00:00:00:c9:32:8b:a8

Step 4

Finally, to put the zoneset into production activate the zoneset using zoneset activate name ZS_Engr_primary vsan 804. This will activate all the zones in the zoneset, not just the one that was just added. For enhanced zoning, in addition to activating the zoneset, it needs to be committed.
ca-9506(config)# zoneset activate name ZS_Engr_primary vsan 804

Step 5 Step 6

If Enhanced zoning is being used, the zoneset must be explicitly committed:


ca-9506(config)# zone commit vsan 804

Display the zoneset:


ca-9506# show zoneset name ZS_Engr_primary vsan 804 zoneset name ZS_Engr_primary vsan 804 zone name Z_host1_fcaw0_symm78FA03ab vsan 804 pwwn 22:35:00:0c:85:e9:d2:c2 pwwn 10:00:00:00:c9:32:8b:a8

Step 7

If enhanced zoning is enabled, then the changes to the pending database should be committed and therefore be distributed to the other switchs full zone database:
ca-9506(config)# zone commit vsan 804

Creating a zone and adding it to a zoneset (CLI inline method)


This procedure will demonstrate how to create a single zone with a solaris host and a disk storage port in it, then add it to the zoneset ZS_Engr_primary. This will utilize the inline method, which will automatically add the zone to the zoneset upon creation of the zone. For enhanced zoning the additional step of committing the changes is required. This example will use pwwns as the zone members, which can be obtained either from the device itself or from the show flogi database vsan 804 command. Zoneset: ZS_Engr_primary Solaris host1, hba instance fcaw0: 22:35:00:0c:85:e9:d2:c2 Symmetrix 78, FA port 03ab: 10:00:00:00:c9:32:8b:a8
Figure 5-12 Inline Zoning Topology

Step 1

Enter the submode of the zoneset


ca-9506# config terminal Enter configuration commands, one per line. End with CNTL/Z.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

5-110

78-xxxxx-xx

Chapter 5

Zoning Zones

ca-9506(config)# zoneset name ZS_Engr_primary vsan 804

Step 2

Create the zone


ca-9506(config-zoneset)# zone name Z_host1_fcaw0_symm78FA03ab

Step 3

Add the members


ca-9506(config-zoneset-zone)# member pwwn 22:35:00:0c:85:e9:d2:c2 ca-9506(config-zoneset-zone)# member pwwn 10:00:00:00:c9:32:8b:a8

Step 4

Display the zoneset:


ca-9506# show zoneset name ZS_Engr_primary vsan 804 zoneset name ZS_Engr_primary vsan 804 zone name Z_host1_fcaw0_symm78FA03ab vsan 804 pwwn 22:35:00:0c:85:e9:d2:c2 pwwn 10:00:00:00:c9:32:8b:a8

Step 5

For basic zoning, put the zoneset into production activate the zoneset using zoneset activate name ZS_Engr_primary vsan 804. This will activate all the zones in the zoneset, not just the one that was just added. For enhanced zoning, in addition to activating the zoneset, it needs to be committed.
ca-9506(config)# zoneset activate name ZS_Engr_primary vsan 804

Step 6

If Enhanced zoning is being used, the zoneset must be explicitly committed:


ca-9506(config)# zone commit vsan 804

Step 7

Display the zoneset:


ca-9506# show zoneset name ZS_Engr_primary vsan 804 zoneset name ZS_Engr_primary vsan 804 zone name Z_host1_fcaw0_symm78FA03ab vsan 804 pwwn 22:35:00:0c:85:e9:d2:c2 pwwn 10:00:00:00:c9:32:8b:a8

Creating a FC Alias based zone (CLI)


Fibre Channel aliases allow the administrator toassign a plain text, human readable name to a pwwn, fcid, interface, ip-address, nwwn or symbolic-nodename. FC Aliases are restricted to the VSAN for which they were created in. The most common and recommended is the pwwn, which will be the basis for this procedure.

Tip

Aliases are distributed with the full zoneset database, thus if zoning is going to be editted on multiple switches, full zoneset distribution should be enabled. An alias can be mapped to more than one device, however it is recommended that a one to one mapping be used.

The following resources will be used: Zoneset: ZS_Engr_primary

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

5-111

Chapter 5 Zones

Zoning

Solaris host1, hba instance fcaw0: 22:35:00:0c:85:e9:d2:c2 Symmetrix 78, FA port 03ab: 10:00:00:00:c9:32:8b:a8
Figure 5-13 Alias Zoning Topology

Step 1

Create the FC Alias to pwwn mappings.


ca-9506# config terminal Enter configuration commands, one per line. End with CNTL/Z. ca-9506(config)# fcalias name host1_fcaw0 vsan 804 ca-9506(config-fcalias)# member pwwn 22:35:00:0c:85:e9:d2:c2 ca-9506(config-fcalias)# exit ca-9506(config)# fcalias name symm78_fa03ab vsan 804 ca-9506(config-fcalias)# member pwwn 10:00:00:00:c9:32:8b:a8 ca-9506(config-fcalias)# end

Step 2

Display the newly created FC Aliases.


ca-9506# show fcalias vsan 804 fcalias name host1_fcaw0 vsan 804 pwwn 22:35:00:0c:85:e9:d2:c2 fcalias name symm78_fa03ab vsan 804 pwwn 10:00:00:00:c9:32:8b:a8

Step 3

Create the zone using the FC Aliases


ca-9506# config terminal Enter configuration commands, ca-9506(config)# zoneset name ca-9506(config-zoneset)# zone ca-9506(config-zoneset-zone)# ca-9506(config-zoneset-zone)# one per line. End with CNTL/Z. ZS_Engr_primary vsan 804 name Z_host1_fcaw0_symm78FA03ab member fcalias host1_fcaw0 member fcalias symm78_fa03ab

Step 4

Optionally, display the zoneset.


ca-9506# show zoneset vsan 804 zoneset name ZS_Engr_primary vsan 804 zone name Z_host1_fcaw0_symm78FA03ab vsan 804 fcalias name host1_fcaw0 vsan 804 pwwn 22:35:00:0c:85:e9:d2:c2 fcalias name symm78_fa03ab vsan 804 pwwn 10:00:00:00:c9:32:8b:a8

Step 5

Activate the zoneset


ca-9506# conf t Enter configuration commands, one per line. End with CNTL/Z. ca-9506(config)# zoneset activate name ZS_Engr_primary vsan 804

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

5-112

78-xxxxx-xx

Chapter 5

Zoning Zones

Zoneset activation initiated. check zone status

Step 6

If enhanced zoning is enabled then the configuration needs to be committed:


ca-9506(config)# zone commit vsan 804

Creating an Interface based zone (CLI method)


This procedure will demonstrate how to create a zone based upon the physical interface(fcX/Y) of the switch. If enhanced zoning is enabled, then the additional step of committing the pending database will be required.

Tip

Interface based zoning should be done when a zone needs to be created prior to the hba being connected to the fabric. After, the hba is connected to the fabric, the zone member should be converted to a pwwn based member.
Figure 5-14 Interface Zoning Topology

Step 1

Create the zone using the interfaces


ca-9506# config terminal Enter configuration commands, ca-9506(config)# zoneset name ca-9506(config-zoneset)# zone ca-9506(config-zoneset-zone)# ca-9506(config-zoneset-zone)# one per line. End with CNTL/Z. ZS_Engr_primary vsan 804 name Z_host1_fcaw0_symm78FA03ab member interface fc1/1 member interface fc1/2

Step 2

Optionally, display the zoneset.


ca-9506# show zoneset vsan 804 zoneset name ZS_Engr_primary vsan 804 zone name Z_host1_fcaw0_symm78FA03ab vsan 804 interface fc1/1 swwn 20:00:00:0c:85:e9:d2:c0 interface fc1/2 swwn 20:00:00:0c:85:e9:d2:c0

Note

The swwn is the switchs wwn as displayed by the show wwn switch command:
ca-9506# show wwn switch Switch WWN is 20:00:00:0c:85:e9:d2:c0

Step 3

Activate the zoneset

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

5-113

Chapter 5 Zones

Zoning

ca-9506# conf t Enter configuration commands, one per line. End with CNTL/Z. ca-9506(config)# zoneset activate name ZS_Engr_primary vsan 804 Zoneset activation initiated. check zone status

Step 4

If enhanced zoning is enabled then the configuration needs to be committed:


ca-9506(config)# zone commit vsan 804

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

5-114

78-xxxxx-xx

C H A P T E R

Inter-VSAN Routing
Inter-VSAN Routing (IVR), first introduced into the MDS platform in SAN-OS 1.3(1), provided the ability for devices in different VSANs to communicate. This facilitated the ability to create VSANs consisting of devices which could be shared with devices in different VSANs. The classic example is to share a tape library with many hosts which are in different VSANs, or to allow disk subsystems to communicate over WAN distances without having to merge large zoneset databases. This initial release of IVR (IVR-1) suffered from two deficiencies in that it required unique domain IDs in the source and destination VSANs and secondly that the two VSANs could not be the same number. Although, these two issues could be easily solved with some simple planning. If the requirement was needed to merge two large fabrics and then IVR between them, there maybe duplicate VSAN IDs or domain IDs in the two fabrics. Introduced in SAN-OS 2.1, IVR obtained the ability to perform Network Address Translation (NAT). IVR with FCNAT (IVR-2) solved both of the issues with IVR-1, as it eliminated the need for unique domain IDs and VSAN IDs. IVR-2 continues to use the same basic principles as in IVR-1 such as IVR topologies, IVR zones and zonesets as well as transit VSANs. In addition to the FCNAT capabilities, IVR also gained the ability to leverage CFS (introduced in SAN-OS 2.0(1)) and auto-topology (introduced in SAN-OS 2.1(1)). Although these two technologies make implementing IVR easier and faster, careful planning should be done prior to configuration.

Tip

The preferred method of configuring IVR either with or without NAT is with Cisco Fabric Services (CFS). This will ease topology configuration and reduce the number of configuration steps and potential configuration errors. See IVR with CFS, page 6-115

IVR with CFS


Prior to SAN-OS 2.X, the IVR topology had to be defined on each switch using either FM or the CLI. If a new switch was added to fabric which was going to perform IVR, then the entire IVR topology had to be manually entered on the new switch, and then the other switches would have to be modified accordingly for any new entries on the new switch. This constant manual intervention can lead to user errors. With the introduction of CFS support for IVR, each switch does not need to be manually modified for each change in the IVR topology. A single topology can be maintained and CFS can then distribute the changes to the other IVR switches in the fabric. Additionally, if a new switch is added to the fabric, CFS can automatically synchronize the new switch with the existing IVR topology.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

6-115

Chapter 6

Inter-VSAN Routing

Modifying the topology is still a manual operation, but the user nolonger needs to go to every switch (using either FM or CLI) and modify those switches. For example, a fabric with three switches in it would require nine entries into the topology database (3 switches with IVR * 3 VSAN Route Switches). Adding a fourth IVR switch would increase this from nine to sixteen (4 switches with IVR * 4 VSAN Route Switches). Using the following topology as a reference:
Figure 6-1 CFS Reference Topology

In the below example, which only contains two switches, Figure 6-2 on page 6-116 shows two switches, 172.22.36.9 and 172.22.36.11 which have duplicate entries for the VSAN Route Switch. Thus if a VSAN needed to be added to the VSAN route switch 172.22.36.11, I would need to modify two rows. One for each unique switch in the master column.

Note

If CFS is to be used with IVR, all IVR enabled switches must have CFS distribution for IVR enabled. Conversely, If CFS is not going to be used for IVR, then all of the IVR enabled switches should have CFS distribution for IVR disabled.

Figure 6-2

IVR Topology without CFS

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

6-116

78-xxxxx-xx

Chapter 6

Inter-VSAN Routing

If the same configuration was done with CFS, then I would have one row for each VSAN Route Switch and would not have to enter the same information for each switch in the fabric as in Figure 6-3 on page 6-117.
Figure 6-3 IVR Topology with CFS

It turns out that only one row per IVR enabled switch is displayed making the topology table easier to comprehend. Looking only at columns two and three. Its easy to tell that 172.22.36.11 routes between VSANs 1, 3000-3002, while switch 172.22.36.9 routes between 1, 3001 and 3003. All of the duplicate information is removed. The first column represents the switch that FM will use to perform any CFS operations from.

Configuring a Three Switch, Two Transit VSAN Topology with CFS


This recipe will cover how to configure the IVR topology for a configuration that requires three IVR switches leveraging two transit VSANs. This recipe will leverage CFS to distribute the topology, and IVR has been enabled on all three switches already. This procedure can be used for IVR-1 or IVR-2 with FC-NAT.

Tip

Instead of using multiple transit VSANs, look to use a single transit VSAN extended over multiple switches. It will simplify the topology while providing the isolation of a transit VSAN.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

6-117

Chapter 6

Inter-VSAN Routing

Figure 6-4

Three Switch, Dual Transit VSAN IVR Topology.

Prior to configuring the topology, the topology needs to be determined. By examining the diagram, the following entries will need to be configured.
Table 6-1 IVR Topology Table

VSAN Route Switch 172.22.36.11 172.22.36.9 172.22.36.142

VSANs to Route 3000, 3001 3001, 3002 3002, 3003

Tip

Creating a table such as Table 6-1 provides simple documentation and can be easily added to a proper implementation plan or detailed design document. In FM, in the Logical Domains pane, select the Fabric, All VSANs then IVR. Click the CFS tab to activate the other tabs, and then Local Topology.

Step 1 Step 2

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

6-118

78-xxxxx-xx

Chapter 6

Inter-VSAN Routing

Figure 6-5

Local Topology Tab

Step 3 Step 4 Step 5

Click the blue Create Row... button. In the resulting pop up box, reference Table 6-1 on page 6-118, in the Switch pulldown menu, select the first switch (172.22.36.11) and its associated VSAN list. Fill in the switchs VSAN List (3000, 3001).
Figure 6-6 Create Topology

Step 6 Step 7

Click Create. Repeat this procedure for the second and third switch. When the third switch has been created. The dialog box can be dismissed by selecting Close. The local topology should look like Figure 6-7 on page 6-120.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

6-119

Chapter 6

Inter-VSAN Routing

Figure 6-7

Local Topology

Step 8 Step 9 Step 10 Step 11

Select the Action tab. Select the Activate Local checkbox, and press the green Apply Changes button. Select the CFS pull down menu and select Commit. The message CFS(ivr):Committed should display in the bottom left hand corner of Fabric Manager. To verify, select the Active Topology tab and the following should be displayed:
Figure 6-8 Active Topology

At this point, the topology is correctly defined and the Active Topology will contain the correct information. Correct configuration and distribution via CFS will result in the Discrepancies tab containing no entries. From this point IVR Zonesets can be defined providing connectivity between the two end devices.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

6-120

78-xxxxx-xx

Chapter 6

Inter-VSAN Routing IVR-1

IVR-1
IVR-1 is the method of Inter-VSAN Routing that has existed since SAN-OS 1.3 and does not perform any Network Address Translation (NAT) functions. It requires unique VSAN and domain IDs across the IVR topology. It has the ability to leverage CFS for its configuration distribution and application locking. Although IVR-1 can be accomplished without the use of CFS, the recipes contained in this chapter will focus on using CFS.

Note

IVR-1 requires unique DomainIDs throughout the IVR topology.

Enable IVR-1
Prior to configuring IVR-1, it needs to be enabled, this can be done via CLI or FM. Any switch that is acting as a border switch or one that will route frames between VSANs should be enabled in the same manner. This includes CFS, such that either all the IVR switches have CFS enabled for IVR, or none of them do.

To enable IVR-1 via the CLI:


Step 1

Enter conf term mode and enable IVR:


172.22.36.11# conf terminal Enter configuration commands, one per line. 172.22.36.11(config)# ivr enable End with CNTL/Z.

Step 2

By default, CFS is not enabled for IVR-1, so enable CFS distribution. Notice that the scope is physical as IVR crosses VSANs.
172.22.36.11(config)# ivr distribute 172.22.36.11(config)# do show cfs application name ivr Enabled Timeout Merge Capable Scope : : : : Yes 30s Yes Physical

Step 3

Verify that all the switches are recognized by IVR using CFS.
172.22.36.11# show cfs peers name ivr Scope : Physical -------------------------------------------------Switch WWN IP Address -------------------------------------------------20:00:00:05:30:00:68:5e 172.22.36.11 [Local] Total number of entries = 1

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

6-121

Chapter 6 IVR-1

Inter-VSAN Routing

To enable IVR-1 via FM:


Step 1

In the Logical Domains pane, select All VSANs, then IVR. Under the control tab, change the command to enable for the switches that should have IVR enabled. Then click the Apply Changes button. The status field should change from disabled to enabled.
Figure 6-9 Enable IVR in Fabric Manager

Step 2

Next, enable CFS distribution for IVR. Click the CFS tab, then under the Enable Admin column, change the column from noSelection to enable and press the green Apply Changes button. The Last result column should display success.
Figure 6-10 Enable CFS Distribution for IVR

Single Switch, Two VSANs


In this recipe, the most basic IVR environment will be set up, involving one MDS switch and two VSANs. CFS will be used.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

6-122

78-xxxxx-xx

Chapter 6

Inter-VSAN Routing IVR-1

Figure 6-11 Single Switch IVR-1, Two VSAN Topology

The first phase is to create the IVR topology:


Step 1 Step 2 Step 3 Step 4

Enable IVR as per Enable IVR-1, page 6-121. In FM, in the Logical Domains window, select the fabric, All VSANs and then IVR. In the Local Topology tab click the Create Row... button. Enter in the VSANs to be routed in the VSAN list: 3000,3002, and select the switch that will perform the routing from the pull down menu. The click Create. The word success should be displayed in the botton of the window, then click Close.
Figure 6-12 Single Switch IVR-1 Create Topology

Step 5 Step 6

To activate the topology, click the Action tab, click the checkbox Activate Local and click the Apply Changes button. The topology is not active until it has been CFS committed. To CFS commit the topology, click the CFS pulldown menu button and select Commit. In the bottom left hand corner of FM, the message CFS(ivr):Committed should be displayed. Also, the Active Topology tabs contents should match that of the Local Topology tabs.

The next phase is to create the IVR zoneset and accompaning zones:
Step 1 Step 2

In FMs Logical Domains pane, select the fabric, All VSANs and then right click on IVR. Select Edit Local Full Zone Database In the resulting dialog box, to create the IVR zoneset right click in the left hand pane on Zonesets and choose Insert. Pick a name for the zoneset and click OK. The resulting zoneset will appear in the left hand pane.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

6-123

Chapter 6 IVR-1

Inter-VSAN Routing

Step 3 Step 4 Step 5

To create the IVR Zone, in the left hand pane, right click on Zones and select Insert. Enter a meaningful zonename and click OK. Add the members to be zoned from the bottom pane into the newly created zone by dragging them into the zone. In the left hand pane, drag the zone into the zoneset:
Figure 6-13 Single Switch IVR-1, Create Zoneset

Step 6 Step 7

To activate the zoneset and implicitly CFS commit it, right click on the zoneset and select Activate. Then Continue Activation. Click Close to return to the main FM window. The IVR members can be displayed on the map by selected the IVR Zone in the Logical Domains pane.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

6-124

78-xxxxx-xx

Chapter 6

Inter-VSAN Routing IVR-2: FC-NAT

Figure 6-14 Single Switch IVR-1, Display IVR Zoneset

IVR-2: FC-NAT
In SAN-OS 2.1(1a), Cisco introduced IVR with Fibre Channel Network Address Translation (NAT). This provides the user with the ability to route between fabrics containing duplicate domainIDs between VSANs, and routing between two VSANs of the same VSAN ID. In addition, instead of representing each native VSANs domain with a unique virtual domain, a single virtual domain is used to represent all of the domains in the native VSAN. Lastly, IVR-FCNAT, leveraging the CFS infrastructure can automatically discover the topology to use thus alleviating the process of configuring and maintaining the IVR topology.

Enabling IVR-2
This recipe provides the steps to enable IVR-2, with CFS and Auto Topology discovery. Prior to using IVR-2, it must be enabled using the following procedure.

Note

In a multi switch topology, these steps should be followed for all the switches in the fabric. Remember, that CFS commands (for example the adding of the NAT and Auto Topology) only have to be done from one switch as CFS will instruct the other switches.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

6-125

Chapter 6 IVR-2: FC-NAT

Inter-VSAN Routing

Step 1

In the Logical Domains pane, select All VSANs, then IVR. Under the control tab, change the command to enable for the switches that should have IVR enabled. Then click the Apply Changes button. The status field should change from disabled to enabled.
Figure 6-15 Enable IVR in Fabric Manager

Step 2

Next, enable CFS distribution for IVR. Click the CFS tab, then under the Enable Admin column, change the column from noSelection to enable and press the green Apply Changes button. The Last result column should display success.
Figure 6-16 Enable CFS Distribution for IVR

Note

Auto topology is not required for IVR-2 with FC-NAT. If it is not enabled, then the topology will need to be manually defined. If Auto Topology is not used then Step 4 can be skipped. Next to enable the FC-NAT and auto topology discovery functions of IVR, select the Action tab and check the Enable IVR Nat checkbox

Step 3

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

6-126

78-xxxxx-xx

Chapter 6

Inter-VSAN Routing IVR-2: FC-NAT

Step 4 Step 5

Check the Automatically Discover Topology check box. Press the Apply Changes button. The configuration is not yet active until it has been CFS committed.
Figure 6-17 Enabling IVR-2s FC-NAT and Auto Topology Discovery

Step 6 Step 7

Goto the CFS tab, and in the Config Changes Action column, change the selection to Commit. Then press the Apply Changes button. If you switch to the Active Topology tab, the current active topology will be displayed:
Figure 6-18 IVR-2 FC-NAT Auto Discovered Topology

At this point IVR-2 is enabled for CFS distribution, NAT and automatic topology discovery.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

6-127

Chapter 6 IVR-2: FC-NAT

Inter-VSAN Routing

How to Upgrade from IVR-1 to IVR-2:


This recipe will provide the steps to upgrade an IVR-1 configuration to IVR-2 leveraging both CFS and Auto Topology.

Warning

Upgrading from IVR-1 to IVR-2 will be disruptive to IVR based traffic but is not disruptive to non-IVR traffic, for example non IVR zones contained in a VSAN. Upgrading from IVR-1 to IVR-2 with NAT may cause virtual devices to change their FCIDs, thus requiring FCID dependant hosts such as HPUX and AIX to rescan for the devices.

Note

Within a single physical fabric, IVR-1 cannot coexist with IVR-2. Either the border switches running IVR are in IVR-1 mode or IVR-2 mode. Mixed configurations are not supported. CFS can be leveraged to make sure all switches are running the same configuration.

Step 1 Step 2

Backup the configuration for all the of MDS switches using a procedure similar to the recipe in Copying files to or from the MDS, page 1-19. Upgrade all IVR border switches to SAN-OS 2.1(1a) or later, as IVR-2 with FC-NAT was first introduced in SAN-OS 2.1(1a). Switches that are not acting as IVR border switches are not required to be upgraded. However, it is recommended that they be upgraded. For the recipe on upgrading SAN-OS firmware see Firmware Upgrades, page 1-22. Deactivate the IVR Zoneset. This will not delete the IVR zoneset from the local database as it will be reactivated once IVR-2 has been enabled. In FM, in the Logical Domains pane, select your Fabric, All VSANs, and then right-click on IVR. Select Deactivate Zoneset from the resulting pop-up menu. Upon selecting OK, all devices will now be isolated from devices in other VSANs. Upon successful deactivation, the pop-up box can be dismissed by clicking Close. Enable CFS for IVR for all switches that will perform IVR. In the Logical Domains pane, select the Fabric, All VSANs, then IVR. Click the CFS tab in the top pane. In the Enable Admin column, change the field to enable and press the green Apply Changes button.
Figure 6-19 Enable CFS

Step 3

Step 4

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

6-128

78-xxxxx-xx

Chapter 6

Inter-VSAN Routing IVR-2: FC-NAT

Step 5

Enable FC-Nat and Auto Topology. First select the CFS tab to determine the CFS master switch. Then select the Action tab. Only one switch should be listed here, it is the switch that FM will use to perform the configuration distribution from. Select both the Enable IVR Nat and the Automatically Discover Topology checkboxes. Press the green Apply Changes button.

Note

Although there may still be a Local Topology present on the switch, Auto Topology takes care of the need to modify the Local Topology database. Select the CFS button and choose Commit. In the bottom left corner of FM, the message CFS(ivr) Committed should be displayed. Activate the IVR Zoneset. In the Logical Domains pane, select the Fabric, All VSANs, then right click on IVR. Select Edit Local Full Zone Database from the resulting menu. Right click on the zoneset that was previously deactivated and select Activate. Select Continue Activation in the resulting confirmation window. It will take a few seconds to commit the changes and save the running-configuration to startup. Click Close to return to the main FM window.

Step 6 Step 7 Step 8 Step 9

At this point the MDSs have been upgrade to IVR-2 with FC-NAT, CFS and Auto Topology. There may be hosts such as HPUX and AIX that have their disks tied to the FCID. Since NAT is being performed, appropriate action should be performed to have the hosts rescan for the new FCIDs.

How to Configure Persistent FCIDs in IVR


Introduced in SAN-OS version 2.1(2), the virtual devices created by an IVR with NATconfiguration can have their associated FCIDs made persistent. This feature, similar to the persistent FC_ID feature discussed in Chapter 4, VSANs for actual devices, enables a virtual device to receive the same FC_ID across reboots of the MDS.

Tip

HPUX and AIX are two operating systems that utilize the FC_ID in the device path to the storage. In order to have the switch always assign the same FC_ID to a virtual device across switch reboots, persistent FC_ID for IVR must be configured. If the FC_ID changes for a device accessed by either an AIX or a HPUX host, the host may lose access to the device. This example will be used to illustrate how to configure a storage device to use a specific FCID in the hosts VSAN. The actual devices are already logged into the fabric and the IVR topology has already been created to include VSANs 3000 and 3002. IVR Features Enabled: CFS IVR with NAT Host PWWN: 10:00:00:00:c9:32:8b:a8 VSAN 3002

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

6-129

Chapter 6 IVR-2: FC-NAT

Inter-VSAN Routing

Storage PWWN: 50:06:0e:80:03:4e:95:33 VSAN 3000 Real FCID: 0xef0002 FCID to be configured in Host VSAN: 0x630063 IVR Topology:
AFID SWITCH WWN Active Cfg. VSANS -------------------------------------------------------------1 20:00:00:05:30:00:68:5e * yes yes 3000,3002

Step 1

Enter the IVR fcdomain configuration mode for the AFID and VSAN for the location of the virtual device. In this case its the AFID and VSAN where the storage will be virtual.
switch# conf t switch(config)# ivr fcdomain database autonomous-fabric-num 1 vsan 3002 switch(config-fcdomain)#

Step 2

Enter the native AFID, VSAN of the storage device and domain to be used in the hosts VSAN. CFS has been enabled for IVR so any changes will need to be committed later. The domainID in this command is in decimal format.
switch(config-fcdomain)# native-autonomous-fabric-num 1 native-vsan 3000 domain 99 fabric is locked for configuration. Please commit after configuration is done. switch(config-fcdomain-fcid)#

Step 3

Specify the pwwn and FCID to be used. Alternatively, a device-alias can be used instead of a pwwn. The virtual domain and FCIDs will not be created until the zoneset is activated.
switch(config-fcdomain-fcid)# pwwn 50:06:0e:80:03:4e:95:33 fcid 0x630063 The FCID should correspond to virtual domain 99 specified earlier for this mode switch(config-fcdomain-fcid)#

Step 4

Create the IVR zonesets and zones:


switch(config-fcdomain-fcid# ivr switch(config-ivr-zoneset)# zone switch(config-ivr-zoneset-zone)# switch(config-ivr-zoneset-zone)# switch(config-ivr-zoneset-zone)# zoneset name IVR_Zoneset1 name IVRZ_host1_lpfc0_Array1_port12 member pwwn 10:00:00:00:c9:32:8b:a8 vsan 3002 member pwwn 50:06:0e:80:03:4e:95:33 vsan 3000 ivr zoneset activate name IVR_Zoneset1

Step 5

CFS commit the changes for IVR to activate the IVR zoneset and the modifications to the IVR persistent FCID database.
switch(config)# ivr commit commit initiated. check ivr status

Step 6

Verify FCIDs and active zoneset:


switch(config)# do show ivr fcdomain database ---------------------------------------------------AFID Vsan Native-AFID Native-Vsan Virtual-domain ---------------------------------------------------1 3002 1 3000 0x63(99) Number of Virtual-domain entries: 1 ---------------------------------------------------AFID Vsan Pwwn Virtual-fcid

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

6-130

78-xxxxx-xx

Chapter 6

Inter-VSAN Routing IVR-2: FC-NAT

---------------------------------------------------1 3002 50:06:0e:80:03:4e:95:33 0x630063 [HDS20117-c20-8] Number of Virtual-fcid entries: 1 switch# show fcns database vsan 3002 VSAN 3002: -------------------------------------------------------------------------FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------0x1c0003 N 10:00:00:00:c9:32:8b:a8 (Emulex) scsi-fcp [ca-sun1_lpfc0] 0x630063 N 50:06:0e:80:03:4e:95:33 scsi-fcp [HDS20117-c20-8] Total number of entries = 2 switch# show zoneset active vsan 3002 zoneset name ZS_test vsan 3002 zone name zone1 vsan 3002 pwwn 50:06:0e:80:03:4e:99:99 pwwn 50:06:0e:80:03:4e:98:98 zone name IVRZ_IVRZ_host1_lpfc0_Array1_port12 vsan 3002 * fcid 0x630063 [pwwn 50:06:0e:80:03:4e:95:33] [HDS20117-c20-8] * fcid 0x1c0003 [pwwn 10:00:00:00:c9:32:8b:a8] [ca-sun1_lpfc0]

Single Switch: Two VSANs


In this example, a simple two VSAN configuration will be done with one MDS. This type of topology could be done with IVR-1, however, upon closer inspection not only are the domainIDs the same between the two VSANs, but the devices themselves have exactly the same FCID. Only IVR-2 with FC-NAT is suitable for this configuration.
Figure 6-20 IVR-2 Single Switch Example Topology

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

6-131

Chapter 6 IVR-2: FC-NAT

Inter-VSAN Routing

Step 1 Step 2

Enable IVR-2 with CFS, FC-NAT and automatic topology discovery as per Enabling IVR-2, page 6-125. Create the IVR Zones and Zoneset. In the Logical Domains pane, select the Fabric, All VSANs and then right click on IVR and select Edit Full Local Zone Database. The resulting dialog box will look exactly like a non-IVR zoning dialog box:
Figure 6-21 IVR-2 Create Zoneset

Step 3

In the resulting dialog box, to create the IVR zoneset right click in the left hand pane on Zonesets and choose Insert. Pick a name for the zoneset and click OK. The resulting zoneset will appear in the left hand pane. To create the IVR Zone, in the left hand pane, right click on Zones and select Insert. Enter a meaningful zonename and click OK. Add the members to be zoned from the bottom pane into the newly created zone by dragging them into the zone. In the left hand pane, drag the zone into the zoneset:

Step 4 Step 5 Step 6

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

6-132

78-xxxxx-xx

Chapter 6

Inter-VSAN Routing IVR-2: FC-NAT

Figure 6-22 Single Switch IVR-2, Create Zoneset

Step 7

To activate the zoneset and implicitly CFS commit it, right click on the zoneset and select Activate. Then Continue Activation. Click Close to return to the main FM window.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

6-133

Chapter 6 IVR-2: FC-NAT

Inter-VSAN Routing

Figure 6-23 Single Switch IVR-2, Display IVR Zoneset

At this point, the host ca-sun1 can access the storage on HDS20017-c20-8.

Adding a new IVR Enabled Switch


This recipe will provide the process necessary to add a new switch to an existing IVR-2 configuration with CFS and auto topology. It will build off of the configuration described in Single Switch: Two VSANs, page 6-131. The new switch is the one with the IP address of 172.22.36.9:

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

6-134

78-xxxxx-xx

Chapter 6

Inter-VSAN Routing IVR-2: FC-NAT

Figure 6-24 Topology for adding a new IVR-2 Switch

In this configuration, the switch 172.22.36.11 is currently performing IVR-FCNat between the host in VSAN 3000 and the storage in VSAN 3002. IVR w/FC-NAT using CFS and auto topology will be enabled on 172.22.36.9 without any impact to the currently running configuration.

Tip

By having multiple IVR-2 capable switches configured, it is possible for a second switch to take over the routing functionality of the other provided it can directly see both the source and destination VSANs. For this recipe, if IVR-2 was disabled on 172.22.36.11, 172.22.36.9 could automatically take over the routing. This is not possible with IVR-1. Enable IVR in FM, by in the Logical Domains pane, select Fabric, All VSANs then IVR. In the control tab, under the Command column, change the 172.22.36.9 switch to enable and press the green Apply Changes button. The status column should change from a yellow progess box, to the word success. Select the CFS tab. In the Enable Admin column, change the field for the 172.22.36.9 switch to enable, and press the green Apply Changes button. The Last Result column should change from inProgress to Success. Click the Active Topology column, and a resulting topology will be displayed that will include both switches, even though only the 172.22.36.11 switch is performing any IVR. Save the configuration of both switches

Step 1 Step 2

Step 3 Step 4 Step 5 Step 6

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

6-135

Chapter 6 IVR-2: FC-NAT

Inter-VSAN Routing

Note

By having CFS already enabled on the first switch, 172.22.36.11, by just enabling IVR and CFS on the second switch, 172.22.36.9, the second switch will learn what configuration parameters namely FC-NAT and auto topology, should be enabled. Therefore, there was no need to enable FC-NAT and auto topology on the second switch.

Tip

This is one of the primary advantages of CFS as the configuration of the first switch could have been very complex and accurately configuring additional switches may be considerably tougher as the number of switches, devices and VSANs grow within the topology. Adding a single switch to an N IVR enabled configuration will require changes to be made to all N switches, therefore leaving room for human error if done manually.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

6-136

78-xxxxx-xx

C H A P T E R

FCIP
FCIP (Fibre Channel over IP) is a IETF standards based protocol for connecting Fibre Channel SANs over IP networks. FCIP encapsulates the FCP frames in a TCP/IP packet which is then sent across an IP network. FCIP is used to interconnect geographically dispersed SANs using the IP backbone network. In short FCIP creates a FC tunnel over an existing IP network. The MDS 9200 and 9500 series switches support FCIP using the IPS-8, IPS-4 and the 14+2 blades. The chapter below documents recipes to create and manage FCIP links between MDS switches.

How to Enable FCIP


The FCIP enable command has to be run before attempting to configure FCIP on the switch.
Warning

If the fcip enable command is not run, further FCIP configuration is not possible. This command will enable further FCIP configuration options in the CLI.
MDS1# conf t Enter configuration commands, one per line. MDS1(config)# FCIP enable MDS1(config)#^Z MDS1#

End with CNTL/Z.

Configuring FCIP on MDS


The recipe below shows the configuration of FCIP on the MDS. The topology is as shown in Figure 7-1
Figure 7-1 FCIP Topology

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-137

Chapter 7 Configuring FCIP on MDS

FCIP

The topology consists of two 9509 with one IPS-8 blade in each of the switches. The MDS switches are both connected to Cisco Catalyst 6509 switches the gigE port 8/1 on each of the MDS switches. The gigE port 8/1 on the switch sjc7-9509-5 has an ip address 172.22.34.83 with a subnet mask of 255.255.254.0 and the gateway address 172.22.34.1. The gigE 8/1 port on the switch sjc7-9509-6 has an ip address 172.22.36.58 with a subnet 255.255.254.0 and the gateway address 172.22.34.1. In the recipe the a FCIP tunnel will be established between the switch sjc7-9509-5 (gigE 8/1) and sjc7-9509-6 (gigE 8/1).

Warning

The ip address for the ports on the IPS blade should be in a different subnet than the management interface. This is critical to get FCIP working on the MDS.

The configuration steps on the MDS switches are as follows.


Step 1

Configure the gigE interface on the MDS switches The gigE interface on the MDS switch sjc7-9509-5 is given an ip address, a subnet mask. This allows the gigE interface to communicate with the network.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# interface gigabitethernet 8/1 sjc7-9509-5(config-if)# ip address 172.22.34.83 255.255.254.0 sjc7-9509-5(config-if)# no shut sjc7-9509-5(config-if)# end sjc7-9509-5#

The gigE interface on the MDS switch sjc7-9509-5 is given an ip address, a subnetmask. This allows the gigE interface to communicate with the network.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# interface gigabitethernet 8/1 sjc7-9509-6(config-if)# ip address 172.22.36.98 255.255.254.0 sjc7-9509-6(config-if)# no shut sjc7-9509-6(config-if)# end sjc7-9509-5#

Step 2

Configure ip route so that the 2 gigE interfaces can communicate with each other IP route needs to be configured to allow the two gigE ports on switches sjc7-9509-5 and sjc7-9509-6 to communicate with each other. In the above recipe the gigE ports in two different subnet. This does require explicit routes to be able to communicate with each other.

Note

The recommended best practice is just to have a host route to each of the two gigE interfaces. Therefore, only allow the two gigE interfaces to communicate with each other, This is done by having a subnet mask of 255.255.255.255 when the route is added. For the gigE port 8/1 on the switch sjc7-9509-5 to communicate with the port gigE 8/1 on switch sjc7-9509-6 the following is the route configuration that needs to be done on switch sjc7-9509-5.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# ip route 172.22.36.98 255.255.255.255 172.22.34.1 interface gigabitethernet 8/1 sjc7-9509-5(config)# end sjc7-9509-5#

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-138

78-xxxxx-xx

Chapter 7

FCIP Configuring FCIP on MDS

The above configuration translates as, in order to reach 172.22.36.98 use the gateway 172.22.34.1 and interface gigE 8/1 on switch sjc7-9509-5. Similarly for the gigE port 8/1 on switch sjc7-9509-6 to communicate with gigE port 8/1 on switch sjc7-9509-5 the following route configuration needs to be done on switch sjc7-9509-6
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# ip route 172.22.34.83 255.255.255.255 172.22.36.1 interface gigabitethernet 8/1 sjc7-9509-6(config-if)# end sjc7-9509-6#

The above configuration translates as, in order to reach 172.22.34.83 use the gateway 172.22.36.1 and interface gigE 8/1 on switch sjc7-9509-6.
Step 3

Ping the gigE interfaces to ensure that the gigE ports can communicating with each other From the switch sjc7-9509-5 ping the ip address of the gigE interface 8/1 on switch sjc7-9509-6. Similarly ping the ip address of the gigE interface 8/1 on switch sjc7-9509-5 from switch sjc7-9509-6. This can be done from the switch prompt.
sjc7-9509-5# ping 172.22.36.98 PING 172.22.36.98 (172.22.36.98) 56(84) bytes of data. 64 bytes from 172.22.36.98: icmp_seq=1 ttl=254 time=0.504 64 bytes from 172.22.36.98: icmp_seq=2 ttl=254 time=0.404 64 bytes from 172.22.36.98: icmp_seq=3 ttl=254 time=0.414 64 bytes from 172.22.36.98: icmp_seq=4 ttl=254 time=0.416 64 bytes from 172.22.36.98: icmp_seq=5 ttl=254 time=0.449 64 bytes from 172.22.36.98: icmp_seq=6 ttl=254 time=0.399 64 bytes from 172.22.36.98: icmp_seq=7 ttl=254 time=0.411

ms ms ms ms ms ms ms

--- 172.22.36.98 ping statistics --7 packets transmitted, 7 received, 0% packet loss, time 5999ms rtt min/avg/max/mdev = 0.399/0.428/0.504/0.036 ms sjc7-9509-5# sjc7-9509-6# ping 172.22.34.83 PING 172.22.34.83 (172.22.34.83) 56(84) bytes of data. 64 bytes from 172.22.34.83: icmp_seq=1 ttl=254 time=0.492 64 bytes from 172.22.34.83: icmp_seq=2 ttl=254 time=0.401 64 bytes from 172.22.34.83: icmp_seq=3 ttl=254 time=0.438 64 bytes from 172.22.34.83: icmp_seq=4 ttl=254 time=0.440 64 bytes from 172.22.34.83: icmp_seq=5 ttl=254 time=0.408 64 bytes from 172.22.34.83: icmp_seq=6 ttl=254 time=0.400 64 bytes from 172.22.34.83: icmp_seq=7 ttl=254 time=0.389

ms ms ms ms ms ms ms

--- 172.22.34.83 ping statistics --7 packets transmitted, 7 received, 0% packet loss, time 5995ms rtt min/avg/max/mdev = 0.386/0.412/0.492/0.037 ms sjc7-9509-6#

Note

It is critical to check the connectivity between the host NIC card and the gigE port on the MDS IPS blade before proceeding further. A successful ping test should be sufficient to check the connectivity. Measure the round trip time between the two gigE interfaces. It is important to measure the round trip time between the interfaces. This value is required for later during the configuration.
sjc7-9509-5# ips measure-rtt 172.22.36.98 interface gigabitethernet 8/1 Round trip time is 407 micro seconds (0.41 milli seconds) sjc7-9509-5#

Step 4

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-139

Chapter 7 Configuring FCIP on MDS

FCIP

sjc7-9509-6# ips measure-rtt 172.22.34.83 interface gigabitethernet 8/1 Round trip time is 407 micro seconds (0.41 milli seconds) sjc7-9509-6#

Note

FCIP uses port 3225. If there is a firewall between the two switches connected through FCIP then port 3225 need to be opened for the FCIP tunnel to come up between the two switches. The VSAN (defaults to VSAN 1) to which the FCIP interfaces belongs to cannot be suspended. This will prevent the FCIP tunnel from coming up. Configure the FCIP profile on both the switches. FCIP profile needs to be created. The profile defines the characteristics for the FCIP tunnel that is setup. The round trip time measured is needed during the profile configuration. In this case the 407 micro seconds. The ip address used in the profile config below is the ip address assigned to the gigE interface on the switch that will be associated with the profile.

Step 5

Tip

It is recommended that the FCIP profile and FCIP interface numbers number be kept same on both the switches for the given FCIP tunnel for easier management.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# fcip profile 100 sjc7-9509-5(config-profile)# ip address 172.22.34.83 sjc7-9509-5(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 500 round-trip-time-us 407 sjc7-9509-5(config-profile)# end sjc7-9509-5# sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# fcip profile 100 sjc7-9509-6(config-profile)# ip address 172.22.36.98 ssjc7-9509-6(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 500 round-trip-time-us 407 sjc7-9509-6(config-profile)# end sjc7-9509-6#

At this stage of the configuration the FCIP profile 100 has been defined on both the switches sjc7-9509-5 and sjc7-9509-6.
Step 6

Configure the fcip interface on both the switches. The fcip interfaces need to be configured on both the switches. The configuration is as shown below.In the fcip interface configuration the profile to used and the peer information (remote gigEs ip address are specified.)are specified. Additionally compression and write acceleration can also be configured.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# interface fcip 100 sjc7-9509-5(config-if)# peer-info ipaddr 172.22.36.98 sjc7-9509-5(config-if)# use-profile 100 sjc7-9509-5(config-if)# no shutdown sjc7-9509-5(config-if)# end sjc7-9509-5# sjc7-9509-6# conf t

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-140

78-xxxxx-xx

Chapter 7

FCIP Configuring multiple FCIP tunnels using a single gigE port

Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# int fcip 100 sjc7-9509-6(config-if)# use-profile 100 sjc7-9509-6(config-if)# peer-info ipaddr 172.22.34.84 sjc7-9509-6(config-if)# no shut sjc7-9509-6(config-if)# end sjc7-9509-6#

The FCIP tunnel should be up and running now. a show interface fcip 100 brief should show the status of the FCIP link between the two switches.
sjc7-9509-5# sh int fcip 100 br ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip100 1 auto on trunking TE 100 GigabitEthernet8/1 -sjc7-9509-5# sjc7-9509-6# sh int fcip 100 br ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip100 1 auto on trunking TE 100 GigabitEthernet8/1 -sjc7-9509-6#

Configuring multiple FCIP tunnels using a single gigE port


Consider the following scenario, MDS1 in location A has to be connected to a MDS 2 in location B and also to MDS3 in location C both using FCIP links. Also there is only one free IPS port on MDS 1 in location A. One way to accomplish this would be to use the same gigE port 8/1 on MDS 1 but use different TCP ports to connect to location B and location C. FCIP uses TCP port 3225 for the first link, say between MDS 1 and MDS 2. For the second link between MDS 1and MDS 3 FCIP can be configured to use an alternate TCP port. The recipe below explains how this is done. The topology is shown below.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-141

Chapter 7 Configuring multiple FCIP tunnels using a single gigE port

FCIP

Figure 7-2

FCIP 3 Way topology

The first FCIP link is configured between gigE port 8/1 on switch sjc7-9509-6 and the gigE port 8/1 on switch sjc7-9509-5. This FCIP connection used the default FCIP TCP port 3225. The second FCIP link between gigE port 8/1 on switch sjc7-9509-5 and gigE port 9/1 on switch sjc7-9509-3 is configured to use an alternate TCP port 3500.

Note

The down side to this configuration is that the bandwidth of the gigE port 8/1 on the switch sjc7-9509-6 has to be shared between the 2 tunnels that will be set up to switch sjc7-9509-3 and sjc7-9509-5. The configuration summary on the three MDS switches are as follows.

Configure FCIP tunnel fcip100 between sjc7-9509-5 and sjc7-9509-6 using the default FCIP port. Configure FCIP tunnel fcip100 between sjc7-9509-5 and sjc7-9509-3 using TCP port 3500.

Configuring tunnel fcip100


Step 1

Configure the gigE interface on the MDS switches sjc7-9509-6 and sjc7-9509-5 The gigE interface on the MDS switch sjc7-9509-5 is given an ip address, a subnetmask. This allows the gigE interface to communicate with the network.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# interface gigabitethernet 8/1 sjc7-9509-5(config-if)# ip address 172.22.34.83 255.255.254.0 sjc7-9509-5(config-if)# no shut sjc7-9509-5(config-if)# ^Z sjc7-9509-5#

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-142

78-xxxxx-xx

Chapter 7

FCIP Configuring multiple FCIP tunnels using a single gigE port

The gigE interface on the MDS switch sjc7-9509-5 is given an ip address, a subnetmask. This allows the gigE interface to communicate with the network.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# interface gigabitethernet 8/1 sjc7-9509-6(config-if)# ip address 172.22.36.98 255.255.254.0 sjc7-9509-6(config-if)# no shut sjc7-9509-6(config-if)# ^Z sjc7-9509-5#

Step 2

Configure ip route so that the two gigE interfaces can communicate with each other IP route needs to be configured to allow the two gigE ports on switches sjc7-9509-5 and sjc7-9509-6 to communicate with each other. In the above recipe the gigE ports in two different subnets. This does require explicit routes to be able to communicate with each other.

Note

The recommended best practice is just to have a host route to each of the two gigE interfaces. I.e. only allow the two gigE interfaces to communicate with each other, This is done by having a subnet mask of 255.255.255.255 when the route is added. Syntax for configuring a route is shown below. For the gigE port 8/1 on the switch sjc7-9509-5 to communicate with the port gigE 8/1 on switch sjc7-9509-6 the following is the route configuration needs to be done on switch sjc7-9509-5.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# ip route 172.22.36.98 255.255.255.255 172.22.34.1 interface gigabitethernet 8/1 sjc7-9509-5(config)# ^Z sjc7-9509-5#

The above configuration translates as, in order to reach 172.22.36.98 use the gateway 172.22.34.1 and interface gigE 8/1 on switch sjc7-9509-5. Similarly for the gigE port 8/1 on switch sjc7-9509-6 to communicate with gigE port 8/1 on switch sjc7-9509-5 the following route configuration needs to be done on switch sjc7-9509-6.

sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# ip route 172.22.34.83 255.255.255.255 172.22.36.1 interface gigabitethernet 8/1 sjc7-9509-6(config-if)# ^Z sjc7-9509-6#

The above configuration translates as, in order to reach 172.22.34.83 use the gateway 172.22.36.1 and interface gigE 8/1 on switch sjc7-9509-6.
Step 3

Ping the gigE interfaces to ensure that the gigE ports can communicating with each other From the switch sjc7-9509-5 ping the ip address of the gigE interface 8/1 on switch sjc7-9509-6. Similarly ping the ip address of the gigE interface 8/1 on switch sjc7-9509-5 from switch sjc7-9509-6. This can be done from the switch prompt.
sjc7-9509-5# ping 172.22.36.98 PING 172.22.36.98 (172.22.36.98) 56(84) bytes of data. 64 bytes from 172.22.36.98: icmp_seq=1 ttl=254 time=0.504 64 bytes from 172.22.36.98: icmp_seq=2 ttl=254 time=0.404 64 bytes from 172.22.36.98: icmp_seq=3 ttl=254 time=0.414 64 bytes from 172.22.36.98: icmp_seq=4 ttl=254 time=0.416

ms ms ms ms

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-143

Chapter 7 Configuring multiple FCIP tunnels using a single gigE port

FCIP

64 bytes from 172.22.36.98: icmp_seq=5 ttl=254 time=0.449 ms 64 bytes from 172.22.36.98: icmp_seq=6 ttl=254 time=0.399 ms 64 bytes from 172.22.36.98: icmp_seq=7 ttl=254 time=0.411 ms --- 172.22.36.98 ping statistics --7 packets transmitted, 7 received, 0% packet loss, time 5999ms rtt min/avg/max/mdev = 0.399/0.428/0.504/0.036 ms sjc7-9509-5# sjc7-9509-6# ping 172.22.34.83 PING 172.22.34.83 (172.22.34.83) 56(84) bytes of data. 64 bytes from 172.22.34.83: icmp_seq=1 ttl=254 time=0.492 64 bytes from 172.22.34.83: icmp_seq=2 ttl=254 time=0.401 64 bytes from 172.22.34.83: icmp_seq=3 ttl=254 time=0.438 64 bytes from 172.22.34.83: icmp_seq=4 ttl=254 time=0.440 64 bytes from 172.22.34.83: icmp_seq=5 ttl=254 time=0.408 64 bytes from 172.22.34.83: icmp_seq=6 ttl=254 time=0.400 64 bytes from 172.22.34.83: icmp_seq=7 ttl=254 time=0.389

ms ms ms ms ms ms ms

--- 172.22.34.83 ping statistics --7 packets transmitted, 7 received, 0% packet loss, time 5995ms rtt min/avg/max/mdev = 0.386/0.412/0.492/0.037 ms sjc7-9509-6#

Step 4

Measure the round trip time between the two gigE interfaces. It is important to measure the round trip time between the interfaces. This value is required for later during the configuration.
sjc7-9509-5# ips measure-rtt 172.22.36.98 interface gigabitethernet 8/1 Round trip time is 407 micro seconds (0.41 milli seconds) sjc7-9509-5# sjc7-9509-6# ips measure-rtt 172.22.34.83 interface gigabitethernet 8/1 Round trip time is 407 micro seconds (0.41 milli seconds) sjc7-9509-6#

Note

FCIP uses port 3225. If there is a firewall between the two switches connected through FCIP then port 3225 need to be opened for the FCIP tunnel to come up between the two switches. The VSAN (defaults to VSAN 1) to which the FCIP interfaces belongs to cannot be suspended. This will prevent the FCIP tunnel from coming up. Configure the FCIP profile on both the switches. FCIP profile needs to be created. The profile defines the characteristics for the FCIP tunnel that is setup. The round trip time measured is needed during the profile configuration. In this case the 407 micro seconds. The ip address used in the profile config below is the ip address assigned to the gigE interface on the switch that will be associated with the profile.

Step 5

Tip

It is recommended that the FCIP profile and FCIP interface numbers number be kept same on both the switches for the given FCIP tunnel for easier management.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# fcip profile 100 sjc7-9509-5(config-profile)# ip address 172.22.34.83 sjc7-9509-5(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 500 round-trip-time-us 407 sjc7-9509-5(config-profile)# ^Z

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-144

78-xxxxx-xx

Chapter 7

FCIP Configuring multiple FCIP tunnels using a single gigE port

sjc7-9509-5# sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# fcip profile 100 sjc7-9509-6(config-profile)# ip address 172.22.36.98 ssjc7-9509-6(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 500 round-trip-time-us 407 sjc7-9509-6(config-profile)# ^Z sjc7-9509-6#

At this stage of the configuration the FCIP profile 100 has been defined on both the switches sjc7-9509-5 and sjc7-9509-6.
Step 6

Configure the fcip interface on both the switches. The fcip interfaces need to be configured on both the switches. The configuration is as shown below.In the fcip interface configuration the profile to used and the peer information (remote gigEs ip address are specified.)are specified. Additionally compression and write acceleration can also be configured.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# interface fcip 100 sjc7-9509-5(config-if)# peer-info ipaddr 172.22.36.98 sjc7-9509-5(config-if)# use-profile 100 sjc7-9509-5(config-if)# no shutdown sjc7-9509-5(config-if)#^Z sjc7-9509-5# sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# int fcip 100 sjc7-9509-6(config-if)# use-profile 100 sjc7-9509-6(config-if)# peer-info ipaddr 172.22.34.84 sjc7-9509-6(config-if)# no shut sjc7-9509-6(config-if)# end sjc7-9509-6#

Step 7

Check to see if the first FCIP tunnel is up and running between switches sjc7-9509-6 and sjc7-9509-5. The FCIP tunnel should be up and running now. a show interface fcip 100 brief should show the status of the FCIP link between the two switches.
sjc7-9509-5# sh int fcip 100 br ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip100 1 auto on trunking TE 100 GigabitEthernet8/1 -sjc7-9509-5# sjc7-9509-6# sh int fcip 100 br ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip100 1 auto on trunking TE 100 GigabitEthernet8/1 --

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-145

Chapter 7 Configuring multiple FCIP tunnels using a single gigE port

FCIP

sjc7-9509-6#

Configuring tunnel fcip200


Step 1

Configure the gigE interface on the MDS switches sjc7-9509-3 The gigE interface on the MDS switch sjc7-9509-3 is given an ip address, a subnetmask. This allows the gigE interface to communicate with the network.
sjc7-9509-3# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-3(config)# interface gigabitethernet 9/1 sjc7-9509-3(config-if)# ip address 172.22.36.109 255.255.254.0 sjc7-9509-3(config-if)# no shut sjc7-9509-3(config-if)# end sjc7-9509-3#

Step 2

Configure ip route so that the two gigE interlaces can communicate with each other IP route needs to be configured to allow the two gigE ports on switches sjc7-9509-3 and sjc7-9509-6 to communicate with each other. In the above recipe the gigE ports in two different subnets. This does require explicit routes to be able to communicate with each other.

Tip

The recommended best practice is just to have a host route to each of the two gigE interfaces. Therefore only allow the two gigE interfaces to communicate with each other, This is done by having a subnet mask of 255.255.255.255 when the route is added. Syntax for configuring a route is shown below. For the gigE port 8/1 on the switch sjc7-9509-5 to communicate with the port gigE 9/1 on switch sjc7-9509-3 the following is the route configuration that needs to be done on switch sjc7-9509-5.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# ip route 172.22.36.109 255.255.255.255 172.22.34.1 interface gigabitethernet 8/1 sjc7-9509-5(config)# ^Z sjc7-9509-5#

The above configuration translates as, in order to reach 172.22.36.109 use the gateway 172.22.36.1 and interface gigE 8/1 on switch sjc7-9509-5. Similarly for the gigE port 9/1 on switch sjc7-9509-3 to communicate with gigE port 8/1 on switch sjc7-9509-5 the following route configuration needs to be done on switch sjc7-9509-3
sjc7-9509-3# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-3(config)# ip route 172.22.34.83 255.255.255.255 172.22.36.1 interface gigabitethernet 9/1 sjc7-9509-3(config)# end sjc7-9509-3#

The above configuration translates as, in order to reach 172.22.34.83 use the gateway 172.22.36.1 and interface gigE 9/1 on switch sjc7-9509-3.
Step 3

Ping the gigE interfaces to ensure that the gigE ports can communicating with each other

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-146

78-xxxxx-xx

Chapter 7

FCIP Configuring multiple FCIP tunnels using a single gigE port

From the switch sjc7-9509-5 ping the ip address of the gigE interface 9/1 on switch sjc7-9509-3. Similarly ping the ip address of the gigE interface 8/1 on switch sjc7-9509-5 from switch sjc7-9509-3. This can be done from the switch prompt.
sjc7-9509-5# ping 172.22.36.109 PING 172.22.36.109 (172.22.36.109) 56(84) bytes 64 bytes from 172.22.36.109: icmp_seq=1 ttl=254 64 bytes from 172.22.36.109: icmp_seq=2 ttl=254 64 bytes from 172.22.36.109: icmp_seq=3 ttl=254 64 bytes from 172.22.36.109: icmp_seq=4 ttl=254 64 bytes from 172.22.36.109: icmp_seq=5 ttl=254 64 bytes from 172.22.36.109: icmp_seq=6 ttl=254 of data. time=0.351 time=0.399 time=0.405 time=0.408 time=0.398 time=0.410

ms ms ms ms ms ms

--- 172.22.36.109 ping statistics --6 packets transmitted, 6 received, 0% packet loss, time 4998ms rtt min/avg/max/mdev = 0.398/5.525/26.018/10.246 ms sjc7-9509-5# sjc7-9509-3# ping 172.22.34.83 PING 172.22.34.83 (172.22.34.83) 56(84) bytes of data. 64 bytes from 172.22.34.83: icmp_seq=1 ttl=254 time=0.542 64 bytes from 172.22.34.83: icmp_seq=2 ttl=254 time=0.408 64 bytes from 172.22.34.83: icmp_seq=3 ttl=254 time=0.424 64 bytes from 172.22.34.83: icmp_seq=4 ttl=254 time=0.430 64 bytes from 172.22.34.83: icmp_seq=5 ttl=254 time=0.404 64 bytes from 172.22.34.83: icmp_seq=6 ttl=254 time=0.448

ms ms ms ms ms ms

--- 172.22.34.83 ping statistics --6 packets transmitted, 6 received, 0% packet loss, time 4995ms rtt min/avg/max/mdev = 0.404/0.442/0.542/0.052 ms sjc7-9509-3#

Step 4

Measure the round trip time between the two gigE interfaces. It is important to measure the round trip time between the interfaces. This value is required for later during the configuration.
sjc7-9509-5# ips measure-rtt 172.22.36.109 interface gigabitethernet 8/1 Round trip time is 408 micro seconds (0.41 milli seconds) sjc7-9509-5# sjc7-9509-3# ips measure-rtt 172.22.34.83 interface gigabitethernet 9/1 Round trip time is 407 micro seconds (0.41 milli seconds) sjc7-9509-3#

Tip

FCIP uses port 3225 by default. Since the FCIP tunnel to sjc7-9509-6 already is up and is using TCP port 3225 this tunnel will use port 3500. Configure the FCIP profile on both the switches. FCIP profile needs to be created. The profile defines the characteristics for the FCIP tunnel that is setup. The round trip time measured is needed during the profile configuration. In this case the 407 micro seconds. The ip address used in the profile config below is the ip address assigned to the gigE interface on the switch that will be associated with the profile.

Step 5

Tip

It is recommended that the FCIP profile and FCIP interface numbers number be kept same on both the switches for the given FCIP tunnel for easier management.
sjc7-9509-5# conf t

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-147

Chapter 7 Configuring multiple FCIP tunnels using a single gigE port

FCIP

Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# fcip profile 200 sjc7-9509-5(config-profile)# port 3500 sjc7-9509-5(config-profile)# ip address 172.22.34.83 sjc7-9509-5(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 500 round-trip-time-us 407 sjc7-9509-5(config-profile)# end sjc7-9509-5# sjc7-9509-3# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-3(config)# fcip profile 200 sjc7-9509-3(config-profile)# port 3500 sjc7-9509-3(config-profile)# ip address 172.22.36.109 sjc7-9509-3(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 500 round-trip-time-us 407 sjc7-9509-3(config-profile)# end sjc7-9509-3#

At this stage of the configuration the FCIP profile 200 has been defined on both the switches sjc7-9509-5 and sjc7-9509-3 on TCP port 3500.
Step 6

Configure the fcip interface on both the switches. The fcip interfaces need to be configured on both the switches. The configuration is as shown below.In the fcip interface configuration the profile to used and the peer information (remote gigEs ip address are specified.)are specified. Additionally compression and write acceleration can also be configured.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# interface fcip 200 sjc7-9509-5(config-if)# peer-info ipaddr 172.22.36.109 port 3500 sjc7-9509-5(config-if)# use-profile 200 sjc7-9509-5(config-if)# no shut sjc7-9509-5(config-if)# end sjc7-9509-5# sjc7-9509-3# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-3(config)# interface fcip 200 sjc7-9509-3(config-if)# peer-info ipaddr 172.22.34.83 port 3500 sjc7-9509-3(config-if)# use-profile 200 sjc7-9509-3(config-if)# no shut sjc7-9509-3(config-if)# end sjc7-9509-3#

Step 7

Check to see if the first FCIP tunnel is up and running between switches sjc7-9509-6 and sjc7-9509-5. The FCIP tunnel should be up and running now. The command show interface fcip 100 brief should show the status of the FCIP link between the two switches.
sjc7-9509-5# show interface fcip 100-200 brief ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip100 1 auto on trunking TE 100 GigabitEthernet8/1 -fcip200 1 auto on trunking TE 200 GigabitEthernet8/1 -sjc7-9509-5 sjc7-9509-3# show interface fcip 200 brief

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-148

78-xxxxx-xx

Chapter 7

FCIP Configuring FCIP with IPsec using FM

------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip200 1 auto on trunking TE 200 GigabitEthernet9/1 -sjc7-9509-3#

Configuring FCIP with IPsec using FM


Fabric Manager (FM) can be used to configure FCIP. The the following recipe will demonstrate how IPsec can be configured for a FCIP link. IPSec is supported only on the 14+2 blade in a MDS9000 series switches. MDS 9216i has the 14+2 built into the supervisor and hence supports IPSec. The topology used in this recipe is shown in Figure 7-3.
Figure 7-3 FCIP and IPSec Topology

Warning

If the fcip enable command is not run, further FCIP configuration is not possible. This command will enable further FCIP configuration options in the CLI.

Step 1

Enable FCIP on the involved switches. In this example sjc7-9216-3 and sjc7-9216-6 are the switches involved. Using Fabric Manager select FCIP in the Physical Attributes on the Lower Left hand Pane. This selection is encircled using a blue box as shown in Figure 7-4. Once this is selected in the upper right hand side pane is populated with the switches seen by FM and the state of FCIP in each of the switches in the Fabric. Then working in the control TAB select the switch sjc7-9216-3 on which the FCIP feature needs to be enabled. Using the drop down box in the command column enable FCIP. This is step is marked using a red box in Figure 7-4. Repeat the above process for the switch sjc7-9216-6 and FCIP on it.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-149

Chapter 7 Configuring FCIP with IPsec using FM

FCIP

Figure 7-4

Enable FCIP through Fabric Manager

Once FCIP is enabled on the two switches it needs to be applied to configuration on the switches. Selecting the green button (highlighted in a blue) applies the changes to the switch.
Figure 7-5 FM Apply Changes to switches.

This is shown in Figure 7-5. Once the command is successfully completed the output is as shown in the red box shown in Figure 7-6.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-150

78-xxxxx-xx

Chapter 7

FCIP Configuring FCIP with IPsec using FM

Figure 7-6

Enable results

Note

IPSec is available only when 14 + 2 module is used in MDS 9216, MDS 9506, MDS 9509 or a MDS 9216i is used. Configure IP address for the gigE interfaces on switches sjc7-9216-3 and sjc7-9216-6. This is done from Physical Attributes --> Switches --> Interfaces --> Gigabit Ethernet. This is shown in Figure 7-7 and is high lighted using a red. Select gigE port 2/1 on sjc7-9216-3 and gigE 2/1 on sjc7-9216-6 and in the ipaddress/Mask column to configure the ip address and the subnet mask of the gigE ports. The gigE 2/1 on sjc7-9216-3 has an ipaddress of 172.22.34.82 and a mask 255.255.254.0 (/23) and gigE 2/1 on sjc7-9216-6 is assigned 172.22.36.108 and a mask of 255/255/254/0 (/23). Also from the pull down menu in the Admin column set the port state to up. This is highlighted in blue in Figure 7-7.

Step 2

Warning

The ip address for the ports on the ips blade should be in a different subnet than the management interface. This is critical to get FCIP working on the MDS.

Then apply the changes to the switches using the green button above the General tab. This then configures the ip address and the maks for the gigE interfaces on the switches.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-151

Chapter 7 Configuring FCIP with IPsec using FM

FCIP

Figure 7-7

Assign IP Address to the GigE ports on the switches

Step 3

Configure ip routes between the two gigE interfaces so that hey can communicate with each other.

Note

The recommended best practice is just to have a host route to each of the two gigE interfaces. I.e. only allow the two gigE interfaces to communicate with each other, This is done by having a subnet mask of 255.255.255.255 when the route is added. On Switch sjc7-9216-3
sjc7-9216-3# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-3(config)# ip route 172.22.36.108 255.255.255.255 172.22.34.1 interface gigabitethernet 2/1 sjc7-9216-3(config)#end sjc7-9216-3#

The above configuration translates as, in order to reach 172.22.36.108 use the gateway 172.22.34.1 and interface gigE 2/1 on switch sjc7-9216-3. On switch sjc7-9216-6
sjc7-9216-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-6(config)#ip route 172.22.34.82 255.255.255.255 172.22.36.1 interface gigabitethernet 2/1 sjc7-9216-6(config)# end sjc7-9216-6#

The above configuration translates as, in order to reach 172.22.34.82 use the gateway 172.22.36.1 and interface gigE 2/1 on switch sjc7-9216-6.
Step 4

Ping the gigE interfaces to ensure that the gigE ports can communicating with each other

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-152

78-xxxxx-xx

Chapter 7

FCIP Configuring FCIP with IPsec using FM

From the switch sjc7-9216-3 ping the ip address of the gigE interface 2/1 on switch sjc7-9216-6. Similarly ping the ip address of the gigE interface 2/1 on switch sjc7-9509-5 from switch sjc7-9509-6. This can be done from the switch prompt.
sjc7-9216-3# ping 172.22.36.108 PING 172.22.36.108 (172.22.36.108) 56(84) bytes of data. 64 bytes from 172.22.36.108: icmp_seq=1 ttl=63 time=0.495 64 bytes from 172.22.36.108: icmp_seq=2 ttl=63 time=0.524 64 bytes from 172.22.36.108: icmp_seq=3 ttl=63 time=0.488 64 bytes from 172.22.36.108: icmp_seq=4 ttl=63 time=0.496 64 bytes from 172.22.36.108: icmp_seq=5 ttl=63 time=0.467

ms ms ms ms ms

--- 172.22.36.108 ping statistics --5 packets transmitted, 5 received, 0% packet loss, time 4012ms rtt min/avg/max/mdev = 0.467/0.493/0.524/0.034 ms sjc7-9216-3# sjc7-9216-6# ping 172.22.34.82 PING 172.22.34.82 (172.22.34.82) 56(84) bytes of data. 64 bytes from 172.22.34.82: icmp_seq=1 ttl=254 time=0.495 64 bytes from 172.22.34.82: icmp_seq=2 ttl=254 time=0.461 64 bytes from 172.22.34.82: icmp_seq=3 ttl=254 time=0.450 64 bytes from 172.22.34.82: icmp_seq=4 ttl=254 time=0.468

ms ms ms ms

--- 172.22.34.82 ping statistics --4 packets transmitted, 4 received, 0% packet loss, time 2997ms rtt min/avg/max/mdev = 0.450/0.468/0.495/0.027 ms sjc7-9216-6#

Note

It is critical to check the connectivity between the host NIC card and the gigE port on the MDS IPS blade before proceeding further. A successful ping test should be sufficient to check the connectivity. Create the FCIP tunnel between the two GigE interfaces.
Figure 7-8 FCIP Tunnel Icon in FM

Step 5

In FM select the FCIP tunnel highlighted in blue in Figure 7-8. This brings up a FCIP config dialogue. The FCIP Tunnel Wizard start screen is shown below in Figure 7-9.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-153

Chapter 7 Configuring FCIP with IPsec using FM

FCIP

Figure 7-9

FCIP tunnel Wizard Start Screen

In the start screen select the two switches that are going to be interconnected using FCIP. In the topology used for this recipe the switches are sjc7-9216-3 and sjc7-9216-6, Select the next button after the switches are selected.
Figure 7-10 Configured and available gigE ports.

The next screen displays the available gigE ports on the two switches as well as the ip address of the gigE port if configured. In the topology gigE 2/1 on both switches are used. This is shown in Figure 7-10. To protect the FCIP tunnel using IPSec select Enforce IPSEC Security check box and then supply IKE Auth Key (a pass phrase or key). These are high lighted in blue boxes in Figure 7-10. Click on the next button to get to the next step.

Tip

For the IKE Auth Key use at least a 8 character pass phrase or key.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-154

78-xxxxx-xx

Chapter 7

FCIP Configuring FCIP with IPsec using FM

Figure 7-11 Dialogue to set tunnel properties

In the specify Tunnel properties screen the Max bandwidth and the min bandwidth of the link needs to be set. this is shown in Figure 7-11. The Max Bandwidth is the set to a value that the FCIP tunnel is allowed to use. If the tunnel is dedicated for FCIP then the max bandwidth is set to the MAX value of the bandwidth available. Min Value is adjusted based on whether the link is shared or is a dedicate for FCIP. The estimated round trip time can be measures using the measure button high lighted using a brown box in Figure 7-11. Selecting the measure button brings up a screen which measures the RTT between the two gigE interfaces. This is shown in Figure 7-12.
Figure 7-12 Measure RTT output screen.

Based on the RTT output the Value is changed on the Estimated RTT. In this topology the RTT was measured to be 423 micro seconds. This is shown using a red circle in Figure 7-11. Also the write acceleration and compression check box is selected to turn these features on. The next button will bring up the Create FCIP ISL screen.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-155

Chapter 7 Configuring FCIP with IPsec using FM

FCIP

Figure 7-13 Create FCIP ISL

In this screen verify the switches and the interfaces that are being interconnected. Also turn on trunk mode so that VSANs can flow through the link. This selection is shown using a red ellipse in. Figure 7-13. In this screen the VSANS allowed through the FCIP link can be changes in the VSAN list dialogue. Selecting finish will bring up another screen prompting to enable ipsec on the two switches. This is shown in Figure 7-14. Select Yes and the tunnel creation will complete. The FCIP link is configured and up. This can be seen either through FM or through command line.

sjc7-9216-3# sh int fcip2 br ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip2 1 auto on trunking TE 1 GigabitEthernet2/1 -sjc7-9216-3# sjc7-9216-6# sh int fcip 2 br ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip2 1 auto on trunking TE 1 GigabitEthernet2/1 -sjc7-9216-6#

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-156

78-xxxxx-xx

Chapter 7

FCIP Configuring FCIP with IPsec using FM

Figure 7-14 Enable ipsec on the two switches.

The FCIP tunnel secured by IPSec should be up and running.


Figure 7-15 FCIP status with Compression, Write Acceleration and IPSec enabled

In FM Physical Attributes --> ISLs -->FCIP and on the right hand side select Tunnels (Advanced) You should see the IP compression mode(brown box), Write Acceleration(blue box) has been turned on on both switches and IPSec set to true (red box). This is shown in Figure 7-15. The link should also show up in FM map topology.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-157

Chapter 7 Tuning FCIP

FCIP

Tuning FCIP
Configuring and bringing up a FCIP tunnel establishes a ISL between the two interconnected switches. To achieve greater efficiency and utilization from the link, some link parameters may need to be optimized. This optimization is specific to each FCIP link between the two connected switches. The optimization for a FCIP link over a slow 1.54 Mb/s connection will be done differently than a FCIP link over a 1Gb/s connection with very low latency. This section will look to provide insight into what parameters can be optimized to increase efficiency and utilization of the FCIP link.

Note

Individual results may vary due to network conditions (existing link utilization / quality) as well as storage array and host type using the FCIP tunnel.

TCP Tuning: Latency and Available Bandwidth


The latency of the link is the amount of time that it takes a packet to go from one end of the FCIP link to the other. This could be due to many factors including distance and the number of devices that it must traverse. Even the fastest routers and switches do incur some amount of latency. Latency cannot be eliminated, however protocols can be tuned and MDS features such as FCIP Write Acceleration can be enabled to eliminate the effects of it. These features are enabled in the FCIP profile. Available Bandwidth is the amount of bandwidth that the FCIP link can use on the network. You will need to define a maximum and a minimum value for the FCIP link to use in the FCIP profile.

The maximum available bandwidth value is the maximum amount of bandwidth that the FCIP link should use on the network. The minimum available bandwidth value is used as a guideline for the minimum value. Of course, if there are serious problems on the network (dropped packets, congestion), the link will go slower than the minimum value. It is recommended that the minimum value be set at least to what the minimum acceptable bandwidth that the application (EMC SRDF, IBM PPRC etc.) requires.

Tip

If the underlying link is dedicated for FCIP, the minimum and maximum available bandwidth values should be set the same. Table 7-1 contains some common WAN links and their speeds. These circuits are most often used as the underlying network for a FCIP link. For example, the underlying network may be a OC3 but you may only be able to use 100Mb of that link.

Tip

When deploying FCIP, you should always involve the LAN and WAN teams to find out about the connectivity that they are providing you. If there are performance issues, they can often help you troubleshoot them from the networks standpoint. Involve them earlier rather than later.
Table 7-1 Common WAN circuit speeds

Circuit Name T1 T3

Link Speed 1.544 Megabits/sec 44.736 Megabits/sec

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-158

78-xxxxx-xx

Chapter 7

FCIP Tuning FCIP

Circuit Name OC3 OC12 OC24 OC48 OC192 OC 768

Link Speed 155 Megabits/sec 622 Megabits/sec 1.244 Gigabits/sec 2.488 Gigabits/sec 10 Gigabits/sec 40 Gigabits/sec

FCIP Write Acceleration


To help alleviate the affects of latency, the IPS-8, IPS-4 and 14+2 blades support write acceleration. The configuration is shown below.

Note

Only in MDS version 2.0.1b and higher should write acceleration be used with Port-Channels. Enable Write Acceleration.
sjc7-9509-5# conf t Enter configuration commands, one per line. sjc7-9509-5(config)# interface fcip 100 sjc7-9509-5(config)# write-accelerator sjc7-9509-5(config)# end sjc7-9509-5# sjc7-9509-6# conf t Enter configuration commands, one per line. sjc7-9509-6(config)# interface fcip 100 sjc7-9509-6(config)# write-accelerator sjc7-9509-6(config)# end sjc7-9509-6# End with CNTL/Z.

Step 1

End with CNTL/Z.

FCIP Compression
The IPS-8 and IPS-4 blades support software based compression. The 14+2 blades supports hardware based compression. The IPS-8 and IPS-4 which do software based compression can do compression at the rate of 155 Mb/sec per port. While the 14+2 blades 2 IPS ports does hardware compression can do compression at line rate(1 Gb/s) per port. In SAN-OS version 2.0, three new modes of compression were introduced along with hardware based compression using the 14+2 module. These modules are summarized in Figure 7-16. IP compression modes.
1. 2. 3.

auto (default) mode picks the appropriate compression scheme based on the bandwidth of the link (the bandwidth of the link configured in the FCIP profiles TCP parameters) mode1 is a fast compression mode for high bandwidth links (> 25 Mbps) mode2 is a moderate compression mode for moderately low bandwidth links (between 10 and 25 Mbps)

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-159

Chapter 7 Tuning FCIP

FCIP

4.

mode3 is a high compression mode for low bandwidth links (< 10 Mbps)

Warning

The compression and write acceleration has to be enabled on both sides and also the compression modes on both sides should also match for the link to come up.

Note

The numbers displayed in Figure 7-16 are derived using the Canterbury Corpus test suite. Your actual performance number may vary depending on device type and data pattern. However, these should be used as a guide to determine which compression mode is most appropriate.
Figure 7-16 Approximate application throughput with MDS 2.0 compression

Step 1

To enable IP compression for an FCIP tunnel perform the following:


sjc7-9509-5# conf t Enter configuration commands, one per line. sjc7-9509-5(config)# interface fcip 100 sjc7-9509-5(config)# ip-compression mode1 sjc7-9509-5(config)# ^Z sjc7-9509-5# sjc7-9509-6# conf t Enter configuration commands, one per line. sjc7-9509-6(config)# interface fcip 100 sjc7-9509-6(config)# ip-compression mode1 sjc7-9509-6(config)# ^Z

End with CNTL/Z.

End with CNTL/Z.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-160

78-xxxxx-xx

Chapter 7

FCIP Tuning FCIP

sjc7-9509-6#

Enabling Tape Acceleration


The tape Acceleration feature works similar to the write Acceleration. It alleviates the utilization and performance issues associated with backing up to tape drives through FCIP tunnels over WAN links over a long distance.

Warning

FCIP Tape acceleration will not work if the fcip port is part of a port channel or even if there are multiple paths with equal costs between backup host and the tape device.

Tip

It is recomended as a best practice to have multiple paths between the backup serve and the tape device but configre the multiple paths with varying costs (i.e. FSPF cost of the fcip link). This will ensure that only one link is used at a given time for tape acceleration and has other higher cost links to fail over is the current link fails. It is enabled in CLI as shown below on both switches..
sjc7-9216-3# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-3(config)# int fcip2 sjc7-9216-3(config-if)# write-accelerator tape-accelerator sjc7-9216-3(config-if)#shut sjc7-9216-3(config-if)#no shut sjc7-9216-3# sjc7-9216-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-6(config)# int fcip2 sjc7-9216-6(config-if)# write-accelerator tape-accelerator sjc7-9216-6(config-if)#shut sjc7-9216-6(config-if)#no shut sjc7-9216-6#

The show interface fcip command shows the status of the link and the configuration of the FCIP tunnel.
sjc7-9216-3# sh int fcip 2 fcip2 is trunking Hardware is GigabitEthernet Port WWN is 20:50:00:0d:ec:02:31:c0 Peer port WWN is 20:50:00:0d:ec:01:b0:80 Admin port mode is auto, trunk mode is on Port mode is TE Port vsan is 1 Speed is 1 Gbps Trunk vsans (admin allowed and active) (1,10,304) Trunk vsans (up) (1,10) Trunk vsans (isolated) (304) Trunk vsans (initializing) () Using Profile id 1 (interface GigabitEthernet2/1) Peer Information Peer Internet address is 172.22.36.108 and port is 3225 FCIP tunnel is protected by IPSec Write acceleration mode is on Tape acceleration mode is on

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-161

Chapter 7 Tuning FCIP

FCIP

Tape Accelerator flow control buffer size is automatic IP Compression is enabled and set for mode1 Special Frame is disabled Maximum number of TCP connections is 2 Time Stamp is disabled QOS control code point is 0 QOS data code point is 0 B-port mode disabled TCP Connection Information 2 Active TCP connections Control connection: Local 172.22.34.82:60584, Remote 172.22.36.108:3225 Data connection: Local 172.22.34.82:60586, Remote 172.22.36.108:3225 2476 Attempts for active connections, 71 close of connections TCP Parameters Path MTU 2200 bytes Current retransmission timeout is 200 ms Round trip time: Smoothed 10 ms, Variance: 5 Advertized window: Current: 148 KB, Maximum: 146 KB, Scale: 6 Peer receive window: Current: 118 KB, Maximum: 118 KB, Scale: 6 Congestion window: Current: 19 KB, Slow start threshold: 211 KB Current Send Buffer Size: 146 KB, Requested Send Buffer Size: 149878 KB CWM Burst Size: 50 KB 5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 15561 frames input, 1705428 bytes 15502 Class F frames input, 1699056 bytes 59 Class 2/3 frames input, 6372 bytes 0 Reass frames 0 Error frames timestamp error 0 15895 frames output, 1652940 bytes 15836 Class F frames output, 1646568 bytes 59 Class 2/3 frames output, 6372 bytes 0 Error frames sjc7-9216-6# sh int fcip 2 fcip2 is trunking Hardware is GigabitEthernet Port WWN is 20:50:00:0d:ec:01:b0:80 Peer port WWN is 20:50:00:0d:ec:02:31:c0 Admin port mode is auto, trunk mode is on Port mode is TE Port vsan is 1 Speed is 1 Gbps Trunk vsans (admin allowed and active) (1,5,10,39,200,301) Trunk vsans (up) (1,10) Trunk vsans (isolated) (5,39,200,301) Trunk vsans (initializing) () Using Profile id 1 (interface GigabitEthernet2/1) Peer Information Peer Internet address is 172.22.34.82 and port is 3225 FCIP tunnel is protected by IPSec Write acceleration mode is on Tape acceleration mode is on Tape Accelerator flow control buffer size is automatic IP Compression is enabled and set for mode1 Special Frame is disabled Maximum number of TCP connections is 2 Time Stamp is disabled QOS control code point is 0 QOS data code point is 0 B-port mode disabled TCP Connection Information 2 Active TCP connections Control connection: Local 172.22.36.108:3225, Remote 172.22.34.82:60584

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-162

78-xxxxx-xx

Chapter 7

FCIP Tuning FCIP

Data connection: Local 172.22.36.108:3225, Remote 172.22.34.82:60586 2458 Attempts for active connections, 72 close of connections TCP Parameters Path MTU 2200 bytes Current retransmission timeout is 200 ms Round trip time: Smoothed 8 ms, Variance: 6 Advertized window: Current: 118 KB, Maximum: 26 KB, Scale: 6 Peer receive window: Current: 148 KB, Maximum: 148 KB, Scale: 6 Congestion window: Current: 14 KB, Slow start threshold: 211 KB Current Send Buffer Size: 26 KB, Requested Send Buffer Size: 26948 KB CWM Burst Size: 50 KB 5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 15336 frames input, 1606556 bytes 15277 Class F frames input, 1600184 bytes 59 Class 2/3 frames input, 6372 bytes 0 Reass frames 0 Error frames timestamp error 0 15565 frames output, 1705796 bytes 15506 Class F frames output, 1699424 bytes 59 Class 2/3 frames output, 6372 bytes 0 Error frames sjc7-9216-6#

In the above outputs from sjc7-9126-3 and sjc7-9216-6 the link properties have been highlighted. In FM select Physical Attributes --> ISLs -->FCIP and on the right hand side select Tunnels (Advanced). Then select the chek box on both the switches for Tape Acceleration(highlighted in a blue box) and then apply the config to the switches by selecting the green apply changes button(highlighted in a red box) in Figure 7-17.
Figure 7-17 Activatiing Tape Acceleration in FM

The tape flowctrl buffer size can be tuned from auto (0 == auto) using the brown box in Figure 7-17.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-163

Chapter 7 SAN Extention Tuner (SET)

FCIP

SAN Extention Tuner (SET)


SAN Extention Tuner (SET) can be used to test and tune the FCIP link for performance. SET can generate SCSI I/O commands and direct then towards virtual targets. The SET allows for variation of the I/O type (read/writes) transfer size and the number of concurrent I/Os that can be generated. The results of using SET allow the admin to determine the resulting I/Os and throughput (MB) per second as well as I/O latency which will help in fine tuning the FCIP throughput. By having these metrics, characteristics of the WAN circuit can be validated as well as what the potential throughput of the FCIP tunnel is without having to involve a host or disk subsystem. Easily creating consistent traffic flows, features such as Write Acceleration, Tape Acceleration, compression and encryption can be enabled or disabled to determine their effect on the throughput of the tunnel. These features as well as tuning the effects of modifying the FCIP tunnels round trip time, maximum and minimum bandwidth can be seen with SET. If the capabilities of a disk subsystems replication abilities (number of outstanding I/Os, size of transfers) SET can be used to model the array to see if the array is performing up to spec.

Tip

SET requires the SAN_EXTN_OVER_IP license to work. SET requires the following.

Two IPS modules. FCIP link between the switches. One unused gigabit ethernet port per switch to act as a initiator or target. Physical layer of the second gigabit ethernet port should be up. Enabling ISCSI on both the switches. Enabling SAN-EXT-TUNER.

SET works by creating virtual initiators and targets behind two gigabit ethernet ports. These virtual devices are created in a VSAN, they have pwwns and obtain FCIDs just like real end devices do. They are required to be zoned together to communicate, and they send standard FCP commands to each other which are handled by the fibre channel infrastructure as they are no different from normal FC traffic. The frames are routed via FSPF to their destination and can travel on E and TE ports through MDS and non-MDS switches. If the minimum requirements are met, SET can be used to test optical networks and not just FCIP links. While the same gigabit ethernet port that is configured for FCIP can be used as a target or initiator, it is not recommended as this may interfere with the ability to generate sufficient bandwidth. Always use an unused gigabit ethernet port to create the initiator and target on.

Note

This SET recipe assumes that the FCIP link is already up and functional. For assistance with creating a FCIP tunnel see Configuring FCIP on MDS, page 7-137 For this recipe the following resources will be used: Switch: sjc7-9216-5 Additional gigE port: gig2/2 VSAN 1000

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-164

78-xxxxx-xx

Chapter 7

FCIP SAN Extention Tuner (SET)

Virtual nwwn: 10:00:00:00:00:00:00:00 Virtual pwwn: 20:00:00:00:00:00:00:01 Switch: sjc7-9216-6 Additional gigE port: gig2/2 VSAN 1000 Virtual nwwn 11:00:00:00:00:00:00:00 Virtual pwwn: 30:00:00:00:00:00:00:01

Step 1

Enable the second gigabit ethernet port on both the switches. In this recipe gigE port 2/2 will be used on both the switches for SET.

Tip

This additional gigE port does not require an IP address to be assigned. Only the physical layer is required to be up.
sjc7-9216-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-5(config-if)#interface gigabitethernet 2/2 sjc7-9216-5(config-if)# no shut sjc7-9216-5(config-if)# end sjc7-9216-5# sjc7-9216-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-6(config-if)#interface gigabitethernet 2/2 sjc7-9216-6(config-if)# no shut sjc7-9216-6(config-if)# end sjc7-9216-6#

Check to ensure that the physical layer of the gigE port is up and running.
sjc7-9216-5# show interface gigabitethernet 2/2 brief ------------------------------------------------------------------------------Interface Status IP Address Speed MTU Port Channel ------------------------------------------------------------------------------GigabitEthernet2/2 up -1 Gbps 1500 --

sjc7-9216-6# sh int gigabitethernet 2/2 brief ------------------------------------------------------------------------------Interface Status IP Address Speed MTU Port Channel ------------------------------------------------------------------------------GigabitEthernet2/2 up -1 Gbps 1500 --

Step 2

Enable ISCSI on both the switches sjc7-9216-5 and sjc7-9216-6 if not already enabled.
sjc7-9216-5# conf t Enter configuration commands, one per line. sjc7-9216-5(config)# enable iscsi sjc7-9216-5(config)# end sjc7-9216-5# sjc7-9216-6# conf t Enter configuration commands, one per line. End with CNTL/Z.

End with CNTL/Z.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-165

Chapter 7 SAN Extention Tuner (SET)

FCIP

sjc7-9216-6(config)# iscsi enable sjc7-9216-6(config)# end sjc7-9216-6#

Step 3

Enable the iscsi on the second gigE interface on both the switches.
sjc7-9216-5# conf t Enter configuration commands, one per line. sjc7-9216-5(config)# interface iscsi 2/2 sjc7-9216-5(config-if)# no shut sjc7-9216-5(config-if)# end sjc7-9216-5# sjc7-9216-6# conf t Enter configuration commands, one per line. sjc7-9216-6(config)# interface iscsi 2/2 sjc7-9216-6(config-if)# no shut sjc7-9216-6(config-if)# end sjc7-9216-6# End with CNTL/Z.

End with CNTL/Z.

Verify that the ISCSI interface is up and running.


sjc7-9216-5# sh interface iscsi 2/2 brief ------------------------------------------------------------------------------Interface Status Oper Mode Oper Speed (Gbps) ------------------------------------------------------------------------------iscsi2/2 up ISCSI 1 sjc7-9216-5# sjc7-9216-6# sh interface iscsi 2/2 brief ------------------------------------------------------------------------------Interface Status Oper Mode Oper Speed (Gbps) ------------------------------------------------------------------------------iscsi2/2 up ISCSI 1 sjc7-9216-6#

Step 4

Enable San Extention Tuner on both switches.


sjc7-9216-5# conf t Enter configuration commands, one per line. sjc7-9216-5(config)# san-ext-tuner enable sjc7-9216-5(config)# end sjc7-9216-5# sjc7-9216-6# conf t Enter configuration commands, one per line. sjc7-9216-6(config)# san-ext-tuner enable sjc7-9216-6(config)# end sjc7-9216-6# End with CNTL/Z.

End with CNTL/Z.

Tip

Using a seperate VSAN for SET ensures that other devices in other VSANs are not impacted by the SET traffic.

Step 5

Create a seperate VSAN for SET nports.


sjc7-9216-5# conf t Enter configuration commands, one per line. sjc7-9216-5(config)#vsan database End with CNTL/Z.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-166

78-xxxxx-xx

Chapter 7

FCIP SAN Extention Tuner (SET)

sjc7-9216-5(config-vsan-db)#vsan 1000 name SETVSAN sjc7-9216-5(config-vsan-db)#end sjc7-9216-5# sjc7-9216-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-6(config)#vsan database sjc7-9216-6(config-vsan-db)#vsan 1000 name SETVSAN sjc7-9216-6(config-vsan-db)#end sjc7-9216-6#

Step 6

Configure nWWN and nport on both the switches. The nport pwwn on switch sjc7-9216-5 is configured as 20:00:00:00:00:00:00:01with a nWWN of 10:00:00:00:00:00:00:00. Similarly on switch sjc7-9216-6 the nport pwwn is configured as 30:00:00:00:00:00:00:01 and nWWN of 11:00:00:00:00:00:00:00. Both the nports are made a part of VSAN 1000 which is created just for SET.
sjc7-9216-5# san-ext-tuner sjc7-9216-5(san-ext)# nwwN 10:00:00:00:00:00:00:00 sjc7-9216-5(san-ext)# nport pWWN 20:00:00:00:00:00:00:01 vsan 1000 interface gigabitethernet 2/2 sjc7-9216-5(san-ext-nport)#end sjc7-9216-5# sjc7-9216-6# san-ext-tuner sjc7-9216-6(san-ext)# nwwN 11:00:00:00:00:00:00:00 sjc7-9216-6(san-ext)# nport pWWN 30:00:00:00:00:00:00:01 vsan 1000 interface gigabitethernet 2/2 sjc7-9216-6(san-ext-nport)#end sjc7-9216-6#

Verify that the created has logged on to the fabric.


sjc7-9216-5# sh flogi database --------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME --------------------------------------------------------------------------iscsi2/2 1000 0x410002 20:00:00:00:00:00:00:01 11:00:00:00:00:00:00:00 Total number of flogi = 1. sjc7-9216-5# sjc7-9216-6# sh flogi database v 1000 --------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME --------------------------------------------------------------------------iscsi2/2 1000 0x0c0003 30:00:00:00:00:00:00:01 11:00:00:00:00:00:00:00 Total number of flogi = 1. sjc7-9216-6#

The command show fcns database vsan 1000 should display both the pwwns in VSAN 1000.
sjc7-9216-5# show fcns database vsan 1000 VSAN 1000: -------------------------------------------------------------------------FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------0x0c0003 N 30:00:00:00:00:00:00:01 scsi-fcp 0x410002 N 20:00:00:00:00:00:00:01 scsi-fcp Total number of entries = 2 sjc7-9216-5#

sjc7-9216-6# show fcns database vsan 1000

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-167

Chapter 7 SAN Extention Tuner (SET)

FCIP

VSAN 1000: -------------------------------------------------------------------------FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------0x0c0003 N 30:00:00:00:00:00:00:01 scsi-fcp 0x410002 N 20:00:00:00:00:00:00:01 scsi-fcp Total number of entries = 2 sjc7-9216-6#

Step 7

Create a zoneset and a zone in vsan 1000 so that both SET nports can communicate with each other. It is not required that the CLI be used for zoning these devices.
sjc7-9216-5# conf t Enter configuration commands, one sjc7-9216-5(config)# zoneset name sjc7-9216-5(config-zoneset)# zone sjc7-9216-5(config-zoneset-zone)# sjc7-9216-5(config-zoneset-zone)# sjc7-9216-5(config-zoneset-zone)# sjc7-9216-5(config-zoneset)# exit per line. End with CNTL/Z. ZS_santune_vsan1000 vsan 1000 name Z_SET_VSAN100 member pwwn 20:00:00:00:00:00:00:01 member pwwn 30:00:00:00:00:00:00:01 exit

Step 8

Activate the zoneset. If enhanced zoning is enabled for this VSAN, then the zone changes will need to be committed as well.
sjc7-9216-5(config)# zoneset activate name ZS_santune_vsan1000 vsan 1000 Zoneset activation initiated. check zone status sjc7-9216-5(config)# end sjc7-9216-5#

Verify that the zoneset is active and the two nports are able to communicate with each other.
sjc7-9216-5# show zoneset active vsan 1000 zoneset name santune vsan 1000 zone name Z_SET_VSAN100 vsan 1000 * fcid 0x410002 [pwwn 20:00:00:00:00:00:00:01] * fcid 0x0c0003 [pwwn 30:00:00:00:00:00:00:01] sjc7-9216-5#

Step 9

Create tasks that will either read or write from one nport to another. Two tasks, one to read and one to write will be created as examples. In this example the nport on switch sjc7-9216-5 will act as an initiator and the nport on switch sjc7-9216-6 will act as a traget.

Note

By default, an all-zero pattern is used as the pattern for data generated by the virtual N ports. One can optionally specify a file as the data pattern to be generated by selecting a data pattern file.
sjc7-9216-5# san-ext-tuner sjc7-9216-5(san-ext)# nport pWWN 20:00:00:00:00:00:00:01 vsan 1000 interface gigabitethernet 2/2 sjc7-9216-5(san-ext-nport)# read command-id 1 target 30:00:00:00:00:00:00:01 transfer-size 1024000 outstanding-ios 5 continuous <-- read command sjc7-9216-5(san-ext-nport)# write command-id 2 target 30:00:00:00:00:00:00:01 transfer-size 1024000 outstanding-ios 5 continuous <-- write command sjc7-9216-5(san-ext-nport)#

Note

The transfer size should be a multiple of 512. The test can be continuous or can be limited to a certain number of transactions. The the throughput and performance data can be gathered on the switch as shown below.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-168

78-xxxxx-xx

Chapter 7

FCIP SAN Extention Tuner (SET)

sjc7-9216-5# show san-ext-tuner interface gigabitethernet 2/2 nport pWWN 20:00:00:00:00:00:00:01 vsan 1000 counters Statistics for nport Node name 10:00:00:00:00:00:00:00 Port name 20:00:00:00:00:00:00:01 I/Os per sec : 18 Reads : 50% Writes : 50% Egress throughput : 9.06 MBs/sec (Max - 9.12 MBs/sec) Ingress throughput : 8.62 MBs/sec (Max - 10.45 MBs/sec) Average response time : Read - 572450 us, Write - 568564 us Minimum response time : Read - 343728 us, Write - 331788 us Maximum response time : Read - 1350666 us, Write - 990794 us Errors : 0 sjc7-9216-5 sjc7-9216-6# show san-ext-tuner interface gigabitethernet 2/2 nport pWWN 30:00:00:00:00:00:00:01 vsan 1000 counters Statistics for nport Node name 11:00:00:00:00:00:00:00 Port name 30:00:00:00:00:00:00:01 I/Os per sec : 17 Reads : 58% Writes : 41% Egress throughput : 8.84 MBs/sec (Max - 10.47 MBs/sec) Ingress throughput : 9.02 MBs/sec (Max - 9.42 MBs/sec) Average response time : Read - 447611 us, Write - 424872 us Minimum response time : Read - 36986 us, Write - 124183 us Maximum response time : Read - 1165843 us, Write - 1386055 us Errors : 0 sjc7-9216-6#

Collecting the above data over the specific period of time will help calibrate the link and further tune the link for optimal throughput and performance.
Step 10

To stop the above tests


sjc7-9216-5(san-ext-nport)# stop command-id 1 sjc7-9216-5(san-ext-nport)# stop command-id 2 sjc7-9216-5(san-ext-nport)#end sjc7-9216-5#

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

7-169

Chapter 7 SAN Extention Tuner (SET)

FCIP

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

7-170

78-xxxxx-xx

C H A P T E R

iSCSI
iSCSI is a transport protocol which is used to transport SCSI packets over TCP/IP. It is an internet protocol based standard that allows hosts to connect and to access storage over a network interface card. iSCSI is used to facilitate data transfers over intranets and to manage storage over long distances. Since iSCSI runs over IP it can be used to transmit data over LAN, WAN, MAN etc. there by enabling data access independent of the location of the storage sub system. The MDS 9200 and 9500 series switch support iSCSI. This is done using the IPS-8, IPS-4 and the 14+2 blades.

How to Enable iSCSI


This command has to be run before attempting to configure iSCSI on the switch.
Warning

If the iscsi enable command is not run further iSCSI configuration is not possible. This command will enable further iSCSI configuration options in the CLI
sjc7-9509-5# conf t Enter configuration commands, one per line. sjc7-9509-5(config)# iscsi enable sjc7-9509-5(config)# end sjc7-9509-5#

End with CNTL/Z.

Configuring iSCSI on MDS in Transparent Mode


The recipe below shows the configuration of iSCSI on the MDS. Transparent mode configures an equivilant fibre channel initiator for each iSCSI initiator. In this process, no lun masking or reassignment is done on the MDS. For larger installations, iSCSI should be configured using the proxy initiator mode, see Configuring iSCSI on MDS in Proxy initiator mode, page 8-176. The topoology for this reciple is displayed in Figure 8-1 on page 8-172.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

8-171

Chapter 8 Configuring iSCSI on MDS in Transparent Mode

iSCSI

Figure 8-1

iSCSI topology.

The topology consists of a Windows 2003 server using a gigabit ethernet NIC for iSCSI. It is connected to a port on the catalyst 6509. The iSCSI interface on the host is assigned the ip address 172.22.38.45. The IPS port 8/1 on the switch sjc7-9509-5 is also connected to the Catalyst and has an ip address 172.22.36.98. The storage port from the array is connected to the FC port 1/16 on sjc7-9509-5. This example shows how to configure an iSCSI initiator using iqn (iSCSI qualified name).

Warning

The ip address for the ports on the ips blade should be in a different subnet than the management interface. This is critical to get iSCSI working on the MDS.

Configuring iSCSI initiator is as shown.


Step 1

Configure the gigE interface on the MDS switch. The gigE interface on the MDS switch is given an ip address, a subnetmask. This allows the gigE interface to communicate with the network.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# interface gigabitethernet 8/1 sjc7-9509-5(config-if)# ip address 172.22.34.83 255.255.254.0 sjc7-9509-5(config-if)# end sjc7-9509-5#

Step 2

Configure IP routes as required. In the recipe the initiator sjc7-pc-1 is in the 172.22.38.0 subnet while the gigE interface is in the 172.22.34.0 sub net. In order to allow the initiator and the gigE port to communicate ip route has to be configured on the switch.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)#ip route 172.22.38.45 255.255.255.255 172.22.34.1 interface gigabitethernet 8/1 sjc7-9509-5(config)# end sjc7-9509-5#

Note

It is critical to check the connectivity between the host NIC card and the gigE port on the MDS IPS blade before proceeding further. A successful ping test should be sufficient. Ping the gigE interface from the host. Similarly ping the host from the switch.
sjc7-9509-5#ping 172.22.38.45 PING 172.22.38.45 (172.22.38.45) 56(84) bytes of data. 64 bytes from 172.22.38.45: icmp_seq=1 ttl=127 time=1.18 ms 64 bytes from 172.22.38.45: icmp_seq=2 ttl=127 time=0.483 ms 64 bytes from 172.22.38.45: icmp_seq=3 ttl=127 time=0.479 ms --- 172.22.38.45 ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 2006ms

Step 3

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

8-172

78-xxxxx-xx

Chapter 8

iSCSI Configuring iSCSI on MDS in Transparent Mode

rtt min/avg/max/mdev = 0.479/0.716/1.186/0.332 ms sjc7-9509-5#

Step 4

Enable the iSCSI interface on the switch sjc7-9509-5. The iSCSI interface 8/1 (same port as the gigE interface) needs to be enabled. Along with the enabling of the iSCSI interface additional iSCSI related TCP tuning can also be done. In this recipe the default values are used.
sjc7-9509-5# conf t Enter configuration commands, one per line. sjc7-9509-5(config)# interface iscsi 8/1 sjc7-9509-5(config-if)# no shut sjc7-9509-5(config-if)# end sjc7-9509-5# End with CNTL/Z.

Step 5

Configure the iSCSI initiator on the switch sjc7-9509-5; This can be done in multiple ways. It can be done using the ip address of the initiator or as a proxy initiator or using an iqn (iSCSI Qualified Name), This example will use iqn name. Most iSCSI drivers / clients can automatically assign an iqn name to the on the host. The iqn name has to be at least 16 characters long. The iqn name can also be manually assigned. If it is being manually assigned care has to taken to ensure that the iqn name is unique. The win2k server used in the The Microsoft driver pre configures the iqn name at the time of install. This can be viewed as shown below. In the iSCSI utility select Initiator Settings, this shows the iqn name for the host. The driver assigned initiator node name is iqn.1991-05.com.microsoft:sjc7-pc-1.
Figure 8-2 iSCSI initiator properties

The Figure 8-2 shows the iqn name in the iSCSI driver interface. it is highlighted in a red box. For linux OS this can be found in the /etc/initiatorname.iscsi file. This is required to configure the iSCSI initiator on the MDS.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

8-173

Chapter 8 Configuring iSCSI on MDS in Transparent Mode

iSCSI

sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# iscsi initiator name iqn.1991-05.com.microsoft:sjc7-pc-1 sjc7-9509-5(config-(iscsi-init))# static pWWN system-assign 1 <-- system assigned sjc7-9509-5(config-(iscsi-init))# vsan 1 <-- Must be a member in the Targets VSAN sjc7-9509-5(config-(iscsi-init))# end sjc7-9509-5#

Tip

If an iSCSI initiator needs to have the same pwwn as was previously used by a host, as in the case if you were converting a host from fibre channel to iSCSI, the pwwn can be manually assigned. This would alleviate the need to modify the lun masking database on the array.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# iscsi initiator name iqn.1991-05.com.microsoft.sjc7-pc-1 sjc7-9509-5(config-iscsi-init)# static pWWN 21:05:00:0d:ec:02:2d:82 <-- manually assigned sjc7-9509-5(config-iscsi-init)# vsan 1 <-- Must be a member of the Target VSAN sjc7-9509-5(config-iscsi-init)# end sjc7-9509-5#

In the recipe the iqn assigned by the driver is used. If it needs to be changed make sure that it is unique and is at least 16 characters long. Optionally a pwwn can also be assigned to the initiator. The pwwn can be statically assigned by the administrator as shown above or the system can automatically assign one. The initiator can be part of multiple VSANs. In order to talk to the Target it has to be a member of the targets VSAN. In the above example the target belongs to VSAN 1.

Note

Alternatively the IP address of the iSCSI initiator can be used in all the configuration. Also assigning a static pwwn is also optional. While doing zoning an iscsi interface, its IP address can be used in place of its pwwn or the iqn name. An example of the using ip address is shown below.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# iscsi initiator ipaddress 172.22.38.45 sjc7-9509-5(config-(iscsi-init))# static pWWN system-assign 1 <-- system assigned sjc7-9509-5(config-(iscsi-init))# vsan 3003 <-- Must be a member in the Targets VSAN sjc7-9509-5(config-(iscsi-init))# end sjc7-9509-5#

Note

For the iSCSI initiator to communicate with a target port, the iSCSI initiator has to made a member of the target ports VSAN. iSCSI initiators can be members of multiple VSANs. Configure the virtual target on the switch sjc7-9509-5.
sjc7-9509-5# config t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# iscsi virtual-target name iscsi-sjc7-jbod-1 sjc7-9509-5(config-iscsi-tgt)# pWWN 22:00:00:20:37:5a:40:26 sjc7-9509-5(config-iscsi-tgt)# end sjc7-9509-5#

Step 6

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

8-174

78-xxxxx-xx

Chapter 8

iSCSI Configuring iSCSI on MDS in Transparent Mode

The virtual target is a name assigned to the storage device. This name has to be 16 characters long. Then the pwwn of the storage port is assigned to the virtual target as shown above. This completes the configuration of a virtual target.
Step 7

Permit the initiator to communicate with the virtual-target already configured.


sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# iscsi virtual-target name iscsi-sjc7-jbod-1 sjc7-9509-5(config-iscsi-tgt)# initiator iqn.1991-05.com.microsoft.sjc7-pc-1 permit sjc7-9509-5(config-iscsi-tgt)# end sjc7-9509-5#

The virtual target can be configured to allow all initiators to communicate with the virtual-target. In the above command the virtual-target is only configured to communicate with the initiator iqn.1991-05.com.microsoft.sjc7-pc-1. This permits the initiator iqn.1991-05.com.microsoft:sjc7-pc-1 to communicate with the virtual-target iscsi-sjc7-jbod-1.
Step 8

Next, create a zone with the initiator and the virtual target configured. Add the zone to the zoneset and activate the zoneset. Create a zone with the iSCSI initiator and the virtual targets as members.This enables the initiator and the target to communicate with each other. The zone can be created either with the iqn name or with the ip address or with the pwwn that was assigned to the initiator. In this recipe it is created using the pwwn. Zoning with pwwn of the virtual-target and initiator.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# zone name Z_iscsi_tst vsan 3003 sjc7-9509-5(config-zone)# mem pwwn 22:00:00:20:37:39:9c:1f sjc7-9509-5(config-zone)# mem pwwn 21:05:00:0d:ec:02:2d:82 sjc7-9509-5(config-zone)#end sjc7-9509-5# sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# zoneset name ZS_ISCSI vsan 3003 sjc7-9509-5(config-zoneset)#member Z_iscsi_tst sjc7-9509-5(config-zoneset)end sjc7-9509-5#

Alternatively: Zoning with the pwwn of the virtual target and the iqn of the iSCSI initiator.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# zone name Z_iscsi_tst vsan 3003 sjc7-9509-5(config-zone)# member pwwn 22:00:00:20:37:39:9c:1f sjc7-9509-5(config-zone)# member symbolic-nodename iqn.1991-05.com.microsoft:sjc7-pc-1 sjc7-9509-5(config-zone)# end sjc7-9509-5#

Activate the zoneset to allow the zone members to communicate.


sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# zoneset activate name ZS_ISCSI vsan 3003 Zoneset activation initiated. check zone status sjc7-9509-5#

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

8-175

Chapter 8 Configuring iSCSI on MDS in Proxy initiator mode

iSCSI

Configuring iSCSI on MDS in Proxy initiator mode


The recipie below details the proxy mode configuration for iSCSI on MDS switch. This method is the prefred mode for configuring a large number of iSCSI clients to work with the switch. In proxy initiator mode the one fibre channel initiator is used for all iSCSI clients that access the MDS via the same iSCSI interface (iscsi3/3 for example). The initiators will use the PWWN assigned to the iSCSI interface. The iSCSI interface to which an iSCSI client will logon to is configured in the client and must be permited by the virtual target configured for that initiator. Proxy mode is advantageous over transparent mode when the configuration requires multiple iSCSI initiators to access the same fibre channel target. For example if 20 iSCSI initiators need to comunicate with a fibre channel disk, in transparent mode 20 iSCSI initators, 20 zones need to be created and also array based lun masking has to be updated for all 20 initiator instances. On the other hand, proxy initiator mode is far easier to manage as it allows for centralized management of the iSCSI configuration, as all iSCSI clients accessing the same switch interface will use a single fibre channel initiator. First, a pwwn is assigned to iSCSI interface. Then this one pwwn, is zoned with the fibre channel target so that the proxy initiator can see the luns presented by the virtual target. All the lun masking and zoning need to be performed only with the proxy initiator. As new hosts (iSCSI clients) are added they only need to be exposed to the luns that they need to see. As no new zones nor modifications to the arrays lun masking needs to be done. A typical practice is to create a virtual target for each hosts and configure the virtual target in such a way that they only expose the required luns to the iSCSI initiator. The proxy initiator is not restricted to a single VSAN, as iSCSI clients are configured and given access to different VSANs, the proxy initiator is created in the new VSAN. Therefore the maximum number of initiators that would need to be zoned would be the number of proxy initiators that have iSCSI clients in a VSAN. This is far less than under transparent mode, whereby a fibre channel initiator is created for every iSCSI client. The topology used for the iSCSI proxy initiator recipe is shown in Figure 8-3. It has 2 windows hosts on the same subnet. Both hosts iSCSI interfaces are on the 172.22.38.0 network.
Figure 8-3 iSCSI Proxy Topology

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

8-176

78-xxxxx-xx

Chapter 8

iSCSI Configuring iSCSI on MDS in Proxy initiator mode

Step 1

Configure the gigE interface on the MDS switch The gigE interface on the MDS switch is given an ip address, a subnetmask. This allows the gigE interface to communicate with the network.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# interface gigabitethernet 8/1 sjc7-9509-6(config-if)# ip address 172.22.36.98 255.255.254.0 sjc7-9509-6(config-if)# end sjc7-9509-6#

Step 2

Configure ip routes if required In the recipe the initiator sjc7-pc-1 and sjc7-pc-2 are in the 172.22.38.0 subnet while the gigE interface is in the 172.22.36.0 sub net. In order to allow the initiator and the gigE port to communicate ip route has to be configured on the switch. A host based route will be configured to allow the gigE port to communicate with sjc7-pc-1 and sjc7-pc-2 hosts.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# ip route 172.22.38.45 255.255.255.255 172.22.36.1 interface gigabitethernet 8/1 sjc7-9509-6(config)# ip route 172.22.38.46 255.255.255.255 172.22.36.1 interface gigabitethernet 8/1 sjc7-9509-6(config)# end sjc7-9509-6#

Step 3

Ping the gigE interface from the hosts. Similarly ping the host from the switch.
sjc7-9509-6#ping 172.22.38.45 PING 172.22.38.45 (172.22.38.45) 56(84) bytes of data. 64 bytes from 172.22.38.45: icmp_seq=1 ttl=127 time=1.18 ms 64 bytes from 172.22.38.45: icmp_seq=2 ttl=127 time=0.483 ms 64 bytes from 172.22.38.45: icmp_seq=3 ttl=127 time=0.479 ms --- 172.22.38.45 ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 2006ms rtt min/avg/max/mdev = 0.479/0.716/1.186/0.332 ms sjc7-9509-6# sjc7-9509-6#ping 172.22.38.46 PING 172.22.38.45 (172.22.38.45) 56(84) bytes of data. 64 bytes from 172.22.38.46: icmp_seq=1 ttl=127 time=1.18 ms 64 bytes from 172.22.38.46: icmp_seq=2 ttl=127 time=0.483 ms 64 bytes from 172.22.38.46: icmp_seq=3 ttl=127 time=0.479 ms --- 172.22.38.46 ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 2006ms rtt min/avg/max/mdev = 0.479/0.716/1.186/0.332 ms sjc7-9509-6#

Step 4

Enable the iSCSI interface on the switch sjc7-9509-6. The iSCSI interface 8/1 (same port as the gigE interface) needs to be enabled. Along with the enabling of the iSCSI interface additional iSCSI related TCP tunning can also be done. In this recipe the default values are used. The switch port command is used to enable proxy-initiator mode for the iscsi interface 8/1. It is also used to assign pwwn to the interface as shown in the recipe below.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# interface iscsi 8/1 sjc7-9509-6(config-if)# switchport proxy-initiator nWWN 21:05:00:0d:ec:02:2d:82 pwwn 21:05:00:0d:ec:02:2d:82 sjc7-9509-6(config-if)# no shut sjc7-9509-6(config-if)# end

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

8-177

Chapter 8 Configuring iSCSI on MDS in Proxy initiator mode

iSCSI

sjc7-9509-6#

Step 5

Add the iSCSI interface to the required VSANs. This needs to be done to allow the iscsi interface to communicate with the virtual target to see the luns in different VSANs. The commands below add the iSCSI interface 8/1 into the 3003 VSAN. This is VSAN to which the FC target is connected. Once this is done the interface iSCSI 8/1 will log on to the fabric.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# iscsi interface vsan-membership sjc7-9509-6(config)# vsan database sjc7-9509-6(config-vsan-db)# vsan 3003 interface iscsi 8/1 sjc7-9509-6(config-vsan-db)#end sjc7-9509-6#

Note

The command iscsi interface vsan-membership is required to make the iscsi interface as a part of the a VSAN. Configure the virtual target on the switch sjc7-9509-6.
sjc7-9509-6# config t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# iscsi virtual-target name sjc7-jbod-iscsi-1 sjc7-9509-6(config-iscsi-tgt)# pWWN 22:00:00:20:37:39:9c:1f sjc7-9509-6(config-iscsi-tgt)# end sjc7-9509-6#

Step 6

The virtual target is a name assigned to the storage device. This name has to be 16 characters long. Then the pwwn of the storage port is assigned to the virtual target as shown above. This completes the configuration of a virtual target.
Step 7

Create a zone with the initiator and the virtual target configured. Add the zone to the zoneset and activate the zoneset. Create a zone with the iSCSI initiator and the virtual targets as members.This enables the initiator and the target to communicate with each other. The zone can be created either with the iqn name or with the ip address or with the pwwn that was assigned to the initiator. In this recipe it is created using the pwwn. Zoning with pwwn of the virtual-target and initiator.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# zone name Z_iscsi_tst vsan 3003 sjc7-9509-6(config-zone)# mem pwwn 22:00:00:20:37:39:9c:1f sjc7-9509-6(config-zone)# mem pwwn 21:05:00:0d:ec:02:2d:82 sjc7-9509-6(config-zone)#end sjc7-9509-6# sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# zoneset name ZS_ISCSI vsan 3003 sjc7-9509-6(config-zoneset)#member Z_iscsi_tst sjc7-9509-6(config-zoneset)end sjc7-9509-6#

Activate the zoneset to allow the zone members to communicate.


sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

8-178

78-xxxxx-xx

Chapter 8

iSCSI Configuring iSCSI on MDS in Proxy initiator mode

sjc7-9509-6(config)# zoneset activate name ZS_ISCSI vsan 3003 Zoneset activation initiated. check zone status sjc7-9509-6# sjc7-9509-6# show zoneset active vsan 3003 zoneset name ZS_iscsi vsan 3003 zone name Z_iscsi_proxy_8-1 vsan 3003 * fcid 0xd90002 [pwwn 21:05:00:0d:ec:02:2d:82] * fcid 0xd90000 [pwwn 22:00:00:20:37:39:9c:1f] sjc7-9509-6#

Tip

As a best practice, to achieve lun security a virtual target with access to specific luns needs to be created for each initiator. Configure a virtual target for each initiator and configure lun masking for the initiator Once the zone is successfully activated the luns made available on the storage port should now be visible to the iscsi interface. As this interface could be a proxy iscsi interface for many iscsi initiators, some form of lun security needs to enabled. To achieve lun security a virtual target with access to specific luns needs to be created for each initiator. In this recipe the iscsi interface can see 10 luns (lun 11 to lun 20 in decimal). The configuration below will allow the host sjc7-pc-1 to see luns 11 - 14 (decimal) on the array.
sjc7-9509-6# config t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# iscsi virtual-target name sjc7-jbod-iscsi-1 sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun b iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun b iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun d iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun e iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# initiator ip address 172.22.38.45 permit sjc7-9509-6(config-(iscsi-tgt))# end sjc7-9509-6#

Step 8

1 2 3 4

Similarly for the hosts sjc7-pc-2, the following needs to be done to make it see luns 16 - 20 (decimal).
sjc7-9509-6# config t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# iscsi virtual-target name sjc7-jbod-iscsi-1 sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun 10 iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun 11 iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun 12 iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun 13 iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun 14 iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# initiator ip address 172.22.38.46 permit sjc7-9509-6(config-(iscsi-tgt))# end sjc7-9509-6#

1 2 3 4 5

After the above configuration changes both the hosts should be able to see the luns allocated to them through the virtual -target created for each. As can be seen there is no need to create additional zones when new iSCSI client are added. If the iSCSI clients need access additional targets the iSCSI interface needs to be zoned to the other targets as shown in Step 7 in Configuring iSCSI on MDS in Proxy initiator mode.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

8-179

Chapter 8 Configuring iSCSI Client (initiators) on hosts

iSCSI

Configuring iSCSI Client (initiators) on hosts


iSCSI on Windows
This section details the configuration of a Microsoft Windows iSCSI driver 2.0 configuration. The steps to configure the iSCSI client are as shown below. The current section assumes that the iscsi driver has already been installed on the windows server.
Figure 8-4 Add Target Portal in iSCSI driver

Step 1

In the iSCSI initiators properties select the Discovery tab. This is shown using a red circle in Figure 8-4. In this tab select the Add button. This brings up the Add Target Portal dialogue. The ip address of the gigE interface on the switch sjc7-9509-5 which is 172.22.34.83 and select OK. This will add the target in the iSCSI client. At this stage the show fcns database command should show the iSCSI initiator logged on to the fabric in the specified vsan. In this case it is vsan 3003. This is shown in the output below. If the initiator does not show up in the output check the and see if the initiator is properly configured. The output in brackets (ex: [dev1-1]) are examples of how Device Aliases are used, see Device Aliases, page 1-43 for more information.
sjc7-9509-5# show fcns database vsan 3003 VSAN 3003: -------------------------------------------------------------------------FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------0x900002 N 21:05:00:0d:ec:02:2d:82 (Cisco) scsi-fcp:init isc.w 0xd900cb NL 22:00:00:20:37:39:9c:1f (Seagate) scsi-fcp:target

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

8-180

78-xxxxx-xx

Chapter 8

iSCSI Configuring iSCSI Client (initiators) on hosts

[dev1-1] 22:00:00:20:37:38:84:9e [dev3-1] 0xd900ce NL 22:00:00:20:37:18:16:c9 [dev5-1] 0xd900d1 NL 22:00:00:20:37:38:7c:a4 [dev6-2] 0xd900d2 NL 22:00:00:20:37:36:00:92 [dev4-1] 0xd900d4 NL 22:00:00:20:37:5a:40:26 [dev7-2] 0xd900d5 NL 22:00:00:20:37:38:88:9c [dev2-1] Total number of entries = 8 sjc7-9509-5# 0xd900cd NL

(Seagate) (Seagate) (Seagate) (Seagate) (Seagate) (Seagate)

scsi-fcp:target scsi-fcp:target scsi-fcp:target scsi-fcp:target scsi-fcp:target scsi-fcp:target

Step 2

Select the target tab in the iSCSI initiator properties. This is shown using a blue ellipse in Figure 8-5.
Figure 8-5 Target Tab in the iSCSI properties

If the virtual-target is properly configured and the initiator is permitted to access the virtual-target, the configured virtual target should be visible under targets and it status should be in active as seen in Figure 8-5 high lighted in a brown square.
Step 3

Select the Log On button. This brings up the Log On to Target window. This is shown in Figure 8-6.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

8-181

Chapter 8 Configuring iSCSI Client (initiators) on hosts

iSCSI

Figure 8-6

Log ON to Target Options

Here there is an option of making the system automatically log on to the target automatically and also to enable multipath if multipathing software is installed on the server. Clicking on OK starts the iSCSI login and storage discovery process for the target listed in the window.An iSCSI initiator can see multiple targets.
Figure 8-7 Target Logged on status

On a successful log on to the target the status of the target changes for inactive to connected as seen in Figure 8-7 highlighted in a brown rectangle. At this stage the host should be able to see the storage made available (directly assigned or made available through LUN masking or otherwise) to the iSCSI initiator. Selecting the Details button in the Targets tab brings up the Target properties window. In this window selecting the Devices Tab lists the disk available to the host on the storage array. This is shown in Figure 8-8.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

8-182

78-xxxxx-xx

Chapter 8

iSCSI Configuring iSCSI Client (initiators) on hosts

Figure 8-8

Disk available to the initiator on the target.

Now from the MS windows Start -->Settings -->Control Panel --> Administrative Tools --> Computer management -->Disk Management the user can scan for new disk and initialize them for use on the system.

iSCSI on Linux
This section talks about the configuration that needs to be performed on a linux client to get iSCSI client going on a Linux server. To enable iSCSI to work on a linux host the iSCSI driver has to be downloaded and configured. The linkdriver is currently maintained on sourceforge and can be downloaded from http://sourceforge.net/project/showfiles.php?group_id=26396&release_id=177564. In some instances the driver may need to be compiled. Once the driver is installed on the host it can be configured by editing the iscsi.conf file. This is usually present in the /etc directory on the linux server. Again as in case of the windows configuration the target portal has to be added and the iSCSI initiator needs to be configured on the MDS. Then the iSCSI initiator on the linux host has to be restarted to log on to the target to access the LUNs. As with the case of the Windows iSCSI driver, the iSCSI initiators name is auto generated by the driver subsystem. The auto generated name can be found in the /etc/initiatorname.iscsi file. The process of configuring the iSCSI initiator and virtual target on the MDS switch and creating zones and activating the zoneset is as shown in the Configuring iSCSI on MDS in Transparent Mode section. The iSCSI initiator name is auto generated by the iSCSI driver in linux and is stored /etc/initiatorname.iscsi. file.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x 78-xxxxx-xx

8-183

Chapter 8 Configuring iSCSI Client (initiators) on hosts

iSCSI

Step 1

The /etc/iscsi.conf file has to be edited to make iSCSI configuration changes on a linux server. Under the DiscoverAddress Settings section add the address of the target portal, in this case it would be 172.22.34.83.
# DiscoveryAddress Settings # ------------------------# Add "DiscoveryAddress=xxx" entries for each iSCSI router instance. # The driver will attempt to discover iSCSI targets at that address # and make as many targets as possible available for use. # 'xxx' can be an IP address or a hostname. A TCP port number can be # specified by appending a colon and the port number to the address. # All entries have to start in column one and must not contain any # whitespace. # # Example: # DiscoveryAddress=172.22.34.83

This is all that is needed to changed to get basic iscsi configured on the linux server.
Step 2

Restaring the iSCSI process. Once the target portal address has been updated the iscsi processes have to be restarted to force the iscsi client to log on to the target and to discover the luns. This is done using the rc scripts. The rc scripts are located in /etc/rc3.d directory. It is S25iscsi script. This script needs to be run with a restart option after the changes to the /etc/iscsi.conf are made. The script also have a status option to check the status of the iscsi processes on the system
[root@sjc7-pc-6 Stopping iSCSI: Starting iSCSI: [root@sjc7-pc-6 iSCSI driver is [root@sjc7-pc-6 rc3.d]# /etc/rc3.d/S25iscsi restart sync umount sync iscsid iscsi iscsi iscsid rc3.d]# /etc/rc3.d/S25iscsi status loaded rc3.d]#

At this stage the iscsi initiator should log on to the target and discover the luns that have been assigned to it.

Cisco MDS 9000 Family Cookbook for SAN-OS 2.x

8-184

78-xxxxx-xx