VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus

10GE Networking
Brad Hedlund, CCIE #5530 Data Center Architect Cisco Systems bhedlund@cisco.com
BRAD HEDLUND .com

v 2.5

- About the Author Brad Hedlund

Technical Solutions Architect, Data Center Cisco Systems, Inc. CCIE #5530
http://bradhedlund.com/about/

Blog: http://bradhedlund.com Twitter: http://twitter.com/bradhedlund E-mail: bhedlund@cisco.com

Comments welcome.

BRAD HEDLUND .com

Cisco UCS Networking Overview

SAN A ETH 1 ETH 2 SAN B

MGMT

MGMT

6100 Fabric A

6100 Fabric B

F E X A

Chassis 1

F E X B

F E X A

Chassis 2

F E X B

CNA B200

CNA CNA B250

BRAD HEDLUND .com

Emulex/Qlogic CNA
FEX FEX

Fabric A

Fabric B

10GE

10GE

CNA

vHBA

vNIC

vNIC

vHBA

vhba0

vmnic0

vmnic1

vhba1

vSwitch / Nexus 1000V

BRAD HEDLUND .com

Emulex/Qlogic CNA Logical View

Fabric Interconnect A
vfc 1

Fabric Interconnect B
vEth 1
vEth 2 vfc 2

FEX

FEX

CNA

vhba0

vmnic0

vmnic1

vhba1

vSwitch / Nexus 1000V

BRAD HEDLUND .com

Service Profile / Fabric Selection

BRAD HEDLUND .com

Cisco VIC Palo

FEX FEX

Fabric A

Fabric B

10GE

10GE

Cisco VIC

vHBA

vNIC

vNIC

vNIC

vNIC

vNIC

vNIC

vNIC

vNIC

vHBA

58

vhba 0

vmnic 0

vmnic 1

vmnic 2

vmnic 3

vmnic 4

vmnic 5

vmnic 6

vmnic 7

vhba 1

vSwitch / vDS / Nexus 1000V

BRAD HEDLUND .com

Cisco VIC QoS: Example

FEX FEX

Fabric A

Fabric B

Cisco VIC
7
VIC Mgmt
Strict Priority

10GE
TX

10GE
TX

Strict Priority

7
VIC Mgmt

Bandwidth Scheduler
10% 40% 10% 40%

Bandwidth Scheduler
10% 40% 10% 40%

1 RR

3
No Drop

0 RR
Limit TX 1GE

3
No Drop

(8) COS Queues

Limit TX 1GE

vNIC 1

vNIC 2

vHBA 1

vNIC 3

vNIC 4

vNIC 5

vNIC 6

vHBA 2

vNIC 7

vNIC 8

58

COS 1

COS 3

COS 4

COS 5

COS 1

COS 3

COS 4

COS 5

BRAD HEDLUND .com

Cisco VIC Logical View

Fabric Interconnect A
vfc 1 vEth 1 vEth 2 vEth 3 vEth 4

Fabric Interconnect B
vEth 5 vEth 6 vEth 7 vEth 8 vfc 2

FEX A Cisco VIC

FEX B

vhba 0

vmnic 0

vmnic 1

vmnic 2

vmnic 3

vmnic 4

vmnic 5

vmnic 6

vmnic 7

vhba 1

vSwitch / Nexus 1000V

BRAD HEDLUND .com

I/O Consolidation with 10GE

VM Data
vmnic 0

VMotion MGMT

vmnic 1

CONCERN: Bandwidth Availability 10GE

vmnic 2

SOLUTIONS: 1. MAX bw per NIC Rate Limits 2. MIN bw per NIC QoS 3. Do Nothing Let it rip

Network Switch

IP Storage

vmnic 3

BRAD HEDLUND .com

I/O Consolidation with 10GE

VM Data
VMotion MGMT IP Storage
vmnic 1

CONCERN: Bandwidth Availability

SOLUTIONS: 1. MAX bw per Class Rate Limits 2. MIN bw per Class QoS 10GE 3. Do Nothing Let it rip

Network Switch

BRAD HEDLUND .com

MAX Bandwidth Rate Limits

10GE VM Data
vmnic 0

2.5G
2.5G Network Switch 2.5G 2.5G Sum of Limits must not exceed physical link speed Inefficient use of unused BW

VMotion MGMT

vmnic 1

vmnic 2

IP Storage

vmnic 3

This slide animated @ http://bradhedlund.com/2010/08/16/cisco-ucs-qos-vs-hp-virtual-connect-rate-limiting/

BRAD HEDLUND .com

MIN Bandwidth QoS

10GE VM Data
vmnic 0

VMotion MGMT

vmnic 1

vmnic 2

Network Switch

IP Storage

vmnic 3

Minimum BW guarantee with no Maximum Efficient: unused BW available to other NICs

This slide animated @ http://bradhedlund.com/2010/08/16/cisco-ucs-qos-vs-hp-virtual-connect-rate-limiting/
BRAD HEDLUND .com

MIN Bandwidth QoS

Nexus 1000V Classifies Traffic VM Data Cisco VIC Enforces Policy 10GE UCS 6100 Nexus 5000 Enforces Policy

VMotion
vmnic 1

Network Switch

MGMT

IP Storage

BRAD HEDLUND .com

Traffic Considerations
Required Guest VM Data Management
VMKernel Service Console

vMotion Central Storage Access

Optional Fault Tolerance (FT) NFS iSCSI FCoE Realtime

Video/Voice, Market Data

BRAD HEDLUND .com

Cisco UCS + VIC Design (1): Required Traffic

Fabric Interconnect A
vfc 1 vEth 1 vEth 2 vEth 3

Fabric Interconnect B
vEth 6 vEth 7 vEth 8 vfc 2

vhba 0

vmnic 0

vmnic 2

vmnic 4

vmnic 1

vmnic 3

vmnic 5

vhba 1

vSwitch or vDS

MGMT
Primary
Backup
Min BW

VM DATA

vMotion

FC

vmnic1
vmnic0 10% 1GE Max
Bronze COS 1

vmnic2
vmnic3 40%
Best Effort COS 0 BRAD HEDLUND .com

vmnic5
vmnic4 10%
Silver COS 2

MP
MP 40%
COS 3

Class

UCS Manager QoS System Class

Design #1

Assign each Class of Service a minimum bandwidth weighting System wide policy

BRAD HEDLUND .com

Apply QoS Policy to vNIC

Create QoS Policy Assign Policy to vNIC

BRAD HEDLUND .com

Cisco UCS + VIC Design (2): Req + 1 Opt

Fabric Interconnect A
vfc 1 vEth 1 vEth 2 vEth 3 vEth 4

Fabric Interconnect B
vEth 5 vEth 6 vEth 7 vEth 8 vfc 2

vhba 0

vmnic 0

vmnic 2

vmnic 4

vmnic 6

vmnic 1

vmnic 3

vmnic 5

vmnic 7

vhba 1

vSwitch or vDS

MGMT
Primary
Backup
Min BW

VM DATA

vMotion

NFS

FC

vmnic1
vmnic0
Bronze COS 1

vmnic2
vmnic3
Best Effort COS 0 BRAD HEDLUND .com

vmnic5
vmnic4 10%
Silver COS 2

vmnic7
vmnic6 10%
Gold COS 4

MP
MP 40%
COS 3

10% 1GE Max 30%

Class

UCS Manager QoS System Class

Design #2

Assign each Class of Service a minimum bandwidth weighting System wide policy
BRAD HEDLUND .com

Cisco UCS + VIC Design (3): Req + 2 Opt + Realtime

Fabric Interconnect A
vfc 1 vEth 1 vEth 2 vEth 3 vEth 4 vEth 5 vEth 6

Fabric Interconnect B
vEth 7 vEth 8 vEth 9 vEth 10 vfc 2

vhba 0

vmnic 0

vmnic 2

vmnic 4

vmnic 6

vmnic 8

vmnic 1

vmnic 3

vmnic 5

vmnic 7

vmnic 9

vhba 1

vSwitch or vDS

MGMT
Primary
Backup
Min BW

vMotion vmnic1
vmnic0 10%
Bronze COS 1

Realtime vmnic2
vmnic3 10%

FT vmnic5
vmnic4 20%

VM DATA vmnic6
vmnic7 10%
Best Effort COS 0

NFS vmnic9
vmnic8 10%
Silver COS 2

FC MP
MP 40%
COS 3

vmnic0
vmnic1 1GE Max
Bronze COS 1

Class

Plat COS 5 Gold COS 4 BRAD HEDLUND .com

UCS Manager QoS System Class

Design #3

Platinum Class for Realtime traffic Lossless Ethernet and Multicast optimized Silver and Bronze for vMotion & NFS with Jumbo Frames
BRAD HEDLUND .com

Design #4: (2) 10GE NICs with Nexus 1000V Cisco UCS or Nexus 5000
Fabric Interconnect A / Nexus 5000 #1
vfc 1 vEth 4

Fabric Interconnect B / Nexus 5000 #2

vEth 7 QoS Enforcement vfc 2

Cisco VIC or Other adapter Cisco UCS server or non-Cisco

vhba 0

vmnic 0

vmnic 1

QoS Enforcement (if Cisco VIC) QoS Enforcement (if N1K v1.4) QoS Classification vhba 1

NEXUS 1000V

MGMT
Primary
Backup
Min BW

vMotion

Realtime

FT

VM DATA

NFS

FC

vmnic0
vmnic1 1GE Max
Bronze COS 1

vmnic1
vmnic0 10%
Bronze COS 1

vmnic0
vmnic1 10%

vmnic1
vmnic0 20%

vmnic0
vmnic1 10%
Best Effort COS 0

vmnic1
vmnic0 10%
Silver COS 2

MP
MP 40%
COS 3

Class

Plat COS 5 Gold COS 4 BRAD HEDLUND .com

vNIC QoS Policy for Design #4

Preserve and use COS markings from Nexus 1000V Same QoS System Class settings as Design #3

BRAD HEDLUND .com

Design #5: (2) 10GE NICs, no Cisco VIC, no N1K Nexus 5000
Nexus 5000 #1
vfc 1 Eth 1/1

Nexus 5000 #2
Eth 1/1 QoS Enforcement QoS Classification vfc 2

Cisco VIC or Other adapter Cisco C-Series or non-Cisco

vhba 0

vmnic 0

vmnic 1

vSwitch or vDS

VMware NetIOC ? FCoE?

vhba 1

MGMT
Primary
Backup
Min BW

vMotion

Realtime

FT

VM DATA

NFS

FC

vmnic0
vmnic1 1GE Max
Bronze COS 1

vmnic1
vmnic0 10%
Bronze COS 1

vmnic0
vmnic1 10%

vmnic1
vmnic0 20%

vmnic0
vmnic1 10%
Best Effort COS 0

vmnic1
vmnic0 10%
Silver COS 2

MP
MP 40%
COS 3

Class

Plat COS 5 Gold COS 4 BRAD HEDLUND .com

Narrative posted at:

http://bradhedlund.com/?p=1650

BRAD HEDLUND .com