Technical Note

Round-Robin Load Balancing

VMware ESX Server 3.5, VMware ESX Server 3i version 3.5, VMware VirtualCenter 2.5
VMwareESXServer3.5andESXServer3iversion3.5enhanceESXServernativemultipathingbyproviding experimentalsupportforroundrobinloadbalancing.Thistechnicalnoteexplainshowroundrobinload balancingworksandhowtosetit. NOTERoundrobinloadbalancingisexperimentalandnotsupportedforproductionuse. RoundrobinloadbalancingisavailablefromtheVIClientbutismarkedasexperimental.

Understanding Round-Robin Load Balancing

ESXServerhostscanusemultipathingforfailover.WhenonepathfromtheESXServerhosttotheSAN becomesunavailable,thehostswitchestoanotherpath.ESXServerhostscanalsousemultipathingforload balancing.Toachievebetterloadbalancingacrosspaths,administratorscanspecifythattheESXServerhost shouldswitchpathsundercertaincircumstances.DifferentsettableoptionsdeterminewhentheESXServer hostswitchespathsandwhatpathsarechosen. WhentoswitchSpecifythattheESXServerhostshouldattemptapathswitchafteraspecifiednumber ofI/Oblockshavebeenissuedonapathorafteraspecifiednumberofreadorwritecommandshavebeen issuedonapath.Ifanotherpathexiststhatmeetsthespecifiedpathpolicyforthetarget,theactivepath tothetargetisswitchedtothenewpath.The--custom-max-commandsand--custom-max-blocks optionsspecifywhentoswitch. WhichtargettouseSpecifythatthenextpathshouldbeonthepreferredtarget,themostrecentlyused target,oranytarget.The--custom-target-policyoptionspecifieswhichtargettouse. WhichHBAtouseSpecifythatthenextpathshouldbeonthepreferredHBA,themostrecentlyused HBA,theHBAwiththeminimumoutstandingI/Orequests,oranyHBA.The--custom-HBA-policy optionspecifieswhichHBAtouse.

Setting the Path Switching Policy

Youcansetthepathswitchingpolicyforfailoverandforloadbalancingbyusingtheesxcfg-mpath command. NOTEYoucansetpathswitchingforfailoverintheVIClient.SeetheSANConfigurationGuide.Youcanset loadbalancingonlyfromthecommandline. YoucansetthepathswitchingpolicyonaperLUNbasisbyusingtheesxcfg-mpathcommands --policy customoption.Ifyouspecify--policy custom,youmustalsospecifyoneofthecustompolicyoptions. BecausethepathswitchingpolicyissetonaperLUNbasis,youmustalwaysspecifytheLUNusingthe--lun option.

Copyright 2007 VMware, Inc. All rights reserved.

Round-Robin Load Balancing

ThefollowingoptionsaresupportedanddiscussedinmoredetailinTable 1.
esxcfg-mpath --lun <lun> [--policy [mru|rr|fixed|custom]] [--custom-hba-policy|-H [mru|preferred|any|minq]] | [--custom-target-policy|-T [mru|preferred|any]] | [--custom-max-commands|-C <max_commands>] | [--custom-max-blocks|-B <max-blocks>]

Table 1. Policy Options for esxcfg-mpath

Option --policy -p [mru|rr|fixed|custom] Description SetthepolicyforagivenLUNtomru,rr,fixed,orcustom.Thisoptionrequiresthat the--lunoptionisalsopassedtoindicatetheLUNtosetthepolicyfor. Mostrecentlyused(mru)selectsthepathmostrecentlyusedtosendI/Otoadevice. Roundrobin(rr)usesthemrutargetselectionpolicyandtheanyHBAselection policytoselectpaths. Fixed(fixed)usesonlytheactivepath. Custom(custom)setstheLUNtoexpectacustompolicy. NOTEAfteryouvesetthepolicytocustomforaLUN,youcanspecifyoneormoreof theotheroptions.ForanyLUN,yousetthepolicytocustomonlyonce. --custom-hba-policy -H SelectionpolicyfortheHBA.Useoneofthefollowing: preferredUsetheHBAthatspartofthepreferredpath. mru UsethemostrecentlyusedHBA. minqHBAthathasthesmallestnumberofoutstandingI/Ocommandswhenthe ESXServersystemdoesthepathselection. anyUseanyavailableHBA. --custom-target-policy -T Selectionpolicyforthetarget.Useoneofthefollowing: preferredUsethetargetthatispartofthepreferredpath. mruUsethemostrecentlyusedtarget. anyUseanyavailabletarget. --custom-max-commands -C MaximumnumberofI/Orequeststobeissuedonagivenpathbeforethesystemtries toselectadifferentpath.Asettingofzero(0)disablesswitchingbasedoncommands. Specifythenumberofcommandsasaninteger.Defaultis50. --custom-max-blocks -B MaximumnumberofI/Oblockstobeissuedonagivenpathbeforethesystemtriesto selectadifferentpath.Asettingofzero(0)disablesswitchingbasedonblocks. Specifythenumberofblocksasaninteger.Defaultis2048.

Notes
Ifyousetthecustom-max-blocksandcustom-max-commands,options,thesystemattemptstoswitchpaths assoonasoneofthelimitsisreached. IfyousetthetargetortheHBApolicytopreferred,thesystemchoosesthetargetortheHBAofthepreferred pathwhenpossible.Ifapreferredpolicyissetonanactive/passiveSANarray,andthepreferredtargetisnot ontheactiveSP(StorageProcessor),thesystemdoesnotselectthepreferredtargetbutatargetontheactiveSP. PathswitchingisnotperformedifanoutstandingSCSIreservationisonthetarget,orifapathfailoveris underway.PathswitchingisdelayeduntilanI/Orequestisperformedwhennoreservationsorpathfailovers arepending.

Copyright 2007 VMware, Inc. All rights reserved.

Round-Robin Load Balancing

Examples
esxcfg-mpath --lun=vmhba0:0:0 -p custom

SetsthisLUNtouseacustompolicy.Afterthepolicyhasbeensetto customforaLUN,youdonothavetoincludethe-p customoption whenyoufurtherspecifythecustompolicy. ChangestoadifferentHBAafter4000blockshavebeenissuedonthe currentpath.UsesanyavailableHBA. Displaysthefollowing.

Disk vmhba0:0:0 /dev/sda (34732MB) has 1 paths and policy of Custom: maxCmds=50 maxBlks=2048 hbaPolicy=any targetPolicy=mru Local 1:4.0 vmhba0:0:0 On active preferred

esxcfg-mpath --lun=vmhba0:0:0 -H any -B 4000 esxcfg-mpath -q --lun=vmhba0:0:0

# esxcfg-mpath --lun=vmhba0:0:0 --custom-max-commands=100 # esxcfg-mpath -q --lun=vmhba0:0:0

Setsto100themaximumnumberofcommandsthatcanbeissued beforethesystemswitchestoadifferentpath. Displaysthefollowing(asaresultoftheprecedingcommand).

Disk vmhba0:0:0 /dev/sda (34732MB) has 1 paths and policy of Custom: maxCmds=100 maxBlks=2048 hbaPolicy=any targetPolicy=mru Local 1:4.0 vmhba0:0:0 On active preferred

Global Disk Configuration Options

Twoglobaldiskconfigurationoptionsdeterminethedefaultsettingsfor--custom-max-blocksand --custom-max-commandswhentheroundrobinpolicyisset: SPBlksToSwitchNumberofblockssentoveragivenpathbeforeapathswitch. SPCmdsToSwitchNumberofI/Ocommandssentoveragivenpathbeforeapathswitch. Youcanuseesxcfg-advcfgtoviewandsettheseoptions,forexample:
# esxcfg-advcfg -g /Disk/SPCmdsToSwitch Value of SPCmdsToSwitch is 50 # esxcfg-advcfg -s 200 /Disk/SPCmdsToSwitch Value of SPCmdsToSwitch is 200 # esxcfg-advcfg -g /Disk/SPCmdsToSwitch Value of SPCmdsToSwitch is 200

VMware, Inc. 3401 Hillview Ave., Palo Alto, CA 94304 www.vmware.com Copyright 2007 VMware, Inc. All rights reserved. Protected by one or more of U.S. Patent Nos. 6,397,242, 6,496,847, 6,704,925, 6,711,672, 6,725,289, 6,735,601, 6,785,886, 6,789,156, 6,795,966, 6,880,022, 6,944,699, 6,961,806, 6,961,941, 7,069,413, 7,082,598, 7,089,377, 7,111,086, 7,111,145, 7,117,481, 7,149, 843, 7,155,558, 7,222,221, 7,260,815, 7,260,820, 7,269,683, 7,275,136, 7,277,998, 7,277,999, 7,278,030, and 7,281,102; patents pending. VMware, the VMware boxes logo and design, Virtual SMP and VMotion are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation. Linux is a registered trademark of Linus Torvalds. All other marks and names mentioned herein may be trademarks of their respective companies. Revision 20071119