Академический Документы
Профессиональный Документы
Культура Документы
You can find the most up-to-date technical documentation on the VMware Web site at: http://www.vmware.com/support/ The VMware Web site also provides the latest product updates. If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com
20062009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware, the VMware boxes logo and design, Virtual SMP, and VMotion are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
Contents
AboutThisBook
1 GettingStartedwithResourceManagement 13
ViewingHostResourceInformation 14 UnderstandingVirtualMachineResourceAllocation 18 ReservingHostResources 20 VirtualMachineAttributes:Shares,Reservation,andLimit AdmissionControl 22 ChangingVirtualMachineAttributes 23 CreatingandCustomizingResourcePools 25 UnderstandingExpandableReservation 29 CreatingandCustomizingClusters 30
20
2 ResourceManagementConcepts 33
WhatAreResources? 33 ResourceProvidersandConsumers 33 HowESXServerManagesResources 35 HowAdministratorsConfigureResources 35 ResourceUtilizationandPerformance 36 UnderstandingESXServerArchitecture 36 VMkernel 37 VMkernelResourceManager 37 VMkernelHardwareInterfaceLayer 37 VirtualMachineMonitor 37 ServiceConsole 38 HowAdministratorsAffectCPUManagement 38 HowAdministratorsCanAffectMemoryManagement UnderstandingCPUandMemoryVirtualization 39 CPUVirtualizationBasics 39 MemoryVirtualizationBasics 40
39
VMware, Inc.
3 UnderstandingandManagingResourcePools 43
WhatAreResourcePools? 44 WhyUseResourcePools? 45 HostResourcePoolsandClusterResourcePools 46 ResourcePoolAdmissionControl 47 CreatingResourcePools 48 UnderstandingExpandableReservations 50 ViewingResourcePoolInformation 50 ResourcePoolSummaryTab 51 ResourcePoolResourceAllocationTab 52 ChangingResourcePoolAttributes 54 MonitoringResourcePoolPerformance 54 AddingVirtualMachinestoResourcePools 55 RemovingVirtualMachinesfromResourcePools 56 ResourcePoolsandClusters 56 ClustersEnabledforDRS 57 ClustersNotEnabledforDRS 58
4 UnderstandingClusters 59
IntroductiontoClusters 59 VMwareDRS 60 VMwareHA 60 ClustersandVirtualCenterFailure 61 UnderstandingVMwareDRS 62 InitialPlacement 62 LoadBalancingandVirtualMachineMigration 66 DistributedPowerManagement 68 DRSClusters,ResourcePools,andESXServer 70 HostMaintenanceandStandbyModes 71 UnderstandingVMwareHA 72 TraditionalandHAFailoverSolutions 72 VMwareHAFeatures 74 FailoverCapacity 75 PlanningforHAClusters 75 VMwareHAandSpecialSituations 77 PrimaryandSecondaryHosts 78 HAClustersandMaintenanceMode 78 HAClustersandDisconnectedHosts 78 HAClustersandHostNetworkIsolation 79 UsingHAandDRSTogether 80
VMware, Inc.
Contents
5 CreatingaVMwareCluster 89
ClusterPrerequisites 89 ClustersEnabledforHA 90 VirtualCenterVMotionRequirements 91 ClusterCreationOverview 92 CreatingaCluster 93 ChoosingClusterFeatures 93 SelectingAutomationLevel 93 SelectingHAOptions 94 SelectingaVirtualMachineSwapfileLocation 95 FinishingClusterCreation 95 ViewingClusterInformation 96 SummaryPage 96 DRSResourceDistributionCharts 98 DRSRecommendationsPage 99
6 ManagingVMwareDRS 101
CustomizingDRS 101 AddingHoststoaDRSCluster 102 AddingManagedHoststoaCluster 102 AddingUnmanagedHoststoaCluster 103 RemovingHostsfromClusters 104 HostRemovalandResourcePoolHierarchies HostRemovalandVirtualMachines 105 HostRemovalandInvalidClusters 105 ApplyingDRSRecommendations 105 RecommendationGrouping 106 UsingtheDRSRecommendationsPage 107 ReconfiguringDRS 107 UsingDRSAffinityRules 108 UnderstandingRuleResults 110 DisablingorDeletingRules 110
104
VMware, Inc.
7 ClustersandVirtualMachines 111
AddingVirtualMachinestoaCluster 111 AddingaVirtualMachineDuringCreation 111 MigratingaVirtualMachinetoaCluster 112 AddingaHostwithVirtualMachinestoaCluster 112 PoweringOnVirtualMachinesinaCluster 112 DRSEnabled 112 HAEnabled 113 RemovingVirtualMachinesfromaCluster 113 MigratingVirtualMachinesoutofaCluster 113 RemovingaHostwithVirtualMachinesfromaCluster 114 CustomizingDRSforVirtualMachines 114 CustomizingHAforVirtualMachines 115
8 ManagingVMwareHA 117
CustomizingHA 117 AddingHoststoanHACluster 118 AddingManagedHoststoaCluster 118 AddingUnmanagedHoststoaCluster 118 ResultsofAddingHoststoaCluster 119 ConfiguringandUnconfiguringHAonaHost WorkingwithVMwareHA 121 SettingAdvancedHAOptions 121
120
9 AdvancedResourceManagement 123
CPUVirtualization 124 SoftwareCPUVirtualization 124 HardwareAssistedCPUVirtualization 124 VirtualizationandProcessorSpecificBehavior 125 PerformanceImplications 125 UsingCPUAffinitytoAssignVirtualMachinestoSpecificProcessors 126 MulticoreProcessors 128 Hyperthreading 129 EnablingHyperthreading 130 HyperthreadingandESXServer 130 AdvancedServerConfigurationforHyperthreading 131 Quarantining 133 HyperthreadingandCPUAffinity 133 MemoryVirtualization 133 SoftwareMemoryVirtualization 133
6 VMware, Inc.
Contents
HardwareAssistedMemoryVirtualization 134 PerformanceImplications 135 UnderstandingMemoryOverhead 136 MemoryAllocationandIdleMemoryTax 138 HowESXServerHostsAllocateMemory 138 HowHostMemoryIsUsed 139 MemoryTaxforIdleVirtualMachines 140 HowESXServerHostsReclaimMemory 140 MemoryBalloon(vmmemctl)Driver 141 SwapSpaceandGuestOperatingSystems 142 Swapping 142 SwapSpaceandMemoryOvercommitment 144 SwapFilesandESXServerFailure 144 SharingMemoryAcrossVirtualMachines 144 AdvancedAttributesandWhatTheyDo 145 SettingAdvancedHostAttributes 145 SettingAdvancedVirtualMachineAttributes 149
10 UsingNUMASystemswithESXServer 151
IntroductiontoNUMA 152 WhatIsNUMA? 152 NUMAChallengesforOperatingSystems 152 ESXServerNUMAScheduling 153 VMwareNUMAOptimizationAlgorithms 154 HomeNodesandInitialPlacement 154 DynamicLoadBalancingandPageMigration 155 TransparentPageSharingOptimizedforNUMA 156 ManualNUMAControls 156 IBMEnterpriseXArchitectureOverview 157 AMDOpteronBasedSystemsOverview 158 ObtainingNUMAConfigurationInformationandStatistics 159 CPUAffinityforAssociatingVirtualMachineswithaSingleNUMANode 159 MemoryAffinityforAssociatingMemoryAllocationswithaNUMANode 160
11 BestPractices 163
ResourceManagementBestPractices 163 CreatingandDeployingVirtualMachines 164 Planning 164 CreatingVirtualMachines 164 DeployingtheGuestOperatingSystem 165
VMware, Inc. 7
DeployingGuestApplications 165 ConfiguringVMkernelMemory 165 VMwareHABestPractices 166 NetworkingBestPractices 166 SettingUpNetworkingRedundancy 167 OtherVMwareHAClusterConsiderations 170
A PerformanceMonitoringUtilities:resxtopandesxtop 171
DecidingtoUseresxtoporesxtop 171 UsingtheresxtopUtility 171 UsingtheesxtopUtility 172 UsingtheUtilitiesinInteractiveMode 173 InteractiveModeCommandLineOptions 173 CPUPanel 176 MemoryPanel 179 StoragePanels 184 NetworkPanel 193 UsingtheUtilitiesinBatchMode 194 UsingtheUtilitiesinReplayMode 195
Index 197
VMware, Inc.
Thismanual,theResourceManagementGuide,discussesresourcemanagementfor VMwareVirtualInfrastructureenvironments.Itsfocusisonthefollowingmajor topics: Resourceallocationandresourcemanagementconcepts Virtualmachineattributesandadmissioncontrol Resourcepoolsandhowtomanagethem Clusters,VMwareDistributedResourceScheduler(DRS), VMwareHighAvailability(HA),andhowtoworkwiththem Advancedresourcemanagementoptions Performanceconsiderations ResourceManagementGuidecoversbothESXServer3.5andESXServer3iversion3.5. Foreaseofdiscussion,thisbookusesthefollowingproductnamingconventions: FortopicsspecifictoESXServer3.5,thisbookusesthetermESXServer3. FortopicsspecifictoESXServer3iversion3.5,thisbookusesthetermESXServer 3i. Fortopicscommontobothproducts,thisbookusesthetermESXServer. Whentheidentificationofaspecificreleaseisimportanttoadiscussion,thisbook referstotheproductbyitsfull,versionedname. WhenadiscussionappliestoallversionsofESXServerforVMwareInfrastructure 3,thisbookusesthetermESXServer3.x.
VMware, Inc.
Intended Audience
Thismanualisforsystemadministratorswhowanttounderstandhowthesystem managesresourcesandhowtheycancustomizethedefaultbehavior.Itsalsoessential foranyonewhowantstounderstandanduseresourcepools,clusters,DRS,orHA. ThismanualassumesyouhaveaworkingknowledgeofESXServerandofthe VirtualCenterServer.
Document Feedback
VMwarewelcomesyoursuggestionsforimprovingourdocumentation.Ifyouhave comments,sendyourfeedbackto: docfeedback@vmware.com
10
VMware, Inc.
Support Offerings
FindouthowVMwaresupportofferingscanhelpmeetyourbusinessneeds.Goto http://www.vmware.com/support/services.
VMware, Inc.
11
12
VMware, Inc.
Thischapterintroducesbasicresourcemanagementconceptsusingasimpleexample. Thechapterstepsyouthroughresourceallocation,firstinasinglehostenvironment, andtheninamorecomplexmultihostenvironment. Thischapterdiscussesthefollowingtopics: ViewingHostResourceInformationonpage 14 UnderstandingVirtualMachineResourceAllocationonpage 18 ChangingVirtualMachineAttributesonpage 23 CreatingandCustomizingResourcePoolsonpage 25 UnderstandingExpandableReservationonpage 29 CreatingandCustomizingClustersonpage 30
VMware, Inc.
13
VM-QA
VM-Marketing
14
VMware, Inc.
VMware, Inc.
15
16
VMware, Inc.
Field MemoryReservation
MemoryReservation Used
Totalmemoryresourcesofthishostthatarereservedbya runningvirtualmachineandvirtualizationoverhead. Note:Virtualmachinesthatarenotpoweredondonot consumememoryresources.Forpoweredonvirtual machines,thesystemreservesmemoryresourcesaccording toeachvirtualmachinesReservationsettingandoverhead. Afteravirtualmachinehasaccesseditsfullreservation, ESXServerallowsthevirtualmachinetoretainthismuch memory,andwillnotreclaimit,evenifthevirtualmachine becomesidleandstopsaccessingmemory.
MemoryReservation Unused
Totalmemoryresourcesofthishostthatarenotcurrently reserved.
ClicktheMemoryorCPUbuttondependingontheinformationyouwant.
VMware, Inc.
17
18
VMware, Inc.
WhenCPUresourcesareovercommitted,theESXServerhosttimeslicesthephysical processorsacrossallvirtualmachinessoeachvirtualmachinerunsasifithasthe specifiednumberofprocessors. WhenanESXServerhostrunsmultiplevirtualmachines,itallocateseachvirtual machineashareofthephysicalresources.Withthedefaultresourceallocationsettings, allvirtualmachinesassociatedwiththesamehostreceive: AnequalshareofCPUpervirtualCPU.Thatmeanssingleprocessorvirtual machinesareassignedonlyhalfoftheresourcesofadualprocessorvirtual machine. AnequalshareperMBofvirtualmemorysize.Thatmeansan8GBvirtualmachine isentitledtoeighttimesasmuchmemoryasa1GBvirtualmachine.
VMware, Inc.
19
Shares
Sharesspecifytherelativepriorityorimportanceofavirtualmachine.Ifavirtual machinehastwiceasmanysharesofaresourceasanothervirtualmachine,itisentitled toconsumetwiceasmuchofthatresource.SharesaretypicallyspecifiedasHigh, Normal,orLowandthesevaluesspecifysharevalueswitha4:2:1ratio,respectively. YoucanalsochooseCustomtoassignaspecificnumberofshares(whichexpressesa proportionalweight)toeachvirtualmachine. Specifyingsharesmakessenseonlywithregardtosiblingvirtualmachinesorresource pools,thatis,virtualmachinesorresourcepoolswiththesameparentintheresource poolhierarchy.Siblingsshareresourcesaccordingtotheirrelativesharevalues, boundedbythereservationandlimit.SeeWhatAreResourcePools?onpage 44for anexplanationofthehierarchyandsiblingconcepts. Whenyouassignsharestoavirtualmachine,youalwaysspecifytherelativepriority forthatvirtualmachine.
20
VMware, Inc.
CPUandmemorysharevalues,respectively,defaultto: High2000sharespervirtualCPUand20sharespermegabyteofvirtual machinememory Normal1000sharespervirtualCPUand10sharespermegabyteofvirtual machinememory Low500sharespervirtualCPUand5sharespermegabyteofvirtualmachine memory YoucanalsospecifyaCustomsharevalue. Forexample,anSMPvirtualmachinewithtwovirtualCPUsand1GBRAMwithCPU andmemorysharessettoNormalhas2x1000=2000sharesofCPUand10x1024=10240 sharesofmemory. NOTEVirtualmachineswithmorethanonevirtualCPUarecalledSMP(symmetric multiprocessing)virtualmachines. Theamountofresourcesrepresentedbyeachsharechangeswhenanewvirtual machineispoweredon.Thisaffectsallvirtualmachinesinthesameresourcepool.For example: Twovirtualmachinesrunonahostwith8GHz.TheirCPUsharesaresetto Normalandget4GHzeach. Athirdvirtualmachineispoweredon.ItsCPUsharesvalueissettoHigh,which meansitshouldhavetwiceasmanysharesasthemachinessettoNormal.Thenew virtualmachinereceives4GHzandthetwoothermachinesgetonly2GHzeach. Notethatthesameresultoccursiftheuserspecifiesacustomsharevalueof2000 forthethirdvirtualmachine.
Reservation
Reservationspecifiestheguaranteedreservationforavirtualmachine.Theserver allowsyoutopoweronavirtualmachineonlyiftheCPUandmemoryreservationis available.Theserverguaranteesthatamountevenwhenthephysicalserverisheavily loaded.Thereservationisexpressedinconcreteunits(megahertzormegabytes).When resourcesarenotused,theESXServerhostmakesthemavailabletoothervirtual machines.
VMware, Inc.
21
Limit
LimitspecifiestheupperlimitforCPUormemoryforavirtualmachine.Aservercan allocatemorethanthereservationtoavirtualmachine,butneverallocatesmorethan thelimit,evenifthereisunutilizedCPUormemoryonthesystem.Thelimitis expressedinconcreteunits(megahertzormegabytes). CPUandmemorylimitdefaulttounlimited.Whenthememorylimitisunlimited,the amountofmemoryconfiguredforthevirtualmachinewhenitwascreatedbecomesits implicitlimitinmostcases. Inmostcases,itisnotnecessarytospecifyalimit.Therearebenefitsanddrawbacks: BenefitsAssigningalimitisusefulifyoustartwithasmallnumberofvirtual machinesandwanttomanageuserexpectations.Performancewilldeteriorateas youaddmorevirtualmachines.Youcansimulatehavingfewerresourcesavailable byspecifyingalimit. DrawbacksYoumightwasteidleresourcesifyouspecifyalimit.Thesystem doesnotallowvirtualmachinestousemoreresourcesthanthelimit,evenwhen thesystemisunderutilizedandidleresourcesareavailable.Specifythelimitonly ifyouhavegoodreasonsfordoingso.
Admission Control
Whenyoupoweronavirtualmachine,thesystemcheckstheamountofCPUand memoryresourcesthathavenotyetbeenreserved.Basedontheavailableunreserved resources,thesystemdetermineswhetheritcanguaranteethereservationforwhich thevirtualmachinehasbeenconfigured(ifany).Thisprocessiscalledadmissioncontrol. IfenoughunreservedCPUandmemoryareavailable,orifthereisnoreservation,the virtualmachineispoweredon.Otherwise,anInsufficient Resourceswarning appears. NOTEInadditiontotheuserspecifiedmemoryreservation,foreachvirtualmachine thereisalsoanamountofoverheadmemory.Thisextramemorycommitmentis includedintheadmissioncontrolcalculation.SeeUnderstandingMemoryOverhead onpage 136.
22
VMware, Inc.
VMware, Inc.
23
RepeatthesestepsintheMemoryResourcespanel,andclickOK.
6 7
Rightclickthemarketingvirtualmachine(VMMarketing). ChangethevalueintheReservationfieldtothenumber,andclickOK.
ClickOKwhenyouredone.
24
VMware, Inc.
SelectthehostsResourceAllocationtabandclickCPU,youseethatsharesfor VMQAaretwicethatoftheothervirtualmachine.
Continuingwiththeexampleabove,assumethatyounolongerwanttoassignone virtualmachineeachtoyourQAandMarketingdepartmentsbutwanttogiveeach departmentapredefinedamountofresources.Dependingondepartmentalneeds,the departmentadministratorcancreatevirtualmachinesforthedepartment. Forexample,ifyoustartedwithahostthatprovides6GHzofCPUand3GBofmemory, youcanchooseshareallocationsofHighforRPQAandsharesallocationsofNormal forRPMarketing.Thatresultsinapproximately4GHzand2GBofmemoryforRPQA, and2GHzand1GBforRPMarketing.Thoseresourcesarethenavailabletothevirtual machinesintherespectiveresourcepools.SeeFigure 14. Figure 1-4. ESX Server Host with Two Resource Pools
ESX Server host 6GHz, 3GB
RP-QA
4GHz, 2GB
VM-QA 1
VM-QA 2
VM-Marketing 1
VM-Marketing 2
VM-Marketing 3
To create and customize resource pools 1 2 3 StartaVIClientandconnecttoaVirtualCenterServer. Intheinventorypanelontheleft,selectahostandchooseNewResourcePoolin theCommandspanelontheright. IntheCreateResourcePooldialogbox,typetheresourcepoolname(forexample, RPQA).
26
VMware, Inc.
SpecifySharesofHighfortheCPUandmemoryresourcesofRPQA.
VMware, Inc.
27
28
VMware, Inc.
6GHz
RP-MOM
VM-K1, 2GHz
VM-K2, 2GHz
VMware, Inc.
29
6GHz
RP-MOM
VM-M2, 2GHz
VM-K1, 2GHz
VM-K2, 2GHz
30
VMware, Inc.
To create and customize a cluster 1 2 3 4 5 6 7 StartaVIClientandconnecttoaVirtualCenterServer. Intheinventorypanelontheleft,rightclickadatacenterandchooseNewCluster. NametheclusterandenableitforHAandDRS. Keepthedefault,fullyautomated,forDRS. KeepthedefaultsforhostfailuresandadmissioncontrolforHA. SelecttheappropriateoptionforSwapfilePolicyforVirtualMachines. ClickFinish. TheVirtualCenterServercreatesanewclusterwiththespecifiedattributes.
ForinformationonDRS,HA,andavailableattributes,seeChapter 5,Creatinga VMwareCluster,onpage 89. Thenexttaskistoaddanumberofhoststothecluster.UsingclustersenabledforDRS makessenseevenifyouhaveonlytwohostsinthecluster. AclusterenabledforHAcansupportamaximumoffourconcurrenthostfailures.In thefollowingsteps,youaddahosttotheclusterthatismanagedbythesame VirtualCenterServer. To add a host to the cluster 1 IntheleftpaneloftheVIClient,selectthehostanddragitovertheclustersicon. IftheclusterisenabledforDRS,youarepromptedwhetheryouwanttoaddthe hostsvirtualmachinesdirectlytotheclusters(invisible)rootresourcepoolor whetheryouwanttocreateanewresourcepooltorepresentthathost.Theroot resourcepoolisatthetoplevelandisnotdisplayedbecausetheresourcesare ownedbythecluster.
VMware, Inc.
31
IftheclusterisnotenabledforDRS,allresourcepoolsareremoved.
SelecttheclusterandchooseitsResourceAllocationtabtoaddmorehostsand lookattheresourceallocationinformationforthecluster.
32
VMware, Inc.
VMware, Inc.
33
Aclusterisagroupofhosts.YoucancreateaclusterusingVMwareVirtualCenter,and addmultiplehoststothecluster.VirtualCentermanagesthesehostsresourcesjointly: theclusterownsalloftheCPUandmemoryofallhosts.Youcanenabletheclusterfor jointloadbalancingorfailover.SeeChapter 4,UnderstandingClusters,onpage 59 foranintroductiontoclusters. Resourcepoolsarealogicalabstractionforflexiblemanagementofresources.Resource poolscanbegroupedintohierarchies.Theycanbeconsideredbothresourceproviders andconsumers.Resourcepoolsprovideresourcestochildresourcepoolsandvirtual machines.Resourcepoolsarealsoresourceconsumersbecausetheyconsumetheir parentsresources.SeeChapter 3,UnderstandingandManagingResourcePools,on page 43. Virtualmachinesareresourceconsumers.Thedefaultresourcesettingsassignedduring creationworkwellformostmachines.Youcanlatereditthevirtualmachinesettingsto allocateasharebasedpercentageofthetotalCPUandmemoryoftheresource provideroraguaranteedreservationofCPUandmemory.Whenyoupoweronthat virtualmachine,theservercheckswhetherenoughunreservedresourcesareavailable andallowspowerononlyifthereareenoughresources.(Thisprocessiscalled admissioncontrol.) Toseehowclusters,resourcepools,andvirtualmachinesaredisplayedintheVIClient, seeFigure 21. Figure 2-1. Clusters, Resource Pools, and Virtual Machines in VI Client
Virtual machine
34
VMware, Inc.
VMware, Inc.
35
36
VMware, Inc.
VMkernel
TheVMkernelisahighperformanceoperatingsystemdevelopedbyVMwarethat runsdirectlyontheESXServerhost.TheVMkernelcontrolsandmanagesmostofthe physicalresourcesonthehardware,including: Memory Physicalprocessors Storageandnetworkingcontrollers TheVMkernelincludesschedulersforCPU,memory,anddiskaccess,andhas fullfledgedstorageandnetworkstacks.ItalsoincludestheVirtualMachineFile System(VMFS).VMFSisadistributedfilesystemoptimizedforlargefileslikevirtual machinedisksandswapfiles.
VMware, Inc.
37
Service Console
TheserviceconsoleisalimiteddistributionofLinuxbasedonRedHatEnterprise Linux3,Update8(RHEL3U8).Theserviceconsoleprovidesanexecutionenvironment formonitoringandadministeringanESXServer3system.ESXServer3idoesnot provideaserviceconsole. NOTEInmostcases,administratorsuseaVIClientconnectedtoeitheranESXServer systemoraVirtualCenterServertomonitorandadministerESXServersystems.
38
VMware, Inc.
VMware, Inc.
39
40
VMware, Inc.
SelectMemory.
VMware, Inc.
41
Reservation. Isaguaranteedlowerboundontheamountofphysicalmemorythatthe hostreservesforthevirtualmachine,evenwhenmemoryisovercommitted.Setthe reservationtoalevelthatensuresthevirtualmachinehassufficientmemorytorun efficiently,withoutexcessivepaging. Limit. Isanupperboundontheamountofphysicalmemorythatthehostwillallocate tothevirtualmachine.Thevirtualmachinesmemoryallocationisalsoimplicitly limitedbyitsconfiguredsize. Overheadmemoryincludesspacereservedforthevirtualmachineframebufferand variousvirtualizationdatastructures.SeeUnderstandingMemoryOverheadon page 136.
Memory Overcommitment
Foreachrunningvirtualmachine,thesystemreservesphysicalmemoryforthevirtual machinesreservation(ifany)andforitsvirtualizationoverhead.Becauseofthe memorymanagementtechniquestheESXServerhostuses,yourvirtualmachinescan usemorememorythanthephysicalmachine(thehost)hasavailable.Forexample,you canhaveahostwith2GBmemoryandrunfourvirtualmachineswith1GBmemory each.Inthatcase,thememoryisovercommitted. Overcommitmentmakessensebecause,typically,somevirtualmachinesarelightly loadedwhileothersaremoreheavilyloaded,andrelativeactivitylevelsvaryovertime. Toimprovememoryutilization,theESXServerhosttransfersmemoryfromidlevirtual machinestovirtualmachinesthatneedmorememory.UsetheReservationorShares parametertopreferentiallyallocatememorytoimportantvirtualmachines.This memoryremainsavailabletoothervirtualmachinesifitisnotinuse.
Memory Sharing
Manyworkloadspresentopportunitiesforsharingmemoryacrossvirtualmachines. Forexample,severalvirtualmachinesmightberunninginstancesofthesameguest operatingsystem,havethesameapplicationsorcomponentsloaded,orcontain commondata.ESXServersystemsuseaproprietarypagesharingtechniquetosecurely eliminateredundantcopiesofmemorypages. Withmemorysharing,aworkloadconsistingofmultiplevirtualmachinesoften consumeslessmemorythanitwouldwhenrunningonphysicalmachines.Asaresult, thesystemcanefficientlysupporthigherlevelsofovercommitment. Theamountofmemorysavedbymemorysharingdependsonworkload characteristics.Aworkloadofmanynearlyidenticalvirtualmachinesmightfreeup morethanthirtypercentofmemory,whileamorediverseworkloadmightresultin savingsoflessthanfivepercentofmemory.
42 VMware, Inc.
ThischapterintroducesresourcepoolsandexplainshowVirtualInfrastructureallows youtoviewandmanipulatethem. Thischapterdiscussesthefollowingtopics: WhatAreResourcePools?onpage 44 ResourcePoolAdmissionControlonpage 47 CreatingResourcePoolsonpage 48 ViewingResourcePoolInformationonpage 50 ChangingResourcePoolAttributesonpage 54 MonitoringResourcePoolPerformanceonpage 54 AddingVirtualMachinestoResourcePoolsonpage 55 RemovingVirtualMachinesfromResourcePoolsonpage 56 ResourcePoolsandClustersonpage 56 Alltasksassumeyouhavepermissiontoperformthem.SeetheonlineHelpfor informationonpermissionsandhowtosetthem.
VMware, Inc.
43
44
VMware, Inc.
VMware, Inc.
45
Forexample,assumeahosthasanumberofvirtualmachines.Themarketing departmentusesthreeofthevirtualmachinesandtheQAdepartmentusestwovirtual machines.BecausetheQAdepartmentneedslargeramountsofCPUandmemory,the administratorcreatesoneresourcepoolforeachgroup.TheadministratorsetsCPU SharestoHighfortheQAdepartmentpoolandtoNormalfortheMarketing departmentpoolsothattheQAdepartmentuserscanrunautomatedtests.Thesecond resourcepoolwithfewerCPUandmemoryresourcesissufficientforthelighterload ofthemarketingstaff.WhenevertheQAdepartmentisnotfullyusingitsallocation,the marketingdepartmentcanusetheavailableresources. ThisscenarioisshowninFigure 32.Thenumbersshowtheeffectiveallocationstothe resourcepools. Figure 3-2. Allocating Resources to Resource Pools
ESX Server host 6GHz, 3GB
RP-QA
4GHz, 2GB
VM-QA 1
VM-QA 2
VM-Marketing 1
VM-Marketing 2
VM-Marketing 3
Becauseallresourcesarecombined,younolongermanageresourcesfor individualhostsbutmanageallresourcesinthecontextofthecluster.Youassign virtualmachinestoresourcepoolswithpredefinedcharacteristics.Ifyoulater changecapacitybyadding,removing,orupgradinghosts,youmighthaveto changetheresourceallocationsyoumadefortheresourcepools. IftheVirtualCenterServerbecomesunavailable,makechangesusingaVIClient connectedtoanESXServerhost.However,theclustermightbecomeyellow (overcommitted)orred(invalid)whentheVirtualCenterServerbecomesavailable again.SeeValid,Yellow,andRedClustersonpage 81.Ifyourclusterisin automaticmode,VirtualCenterreappliesthelastknownclusterconfiguration(and potentiallyundoesyourchanges)whentheVirtualCenterServerbecomes availableagain.
Unreserved
Reservation type
VMware, Inc.
47
HowunreservedCPUandmemoryarecomputedandwhetheractionsareperformed dependsonthereservationtype: Fixedreservationtype.Thesystemcheckswhethertheresourcepoolhassufficient unreservedresources.Ifitdoes,theactioncanbeperformed.Ifitdoesnot,a messageappearsandtheactioncannotbeperformed. Expandablereservationtype.Thesystemcheckswhethertheresourcepoolhas sufficientresourcestofulfilltherequirements. Iftherearesufficientresources,theactionisperformed. Iftherearenotsufficientresources,themanagingservercheckswhether resourcesareavailableinaparentresourcepool(directparentorancestor).If theyare,theactionisperformedandtheparentresourcepoolresourcesare reserved.Ifnoresourcesareavailable,amessageappearsandtheactionisnot performed.SeeUnderstandingExpandableReservationonpage 29. ThesystemdoesnotallowyoutoviolatepreconfiguredReservationorLimitsettings. Eachtimeyoureconfigurearesourcepoolorpoweronavirtualmachine,thesystem validatesallparameterssoallservicelevelguaranteescanstillbemet.
48
VMware, Inc.
Description Nameofthenewresourcepool.
NumberofCPUsharestheresourcepoolhaswithrespecttothe parentstotal.Siblingresourcepoolsshareresourcesaccording totheirrelativesharevaluesboundedbythereservationand limit.YoucanchooseLow,Normal,orHigh,orchooseCustom tospecifyanumberthatassignsasharevalue. GuaranteedCPUallocationforthisresourcepool. Indicateswhetherexpandablereservationsareconsidered duringadmissioncontrol.Ifyoupoweronavirtualmachinein thisresourcepool,andthereservationsofthevirtualmachines combinedarelargerthanthereservationoftheresourcepool, theresourcepoolcanuseresourcesfromitsparentorancestors ifthischeckboxisselected(thedefault). UpperlimitfortheamountofCPUthehostmakesavailableto thisresourcepool.DefaultisUnlimited.Tospecifyalimit, deselecttheUnlimitedcheckboxandtypeinthenumber.
Limit
Memory Resources Shares Numberofmemorysharestheresourcepoolhaswithrespectto theparentstotal.Siblingresourcepoolsshareresources accordingtotheirrelativesharevaluesboundedbythe reservationandlimit.YoucanchooseLow,Normal,orHigh,or chooseCustomtospecifyanumberthatassignsasharevalue. Guaranteedmemoryallocationforthisresourcepool. Indicateswhetherexpandablereservationsareconsidered duringadmissioncontrol.Ifyoupoweronavirtualmachinein thisresourcepool,andthereservationsofthevirtualmachines combinedarelargerthanthereservationoftheresourcepool, theresourcepoolcanuseaparentsorancestorsresourcesifthis checkboxisselected(thedefault). Upperlimitforthisresourcepoolsmemoryallocation.Default isUnlimited.Tospecifyadifferentlimit,deselecttheUnlimited checkbox.
Limit
VMware, Inc.
49
50
VMware, Inc.
Commands
VMware, Inc.
51
ThetopportionofFigure 34specifiestheinformationinTable 32abouttheresource poolitself. Table 3-2. Resource Allocation Tab Fields
Field CPUReservation/ MemoryReservation CPUReservationUsed/ MemoryReservationUsed Description AmountofCPUormemoryspecifiedinthereservationforthis resourcepool.Reservationcanbespecifiedduringresource poolcreation,orlaterbyeditingtheresourcepool. CPUormemoryreservationused.Reservationsareusedby runningvirtualmachinesorbychildresourcepoolswith reservations.
52
VMware, Inc.
InFigure 34,belowtheinformationspecifictotheresourcepoolisalistofthevirtual machinesandchildresourcepoolsofthisresourcepool.Thislistdoesnotcontain virtualmachinesassignedtochildresourcepoolsofthisresourcepool. ClicktheCPUorMemorytabtodisplaytheinformationdescribedinTable 33. Table 3-3. Resource Allocation CPU and Memory Fields
Field Name Reservation Limit Shares Description Nameoftheresourcepoolorvirtualmachine. Specifiedreservationforthisvirtualmachineorresourcepool.Defaultis0,that is,thesystemreservesnoresourcesforthisresourcepool. Specifiedlimitforthisvirtualmachineorresourcepool.DefaultisUnlimited, thatis,thesystemallocatesasmanyresourcestothisvirtualmachineasitcan. Specifiedsharesforthisvirtualmachineorresourcepool.OneofHigh, Normal,Lowifoneofthedefaultsettingshasbeenselected.Customifyou selectacustomsetting. Numberofsharesallocatedtothisvirtualmachineorresourcepool.This numberdependsonthesharessetting(High,Normal,Low,orCustom).See Sharesonpage 20. Sharesvalueforthisresourcepoolorvirtualmachinedividedbythetotal numberofsharesallocatedtoallchildrenoftheparentresourcepool.This valueisunrelatedtotheparentresourcepoolslocalsharesallocation. Reservationtype.EitherFixedorExpandable.SeeUnderstanding ExpandableReservationonpage 29.
Type
VMware, Inc.
53
54
VMware, Inc.
VMware, Inc.
55
56
VMware, Inc.
cluster
Host1
cluster
X Y
X Y
VMware, Inc.
57
cluster
Host1 RP
Host2 RP VM1
cluster
VM2
VM3
VM4
RP
RP
RP
VM1
VM2
VM3
VM4
58
VMware, Inc.
Understanding Clusters
ThischapterpresentsaconceptualintroductiontoclustersandtotheVMware DistributedResourceScheduler(DRS)andHighAvailability(HA)features. Thischapterdiscussesthefollowingtopics: IntroductiontoClustersonpage 59 UnderstandingVMwareDRSonpage 62 UnderstandingVMwareHAonpage 72 UsingHAandDRSTogetheronpage 80 Valid,Yellow,andRedClustersonpage 81 NOTEAlltasksdescribedassumeyouhavepermissiontoperformthem.Seetheonline Helpforinformationonpermissionsandhowtosetthem.
Introduction to Clusters
AclusterisacollectionofESXServerhostsandassociatedvirtualmachineswithshared resourcesandasharedmanagementinterface.Whenyouaddahosttoacluster,the hostsresourcesbecomepartoftheclustersresources.Whenyoucreateacluster,you canchoosetoenableitforDRS,HA,orboth. SeeClusterPrerequisitesonpage 89forinformationonthevirtualmachinesin clustersandonhowtoconfigurethem. NOTEYoucancreateaclusterwithoutaspeciallicense,butyoumusthavealicenseto enableaclusterforDRSorHA.
VMware, Inc.
59
VMware DRS
TheDRSfeatureimprovesresourceallocationacrossallhostsandresourcepools.DRS collectsresourceusageinformationforallhostsandvirtualmachinesintheclusterand generatesrecommendationsforvirtualmachineplacementandhostmachinepower state.Theserecommendationscanbeappliedautomatically.Dependingonthe configuredDRSautomationlevel,DRSdisplaysorappliesthefollowingtypesof recommendations: InitialplacementWhenyoufirstpoweronavirtualmachineinthecluster,DRS eitherplacesthevirtualmachineonanappropriatehostormakesa recommendation.SeeInitialPlacementonpage 62. MigrationAtruntime,DRStriestofixruleviolationsandimproveresource utilizationacrosstheclustereitherbyperformingmigrationsofvirtualmachines, orbyprovidingrecommendationsforvirtualmachinemigrations.SeeLoad BalancingandVirtualMachineMigrationonpage 66. PowermanagementWhentheDistributedPowerManagementfeatureis enabled,DRScomparesclusterandhostlevelcapacitytothedemandsofrunning theclustersvirtualmachines,includingrecenthistoricaldemand.Itmakes recommendationsforplacinghostsinstandbypowermodeifsufficientexcess capacityisfoundorpoweringonhostsifcapacityisneeded.Dependingonthe resultinghostpowerstaterecommendations,virtualmachinesmightneedtobe migratedtoandfromthehostsaswell.SeeDistributedPowerManagementon page 68.
VMware HA
AclusterenabledforHAmonitorsforhostfailure.Ifahostbecomesunavailable,all virtualmachinesthatwereonthehostarepromptlyrestartedondifferenthosts. WhenyouenableaclusterforHA,youcanspecifythenumberofhostfailuresallowed. Ifyousetthenumberofhostfailuresas1,HAmaintainsenoughcapacityacrossthe clustertotoleratethefailureofonehost,sothatallrunningvirtualmachinesonthat hostcanberestartedonremaininghosts.Bydefault,youcannotpoweronavirtual machineifdoingsoviolatesrequiredfailovercapacity(strictadmissioncontrol).See UnderstandingVMwareHAonpage 72.
60
VMware, Inc.
VM4
VM5
VM6
VM4
VM5
VM6
Host B
Host B
VM7
VM8
VM9
VM7
VM8
VM9
VM7 VM5
VM8
VM9
Host C
Host C
Host C
The affected virtual machines have been restarted on the remaining hosts
Initial Placement
Whenyouattempttopoweronasinglevirtualmachineoragroupofvirtualmachines inaDRSenabledcluster,VirtualCenterchecksthatthereareenoughresourcesinthe clustertosupportthevirtualmachine(s).Then,foreachvirtualmachine,VirtualCenter identifiesahostonwhichitcanrunanddoesoneofthefollowing: Automatically places it. Ifallplacementrelatedactions(virtualmachinesbeing poweredonormigrated,orhostsbeingpoweredon)areinautomaticmode,thesesteps aretakenautomaticallywithnorecommendationbeingshowntotheuser. Makes an initial placement recommendation. Iftheautomationlevelofanyofthe placementrelatedactionsareinmanualmode.Initialplacementrecommendations receivedbyusersinnonautomaticplacementscenariosdifferbasedonwhetheroneor morethanonevirtualmachineisbeingpoweredon. NOTENoinitialplacementrecommendationsaregivenforvirtualmachineson standalonehostsorinnonDRSclusters.Whenpoweredon,theyareplacedonthehost wheretheycurrentlyreside.
62
VMware, Inc.
Asinglevirtualmachineisbeingpoweredon,butprerequisiteactionsare required. Theseactionsincludepoweringonahostinstandbymodeorthemigrationof othervirtualmachinesfromonehosttoanother.Inthiscase,onlyone recommendationisprovidedandithasmultiplelines,showingeachofthe prerequisiteactions.Theusercaneitheracceptthisentirerecommendationor cancelpoweringonthevirtualmachine(Figure 43). Figure 4-3. Single Virtual Machine, with Prerequisites
VMware, Inc.
63
Group Power On
Youcanattempttopoweronmultiplevirtualmachinesatthesametime(grouppower on).Itisnotrequiredthatthevirtualmachinesselectedforagrouppoweronattempt beinthesameDRScluster.Theycanbeselectedacrossclustersbutmustbewithinthe samedatacenter.ItisalsopossibletoincludevirtualmachineslocatedinnonDRS clustersoronstandalonehosts,butthesearepoweredonautomaticallyandnot includedinanyinitialplacementrecommendation. Theinitialplacementrecommendationsforgrouppoweronattemptsareprovidedon aperclusterbasis.Ifalloftheplacementrelatedactionsforagrouppoweronattempt areinautomaticmode,thevirtualmachinesarepoweredonwithnoinitialplacement recommendationgiven.Ifplacementrelatedactionsforanyofthevirtualmachinesare inmanualmode,thepoweringonofallofthevirtualmachines(includingthosethat areinautomaticmode)ismanualandisincludedinaninitialplacement recommendation. ForeachDRSclusterthatthevirtualmachinesbeingpoweredonbelongto,thereisa singlerecommendation,whichcontainsalloftheneededprerequisites(orno recommendation).Allsuchclusterspecificrecommendationsarepresentedtogether underthePowerOnRecommendationstab(Figure 44). Figure 4-4. Group Power On Recommendation
64
VMware, Inc.
Whenanonautomaticgrouppoweronattemptismade,andvirtualmachinesnot subjecttoaninitialplacementrecommendation(thatis,thoseonstandalonehostsorin nonDRSclusters)areincluded,VirtualCenterattemptstopowerthemon automatically.Ifthesepoweronsaresuccessful,theyarelistedundertheStarted PowerOnstab.Anyvirtualmachinesthatfailtopoweroninthiswayarelistedunder theFailedPowerOnstab.SeeFigure 45. Figure 4-5. Automatic Group Power On
VMware, Inc.
65
VM1 VM4
VM3 VM6
VM1
VM2
VM3
Host 1
VM7
VM7
VM4
VM5
Host 2
Host 2
VM8
VM9
VM8
VM9
VM6
Host 3
Host 3
66
VMware, Inc.
Whenaclusterbecomesunbalanced,DRSmakesrecommendationsormigratesvirtual machines,dependingonthedefaultautomationlevel: Iftheclusteroranyofthevirtualmachinesinvolvedaremanualorpartially automated,VirtualCenterdoesnottakeautomaticactionstobalanceresources. Instead,theSummarypageindicatesthatmigrationrecommendationsare availableandtheDRSRecommendationspagedisplaysrecommendationsfor changesthatmakethemostefficientuseofresourcesacrossthecluster. Iftheclusterandvirtualmachinesinvolvedareallfullyautomated,VirtualCenter migratesrunningvirtualmachinesbetweenhostsasneededtoensureefficientuse ofclusterresources. NOTEEveninanautomaticmigrationsetup,userscanexplicitlymigrate individualvirtualmachines,butVirtualCentermightmovethosevirtualmachines tootherhoststooptimizeclusterresources. Bydefault,automationlevelisspecifiedforthewholecluster.Youcanalsospecifya customautomationlevelforindividualvirtualmachines.
Migration Threshold
Themigrationthresholdallowsyoutospecifywhichrecommendationsareapplied whentheclusterisinfullyautomatedmode.SeeFigure 47.Youcanmovethesliderto useoneoffivelevels,rangingfromConservative,whichmakesthesmallestnumber ofmigrations,toAggressive,whichmakesthelargestnumberofmigrations.Thefive migrationlevelsapplyrecommendationsbasedontheirassignedstarratings. Figure 4-7. Migration Threshold Choices
VMware, Inc.
67
Migration Recommendations
Ifyoucreateaclusterwithadefaultmanualorpartiallyautomatedmode, VirtualCenterdisplaysmigrationrecommendationsontheDRSRecommendations page.Thesystemsuppliesasmanyrecommendationsasnecessarytoenforcerulesand balancetheresourcesofthecluster.Eachrecommendationincludesthevirtualmachine tobemoved,current(source)hostanddestinationhost,andareasonforthe recommendation.Thereasoncanbeoneofthefollowing: BalanceaverageCPUloads. Balanceaveragememoryloads. Satisfyresourcepoolreservations. Satisfyaffinity(orantiaffinity)rule.SeeUsingDRSAffinityRulesonpage 108. Hostisenteringmaintenancemode.SeeHostMaintenanceandStandbyModes onpage 71.
68
VMware, Inc.
BeforeDistributedPowerManagementcanbeenabledforaDRScluster,youmust ensurethattheESXServerhostshavetheappropriatehardwaresupportand configuration.Specifically,theNICsusedbytheVMkernelnetworkmusthave WakeonLAN(WOL)functionality,whichisusedtobringanESXServerhostupfrom apoweredoffstate.YoushouldtestwakecapabilityoperationoneachESXServer3.5 (orESXServer3iversion3.5)hostonwhichDistributedPowerManagementistobe deployed.Todothis,ensurethatthereisatleastoneotherESXServerhostpoweredon inthecluster(tosendthewakepackets),explicitlyputthehosttobetestedintostandby stateandonceithasentered,ensurethatexplicitlyrequestingthatthestandbyhost powerbackonissuccessful.Ifnot,VMwarerecommendsthatyoudonotconfigureit tobepowermanagedbyDistributedPowerManagement. CAUTIONBeforeimplementingDistributedPowerManagement,testthe WakeonLANcapabilityofyourhosts.IfWOLfunctionalityfails,thepower managementfeaturemaypoweroffhostsandnotbeabletopowerthembackonlater. ThedefaultpowermanagementautomationlevelforaDRSclusterisselectedfromthe PowerManagementtaboftheclustersSettingsdialogbox.SeeFigure 48.Thefeature isenabledwheneverthissettingisnotoff.Theoptionsavailableare: OffThefeatureisdisabledandnorecommendationswillbemade. ManualHostpoweroperationandrelatedvirtualmachinemigration recommendationsaremade,butnotautomaticallyexecuted. AutomaticHostpoweroperationsareautomaticallyexecutedifrelatedvirtual machinemigrationscanallbeexecutedautomatically. Inadditiontotheseclusterlevelsettings,youcanalsosetoverridesforindividualhosts sothattheirautomationleveldiffersfromthatofthecluster.Suchoverridesonlyapply ifthefeatureisenabled(notsettooff)forthecluster.
VMware, Inc.
69
70
VMware, Inc.
Maintenance Mode
Standalonehostsandhostswithinaclustersupportamaintenancemode,which restrictsthevirtualmachineoperationsonthehosttoallowyoutoshutdownrunning virtualmachinesinpreparationforhostshutdown.OnlyESXServer3.0andlater supportsmaintenancemodeforstandalonehosts. Ahostentersorleavesmaintenancemodeonlyastheresultofauserrequest.Ifthehost isinacluster,whenitentersmaintenancemodetheuserisgiventheoptiontoevacuate poweredoffvirtualmachines.Ifthisoptionisselected,eachpoweredoffvirtual machineismigratedtoanotherhost,unlessthereisnocompatiblehostavailableforthe virtualmachineinthecluster.Whileinmaintenancemode,thehostdoesnotallowyou todeployorpoweronavirtualmachine.Virtualmachinesthatarerunningonahost enteringmaintenancemodeneedtobeeithermigratedtoanotherhostorshutdown (eithermanuallyorautomaticallybyDRS). NOTEIfDRScanmakenomigrationrecommendationsforavirtualmachine,anevent isgenerated(checkthevirtualmachinesTasks&Eventstab).Thevirtualmachinemust bemigratedmanuallyorpoweredoffbeforethehostcanentermaintenancemode. Whennomorerunningvirtualmachinesareonthehost,thehostsiconchangesto includeundermaintenanceandthehostsSummarypanelindicatesthenewstate. Thedefaultautomationmodeofavirtualmachinedeterminesitsbehaviorwhenthe host(inaDRScluster)itisrunningonentersmaintenancemode: Anyfullyautomatedvirtualmachineismigratedautomatically. NOTEIfnoappropriatehostisavailable,DRSdisplaysinformationontheTasks &Eventstab. Forapartiallyautomatedormanualvirtualmachine,arecommendationfor furtheruseractionisgeneratedanddisplayed.
VMware, Inc.
71
Standby Mode
Whenahostmachineisplacedinstandbymode,itispoweredoff.Normally,hostsare placedinstandbymodebythedistributedpowermanagementfeature.Youcanalso placeahostinstandbymodemanually;however,DRSmightundo(orrecommend undoing)yourchangethenexttimeitruns.Toforceahosttoremainoff,placeitin maintenancemodeandpoweritoff.Youcanalsodisabledistributedpower management(orsetittomanualmode)onthehosttopreventthehostfrombeing automaticallypoweredon. Ifthedistributedpowermanagementfeaturedeterminesthatthehostneedstobe broughtoutofstandbymode(thatis,tobepoweredbackon),thisisdoneusing WakeonLANtechnology. Toaccommodatethis,youmustensurethatthefollowingstepsaretaken: TheNICthattheVMkernelnetworkingstackisattachedto(selectedasthe VMotionNIC)mustbeWOLcompatible.TodisplaytheWOLcompatibilitystatus foreachNIConahost,selectthehostintheinventorypaneloftheVIClient, choosetheConfigurationtab,andclickNetworkAdapters. TheVMotionnetworkmustbeonasingleIPsubnetpercluster. NOTEHoststhatdonothaveWOLcompatibleNICsareneverselectedforstandby mode.
Understanding VMware HA
TheHAclusterfeatureallowsthevirtualmachinesrunningonESXServersystemsto automaticallyrecoverfromhostfailures.Whenahostbecomesunavailable,all associatedvirtualmachinesareimmediatelyrestartedonotherhostsinthesystem. ThissectionfirstconsidersdifferencesbetweenVMwareHAclustersandtraditional clusteringsolutions,andpresentsHAclusterconcepts.
72
VMware, Inc.
cluster in a box
VMware, Inc.
73
VMware HA Solution
InaVMwareHAsolution,asetofESXServerhostsiscombinedintoaclusterwitha sharedpoolofresources.VirtualCentermonitorsallhostsinthecluster.Ifoneofthe hostsfails,VirtualCenterimmediatelyrespondsbyrestartingeachassociatedvirtual machineonadifferenthost. UsingVMwareHAhasanumberofadvantages: MinimalsetupTheNewClusterWizardisusedforinitialsetup.Youcanadd hostsandnewvirtualmachinesusingtheVIClient.Allvirtualmachinesinthe clustergetfailoversupportwithoutadditionalconfiguration. ReducedhardwarecostandsetupInatraditionalclusteringsolution,duplicate hardwareandsoftwaremustbeconnectedandconfiguredproperly.Thevirtual machineactsasaportablecontainerfortheapplications,andcanbemoved around.Duplicateconfigurationsonmultiplemachinescanbeavoided.Whenyou useVMwareHA,youmusthavesufficientresourcestofailoverthenumberof hostsyouwanttoguarantee.However,theVirtualCenterServertakescareofthe resourcemanagementandclusterconfiguration. IncreasedapplicationavailabilityAnyapplicationrunninginsideavirtual machinehasaccesstoincreasedavailability.Becausethevirtualmachinecan recoverfromhardwarefailure,allapplicationsthataresetuptostartduringthe bootcyclehaveincreasedavailabilityatnoextracosteveniftheapplicationisnot itselfaclusteredapplication. PotentiallimitationsofusingHAclustersincludelossofruntimestateandalonger applicationdowntimethaninatraditionalclusteringenvironmentwithhotstandby.If thoselimitationsbecomeissues,considerusingthetwoapproachestogether.
VMware HA Features
AclusterenabledforHA: SupportseasytouseconfigurationusingtheVIClient. Providesfailoveronhardwarefailureforallrunningvirtualmachineswithinthe boundsoffailovercapacity(seeFailoverCapacityonpage 75). Workswithtraditionalapplicationlevelfailoverandenhancesit. IsfullyintegratedwithDRS.Ifahosthasfailedandvirtualmachineshavebeen restartedonotherhosts,DRScanprovidemigrationrecommendationsormigrate virtualmachinesforbalancedresourceallocation.Ifoneorbothofthesourceand destinationhostsofamigrationfail,HAcanhelprecoverfromthatfailure.
74
VMware, Inc.
Failover Capacity
WhenyouenableaclusterforHA,theNewClusterwizardpromptsyouforthe numberofhostsforwhichyouwantfailovercapacity.Thisnumberisshownasthe ConfiguredFailoverCapacityintheVIClient.HAusesthisnumbertodetermineif thereareenoughresourcestopoweronvirtualmachinesinthecluster. Youneedtospecifyonlythenumberofhostsforwhichyouwantfailovercapacity.HA computestheresourcesitrequirestofailovervirtualmachinesforthatmanyhosts, usingaconservativeestimate,anddisallowspoweringonvirtualmachinesiffailover capacitycannolongerbeguaranteed. NOTE(SEEUPDATE)Youcanallowtheclustertopoweronvirtualmachinesevenwhen theyviolateavailabilityconstraints.Ifyoudothat,theresultisaredcluster,which meansthatfailoverguaranteesmightnolongerbevalid.SeeValid,Yellow,andRed Clustersonpage 81. Afteryouhavecreatedacluster,youcanaddhoststoit.WhenyouaddahosttoanHA clusterthatisnotenabledforDRS,allresourcepoolsareimmediatelyremovedfrom thehost,andallvirtualmachinesbecomedirectlyassociatedwiththecluster. NOTEIftheclusterisalsoenabledforDRS,youcanchoosetokeeptheresourcepool hierarchy.SeeResourcePoolsandClustersonpage 56.
VMware, Inc.
75
Forexample,ifyouhaveavirtualmachinewith1GHzCPUreservationand1GB memoryreservationandanothervirtualmachinewith2GHzCPUreservationand 512MBmemoryreservation,theslotisdefinedas2GHzCPUreservationand1GB memoryreservation.HAdetermineshowmanyslotscanfitintoeachhostbasedon thehostsCPUandmemorycapacity.HAthendetermineshowmanyhostscouldfail withtheclusterstillhavingatleastasmanyslotsaspoweredonvirtualmachines.This numberisthecurrentfailoverlevel. Duringplanning,decideonthenumberofhostsforwhichyouwanttoguarantee failover.HAtriestoreserveresourcesforatleastasmanyhostfailuresbylimitingthe numberofvirtualmachinesthatarepoweredon,whichwillconsumetheseresources. IfyoulefttheAllowvirtualmachinetobestartedeveniftheyviolateavailability constraintsoptionunselected(strictadmissioncontrol),VMwareHAdoesnotallow youtopoweronvirtualmachinesiftheywouldcausethecurrentfailoverlevelto becomelessthantheconfiguredfailoverlevel.VMwareHAalsodoesnotallowthe followingoperationsiftheywouldcausethecurrentfailoverleveltoexceedthe configuredfailoverlevel: Revertingapoweredoffvirtualmachinetoapoweredonsnapshot. Migratingarunningvirtualmachineintothecluster. ReconfiguringarunningvirtualmachinetoincreaseitsCPUormemory reservation. IfyouselectedtheAllowvirtualmachinetobestartedeveniftheyviolateavailability constraintsoptionwhenyouenableHA,youcanpoweronmorevirtualmachinesthan HAwouldadvise.Becauseyouconfiguredthesystemtopermitthis,theclusterdoes notturnred.Thecurrent(available)failoverlevelcanalsofallbelowtheconfigured failoverlevelifthenumberofhoststhathavefailedexceedstheconfigurednumber.For example,ifyouhaveconfiguredtheclusterforaonehostfailure,youareatcapacity (thecurrentfailoverlevelisequaltotheconfiguredfailoverlevel),andtwohostsfail, theclusterturnsred. Aclusterthatisbeneaththeconfiguredfailoverlevelcanstillperformvirtualmachine failoverincaseofhostfailure,usingvirtualmachineprioritytodeterminewhich virtualmachinestopoweronfirst.SeeCustomizingHAforVirtualMachineson page 115. CAUTIONItisnotrecommendedthatyouworkwithredclusters.Ifyoudo,failoveris notguaranteedatthespecifiedlevel.
76
VMware, Inc.
VMware, Inc.
77
78
VMware, Inc.
Thedifferencebetweenadisconnectedhostandahostthatisnotrespondingis describedinthelist. Adisconnectedhosthasbeenexplicitlydisconnectedbytheuser.Aspartof disconnectingahost,VirtualCenterdisablesHAonthathost.Thevirtualmachines onthathostarenotfailedoverandnotconsideredwhenVirtualCentercomputes thecurrentfailoverlevel. Ifahostisnotresponding,theVirtualCenterServernolongerreceivesheartbeats fromit.Thismighthappen,forexample,becauseofanetworkproblem,because thehostfailed,orbecausetheVirtualCenteragentfailed. VirtualCenterdoesnotincludesuchhostswhencomputingthecurrentfailover level,butassumesthatanyvirtualmachinesrunningonanonrespondinghostwill befailedoverifthehostfails.Thevirtualmachinesonahostthatisnotresponding affecttheadmissioncontrolcheck.
VMware, Inc.
79
80
VMware, Inc.
Valid Cluster
Avalidclusterhasenoughresourcestomeetallreservationsandtosupportallrunning virtualmachines.Aclusterisvalidunlesssomethinghappensthatmakesit overcommittedorinvalid. ADRSclustercanbecomeovercommittedifahostfails. ADRSclustercanbecomeinvalidifVirtualCenterisunavailableandyoupower onvirtualmachinesusingaVIClientconnecteddirectlytoanESXServerhost. AnHAclusterbecomesinvalidifthecurrentfailovercapacityislowerthanthe configuredfailovercapacityorifalltheprimaryhostsintheclusterarenot responding.SeePrimaryandSecondaryHostsonpage 78. ADRSorHAclustercanbecomeinvalidiftheuserreducesthereservationona parentresourcepoolwhileavirtualmachineisintheprocessoffailingover. Beforeconsideringthefollowingexamples,notethedefinitionoftheseterms: Reservation(foraresourcepool)Afixed,guaranteedallocationfortheresource poolinputbytheuser. ReservationUsed(foraclusterorresourcepool)Thesumofthereservationor reservationused(whicheverislarger)foreachchild,addedrecursively. Unreserved(foraclusterorresourcepool)Anonnegativenumberthatalso differsaccordingtoresourcepooltype: Foraclusteritequalstotalcapacityreservationused. Fornonexpandableresourcepoolsitequalsreservationreservationused. Forexpandableresourcepoolsitisequalto(reservationreservationused)+ anyunreservedresourcesthatcanbeborrowedfromitsancestorresource pools.
VMware, Inc.
81
VM1, 2G VM7, 2G
VM6, 2G
VM2, 2G VM4, 1G
VM3, 3G VM8, 2G
VM5, 2G
82
VMware, Inc.
VM1, 2G VM7, 2G
VM6, 2G
VM2, 2G VM4, 1G
VM3, 3G VM8, 2G
VM5, 2G
VMware, Inc.
83
Yellow Cluster
Aclusterbecomesyellowwhenthetreeofresourcepoolsandvirtualmachinesis internallyconsistentbutthereisnotenoughcapacityintheclustertosupportall resourcesreservedbythechildresourcepools.Therewillalwaysbeenoughresources tosupportallrunningvirtualmachinesbecause,whenahostbecomesunavailable,all itsvirtualmachinesbecomeunavailable. Aclustertypicallyturnsyellowwhenclustercapacityissuddenlyreduced,for example,whenahostintheclusterbecomesunavailable.VMwarerecommendsthat youleaveadequateadditionalclusterresourcestoavoidyourclusterturningyellow. Considerthefollowingexample,asshowninFigure 412: Aclusterwithtotalresourcesof12GHzcomingfromthreehostsof4GHzeach. Threeresourcepoolsreservingatotalof12GHz. Thetotalreservationusedbythethreeresourcepoolscombinedis12GHz(4+5+3 GHz).ThatshowsupastheReservationUsedinthecluster. Oneofthe4GHzhostsbecomesunavailable,sototalresourcesreduceto8GHz. Atthesametime,VM4(1GHz)andVM3(3GHz),whichwererunningonthehost thatfailed,arenolongerrunning. Theclusterisnowrunningvirtualmachinesthatrequireatotalof6GHz.The clusterstillhas8GHzavailable,whichissufficienttomeetvirtualmachine requirements. Theresourcepoolreservationsof12GHzcannolongerbemet,sotheclusteris markedasyellow.
84
VMware, Inc.
VM1, 2G
VM6, 2G
VM2, 2G VM4, 1G
VM3, 3G VM7, 0G
VM5, 5G
Red Cluster
AclustercanbecomeredbecauseofaDRSviolationoranHAviolation.Thebehavior oftheclusterdependsonthetypeofviolation,asdiscussedinthissection.
VMware, Inc.
85
Aclustercanalsoturnredifyoureconfigurearesourcepoolwhileavirtualmachineis intheprocessoffailingover.Avirtualmachinethatisintheprocessoffailingoveris disconnectedanddoesnotcounttowardthereservationusedbytheparentresource pool.Itispossiblethatyoureducethereservationoftheparentresourcepoolbeforethe failovercompletes.Afterthefailoveriscomplete,thevirtualmachineresourcesare againchargedtotheparentresourcepool.Ifthepoolsusagebecomeslargerthanthe newreservation,theclusterturnsred. ConsidertheexampleinFigure 413. Figure 4-13. Red Cluster
VM1, 1G
VM2, 3G
VM3, 1G
VM4, 1G VM7, 3G
VM5, 1G
VM6, 1G
Red HA Cluster
AclusterenabledforHAbecomesredwhenthenumberofvirtualmachinespowered onexceedsthefailoverrequirements,thatis,thecurrentfailovercapacityissmaller thanconfiguredfailovercapacity.Ifstrictadmissioncontrolisdisabled,clustersdonot becomered,regardlessofwhetherthehostscanguaranteefailover.
86
VMware, Inc.
Inadequatefailovercapacitycanhappen,forexample,ifyoupoweronsomanyvirtual machinesthattheclusternolongerhassufficientresourcestoguaranteefailoverforthe specifiednumberofhosts. ItcanalsohappenifHAissetupfortwohostfailureinafourhostclusterandonehost fails.Theremainingthreehostsmightnolongerbeabletosatisfyatwohostfailure. IfaclusterenabledforHAbecomesredorifcurrentfailovercapacityfallsbelowthe configuredfailovercapacity,itcannolongerguaranteefailoverforthespecified numberofhostsbutcontinuestoperformfailover.Incaseofhostfailure,HAfirstfails overthevirtualmachinesofonehostinorderofpriority,andthenthevirtualmachines ofthesecondhostinorderofpriority,andsoon.SeeCustomizingHAforVirtual Machinesonpage 115. TheSummarypagedisplaysalistofconfigurationissuesforredandyellowclusters. Thelistexplainswhatcausestheclustertobecomeovercommittedorinvalid. NOTEDRSbehaviorisnotaffectedifaclusterisredbecauseofanHAissue.
VMware, Inc.
87
88
VMware, Inc.
Cluster Prerequisites
YoursystemmustmeetcertainprerequisitestouseVMwareclusterfeatures successfully. Ingeneral,DRSandHAworkbestifthevirtualmachinesmeetVMotion requirements,asdiscussedinthenextsection. IfyouwanttouseDRSforloadbalancing,thehostsinyourclustermustbepartof aVMotionnetwork.IfthehostsarenotintheVMotionnetwork,DRScanstill makeinitialplacementrecommendations.
VMware, Inc.
89
NOTEAllhostsinanHAclustermusthaveDNSconfiguredsothattheshorthost name(withoutthedomainsuffix)ofanyhostintheclustercanberesolvedtothe appropriateIPaddressfromanyotherhostinthecluster.Otherwise,the Configuring HAtaskfails.IfyouaddthehostusingtheIPaddress,alsoenable reverseDNSlookup(theIPaddressshouldberesolvabletotheshorthostname). ForVMwareHAusagewithinESXServer3,redundantconsolenetworkingis recommended(thoughnotrequired).Similarly,withESXServer3i,redundant VMkernelnetworkingisrecommended.Ifredundancyisnotprovided,thereisasingle pointoffailureinyourfailoversetup.Whenahostsnetworkconnectionfails,the secondconnectioncansendheartbeatstootherhosts. Tosetupredundancy,youneedtwophysicalnetworkadaptersoneachhost.You connectthemtothecorrespondingserviceconsole(orVMkernelnetwork,in ESXServer3i),usingtwoserviceconsoleinterfaces(VMkernelnetworkinterfacesin ESXServer3i)orusingasingleinterfaceusingNICteaming. NOTEAfteryouhaveaddedaNICtoahostinyourHAcluster,youmustreconfigure HAonthathost.
90
VMware, Inc.
Shared Storage
Ensurethatthemanagedhostsusesharedstorage.Sharedstorageistypicallyona storageareanetwork(SAN).(SEEUPDATE)SeetheiSCSISANConfigurationGuideand theFibreChannelSANConfigurationGuideforadditionalinformationonSANandthe ESXServerConfigurationGuideforinformationonothersharedstorage.
Processor Compatibility
Ensurethatthesourceanddestinationhostshaveacompatiblesetofprocessors. VMotiontransferstherunningarchitecturalstateofavirtualmachinebetween underlyingVMwareESXServersystems.VMotioncompatibilitymeansthatthe processorsofthedestinationhostmustbeabletoresumeexecutionusingthe equivalentinstructionswheretheprocessorsofthesourcehostweresuspended. Processorclockspeedsandcachesizesmightvary,butprocessorsmustcomefromthe samevendorclass(IntelversusAMD)andsameprocessorfamilytobecompatiblefor migrationwithVMotion.
VMware, Inc.
91
ProcessorfamiliessuchasXeonMPandOpteronaredefinedbytheprocessorvendors. Youcandistinguishdifferentprocessorversionswithinthesamefamilybycomparing theprocessorsmodel,steppinglevel,andextendedfeatures. Inmostcases,differentprocessorversionswithinthesamefamilyaresimilar enoughtomaintaincompatibility. Insomecases,processorvendorshaveintroducedsignificantarchitectural changeswithinthesameprocessorfamily(suchas64bitextensionsandSSE3). VMwareidentifiestheseexceptionsifitcannotguaranteesuccessfulmigration withVMotion. NOTEVMwareisworkingtomaintainVMotioncompatibilityacrossthewidestrange ofprocessorsthroughpartnershipswithprocessorandhardwarevendors.Forcurrent information,contactyourVMwarerepresentative.
Other Requirements
OtherVMotionrequirementsyoumustobserve: ForESXServer3.x,thevirtualmachineconfigurationfileforESXServerhostsmust resideonaVMFS. VMotiondoesnotsupportrawdisksormigrationofapplicationsclusteredusing MicrosoftClusterService(MSCS). VMotionrequiresaprivateGigabitEthernetmigrationnetworkbetweenallofthe VMotionenabledmanagedhosts.WhenVMotionisenabledonamanagedhost, configureauniquenetworkidentityobjectforthemanagedhostandconnectitto theprivatemigrationnetwork.
Creating a Cluster
ThissectiondiscusseseachofthepagesintheNewClusterwizard.
VMware, Inc.
93
Selecting HA Options
IfyouenabledHA,theNewClusterwizardallowsyoutosettheoptionslistedin Table 52.SeeWorkingwithVMwareHAonpage 121. Table 5-2. VMware HA Options
Option HostFailures RestartPriority Description Specifiesthenumberofhostfailuresforwhichyouwanttoguarantee failover. Determinestheorderinwhichvirtualmachinesarerestarteduponhost failure.Valuesare:Disabled,Low,Medium,High.Thedefaultis Medium.IfDisabledisselected,HAisdisabledforthevirtual machines. SeeRestartpriority.onpage 115.
94
VMware, Inc.
AdmissionControl
VMware, Inc.
95
Summary Page
TheclusterSummarypagedisplayssummaryinformationforthecluster.See Figure 51. Figure 5-1. Cluster Summary Tab
96
VMware, Inc.
VMware, Inc.
97
98
VMware, Inc.
VMware, Inc.
99
100
VMware, Inc.
ThischapterexplainshowtoaddhoststoaDRScluster,howtoremovethem,andhow tocustomizeDRS. Thischapterdiscussesthefollowingtopics: CustomizingDRSonpage 101 AddingHoststoaDRSClusteronpage 102 RemovingHostsfromClustersonpage 104 ApplyingDRSRecommendationsonpage 105 ReconfiguringDRSonpage 107 UsingDRSAffinityRulesonpage 108 NOTEAlltasksdescribedassumeyouarelicensedandyouhavepermissionto performthem.SeetheonlineHelpforinformationonpermissionsandhowtosetthem.
Customizing DRS
Afteryouhavecreatedacluster,youcanenableitforDRS,HA,orboth.Youcan proceedtoaddorremovehosts,andcustomizetheclusterinotherways. YoucancustomizeDRSasfollows: Specifythedefaultautomationlevelandmigrationthresholdduringcluster creation.SeeSelectingAutomationLevelonpage 93. Addhoststothecluster.SeeAddingHoststoaDRSClusteronpage 102
VMware, Inc.
101
Changethedefaultautomationlevelormigrationthresholdforexistingclusters, asdiscussedinReconfiguringDRSonpage 107. Setcustomautomationmodesforindividualvirtualmachinesinyourclusterto overridetheclusterwidesettings.Forexample,youcansettheclustersdefault automationleveltoautomaticbutthemodeofsomeindividualmachinesto manual.SeeCustomizingDRSforVirtualMachinesonpage 114. Groupvirtualmachinesbyusingaffinityrules.Affinityrulesspecifythatselected virtualmachineshouldalwaysbeplacedonthesamehost.Antiaffinityrules specifythatselectedvirtualmachinesshouldalwaysbeplacedondifferenthosts. SeeUsingDRSAffinityRulesonpage 108.
102
VMware, Inc.
IfyouchooseCreateanewresourcepoolforthishostsvirtualmachinesand resourcepools,VirtualCentercreatesatoplevelresourcepoolthatbecomesa directchildoftheclusterandaddsallchildrenofthehosttothatnewresource pool.Youcansupplyanameforthatnewtoplevelresourcepool.Thedefault isGraftedfrom<host_name>. NOTEIfthehosthasnochildresourcepoolsorvirtualmachines,thehostsresources areaddedtotheclusterbutnoresourcepoolhierarchywithatoplevelresourcepool iscreated. Totakeadvantageofautomaticmigrationfeatures,youmustalsosetupthehosts VMotionnetwork. NOTEWhenyoulaterremovethehostfromthecluster,theresourcepoolhierarchy remainspartofthecluster.Thehostlosestheresourcepoolhierarchy.Thislossmakes sensebecauseoneofthegoalsofresourcepoolsistosupporthostindependent resourceallocation.Youcan,forexample,removetwohostsandreplacethemwitha singlehostwithsimilarcapabilitieswithoutadditionalreconfiguration.
VMware, Inc.
103
104
VMware, Inc.
VMware, Inc.
105
Recommendation Grouping
Therecommendationspageisorganizedintoboxes.Eachboxcontains recommendationsthatsharesomedegreeofinterdependence,whilerecommendations thatappearindifferentboxesareconsideredindependentfromoneanother.Withina box,recommendationsthataredependentonotherrecommendationsareplacedbelow theirprerequisites.Theseinterdependenciesleadtothefollowingactionswhenapplying recommendations: Whenadependentrecommendationisselected(theApplycheckboxisselected), anyrecommendationsappearingaboveitinthesameboxthatareitsprerequisite arealsoselected.Thedependentrecommendationcannotbeappliedwithout them.Notallrecommendationsaboveareprerequisites. Whenaprerequisiterecommendationisdeselected,alloftherecommendations thatappearbelowitinthesameboxanddependonitarealsodeselected.Ifthe prerequisiteisnotapplied,theywillnotbeappliedeither.Notall recommendationsbelowaredependent. Anothertypeofinterdependenceamongstrecommendationsthatisdisplayedonthis pageiswhenactionsareatomicandcanonlybeappliedasasingleunit.Thesetypesof recommendationsmightbenecessarytosatisfyanaffinity(orantiaffinity)ruleand theyareindicatedbyalinkiconandasingleApplycheckbox.
106
VMware, Inc.
Reconfiguring DRS
YoucanturnoffDRSforacluster,oryoucanchangetheconfigurationoptions. To turn off DRS 1 2 3 Selectthecluster. ChooseEditSettingsfromtherightclickmenu. Intheleftpanel,selectGeneral,anddeselecttheVMwareDRScheckbox. YouarewarnedthatturningoffDRSdestroysallresourcepoolsinthecluster. 4 ClickOKtoturnoffDRSanddestroyallresourcepools.
VMware, Inc.
107
NOTETheAdvancedOptionsdialogboxishelpfulwhenyouareworkingwith VMwarecustomersupportonresolvinganissue.Settingadvancedoptionsisnot otherwiserecommended. TosuspendDRSrecommendedVMotionmigrationswithoutchangingtheresource poolhierarchy,settheDRSautomationleveltomanualorpartiallyautomatic.When thisisdone,DRScontinuestorecommendVMotionmigrations,butVirtualCenterdoes notexecutethemwithoutuserapproval.Ifyouhavesettheautomationlevelof individualvirtualmachinestofullyautomatic,returnthemtotheclusterdefault.See CustomizingDRSforVirtualMachinesonpage 114forinstructions.
108
VMware, Inc.
3 4
ClickAddtoaddvirtualmachines,andclickOKwhenyouaredone.
VMware, Inc.
109
110
VMware, Inc.
Thischapterexplainshowtoadd,remove,andcustomizevirtualmachines. Thischapterdiscussesthefollowingtopics: AddingVirtualMachinestoaClusteronpage 111 PoweringOnVirtualMachinesinaClusteronpage 112 RemovingVirtualMachinesfromaClusteronpage 113 CustomizingDRSforVirtualMachinesonpage 114 CustomizingHAforVirtualMachinesonpage 115 NOTEAlltasksassumeyouhavepermissiontoperformthem.SeetheonlineHelpfor informationonpermissionsandhowtosetthem.
DRS Enabled
IfyoupoweronavirtualmachineorgroupofvirtualmachinesandDRSisenabled, VirtualCenterfirstperformsadmissioncontrol.Itcheckswhethertheclusterand resourcepoolhasenoughresourcesforthevirtualmachine(s).Iftheclusterdoesnot havesufficientresourcestopoweronasinglevirtualmachine,oranyofthevirtual machinesinagrouppoweronattempt,amessageappears.
112
VMware, Inc.
HA Enabled
IfyoupoweronavirtualmachineandHAisenabled,VirtualCenterperformsHA admissioncontrol.Itcheckswhetherenoughresourcesexisttoallowforthespecified numberofhostfailoversifyoupoweronthevirtualmachine. Ifenoughresourcesexist,thevirtualmachineispoweredon. Ifnotenoughresourcesexist,andifstrictadmissioncontrolisused(thedefault), amessageinformsyouthatthevirtualmachinecannotbepoweredon.Ifyouare notusingstrictadmissioncontrol,thevirtualmachineispoweredonwithout warnings.
VMware, Inc.
113
3 4
114
VMware, Inc.
VMware, Inc.
115
Foreachvirtualmachine,selectfromtheRestartPriorityorIsolationResponse menutocustomizeitssettings. RestartPriorityIndicatesrelativepriorityforrestartingthevirtualmachine incaseofhostfailure.Higherpriorityvirtualmachinesarestartedfirst. NOTEThispriorityappliesonlyonaperhostbasis.Ifmultiplehostsfail, VirtualCenterfirstmigratesallvirtualmachinesfromthefirsthostinorderof priority,andthenallvirtualmachinesfromthesecondhostinorderof priority,andsoon. IsolationResponseSpecifieswhattheESXServerhostthathaslost connectionwithitsclustershoulddowithrunningvirtualmachines.By default,eachvirtualmachineissettobeshutdownintheeventofahost isolationincident. YoucanchooseLeavepoweredontoindicatethatthevirtualmachineson isolatedhostsshouldcontinuetorunevenifthehostcannolonger communicatewithotherhostsinthecluster.Youmightdothis,forexample, ifthevirtualmachinenetworkisonadifferentnetworkthatisrobustand redundant,orifyouwanttokeepthevirtualmachinesrunning.
116
VMware, Inc.
Managing VMware HA
ThischapterexplainshowtoaddhoststoanHAcluster,howtoremovethem,andhow tocustomizeHAclusters. Thischapterdiscussesthefollowingtopics: CustomizingHAonpage 117 AddingHoststoanHAClusteronpage 118 WorkingwithVMwareHAonpage 121 SettingAdvancedHAOptionsonpage 121 NOTEAlltasksdescribedassumeyouarelicensedandyouhavepermissionto performthem.SeetheonlineHelpforinformationonpermissions.
Customizing HA
Afteryoucreateacluster,youcanenableitforDRS,HA,orboth.Youcanthenaddor removehosts,andcustomizethecluster. YoucancustomizeHAasfollows: Duringclustercreation,acceptorchangetheclustersdefaultrestartpriorityand isolationresponse,choosethenumberofhostfailuresfortheclusterandindicate whetheryouwanttoenforcestrictadmissioncontrol.SeeSelectingHAOptions onpage 94. Addhosts,asdiscussedinAddingHoststoanHAClusteronpage 118.
VMware, Inc.
117
Changethenumberofhostfailuresortheadmissioncontrolforexistingclusters, asdiscussedinWorkingwithVMwareHAonpage 121. Setapriorityforindividualvirtualmachines.HAusesvirtualmachinepriorityto decidetheorderofrestartsothatvirtualmachineswithhigherpriorityfromthe samehostgetprecedenceincaseofinsufficientresources.SeeCustomizingHA forVirtualMachinesonpage 115. SetanIsolationResponseforindividualvirtualmachines.Bydefault,allvirtual machinesarepoweredoffifahostbecomesisolatedfromthenetwork.See CustomizingHAforVirtualMachinesonpage 115.
118
VMware, Inc.
VMware, Inc.
119
120
VMware, Inc.
VMware, Inc.
121
das.usedefaultisolationaddress
das.defaultfailoverhost
das.failuredetectioninverval das.vmMemoryMinMB
das.vmCpuMinMHz
122
VMware, Inc.
Thischapterdiscussessomeadvancedresourcemanagementtopics.Itincludes conceptualinformationandadiscussionoftheadvancedparametersyoucanset.In mostsituations,youdonotneedtousetheadvancedsettingsandusingtheadvanced settingsincorrectlymightbedetrimentaltoyoursystemsperformance.However, experiencedadministratorsmightfindtheseadvancedconfigurationoptionshelpful forfinetuningtheperformanceoftheESXServerenvironment. NOTELicensesforDRSandHAarenotrequiredforanyofthetopicsandtasks discussedinthischapter. Thischapterdiscussesthefollowingtopics: CPUVirtualizationonpage 124 UsingCPUAffinitytoAssignVirtualMachinestoSpecificProcessorson page 126 MulticoreProcessorsonpage 128 Hyperthreadingonpage 129 MemoryVirtualizationonpage 133 UnderstandingMemoryOverheadonpage 136 MemoryAllocationandIdleMemoryTaxonpage 138 HowESXServerHostsReclaimMemoryonpage 140 SharingMemoryAcrossVirtualMachinesonpage 144 AdvancedAttributesandWhatTheyDoonpage 145
VMware, Inc.
123
CPU Virtualization
TounderstandCPUrelatedissues,thedifferencebetweenemulationand virtualization. Withemulation,alloperationsareexecutedinsoftwarebyanemulator.Asoftware emulatorallowsprogramstorunonacomputersystemotherthantheoneforwhich theywereoriginallywritten.Theemulatordoesthisbyemulating,orreproducing,the originalcomputersbehaviorbyacceptingthesamedataorinputsandachievingthe sameresults.Emulationprovidesportabilityandisoftenusedtorunsoftwaredesigned foroneplatformacrossseveraldifferentplatforms. Withvirtualization,theunderlyingphysicalresourcesareusedwheneverpossibleand thevirtualizationlayerexecutesinstructionsonlyasneededtomakethevirtual machinesoperateasiftheywererunningdirectlyonaphysicalmachine.Virtualization emphasizesperformanceandrunsdirectlyontheprocessorwheneverpossible.
124
VMware, Inc.
Performance Implications
CPUvirtualizationaddsvaryingamountsofoverheaddependingontheworkloadand thetypeofvirtualizationused. AnapplicationisCPUboundifmostoftheapplicationstimeisspentexecuting instructionsratherthanwaitingforexternaleventssuchasuserinteraction,device input,ordataretrieval.Forsuchapplications,theCPUvirtualizationoverheadrequires additionalinstructionstobeexecuted,whichtakesCPUprocessingtimethatcouldbe usedbytheapplicationitself.CPUvirtualizationoverheadusuallytranslatesintoa reductioninoverallperformance. ForapplicationsthatarenotCPUbound,CPUvirtualizationlikelytranslatesintoan increaseinCPUutilization.IfspareCPUcapacityisavailabletoabsorbtheoverhead, itcanstilldelivercomparableperformanceintermsofoverallthroughput. ESXServer3supportsuptofourvirtualprocessors(CPUs)foreachvirtualmachine. NOTEDeploysinglethreadedapplicationsonuniprocessorvirtualmachines(instead ofSMPvirtualmachines)forbestperformanceandresourceutilization. SinglethreadedapplicationscantakeadvantageonlyofasingleCPU.Deployingsuch applicationsindualprocessorvirtualmachinesdoesnotspeeduptheapplication. Instead,itcausesthesecondvirtualCPUtousephysicalresourcesthatcouldotherwise beusedbyothervirtualmachines.
VMware, Inc.
125
126
VMware, Inc.
ClicktheRunonprocessor(s)button.
SelecttheprocessorsonwhichyouwantthevirtualmachinetorunandclickOK.
Potential Issues with Affinity Virtualmachineaffinityassignseachvirtualmachine toprocessorsinthespecifiedaffinityset.Beforeusingaffinity,considerthefollowing issues: Formultiprocessorsystems,ESXServersystemsperformautomaticload balancing.Avoidmanualspecificationofvirtualmachineaffinitytoimprovethe schedulersabilitytobalanceloadacrossprocessors. AffinitycaninterferewiththeESXServerhostsabilitytomeetthereservationand sharesspecifiedforavirtualmachine. BecauseCPUadmissioncontroldoesnotconsideraffinity,avirtualmachinewith manualaffinitysettingsmightnotalwaysreceiveitsfullreservation. Virtualmachinesthatdonothavemanualaffinitysettingsarenotadversely affectedbyvirtualmachineswithmanualaffinitysettings. Whenyoumoveavirtualmachinefromonehosttoanother,affinitymightno longerapplybecausethenewhostmighthaveadifferentnumberofprocessors. TheNUMAschedulermightnotbeabletomanageavirtualmachinethatsalready assignedtocertainprocessorsusingaffinity.SeeChapter 10,UsingNUMA SystemswithESXServer,onpage 151foradditionalinformationonusing NUMAwithESXServerhosts. AffinitycanaffectanESXServerhostsabilitytoschedulevirtualmachineson multicoreorhyperthreadedprocessorstotakefulladvantageofresourcesshared onsuchprocessors.
VMware, Inc.
127
Multicore Processors
IntelandAMDhaveeachdevelopedprocessorswhichcombinetwoormoreprocessor coresintoasingleintegratedcircuit(oftencalledapackageorsocket).VMwareusesthe termphysicalprocessororsockettodescribeasinglepackagewhichcanhaveoneormore processorcoreswithoneormorelogicalprocessorsinsideeachcore.Multicore processorsprovidemanyadvantagesforanESXServerhostperformingmultitasking ofvirtualmachines. Adualcoreprocessor,forexample,canprovidealmostdoubledperformance, comparedwithasinglecoreprocessor,byallowingtwovirtualmachinestobe executedatthesametime.Eachcorecanhaveitsownmemorycaches,orcanshare someofitscacheswithothercores,potentiallyreducingtherateofcachemissesand theneedtoaccessslowermainmemory.Asharedmemorybusthatconnectsaphysical processortomainmemorycanlimitperformanceofitslogicalprocessorsifthevirtual machinesrunningonthemarerunningmemoryintensiveworkloadswhichcompete forthesamememorybusresources. EachlogicalprocessorofeachprocessorcorecanbeusedindependentlybytheESX ServerCPUschedulertoexecutevirtualmachines,providingcapabilitiessimilarto traditionalsymmetricmultiprocessing(SMP)systems.Forexample,atwowayvirtual machinecanhaveitsvirtualprocessorsrunningonlogicalprocessorsthatbelongtothe samecore,oronlogicalprocessorsondifferentphysicalprocessors. Table 91providesalistofprocessorsandtheirattributes.
Cores 1 1 1 2 2 2
Threads/Core 1 1 2 1 2 1
Logical Processors 1 1 2 2 4 2
VMware, Inc.
Hyperthreading
IntelCorporationdevelopedhyperthreadingtechnologytoenhancetheperformanceof itsPentiumIVandXeonprocessorlines.Thetechnologyallowsasingleprocessorcore toexecutetwoindependentthreadssimultaneously.Whilethisfeaturedoesnot providetheperformanceofatruedualprocessorsystem,itcanimproveutilizationof onchipresources,leadingtogreaterthroughputforcertainimportantworkloadtypes. SeetheIntelWebsiteforanindepthdiscussionofhyperthreadingtechnology. Foradditionalinformation,seethewhitepaperHyperThreadingSupportinESX Server2,whichisavailableattheVMwareWebsite. Hyperthreadingtechnologyallowsasinglephysicalprocessorcoretobehaveliketwo logicalprocessors.Theprocessorcanruntwoindependentapplicationsatthesame time.Toavoidconfusionbetweenlogicalandphysicalprocessors,Intelreferstoa physicalprocessorasasocket,andthediscussioninthischapterusesthatterminology aswell. Whilehyperthreadingdoesnotdoubletheperformanceofasystem,itcanincrease performancebybetterutilizingidleresources.Anapplicationrunningononelogical processorofabusycorecanexpectslightlymorethanhalfofthethroughputthatit obtainswhilerunningaloneonanonhyperthreadedprocessor.However, hyperthreadingperformanceimprovementsarehighlyapplicationdependent,and someapplicationsmightseeperformancedegradationwithhyperthreadingbecause manyprocessorresources(suchasthecache)aresharedbetweenbothlogical processors.
VMware, Inc.
129
Enabling Hyperthreading
Bydefault,hyperthreadingisenabled.Ifitisdisabled,youcanenableit. AllIntelXeonMPprocessorsandallIntelXeonDPprocessorswith512KL2cache supporthyperthreading;however,noteveryIntelXeonsystemshipswithaBIOSthat supportshyperthreading.ConsultyoursystemdocumentationtoseeiftheBIOS includessupportforhyperthreading.VMwareESXServercannotenable hyperthreadingonasystemwithmorethan16physicalCPUs,becauseESXServerhas alogicallimitof32CPUs. To enable hyperthreading 1 2 Ensurethatyoursystemsupportshyperthreadingtechnology. EnablehyperthreadinginthesystemBIOS. SomemanufacturerslabelthisoptionLogicalProcessorwhileotherscallitEnable Hyperthreading. MakesurehyperthreadingforyourESXServerhostisturnedon. a b c IntheVIClient,selectthehostandclicktheConfigurationtab. SelectProcessorsandclickProperties. Inthedialogbox,youcanviewhyperthreadingstatusandturn hyperthreadingofforon(default).
130
VMware, Inc.
VMwareESXServersystemsmanageprocessortimeintelligentlytoguaranteethat loadisspreadsmoothlyacrossprocessorcoresinthesystem.Virtualmachinesare preferentiallyscheduledontwodifferentcoresratherthanontwologicalprocessorson thesamecore. Ifthereisnoworkforalogicalprocessor,itisputintoahaltedstate,whichfreesits executionresourcesandallowsthevirtualmachinerunningontheotherlogical processoronthesamecoretousethefullexecutionresourcesofthecore.TheVMware schedulerproperlyaccountsforthishalttime,sothatavirtualmachinerunningwith thefullresourcesofacoreischargedmorethanavirtualmachinerunningonahalf core.Thisapproachtoprocessormanagementensuresthattheserverdoesnotviolate anyofthestandardESXServerresourceallocationrules.
VMware, Inc.
131
Youhavethefollowingchoices.
Option Any Description Thedefaultforallvirtualmachinesonahyperthreadedsystem.Thevirtual CPUsofavirtualmachinewiththissettingcanfreelysharecoreswithother virtualCPUsfromthisoranyothervirtualmachineatanytime. VirtualCPUsofavirtualmachineshouldnotsharecoreswitheachotherorwith virtualCPUsfromothervirtualmachines.Thatis,eachvirtualCPUfromthis virtualmachineshouldalwaysgetawholecoretoitself,withtheotherlogical CPUonthatcorebeingplacedintothehaltedstate. Thisoptionissimilartonone.VirtualCPUsfromthisvirtualmachinearenot allowedtosharecoreswithvirtualCPUsfromothervirtualmachines.Theycan sharecoreswiththeothervirtualCPUsfromthesamevirtualmachine. ThisoptionispermittedonlyforSMPvirtualmachines.Ifappliedtoa uniprocessorvirtualmachine,thesystemchangesthisoptiontonone.
None
Internal
TheseoptionshavenoeffectonfairnessorCPUtimeallocation.Regardlessofavirtual machineshyperthreadingsettings,itstillreceivesCPUtimeproportionaltoitsCPU shares,andconstrainedbyitsCPUreservationandCPUlimitvalues. Fortypicalworkloads,customhyperthreadingsettingsshouldnotbenecessary.The optionscanhelpincaseofunusualworkloadsthatinteractbadlywithhyperthreading. Forexample,anapplicationwithcachethrashingproblemsmightslowdownan applicationsharingitsphysicalcore.Youcanplacethevirtualmachinerunningthe applicationinthenoneorinternalhyperthreadingstatustoisolateitfromothervirtual machines. IfavirtualCPUhashyperthreadingconstraintsthatdonotallowittoshareacorewith anothervirtualCPU,thesystemmightdescheduleitwhenothervirtualCPUsare entitledtoconsumeprocessortime.Withoutthehyperthreadingconstraints,both virtualCPUscouldhavebeenscheduledonthesamecore. Theproblembecomesworseonsystemswithalimitednumberofcores(pervirtual machine).Insuchcases,theremightbenocoretowhichthevirtualmachinethatis descheduledcanbemigrated.Asaresult,itispossiblethatvirtualmachineswith hyperthreadingsettononeorinternalcanexperienceperformancedegradation, especiallyonsystemswithalimitednumberofcores.
132
VMware, Inc.
Quarantining
Incertain,rarecircumstances,anESXServersystemmightdetectthatanapplicationis interactingbadlywithhyperthreadingtechnology.Certaintypesofselfmodifying code,forexample,candisruptthenormalbehaviorofthePentiumIVtracecacheand leadtosubstantialslowdowns(upto90percent)foranapplicationsharingacorewith theproblematiccode.Inthosecases,theESXServerhostquarantinesthevirtualCPU runningthiscodeandplacesitsvirtualmachineinthenoneorinternalmode,as appropriate.Quarantiningisnecessaryonlyrarelyandistransparenttotheuser. SettheCpu.MachineClearThresholdadvancedsettingforthehostto0todisable quarantining.SeeSettingAdvancedHostAttributesonpage 145.
Memory Virtualization
Allmodernoperatingsystemsprovidesupportforvirtualmemory,allowingsoftware tousemorememorythanthemachinephysicallyhas.Thevirtualmemoryspaceis dividedintoblocks,typically4KB,calledpages.Thephysicalmemoryisalsodivided intoblocks,alsotypically4KB.Whenphysicalmemoryisfull,thedataforvirtualpages thatarenotpresentinphysicalmemoryarestoredondisk.
VMware, Inc.
133
Eachvirtualmachineseesacontiguous,zerobased,addressablephysicalmemory space.Theunderlyingmachinememoryontheserverusedbyeachvirtual machineisnotnecessarilycontiguous. TheVMMinterceptsvirtualmachineinstructionsthatmanipulateguestoperating systemmemorymanagementstructuressothattheactualmemorymanagement unit(MMU)ontheprocessorisnotupdateddirectlybythevirtualmachine. TheESXServerhostmaintainsthevirtualtomachinepagemappingsinashadow pagetablethatiskeptuptodatewiththephysicaltomachinemappings (maintainedbytheVMM,seeabove). Theshadowpagetablesareuseddirectlybytheprocessorspaginghardware. Thisapproachtoaddresstranslationallowsnormalmemoryaccessesinthevirtual machinetoexecutewithoutaddingaddresstranslationoverhead,aftertheshadow pagetablesaresetup.Becausethetranslationlookasidebuffer(TLB)ontheprocessor cachesdirectvirtualtomachinemappingsreadfromtheshadowpagetables,no additionaloverheadisaddedbytheVMMtoaccessthememory.
a b a b
c b b c
b b
machine memory
Theboxesrepresentpages,andthearrowsshowthedifferentmemorymappings.
134
VMware, Inc.
Thearrowsfromguestvirtualmemorytoguestphysicalmemoryshowthe mappingmaintainedbythepagetablesintheguestoperatingsystem.(The mappingfromvirtualmemorytolinearmemoryforx86architectureprocessorsis notshown.) Thearrowsfromguestphysicalmemorytomachinememoryshowthemapping maintainedbytheVMM. Thedashedarrowsshowthemappingfromguestvirtualmemorytomachine memoryintheshadowpagetablesalsomaintainedbytheVMM.Theunderlying processorrunningthevirtualmachineusestheshadowpagetablemappings. Becauseoftheextralevelofmemorymappingintroducedbyvirtualization,ESXServer canefficientlymanagememoryacrossallvirtualmachines.Someofthephysical memoryofavirtualmachinemightbemappedtosharedpagesortopagesthatare unmapped,orswappedout. AnESXServerhostperformsvirtualmemorymanagementwithouttheknowledgeof theguestoperatingsystemandwithoutinterferingwiththeguestoperatingsystems ownmemorymanagementsubsystem.
Performance Implications
Thissectiondiscussestheperformanceimplicationsofbothsoftwarebasedand hardwareassistedmemoryvirtualization.
VMware, Inc.
135
VMware, Inc.
137
138
VMware, Inc.
Thefollowinginformationappears,asdiscussedinTable 93.
VMware, Inc.
139
140
VMware, Inc.
memory
2
swap space
memory
3
swap space
memory
VMware, Inc.
141
Swapping
AswapfileiscreatedbytheESXServerhostwhenavirtualmachineispoweredon.If thisfilecannotbecreated,thevirtualmachinecannotpoweron.Bydefault,theswap fileiscreatedinthesamelocationasthevirtualmachinesconfigurationfile.Insteadof acceptingthisdefault,youcanalso: Usepervirtualmachineconfigurationoptionstochangethedatastoretoanother sharedstoragelocation.SeeSettingAdvancedVirtualMachineAttributeson page 149andTable 97.
142
VMware, Inc.
Usehostlocalswap,whichallowsyoutospecifyadatastorestoredlocallyonthe host.Thisallowsyoutoswapataperhostlevel,savingspaceontheSAN. However,itcanleadtoaslightdegradationinperformanceforVMotion. To enable host-local swap for a cluster 1 2 3 4 5 6 7 RightclicktheclusterintheVIClientinventorypanelandclickEditSettings. IntheleftpaneoftheclusterSettingsdialogboxthatappears,clickSwapfile Location. SelecttheStoretheswapfileinthedatastorespecifiedbythehostoptionand clickOK. SelectoneoftheclustershostsintheVIClientinventorypanelandclickthe Configurationtab. SelectVirtualMachineSwapfileLocation. ClicktheSwapfileDatastoretabandfromthelistprovidedselectthelocal datastoretouseandclickOK. RepeatStep 4throughStep 6foreachhostinthecluster.
To enable host-local swap for a standalone host 1 2 3 4 SelectthehostintheVIClientinventorypanelandclicktheConfigurationtab. SelectVirtualMachineSwapfileLocation. UndertheSwapfilelocationtaboftheVirtualMachineSwapfileLocationdialog boxthatappears,selecttheStoretheswapfileintheswapfiledatastoreoption. ClicktheSwapfileDatastoretab,fromthelistprovidedselectthelocaldatastore touseandclickOK.
VMware, Inc.
143
NOTETheseswapfilescanconsumemanygigabytesofdiskspacesoensurethatyou deletethemproperly.
144
VMware, Inc.
UsetheMem.ShareScanTimeandMem.ShareScanGHzadvancedsettingstocontrol therateatwhichthesystemscansmemorytoidentifyopportunitiesforsharing memory. Youcanalsodisablesharingforindividualvirtualmachinesbysettingthe sched.mem.pshare.enableoptiontoFALSE(thisoptiondefaultstoTRUE).See SettingAdvancedVirtualMachineAttributesonpage 149. ESXServermemorysharingrunsasabackgroundactivitythatscansforsharing opportunitiesovertime.Theamountofmemorysavedvariesovertime.Forafairly constantworkload,theamountgenerallyincreasesslowlyuntilallsharing opportunitiesareexploited. Todeterminetheeffectivenessofmemorysharingforagivenworkload,tryrunningthe workload,anduseresxtoporesxtoptoobservetheactualsavings.Findthe informationinthePSHAREfieldoftheinteractivemodeintheMemorypage.SeeUsing theUtilitiesinInteractiveModeonpage 173.
VMware, Inc.
145
IntheAdvancedSettingsdialogboxselecttheappropriateitem(forexample,CPU orMemory),andscrollintherightpaneltofindandchangetheattribute.
Table 95,Table 96,andTable 97listtheadvancedresourcemanagementattributes discussedinthisdocument. CAUTIONSettingtheseattributesisrecommendedonlyforadvanceduserswith experienceusingESXServerhosts.Inmostcases,thedefaultsettingsproducethe optimumresult. Table 9-4. Advanced CPU Attributes
Attribute CPU.MachineClearThreshold Description Ifyouareusingahostenabledforhyperthreadingandset thisattributeto0,quarantiningisdisabled.See Quarantiningonpage 133.
146
VMware, Inc.
Mem.ShareScanTime
60
Mem.ShareScanGHz
Mem.SamplePeriod
60
Mem.BalancePeriod
15
Mem.AllocGuestLargePage
VMware, Inc.
147
Numa.PageMigEnable
Numa.AutoMemAffinity
Numa.MigImbalanceThreshold
10
Numa.RebalancePeriod
2000
Numa.RebalanceCoresTotal
Numa.RebalanceCoresNode
148
VMware, Inc.
Inthedialogboxthatappears,clickAddRowtoenteranewparameterandits value.
VMware, Inc.
149
sched.mem.pshare.enable
sched.swap.dir
sched.swap.file
150
VMware, Inc.
10
10
ESXServersupportsmemoryaccessoptimizationforIntelandAMDOpteron processorsinserverarchitecturesthatsupportNUMA(nonuniformmemoryaccess). ThischaptergivesbackgroundinformationonNUMAtechnologiesanddescribes optimizationsavailablewithESXServer. Thischapterdiscussesthefollowingtopics: IntroductiontoNUMAonpage 152 ESXServerNUMASchedulingonpage 153 VMwareNUMAOptimizationAlgorithmsonpage 154 ManualNUMAControlsonpage 156 IBMEnterpriseXArchitectureOverviewonpage 157 AMDOpteronBasedSystemsOverviewonpage 158 ObtainingNUMAConfigurationInformationandStatisticsonpage 159 CPUAffinityforAssociatingVirtualMachineswithaSingleNUMANodeon page 159 MemoryAffinityforAssociatingMemoryAllocationswithaNUMANodeon page 160
VMware, Inc.
151
Introduction to NUMA
NUMAsystemsareadvancedserverplatformswithmorethanonesystembus.They canharnesslargenumbersofprocessorsinasinglesystemimagewithsuperiorprice toperformanceratios.ThesystemsthatofferaNUMAplatformtosupport industrystandardoperatingsystemsincludethosebasedoneitherAMDCPUsorthe IBMEnterpriseXArchitecture.
What Is NUMA?
Forthepastdecade,processorclockspeedhasincreaseddramatically.A multigigahertzCPU,however,needstobesuppliedwithalargeamountofmemory bandwidthtouseitsprocessingpowereffectively.EvenasingleCPUrunninga memoryintensiveworkload,suchasascientificcomputingapplication,canbe constrainedbymemorybandwidth. Thisproblemisamplifiedonsymmetricmultiprocessing(SMP)systems,wheremany processorsmustcompeteforbandwidthonthesamesystembus.Somehighend systemsoftentrytosolvethisproblembybuildingahighspeeddatabus.However, suchasolutionisexpensiveandlimitedinscalability. NUMAisanalternativeapproachthatlinksseveralsmall,costeffectivenodesviaa highperformanceconnection.Eachnodecontainsprocessorsandmemory,muchlike asmallSMPsystem.However,anadvancedmemorycontrollerallowsanodetouse memoryonallothernodes,creatingasinglesystemimage.Whenaprocessoraccesses memorythatdoesnotliewithinitsownnode(remotememory),thedatamustbe transferredovertheNUMAconnection,whichisslowerthanaccessinglocalmemory. Memoryaccesstimesarenotuniformanddependonthelocationofthememoryand thenodefromwhichitisaccessed,asthetechnologysnameimplies.
152
VMware, Inc.
Furthermore,performanceonsuchasystemcanbehighlyvariable.Itvaries,for example,ifanapplicationhasmemorylocatedlocallyononebenchmarkingrun,buta subsequentrunhappenstoplaceallofthatmemoryonaremotenode.This phenomenoncanmakecapacityplanningdifficult.Finally,processorclocksmightnot besynchronizedbetweenmultiplenodes,soapplicationsthatreadtheclockdirectly mightbehaveincorrectly. SomehighendUNIXsystemsprovidesupportforNUMAoptimizationsintheir compilersandprogramminglibraries.Thissupportrequiressoftwaredevelopersto tuneandrecompiletheirprogramsforoptimalperformance.Optimizationsforone systemarenotguaranteedtoworkwellonthenextgenerationofthesamesystem. Othersystemshaveallowedanadministratortoexplicitlydecideonthenodeonwhich anapplicationshouldrun.Whilethismightbeacceptableforcertainapplicationsthat demand100percentoftheirmemorytobelocal,itcreatesanadministrativeburden andcanleadtoimbalancebetweennodeswhenworkloadschange. Ideally,thesystemsoftwareprovidestransparentNUMAsupport,sothatapplications canbenefitimmediatelywithoutmodifications.Thesystemshouldmaximizetheuse oflocalmemoryandscheduleprogramsintelligentlywithoutrequiringconstant administratorintervention.Finally,itmustrespondwelltochangingconditions withoutcompromisingfairnessorperformance.
2 3
VMware, Inc.
153
SomevirtualmachinesarenotmanagedbytheESXServerNUMAscheduler.For example,ifyoumanuallysettheprocessoraffinityforavirtualmachine,theNUMA schedulermightnotbeabletomanagethisvirtualmachine.Virtualmachinesthathave morevirtualprocessorsthanthenumberofphysicalprocessorcoresavailableona singlehardwarenodecannotbemanagedautomatically.Virtualmachinesthatarenot managedbytheNUMAschedulerstillruncorrectly.However,theydontbenefitfrom ESXServersNUMAoptimizations. TheNUMAschedulingandmemoryplacementpoliciesinVMwareESXServercan manageallvirtualmachinestransparently,sothatadministratorsdonotneedto addressthecomplexityofbalancingvirtualmachinesbetweennodesexplicitly. Theoptimizationsworkseamlesslyregardlessofthetypeofguestoperatingsystem. ESXServerprovidesNUMAsupporteventovirtualmachinesthatdonotsupport NUMAhardware,suchasWindowsNT4.0.Asaresult,youcantakeadvantageofnew hardwareevenwithlegacyoperatingsystems.
154
VMware, Inc.
VMware, Inc.
155
Whenavirtualmachinemovestoanewnode,theESXServerhostimmediatelybegins tomigrateitsmemoryinthisfashion.Itmanagestheratetoavoidovertaxingthe system,particularlywhenthevirtualmachinehaslittleremotememoryremainingor whenthedestinationnodehaslittlefreememoryavailable.Thememorymigration algorithmalsoensuresthattheESXServerhostdoesnotmovememoryneedlesslyifa virtualmachineismovedtoanewnodeforonlyashortperiod. Wheninitialplacement,dynamicrebalancing,andintelligentmemorymigrationwork inconjunction,theyensuregoodmemoryperformanceonNUMAsystems,eveninthe presenceofchangingworkloads.Whenamajorworkloadchangeoccurs,forinstance whennewvirtualmachinesarestarted,thesystemtakestimetoreadjust,migrating virtualmachinesandmemorytonewlocations.Afterashortperiod,typicallyseconds orminutes,thesystemcompletesitsreadjustmentsandreachesasteadystate.
ESXServerprovidestwosetsofcontrolsforNUMAplacement,sothatadministrators cancontrolmemoryandprocessorplacementofavirtualmachine. TheVIClientallowsyoutospecify: CPUAffinityAvirtualmachineshoulduseonlytheprocessorsonagivennode. SeeCPUAffinityforAssociatingVirtualMachineswithaSingleNUMANode onpage 159. MemoryAffinityTheservershouldallocatememoryonlyonthespecifiednode. SeeMemoryAffinityforAssociatingMemoryAllocationswithaNUMANode onpage 160. Ifbothoptionsaresetbeforeavirtualmachinestarts,thevirtualmachinerunsonlyon theselectednodeandallofitsmemoryisallocatedlocally. Anadministratorcanalsomanuallymoveavirtualmachinetoanothernodeafterthe virtualmachinehasstartedrunning.Inthiscase,thepagemigrationrateofthevirtual machineshouldalsobesetmanually,sothatmemoryfromthevirtualmachines previousnodecanbemovedtoitsnewnode. ManualNUMAplacementmightinterferewiththeESXServerresourcemanagement algorithms,whichattempttogiveeachvirtualmachineafairshareofthesystems processorresources.Forexample,iftenvirtualmachineswithprocessorintensive workloadsaremanuallyplacedononenode,andonlytwovirtualmachinesare manuallyplacedonanothernode,itisimpossibleforthesystemtogivealltwelve virtualmachinesequalsharesofthesystemsresources.Youmustconsidertheseissues whenmakingmanualNUMAplacementdecisions.
VMware, Inc.
157
158
VMware, Inc.
CPU Affinity for Associating Virtual Machines with a Single NUMA Node
Youmightbeabletoimprovetheperformanceoftheapplicationsonavirtualmachine byassociatingittotheCPUnumbersonasingleNUMAnode(manualCPUaffinity). CAUTIONThereareanumberofpotentialissuesifyouuseCPUaffinity.SeePotential IssueswithAffinityonpage 127. To set CPU affinity for a single NUMA node 1 2 3 UsingaVIClient,rightclickavirtualmachineandchooseEditSettings. IntheVirtualMachinePropertiesdialogbox,selecttheResourcestabandchoose AdvancedCPU. IntheSchedulingAffinitypanel,setCPUaffinityfordifferentNUMAnodes. NOTEYoumustmanuallyselecttheboxesforallprocessorsintheNUMAnode. CPUaffinityisspecifiedonaperprocessor,notonapernode,basis.
VMware, Inc.
159
Example: Binding a Virtual Machine to a Single NUMA Node Thefollowing exampleillustratesmanuallybindingfourCPUstoasingleNUMAnodeforavirtual machineonaneightwayserver.Youwantthisvirtualmachinetorunonlyonnode1. TheCPUsforexample,4,5,6,and7arethephysicalCPUnumbers. To bind a two-way virtual machine to use the last four physical CPUs of an eight-processor machine 1 2 3 4 IntheVIClientinventorypanel,selectthevirtualmachineandchooseEdit Settings. SelectOptionsandclickAdvanced. ClicktheConfigurationParametersbutton. IntheVIClient,turnonCPUaffinityforprocessors4,5,and6.
160
VMware, Inc.
To set the virtual machines memory to specify that all of the virtual machines memory should be allocated on node 1 1 2 3 4 IntheVIClientinventorypanel,selectthevirtualmachineandchooseEdit Settings. SelectOptionsandclickAdvanced. ClicktheConfigurationParametersbutton. IntheVIClient,setmemoryaffinityfortheNUMAnodeto1.
VMware, Inc.
161
162
VMware, Inc.
11
Best Practices
11
VMware, Inc.
163
Whenspecifyingthereservationsforvirtualmachines,donotcommitall resources.Asyoumoveclosertofullyreservingallcapacityinthesystem,it becomesincreasinglydifficulttomakechangestoreservationsandtotheresource poolhierarchywithoutviolatingadmissioncontrol.InaDRSenabledcluster, reservationsthatfullycommitthecapacityoftheclusterorofindividualhostsin theclustercanpreventDRSfrommigratingvirtualmachinesbetweenhosts. Useresourcepoolsfordelegatedresourcemanagement.Tofullyisolatearesource pool,maketheresourcepooltypeFixedanduseReservationandLimit. Groupvirtualmachinesforamultitierserviceinaresourcepool.Resourcepools allowtheESXServerhosttoassignresourcesfortheserviceasawhole.
Planning
Beforeyoudeployavirtualmachine,youneedto: Planyourloadmix. Understandgoalsandexpectations. Understandtherequirements,andwhatitmeanstobesuccessful. Avoidmixingvirtualmachinesthathavecompetingresourcerequirements. Testbeforeyoudeployifyouhavespecificperformanceexpectations,. Virtualizationallowsanumberofvirtualmachinestosharethehostsresources.Itdoes notcreatenewresources.Virtualizationcanresultinoverheads.
164
VMware, Inc.
InstallVMwareTools,whichhelpsyouachievehigherperformance,canresultinmore efficientCPUutilization,andincludesdisk,network,andmemoryreclamationdrivers.
VMware, Inc.
165
166
VMware, Inc.
NIC Teaming
AsshowninFigure 111,usingateamoftwoNICsconnectedtoseparatephysical switchesimprovesthereliabilityofaserviceconsole(or,inESXServer3i,VMkernel) network.BecauseserversconnectedthroughtwoNICs(andthroughseparateswitches) havetwoindependentpathsforsendingandreceivingheartbeats,theclusterismore resilient. ToconfigureaNICteamfortheserviceconsole,configurethevNICsinvSwitch configurationforActive/standbyconfiguration.Therecommendedparametersettings forthevNICsare: RollingFailover=Yes DefaultLoadbalancing=routebasedonoriginatingportID NOTEAfteryouhaveaddedaNICtoahostinyourVMwareHAcluster,youmust reconfigureHAonthathost.
VMware, Inc.
167
NIC Teaming Scenario Thefollowingscenarioillustratestheuseofasingleservice consolenetworkwithNICteamingfornetworkredundancy: Youassumesomeriskwhenyouconfigurehostsintheclusterwithonlyone serviceconsolenetwork(subnet10.20.XX.XX).UsetwoteamedNICstoprotect againstNICfailure. Thedefaulttimeoutisincreasedto60seconds(das.failuredetectiontime= 60000).
168
VMware, Inc.
Youcanfurtheroptimizeyournetwork(ifyouhavealreadyconfiguredaVMotion network)byaddingasecondaryserviceconsolenetworktotheVMotionvswitch. As showninFigure 112,avirtualswitchcanbesharedbetweenVMotionnetworksanda secondaryserviceconsolenetwork. Figure 11-2. Network Redundancy with a Secondary Service Console
Redundant Service Console Network Scenario Thefollowingscenarioillustrates theuseofaredundantserviceconsolenetwork: Configureeachhostintheclusterwithtwoserviceconsolenetworksbyleveraging anexistingVMotionnetwork(subnets10.20.YY.YYand192.168.ZZ.ZZ). Usethedefaultgatewayforthefirstnetworkandspecify das.isolationaddress2=192.168.1.103astheadditionalisolationaddressfor thesecondnetwork. Increasethedefaulttimeoutto20seconds(das.failuredetectiontime=20000).
VMware, Inc.
169
170
VMware, Inc.
Theresxtopandesxtopcommandlineutilitiesprovideadetailedlookathow ESXServerusesresourcesinrealtime.Youcanstarteitherutilityinoneofthreemodes: interactive(default),batch,orreplay.Thisappendixexplainshowtouseresxtopand esxtopineachofthesemodesandgivereferencestoavailablecommandsanddisplay statistics.Unlessspecifiedotherwise,thecommandsandstatisticsfortheutilitiesare thesame.Thefollowingtopicsarediscussed: DecidingtoUseresxtoporesxtoponpage 171 UsingtheUtilitiesinInteractiveModeonpage 173 UsingtheUtilitiesinBatchModeonpage 194 UsingtheUtilitiesinReplayModeonpage 195
youareusingcanconnecttoandbeauthenticatedbytheremoteserver,usethe followingadditionaloptions: [server]Nameoftheremoteserverhosttoconnectto(required). [portnumber]Portnumbertoconnecttoontheremoteserver.Thedefaultportis 443,unlessthishasbeenchangedontheserver,thisoptionisnotneeded. [username]Usernametobeauthenticatedwhenconnectingtotheremotehost.You willbepromptedbytheremoteserverforapassword,aswell. NOTEresxtopdoesnotusealltheoptionssharedbyotherRemoteCLIcommands. YoucanalsouseresxtoponalocalESXServerhost.Todoso,omittheserveroption onthecommandlineandthecommandwilldefaulttolocalhost.
esxtop [-] [h] [v] [b] [s] [a] [c filename] [R [d delay] [n iter]
Theesxtoputilityreadsitsdefaultconfigurationfrom.esxtop310rc.This configurationfileconsistsofsevenlines. Thefirstsixlinescontainlowercaseanduppercaseletterstospecifywhichfieldsappear inwhichorderontheCPU,memory,storageadapter,storagedevice,virtualmachine storage,andnetworkpanel.TheletterscorrespondtothelettersintheFieldsorOrder panelsfortherespectiveesxtoppanel. Theseventhlinecontainsinformationontheotheroptions.Mostimportant,ifyou savedaconfigurationinsecuremode,youdonotgetaninsecureesxtopwithout removingthesfromtheseventhlineofyour.esxtop310rcfile.Anumberspecifies thedelaytimebetweenupdates.Asininteractivemode,typingc,m,d,u,v,orn determinesthepanelwithwhichesxtopstarts. NOTEEditingthisfileisnotrecommended.Instead,selectthefieldsandtheorderin arunningesxtopprocess,makechanges,andsavethisfileusingtheWinteractive command.
172
VMware, Inc.
n server portnumber
username
c <filename>
VMware, Inc.
173
174
VMware, Inc.
q c m d u v n
VMware, Inc.
175
CPU Panel
TheCPUpaneldisplaysserverwidestatisticsaswellasstatisticsforindividualworld, resourcepool,andvirtualmachineCPUutilization.Resourcepools,runningvirtual machines,orotherworldsareattimesreferredtoasgroups.Forworldsbelongingtoa virtualmachine,statisticsfortherunningvirtualmachinearedisplayed.Allother worldsarelogicallyaggregatedintotheresourcepoolsthatcontainthem. Figure A-2. CPU Panel
176
VMware, Inc.
CCPU(%)
%WAIT
%IDLE
%RDY
VMware, Inc.
177
ASHRS SUMMARY STATS AFFINITYBIT MASK HTSHARING CPU HTQ TIMER/s %OVRLP
178
VMware, Inc.
%CSTP
Memory Panel
TheMemorypaneldisplaysserverwideandgroupmemoryutilizationstatistics.Ason theCPUpanel,groupscorrespondtoresourcepools,runningvirtualmachines,or otherworldsthatareconsumingmemory.Fordistinctionsbetweenmachinememory andphysicalmemoryseeMemoryVirtualizationonpage 133. Thefirstline,foundatthetopoftheMemorypanel(seeFigure A3)displaysthe currenttime,timesincelastreboot,numberofcurrentlyrunningworlds,andmemory overcommitmentaverages.Thememoryovercommitmentaveragesoverthepastone, five,andfifteenminutesappear.Memoryovercommitmentof1.00meansamemory overcommitof100percent.SeeMemoryOvercommitmentonpage 42.
VMware, Inc. 179
180
VMware, Inc.
VMware, Inc.
181
MCTL? MCTLSZ (MB) MCTLTGT (MB) MCTLMAX (MB) SWCUR(MB) SWTGT(MB) SWR/s(MB) SWW/s(MB) CPTRD(MB)
182
VMware, Inc.
VMware, Inc.
183
Storage Panels
Threestoragepanelsdisplayserverwidestorageutilizationstatistics. Thissectiondescribesthethreestoragepanels: StorageAdapterPanelonpage 184 StorageDevicePanelonpage 187 VirtualMachineStoragePanelonpage 190
Description Nameofthestorageadapter. StorageadapterchannelID.ThisIDisvisibleonlyifthecorresponding adapterisexpanded.Seetheinteractivecommandebelow. StorageadapterchanneltargetID.ThisIDisvisibleonlyifthecorresponding adapterandchannelareexpanded.Seetheinteractivecommandseanda below. StorageadapterchanneltargetLUNID.ThisIDisvisibleonlyifthe correspondingadapter,channelandtargetareexpanded.Seetheinteractive commandse,a,andtbelow. StorageadapterchanneltargetLUNworldID.ThisIDisvisibleonlyifthe correspondingadapter,channel,targetandLUNareexpanded.Seeinteractive commandse,a,t,and lbelow. Numberofchannels. Numberoftargets. NumberofLUNs. Numberofworlds. Numberofshares.
LID
WID
184
VMware, Inc.
%USD LOAD ACTV QUED CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd DAVG/rd KAVG/rd GAVG/rd DAVG/wr KAVG/wr GAVG/wr
VMware, Inc.
185
186
VMware, Inc.
r w R T N
PARTITION
VMware, Inc.
187
CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd DAVG/rd KAVG/rd
188
VMware, Inc.
r w R
VMware, Inc.
189
190
VMware, Inc.
WQLEN
CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd DAVG/rd KAVG/rd GAVG/rd QAVG/rd DAVG/wr KAVG/wr
VMware, Inc.
V r w R T N
192
VMware, Inc.
Network Panel
ThepanelshowninFigure A7displaysserverwidenetworkutilizationstatistics. Statisticsarearrangedbyportforeachvirtualnetworkdeviceconfigured.Forphysical networkadapterstatistics,seetherowcorrespondingtotheporttowhichthephysical networkadapterisconnected.Forstatisticsonavirtualnetworkadapterconfiguredin aparticularvirtualmachine,seetherowcorrespondingtotheporttowhichthevirtual networkadapterisconnected. Figure A-7. Network Panel
VMware, Inc.
193
194
VMware, Inc.
b c <filename>
n server portnumber
username
Unzipanduntartheresultingtarfilesothatresxtop(oresxtop)canuseitin replaymode.
VMware, Inc.
195
AdditionalcommandlineoptionsarelistedinTable A16. YoudonothavetorunreplaymodeontheESXServerserviceconsole. Replaymodecanberuntoproduceoutputinthesamestyleasbatchmode(seethe commandlineoptionb,below). Inreplaymode,resxtop(oresxtop)acceptsthesamesetofinteractivecommandsas ininteractivemodeandrunsuntiltherearenomoresnapshotscollectedby vm-supporttobereadoruntiltherequestednumberofiterationsarecompleted(see thecommandlineoptionnformoredetails). Table A16liststhecommandlineoptionsavailableforresxtop(oresxtop)replay mode. Table A-16. Command-Line Options in Replay Mode
Option R a Description Pathtothevmsupportcollectedsnapshotsdirectory. Showallstatistics.Thisoptionoverridesconfigurationfilesetupsand showsallstatistics.Theconfigurationfilecanbethedefault ~/.esxtop310rcconfigurationfileorauserdefinedconfiguration file. Runsresxtop(oresxtop)inBatchmode. Loadauserdefinedconfigurationfile.Ifthe-coptionisnotused,the defaultconfigurationfilenameis~/.esxtop310rc.Createyourown configurationfileandspecifyadifferentfilenameusingtheW singlekeyinteractivecommand.SeeInteractiveModeSingleKey Commandsonpage 174forinformationaboutW. Specifiesthedelaybetweenpanelupdates.Thedefaultisfiveseconds. Theminimumistwoseconds.Ifadelayoflessthantwosecondsis specified,thedelayissettotwoseconds. Numberofiterations.resxtop(oresxtop)updatesthedisplaythis numberoftimesandthenexits.
b c <filename>
196
VMware, Inc.
Index
A
admission control 22, 23 CPU 127 DRS 112 HA 95, 113 resource pools 47 strict 60, 76, 86, 115, 117 with expandable resource pools 29 advanced attributes 145 CPU 146 HA 121 hosts 145 memory 147 NUMA 148 virtual machines 149 affinity CPU 38, 126 CPU, and hyperthreading 133 defined 80 memory, NUMA nodes 160 potential issues 127 rules, using 108 algorithms, NUMA 154 AMD Opteron-based systems 158 anti-affinity 80 applications CPU-bound 125 deploying 165 single-threaded 125 architecture, ESX Server 36
automation level 93 and DRS recommendations 60 distributed power management 68, 94 automation modes, virtual machines 114 available memory 15
B
ballooning, memory 141 best practices 163
C
cluster creation overview 92 cluster features, choosing 93 cluster resource pools 46 clusters adding hosts 31, 112 adding managed hosts 102, 118 adding unmanaged hosts 103, 118 adding virtual machines 111, 112 creating 30, 31, 89, 93, 95 customizing 30, 31 distributed power management 23, 30, 60, 68 DRS 57 DRS, adding hosts 102 HA 90 introduction 59 invalid 105 powering on virtual machines 112 prerequisites 89 processor compatibility 91 removing hosts 114 removing virtual machines 113
197
VMware, Inc.
resource pools 56 shared storage 91 shared VMFS volume 91 summary page 96 virtual machines 111 VirtualCenter failure 61 CPU admission control 127 advanced attributes 146 managing allocation 38 overcommitment 19 virtual machines 19 CPU affinity 38, 126 hyperthreading 133 NUMA 159 NUMA nodes 159 potential issues 127 CPU panel esxtop 176 resxtop 176 CPU Reservation 16 CPU Unreserved 16 CPU virtualization 39, 124 CPU.MachineClearThreshold 133, 146 CPU-bound applications 125 custom automation mode 114
D
delegation of control through resource pools 45 device drivers 37 disabled virtual machine 114 disk resources 35 distributed power management 30, 60 and admission control 23 automation level 68, 94 enabling 68
DNS 90, 120 short name 90 DRS action history 100 adding managed hosts 102 adding unmanaged hosts 103 admission control 112 affinity rules 108 automation level 93 clusters, adding hosts 102 custom automation mode 114 customizing virtual machines 114 disabled virtual machine 114 fully automatic 94 host removal and virtual machines 105 initial placement 60, 62 introduction 62, 101 load balancing 60 maintenance mode 71 manual 94 migration 60 migration recommendations 68 overview 60 partially automatic 94 recommendation dependencies 106 recommendations 105 recommendations page 99 recommendations, and automation level 60 reconfiguring 107 red clusters 85 rules 110 turning off 107 using together with HA 80 virtual machine migration 66 VMotion network 89 DRS clusters 70 DRS recommendations page 99 DRS Resource Distribution histograms 98
VMware, Inc.
198
Index
DRS rules 109 dual-processor virtual machine 19 dynamic load balancing, NUMA 155
E
emulation 124 entering Maintenance Mode 104 ESX Server architecture 36 memory allocation 138 memory reclamation 140 resource management 35 esxtop batch mode 194 command-line options 172 common statistics description 174 CPU panel 176 CPU panel single-key commands 179 CPU panel statistics 177 interactive mode 173 interactive mode command-line options 173 interactive mode single-key commands 174 invoking 172 memory panel 179 network panel 193 network panel statistics 193 order pages 175 performance monitoring 172 replay mode 195 statistics column 175 storage adapter panel interactive commands 186 storage adapter panel statistics 184 storage device panel interactive commands 189 storage device panel statistics 187 storage panels 184
examples expandable reservations 50 memory overhead 136 NUMA 160 red cluster 85 reservation 22 resource pools 26 shares 21 valid cluster using resource pools of type expandable 83 valid cluster, all resource pools of type fixed 82 yellow cluster 84 expandable reservations 29, 50 example 50
F
failover capacity 75 fully automatic DRS 94
G
grafted, resource pool 103
H
HA 117, 121 adding managed hosts 118 adding unmanaged hosts 118 admission control 95, 113 advanced attributes 121 and host power off 77 and traditional cluster solutions 72 best practices 166 customizing virtual machines 115 DNS connectivity 90 failover capacity 75 host network isolation 77 introduction 72, 117 iSCSI storage 115 migration with VMotion 77
VMware, Inc.
199
NAS storage 115 networking redundancy 167 NIC teaming 167 options 94 red clusters 86 redundant network paths 90 shared storage 90 turning off 121 using together with DRS 80 HA clusters adding hosts 118 maintenance mode 78 planning 75 high, shares 20, 21 histograms,DRS Resource Distribution 98 home nodes, NUMA 154 host network isolation 77 host resource pools 46 hosts adding to cluster 112 adding to DRS clusters 102, 103 adding to HA clusters 118 entering maintenance mode 104 losing resource pool hierarchy 103 memory use 139 removing and invalid clusters 105 removing and resource pool hierarchies 104 removing from clusters 114 resource information 14 hyperthreading 129, 130, 131 CPU affinity 133 CPU.MachineClearThreshold 133 disabling 40 disabling quarantining 146 performance implications 129 quarantining 133
I
idle memory tax 138, 140 initial placement 60, 62 NUMA 154 invalid clusters, host removal 105 iSCSI storage HA 115 isolation response 115 default 119 isolation through resource pools 45
L
limit attribute 22 hyperthreading 132 pros and cons 22 resource pools 25 load balancing 60 DRS 30 migration recommendations 110 virtual machines, migration 66 logical processors 40, 128 low, shares 20, 21
M
maintenance mode 71, 104 entering 104 HA clusters 78 manual DRS 94 Mem.AllocGuestLargePage 147 Mem.AllocUseGuestPool 147 Mem.AllocUsePSharePool 147 Mem.BalancePeriod 147 Mem.CtlMaxPercent 147
200
VMware, Inc.
Index
Mem.IdleTax 140, 147 Mem.SamplePeriod 139, 147 Mem.ShareScanGHz 145, 147 Mem.ShareScanTime 145, 147 memory advanced attributes 147 available 15 managing allocation 39 overhead 42 reclaiming unused 140 service console 15, 40 sharing across virtual machines 144 virtual machines 18 virtualization basics 40 VMkernel memory 165 memory affinity, NUMA 160 memory balloon driver 141 memory idle tax 138, 140 Mem.IdleTax 147 memory overcommitment 42, 144 memory overhead 136 examples 136 Memory Reservation 17 memory sharing 42 Memory Unreserved 17 memory virtualization 39 migration recommendations 68 migration threshold 67 migration with VMotion, failure, and HA 77 multicore processors 128
NUMA advanced attributes 148 AMD Opteron-based systems 158 CPU affinity 159 CPU assignment 160 dynamic load balancing 155 example 160 home nodes and initial placement 154 introduction 152 manual controls 156 memory affinity 160 optimization algorithms 154 page migration 155 transparent page sharing 156 using with ESX server 151 NUMA scheduling 153 Numa.AutoMemAffinity 148 Numa.MigImbalanceThreshold 148 Numa.PageMigEnable 148 Numa.RebalanceCoresNode 148 Numa.RebalanceCoresTotal 148 Numa.RebalanceEnable 148 Numa.RebalancePeriod 148
O
Opteron 158 overcommitment 42, 144 overcommitted cluster 84 overhead 136 examples 136 overhead memory 42
N
NAS storage, HA 115 network resources 35 NIC teaming 167 normal, shares 20, 21
P
page migration, NUMA 155 partially automatic DRS 94 performance 36 CPU-bound applications 125 monitoring 54 performance monitoring, esxtop 172
201
VMware, Inc.
performance monitoring, resxtop 172 physical and logical processors 40 physical memory usage 139 physical processors 40 processors logical 40 multicore 128 physical 40 processor-specific behavior 125
Q
quarantining, hyperthreading 133
R
red clusters 85 red DRS cluster 85 red HA cluster 86 redundant network paths for HA 90 Remote CLI 171 reservation 16 attribute 21 example 22 hyperthreading 132 resource pools 25 reservation type 25 resource management best practices 163 concepts 33 resource pools 46 adding virtual machines 55 admission control 47 attributes, changing 54 clusters 56 creating 25, 26, 48 customizing 25, 26 delegation of control 45 DRS clusters 70 example 26 grafted 103
202
hierarchies, host removal 104 information 50 introduction 43, 44 isolation 45 performance 54 removing virtual machines 56 reservation type 25 resource allocation tab 52 root resource pool 44 siblings 44 summary tab 51 resources, reserving 20 restart priority 115 default 119 resxtop batch mode 194 common statistics description 174 CPU panel 176 CPU panel single-key commands 179 CPU panel statistics 177 interactive mode 173 interactive mode command-line options 173 interactive mode single-key commands 174 memory panel 179 network panel 193 network panel statistics 193 options 172 order pages 175 performance monitoring 172 replay mode 195 statistics column 175 storage adapter panel interactive commands 186 storage adapter panel statistics 184 storage device panel interactive commands 189
VMware, Inc.
Index
storage device panel statistics 187 storage panels 184 root resource pool 44 rules 109 deleting 110 disabling 110 DRS 110 editing 109 results 110
swap space 142, 144 Linux systems 142 Windows systems 142 swapping 142
T
threads 128 threshold, migration 67 traditional clustering solutions 73
S
SAN and HA 90 sched.mem.maxmemctl 141, 150 sched.mem.pshare.enable 150 sched.swap.dir 150 sched.swap.file 150 sched.swap.persist 150 secondary service console 168 server configuration for hyperthreading 131 service console memory use 15, 40 redundancy 168 shares 20 example 21 high 20 low 20 normal 20 ratio 20 resource pools 25 sharing memory 42 siblings 44 single-processor virtual machine 19 single-threaded applications 125 SMP virtual machines 125 standby mode 60, 63, 68, 72 stars, migration threshold 67 statistics, esxtop 174 statistics, resxtop 174
V
valid clusters 81 example 83 Virtual Infrastructure SDK 38 virtual machine attributes changing 23 shares, reservation, and limit 20 Virtual Machine File System (VMFS) 37, 77, 91 virtual machine migration 66 virtual machines adding during cluster creation 111 adding to cluster 111, 112 adding to resource pools 55 advanced attributes 149 assigning to a specific processor 126 automation modes 114 changing resource allocation 23 configuration file 92 CPU 19 creating (best practice) 164 customizing for DRS 114 customizing for HA 115 deploying (best practice) 164 deploying applications 165 deploying operating system 165 disabled (DRS) 114 dual-processor 19 host removal 105
VMware, Inc.
203
memory 18, 41 memory overhead 136 monitor 133 number of virtual processors 125 removing from cluster 113 removing from resource pools 56 resource allocation 18 single-processor 19 virtual memory 133 virtual memory in virtual machines 133 virtual processors per virtual machine 125 VMFS (Virtual Machine File System) 37, 77, 91 VMkernel 37 hardware interface layer 37 memory 165 resource manager 37
VMkernel port redundancy 168 VMM 37, 133 vmmemctl 141 Mem.CtlMaxPercent 147 sched.mem.maxmemctl 150 VMotion requirements 91
W
wake on LAN (WOL) 72 working set size 138
Y
yellow cluster 84
204
VMware, Inc.
LastUpdated:June12,2009 ThisdocumentprovidesupdatestotheESXServer3.5,ESXServer3iversion3.5, VirtualCenter2.5versionoftheResourceManagementGuide.Updateddescriptions, procedures,andgraphicsareorganizedbypagenumbersothatyoucaneasilylocate theareasoftheguidethathavechanges.Ifthechangespansmultiplesequentialpages, thisdocumentprovidesthestartingpagenumberonly. ThefollowingisalistofupdatestotheResourceManagementGuide: UpdatefortheFailoverCapacitySectiononPage 75 UpdatefortheSharedStorageSectiononPage 91 UpdatefortheSharedVMFSVolumeSectiononPage 91 UpdatefortheTopDRSResourceDistributionChartSectiononPage 98 UpdatefortheVirtualizationandProcessorSpecificBehaviorSectiononPage 125 UpdatefortheSettingAdvancedHostAttributesSectiononPage 145 UpdatefortheCPUPanelSectiononPage 177
VMware, Inc.
Update1
Update for the Top DRS Resource Distribution Chart Section on Page 98
ThedescriptionoftheTopDRSResourceDistributionChartclaims:Thischartisa histogramthatshowsthenumberofhostsontheXaxisandtheutilizationpercentage ontheYaxis. Thisisincorrectbecausetheaxesaretheoppositeofwhatisdescribed.Thesentence shouldread:ThischartisahistogramthatshowsthenumberofhostsontheYaxis andtheutilizationpercentageontheXaxis.
Update for the Virtualization and Processor-Specific Behavior Section on Page 125
TheVirtualizationandProcessorSpecificBehaviorsectionstates:Becauseofthe differentkernelversions,itisnotpossibletomigratevirtualmachinesinstalledona systemrunningoneprocessormodel(forexample,AMD)toasystemrunningona differentprocessor(forexample,Intel).Thissentenceshouldbequalifiedtosay: Becauseofthedifferentkernelversions,itisnotpossibletouseVMotiontomigrate virtualmachinesinstalledonasystemrunningoneprocessormodel(forexample, AMD)toasystemrunningonadifferentprocessor(forexample,Intel).
Update for the Setting Advanced Host Attributes Section on Page 145
TheprocedureTosetadvancedattributesforahostisincorrectandshouldbereplaced bythefollowingprocedure: To set advanced attributes for a host 1 2 3 4 IntheVIClientinventorypanel,selectthehosttocustomize. ClicktheConfigurationtab. IntheSoftwaremenu,clickAdvancedSettings. IntheAdvancedSettingsdialogbox,selecttheappropriateitem(forexample,CPU orMemory)andscrollintherightpaneltofindandchangetheattribute.
VMware, Inc.
Update2
VMware, Inc.
Update3
Update4
VMware, Inc.