4/27/2015

What'supwithLDoms:Part1Introduction&BasicConcepts(SPARC,Solaris,PerformanceundderganzeRest)

Oracle

BlogsHome
Products&Services
Downloads
Support
Partners
Communities
About
Einloggen
OracleBlog
SPARC,Solaris,PerformanceundderganzeRest
WebNotizenvonStefanHinker

OracleVMServerfor...|Main|SecureDeploymentof...

What'supwithLDoms:Part1Introduction&BasicConcepts
ByStefanHinkeronJun29,2012
LDomsthecorrectnameisOracleVMServerforSPARChavebeenaroundforquiteawhilenow.
Buttomysurprise,Igetmoreandmorerequeststoexplainhowtheyworkortogiveadviseonhowto
makegooduseofthem.Thismademethinkthatwritingupafewarticlesdiscussingthedifferent
featureswouldbeagoodidea.NowIdon'tintendtorewritetheLDomsAdminGuideortocopyand
reformatthe(hopefully)wellknown"BeginnersGuidetoLDoms"byTonyShoumackfrom2007.
ThosedocumentsareveryrecommendableespeciallytheBeginnersGuide,althoughbasedonLDoms
1.0,isstillagoodplacetobeginwith.However,LDomshavecomealongwaysincethen,andIhopeto
contributetotheiradoptionbydiscussinghowtheyworkandwhatfeaturestherearetoday.
Inthisandthefollowingposts,Iwillusetheterm"LDoms"asacommonabbreviationforOracleVM
ServerforSPARC,justbecauseit'salotshorterandeasiertotype(andpresumably,read).
So,justtogeteveryoneonthesamebaseline,letsbrieflydiscussthebasicconceptsofvirtualizationwith
LDoms.LDomsmakeuseofahypervisorasalayerofabstractionbetweenreal,physicalhardwareand
virtualhardware.Thisvirtualhardwareisthenusedtocreateanumberofguestsystemswhicheach
behaveverysimilartoasystemrunningonbaremetal:EachhasitsownOBP,eachwillinstallitsown
copyoftheSolarisOSandeachwillseeacertainamountofCPU,memory,diskandnetworkresources
availabletoit.Unlikesomeothertype1hypervisorsrunningonx86hardware,theSPARChypervisoris
embeddedinthesystemfirmwareandmakesusebothofsupportingfunctionsinthesun4vSPARC
instructionsetaswellastheoverallCPUarchitecturetofulfillitsfunction.
TheCMTarchitectureofthesupportingCPUs(T1throughT4)providealargenumberofcoresand
threadstotheOS.Forexample,thecurrentT4CPUhaseightcores,eachrunning8threads,foratotalof
64threadspersocket.TotheOS,thislookslike64CPUs.
https://blogs.oracle.com/cmt/entry/what_s_up_with_ldoms

1/7

4/27/2015

What'supwithLDoms:Part1Introduction&BasicConcepts(SPARC,Solaris,PerformanceundderganzeRest)

TheSPARChypervisor,whencreatingguestsystems,
simplyassignsacertainnumberofthesethreads
exclusivelytooneguest,thusavoidingtheoverheadof
havingtoscheduleOSthreadstoCPUs,asdotypicalx86
hypervisors.ThehypervisoronlyassignsCPUsandthen
stepsaside.Itisnotinvolvedintheactualworkbeing
dispatchedfromtheOStotheCPU,allitdoesismaintain
isolationbetweendifferentguests.

Likewise,memoryisassignedexclusivelytoindividual
guests.Here,thehypervisorprovidesgenericmappings
betweenthephysicalhardwareaddressesandtheguest's
viewsonmemory.Again,thehypervisorisnotinvolved
intheactualmemoryaccess,itonlymaintainsisolation
betweenguests.
DuringtheinitalsetupofasystemwithLDoms,youstart
withonespecialdomain,calledtheControlDomain.
Initially,thisdomainownsallthehardwareavailablein
thesystem,includingallCPUs,allRAMandallIO
resources.Ifyou'dberunningthesystemunvirtualized,thiswouldbewhatyou'dbeworkingwith.To
allowforguests,youfirstresizethisinitialdomain(alsocalledaprimarydomaininLDomsspeak),
assigningitasmallamountofCPUandmemory.ThisfreesupmostoftheavailableCPUandmemory
resourcesforguestdomains.
IOisalittlemorecomplex,butverystraightforward.WhenLDoms1.0firstcameout,theonlywayto
provideIOtoguestsystemswastocreatevirtualdiskandnetworkservicesandattachgueststothese
services.Inthemeantime,severaldifferentwaystoconnectguestdomainstoIOhavebeendeveloped,
themostrecentonebeingSRIOVsupportfornetworkdevicesreleasedinversion2.2ofOracleVM
ServerforSPARC.Iwillcoverthesemoreadvancedfeaturesindetaillater.Fornow,letshaveashort
lookattheinitialwayIOwasvirtualizedinLDoms:
ForvirtualizedIO,youcreatetwoservices,one"Virtual
DiskService"orvds,andone"VirtualSwitch"or
vswitch.Youcan,ofcourse,alsocreatemoreofthese,
butthat'smoreadvancedthanIwanttocoverinthis
introduction.TheseIOservicesnowconnectreal,
physicalIOresourceslikeadiskLUNoranetwortportto
thevirtualdevicesthatareassignedtoguestdomains.For
diskIO,thenormalcasewouldbetoconnectaphysical
LUN(orsomeotherstorageoptionthatI'lldiscusslater)
toonespecificguest.Thatguestwouldbeassigneda
virtualdisk,whichwouldappeartobejustlikeareal
LUNtotheguest,whiletheIOisactuallyroutedthrough
thevirtualdiskservicedowntothephysicaldevice.For
network,thevswitchactsverymuchlikeareal,physicalethernetswitchyouconnectonephysicalport
toitforoutsideconnectivityanddefineoneormoreconnectionsperguest,justlikeyouwouldplug
https://blogs.oracle.com/cmt/entry/what_s_up_with_ldoms

2/7

4/27/2015

What'supwithLDoms:Part1Introduction&BasicConcepts(SPARC,Solaris,PerformanceundderganzeRest)

cablesbetweenarealswitchandarealsystem.Forcompleteness,thereisanotherservicethatprovides
consoleaccesstoguestdomainswhichmimicsthebehaviorofserialterminalservers.
Theconnectionsbetweenthevirtualdevicesontheguest'ssideandthevirtualIOservicesintheprimary
domainarecreatedbythehypervisor.Itusessocalled"LogicalDomainChannels"orLDCstocreate
pointtopointconnectionsbetweenallofthesedevicesandservices.TheseLDCsworkverysimilarto
highspeedserialconnectionsandareconfiguredautomaticallywhenevertheControlDomainaddsor
removesvirtualIO.
Toseeallthisinaction,nowletslookatafirstexample.Iwillstartwithanewlyinstalledmachineand
configurethecontroldomainsothatit'sreadytocreateguestsystems.
Inafirststep,afterwe'veinstalledthesoftware,let'sstartthevirtualconsoleserviceanddownsizethe
primarydomain.

root@sun#ldmlist
NAMESTATEFLAGSCONSVCPUMEMORYUTILUPTIME
primaryactivencUART512261632M0.3%2d13h58m
root@sun#ldmaddvconsconportrange=50005100\
primaryconsoleprimary
root@sun#svcadmenablevntsd
root@sun#svcsvntsd
STATESTIMEFMRI
online9:53:21svc:/ldoms/vntsd:default
root@sun#ldmsetvcpu16primary
root@sun#ldmsetmau1primary
root@sun#ldmstartreconfprimary
root@sun#ldmsetmemory7680mprimary
root@sun#ldmaddconfiginitial
root@sun#shutdownyg0i6

SowhathaveIdone:
I'vedefinedarangeofports(50005100)forthevirtualnetworkterminalserviceandthenstarted
thatservice.Thevntswilllaterprovideconsoleconnectionstoguestsystems,verymuchlike
serialNTS'sdointhephysicalworld.
Next,Iassigned16vCPUs(onthisplatform,aT34,that'stwocores)totheprimarydomain,
freeingtherestupforfutureguestsystems.IalsoassignedoneMAUtothisdomain.AMAUisa
cryptounitintheT3CPU.Theseneedtobeexplicitlyassignedtodomains,justlikeCPUor
memory.(ThisisnolongerthecasewithT4systems,wherecryptoisalwaysavailable
everywhere.)
BeforeIreassignedthememory,Istartedwhat'scalleda"delayedreconfiguration"session.That
avoidsactuallydoingthechangerightaway,whichwouldtakeaconsiderableamountoftimein
thiscase.Instead,I'llneedtorebootonceI'malldone.I'veassigned7680MBofRAMtothe
primary.That's8GBlessthe512MBwhichthehypervisorusesforit'sownprivatepurposes.You
can,dependingonyourneeds,workwithless.I'llspendadedicatedarticleonsizing,discussing
theprosandconsindetail.
Finally,justbeforethereboot,IsavedmyworkontheILOM,tomakethisconfigurationavailable
afterapowercycleofthebox.(It'llalwaysbeavailableafterasimplereboot,buttheILOMneeds
https://blogs.oracle.com/cmt/entry/what_s_up_with_ldoms

3/7

4/27/2015

What'supwithLDoms:Part1Introduction&BasicConcepts(SPARC,Solaris,PerformanceundderganzeRest)

toknowtheconfigurationofthehypervisorafterapowercycle,beforetheprimarydomainis
booted.)
Now,letscreateafirstdiskserviceandafirstvirtualswitchwhichisconnectedtothephysicalnetwork
deviceigb2.Wewilllaterusethesetoconnectvirtualdisksandvirtualnetworkportsofourguest
systemstorealworldstorageandnetwork.

root@sun#ldmaddvdsprimaryvdsprimary
root@sun#ldmaddvswitchnetdev=igb2switchprimaryprimary

Youarefreetochoosewhatevernamesyoulikeforthevirtualdiskserviceandthevirtualswitch.I
stronglyrecommendthatyouchoosenamesthatmakesensetoyouanddescribethefunctionofeach
serviceinthecontextofyourimplementation.Forthevswitch,forexample,youcouldchoosenames
like"adminvswitch"or"productionnetwork"etc.
Thisalreadyconcludestheconfigurationofthecontroldomain.We'vefreedupconsiderableamountsof
CPUandRAMforguestsystemsandcreatedthenecessaryinfrastructureconsole,vtsandvswitchso
thatguestssystemscanactuallyinteractwiththeoutsideworld.Thesystemisnowreadytocreate
guests,whichI'lldescribeinthenextsection.
Forfurtherreading,herearesomerecommendablelinks:
TheLDoms2.2AdminGuide
The"BeginnersGuidetoLDoms"
TheLDomsInformationCenteronMOS
LDomsonOTN
Category:CMT
Tags:ldomsparcvirtualization
Permanentlinktothisentry
OracleVMServerfor...|Main|SecureDeploymentof...
Kommentare:
SendenSieeinenKommentar:
Kommentaresindausgeschaltet.
About

Neuigkeiten,TippsundWissenswertesrundumSPARC,CMT,PerformanceundihreAnalysesowie
ErfahrungenmitSolarisaufdemServerunddemLaptop.
Thisisabilingualblog(mostofthetime).Pleaseselectyourpreferedlanguage:

.
TheviewsexpressedonthisblogaremyownanddonotnecessarilyreflecttheviewsofOracle.
https://blogs.oracle.com/cmt/entry/what_s_up_with_ldoms

4/7