2017/6/2 AdvancedStoragePerformanceMonitoringwithNutanix|CloudXC

AdvancedStoragePerformanceMonitoringwith
Nutanix
PostedonJune25,2015byJoshOdgers

NutanixprovidesexcellentperformancemonitoringandanalyticcapabilitiesthroughourHTML5basedPRISMUI,butwhatifyou
wanttodelvedeeperintotheperformanceofaspecificbusinesscriticalapplication?

Nutanixalsoprovidesadvancedstorageperformancemonitoringandworkloadprofilingthroughport2009onanyCVMwhichshows
verygranulardetailsforVirtualdisks.

Bydefault,NutanixsecuresourCVMandthehttp://CVM_IP:2009pageisnotaccessible,butforadvancedtroubleshootingthiscanbe
enabledbyusingthefollowingcommand.

sudoiptablestfilterAWORLDLISTptcpmtcpdport2009jACCEPT

Whenaccessingthe2009page(whichispartoftheNutanixprocesscalledStargate)youwillseethingslikeExtent(InMemory
Read)cacheusagesandhitsaswellasmuchmore.

Onthemain2009pageyouwillseeasectioncalledHostedVDisks(shownbelow)whichshowsallthecurrentVDisks(equivalentof
aVMDKinESXi)whicharecurrentlyrunningonthatnode.

TheHostedVDisksshowshighleveldetailsabouttheVDisksuchasOutstandingOperations,capacityusage,Read/Writebreakdown
andhowmuchdataisintheOpLog(PersistentWriteCache).

Ifyouneedmoreinformation,youcanclickontheVDiskIdandyouwillgettoapagetitledVDiskXXXXXStatswheretheXXXXX
istheVDiskID.

ThebelowissomeoftheinformationwhichcanbediscoveredintheVDiskStatsPage.

VDiskWorkingSetSize(WWS)

TheworkingsetsizecanbethoughtofasthedatawhichyouwouldideallywanttofitwithintheSSDtierofaNutanixnode,which
wouldresultinallflashtypeperformance.

Inthebelowexample,inthelast2mins,theVDiskhadacombined(orUnion)workingsetof6.208GBandoverthelast1hrover
111GB.

http://www.joshodgers.com/2015/06/25/advancedstorageperformancemonitoringwithnutanix/ 1/5
2017/6/2 AdvancedStoragePerformanceMonitoringwithNutanix|CloudXC

VDiskReadSource

TheReadSourceissimplywhattierofstorageisservicingtheVDisksIOrequests.Inthebelowexample,41%wasfromExtent
Cache(InMemory),7%wasfromtheSSDExtentStoreand52%wasfromtheSATAExtentStore.

Intheaboveexample,thiswasanExchange2013workloadwherethetotaldatasetwasapprox5xthesizeoftheSSDtier.The
importantpointhereisitsnotalwayspossibletohavealldataintheSSDtier,butitscriticaltoensureconsistentperformance.If90%
wasbeingservedfromSATAandperformancewasnotacceptable,youcouldusethisinformationtoselectabetternodetomigrate
(vMotion)theVMtoo,orhelpchoosetopurchaseanewnode.

VDiskWriteDestination

TheWriteDestinationisfairlyselfexplanatory,ifitsOplogitmeansitsRandomIOanditsbeingwrittentoSSD,ifitsstraighttothe
extentstore(SSD)itmeanstheIOiseithersequential,ORinrarecasestheOpLogisbeingbypassediftheSSDtierreached95%full
(whichisgenerallypreventedbyNutanixILMtieringprocess).

VDiskWriteSizeDistribution

TheWriteSizeDistributioniskeytodeterminingthingsliketheWindowsAllocationSizewhenformattingdrivesaswellas
understandingtheworkload.

http://www.joshodgers.com/2015/06/25/advancedstorageperformancemonitoringwithnutanix/ 2/5
2017/6/2 AdvancedStoragePerformanceMonitoringwithNutanix|CloudXC

VDiskReadSizeDistribution

TheReadSizeDistributionissimilartoWriteSizeinthatitskeytodeterminingthingsliketheWindowsAllocationSizewhen
formattingdrivesaswellasunderstandingtheworkload.Inthiscase,a64kallocationsizewouldbeidealasboththeWrite(shown
above)andtheRead(below)are>32Kand<64K86%ofthetime.(WhichisexpectedasthiswasanExchange2013workload).

VDiskWriteLatency

TheWriteLatencyshowsthepercentageofWriteI/Owhichareservicedwithinthelatencyrangesshown.Inthiscase,52%ofwrites
aresubmillisecond.ItalsoshowsforthisvDisk1%ofIObeingoutliersbeingservedbetween510ms.Thisissomethingthatoutside
ofalab,iftheoutlierswereasignificantpercentagethatcouldbeinvestigatedtoensuretheVMdiskconfiguration(e.g.:PVSCSIand
numberofVMDKs)isoptimal.

VDiskOpsandRandomness

HereweseethenumberofIOPS,theRead/Writesplit,MB/sandthesplitbetweenRandomandSequential.

http://www.joshodgers.com/2015/06/25/advancedstorageperformancemonitoringwithnutanix/ 3/5
2017/6/2 AdvancedStoragePerformanceMonitoringwithNutanix|CloudXC

Summary

Foranyenterprisegradestoragesolution,itisimportantthatperformancemonitoringbeeasyasitiswithNutanixviaPRISMUI,but
alsotobeabletoquicklyandeasilydivedeepintoverygranulardetailsaboutaspecificVMorVDisk.Theaboveshowsjustaglimpse
oftheinformationwhichistrackedbydefaultforallVDisksallowingcustomers,partnersandNutanixsupporttoquicklyandeasily
monitor&profileworkloads.

Importantlythesecapabilitiesarehypervisoragnosticgivingcustomersthesamecapabilitiesnomatterwhatchoice/stheymake.

RelatedPosts:

1.ScalingHyperconvergedsolutionsComputeonly.

2.AcropolisHypervisor(AHV)I/OFailover&LoadBalancing

3.AdvancedStoragePerformanceMonitoringwithNutanix

4.NutanixImprovingResiliencyofLargeClusterswithErasureCoding(ECX)

5.NutanixErasureCoding(ECX)DeepDive

6.Acropolis:VMHighAvailability(HA)

7.Acropolis:Scalability

8.NOS&HypervisorUpgradeResiliencyinPRISM

SH A R ETH IS:

LinkedIn 187 Twitter Facebook Google Pinterest

ThisentrywaspostedinAcropolis,Nutanixandtaggedadvanced,disk,hyperv,hyperv,KVM,Nutanix,performance,scaleout,
vdisk,VMDK,vmware,vSpherebyJoshOdgers.Bookmarkthepermalink[http://www.joshodgers.com/2015/06/25/advanced
storageperformancemonitoringwithnutanix/].

6THOUGHTSONADVANCEDSTORAGEPERFORMANCEMONITORINGWITHNUTANIX

KyleMartin
onJune29,2015at4:49pmsaid:

HiJosh,

Justnoticedyourenablecommandismissinganextrainfrontofdport.sudoiptablestfilterAWORLDLISTp
tcpmtcpdport2009jACCEPT

KyleMartin
onJune29,2015at4:51pmsaid:

GuessingitsaWordPressformattingissueasitjustdidthesametomycommand

http://www.joshodgers.com/2015/06/25/advancedstorageperformancemonitoringwithnutanix/ 4/5
2017/6/2 AdvancedStoragePerformanceMonitoringwithNutanix|CloudXC

Pingback:NutanixAllFlash,OnlyWhenRequiredVMPinningmyvirtualcloud.net

derek
onAugust23,2015at5:23pmsaid:

Awesomefeature.
Ihavesearchingthenetformorethanayearforadecentworkingseatmeasurementtool,andNutanixhadis
alreadybuiltin

Sami
onMay7,2016at8:55pmsaid:

WhilerunningdiskspdtestInoticedthatmajorityoftheread/writesareservedbyeitherOplogZeroorEstore
Zero.IamguessingthatNutanixrecognizesallzerocontentwhichisthendirectedtothesedestinationsandnot
reallywrittenonSSD/HDD

JoshOdgers
onMay7,2016at9:14pmsaid:

HiSami,

Ratherthanbruteforcecompressingordeduplicatingzeros/whitespace,Nutanixinsteadsimplywrites
metadatawhichavoidswastingspace,controllerCPUandimpactonSSDs.

IfyouusetoolslikeJetstresswhichalsowriteslotsofzeros(whicharehighlycompressible)youwillsee
similartowhatyourobservingwithdiskspd.

Icallthisdataavoidance.

Hopethatclearsthingsup.

Cheers

Josh

Youmustlogintopostacomment.

http://www.joshodgers.com/2015/06/25/advancedstorageperformancemonitoringwithnutanix/ 5/5