Yahoo!

WhitePaper

Oracle11gR2RealApplicationClusters InstallationusingAutomaticStorage Management

By AshwinNellore RiteshRajkaranChhajer SE&ODBATeam

Yahoo!WhitePaper

INTRODUCTION
OracleoffersDatabase11gR2EnterpriseEditionforfulfillingspecificbusinessrequirementsbyEnterprise levelcustomers.OracleRealApplicationClustersenhancesthealreadyexistingdatabasespecificfeatures byaddinghighavailability,scalabilityandfailovercapabilitieswhichenableYahoo!toexploitthemasa backendforitsmissioncriticalapplicationsandreportingenvironments. Oraclereleased11gRelease2inSeptember,2009tocustomersintroducingnewfeaturesacrosstheboard forApplicationDevelopment,BusinessIntelligenceandDataWarehousing,Clustering,Database, Diagnostics,InformationIntegration,Performance,SecurityandServerManageability.Filteringthesefrom adatabaseperspective,featureslikeAdvancedCompression,ActiveDataguardandmostimportantlyGrid InfrastructureServicesforOracleRealApplicationClusters(RAC)andadditionalfeaturesinOracles volumemanagerAutomaticStorageManagement(ASM)werethekeychangeswhichinvolvedan entirelydifferentlearningcurvefromaDatabaseAdministratorsperspective. WearealreadyfamiliarwiththeconceptofOracleRealApplicationClustersintermsofrunningmultiple instancesofthesamedatabaseacrossmultipleserversmonitoredbyasetofglobalandlocaldaemonsto ensurehighavailability,linearscalabilityandfailoverincaseofservercrashesorhardwareissues.We havebeenabletosuccessfullydeploysuchclusteredenvironmentsacrosstheboardfor95%+ofOracle databasesinProductionaswellasNonProduction(DEV,QA,INTandSTG)currentlybeingsupportedby theSE&ODBATeam. OracleASMisavolumemanagerandafilesystemforOracleDatabasefilesintroducedwithOracle10gin 2004.ItsupportssingleinstanceaswellasRACconfigurationandprovidesanalternativetoconventional volumemanagers,filesystemsandrawdeviceswhichhavebeeninuse. AtYahoo!ourASMdeploymentsresideprimarilyonEMCsDMXandClariionfamilyofSANproviders.We havesuccessfullyutilizedthissolutionforbothourOLTPaswellasDataWarehouseenvironments,casein pointbeingRightmedia(NGD)running2activeactive5nodeOracle10.2.0.4RACandASMonEMCCX960 inSP2andAC4forOLTPandourDataWarehouseenvironmentEdwardrunninga10nodeOracle10.2.0.4 RACandASMonEMCDMXinSP1andAC4colos. Understandably,wewerequiteexcitedbyOracles11gR2RACandASMenhancementsanddecidedto exploitthesefeaturesforourDataandAnalytics(DNA)projectwhichisexpectedtoempowerY! employeesaroundtheglobetomeasureandunderstandthehealthofYahoo!Products,Audienceand NetworksbyprocessingfeedscomingfromYahoo!Gridwithina11gR2databaseandgeneratingMSTRand MicrostrategyReportsforProductmanagersatYahoo!makedatadrivendecisions. Withthis,wesetouttoarchitectourfirstmajor11gR2deployment.

Yahoo!WhitePaper

ARCHITECTURE
Theproductiondatabasearchitectureiscomprisedof6DellPowerEdgeR710serversrunningRHEL5.3 Linuxx86_64architecturewith72GBRAMeachand2quadportQLogicNICcardsconnectedviaBrocade 5300FiberswitchtoanEMCClariionCX4960SANdevicewithacapacityof41Tusablespacecarvedinto LUNsof1Teach. InfinibandswitcheswereconfiguredtouseIPoverInfiniband(IPoIB)resultinginimprovedbandwith, latencyandreliabilitycharacteristicsascomparedtoGigabitEthernet.Inourcase,thesewereconfigured toprovideathroughputof20GB/sec. O/SspecificOracleASMLibrarieshttp://www.oracle.com/technology/tech/linux/asmlib/index.htmlwere installedandconfiguredoneachhost.TheselibrariesenableO/SdaemonsofOracleASMtowrite informationintotheDiskheadersofeachLUNandmarkitasanASMspecificdiskanddetectingthem duringstartup. ForRealApplicationClusters,theentireOracleClusterwarestackhasbeenrearchitectedwiththemajor notablepointsbeing OracleClusterwareandOracleASMresideunderasingleGridInfrastructureHome(GridHome) whichisindependentofOracleDatabaseHome. OracleClusterRegistry(OCR)FileresponsibleformanagingallresourcesinaclusterandVotingDisk whichisresponsiblefornodemembershipcanbestoredwithinOracleASM. Newdaemonsintroducedwith11gR2gridstackformanagingresources.Theirdependenciesand runtimesequencearehighlightedinthediagrambelow. OracleHighAvailabilityServicesDaemon(OHASD)isthelowerlevelstackwhileClusterReady Services(CRS)isthehigherlevelstackinOracleClusterware. ClusterTimesynchronizationservicecanbeusedinsteadofNetworkTimeProtocol(NTP). OnASMside,themajorchangesareasbelow CRSandVotingfilesarestoredwithinASMunderanydiskgroupconfiguredduringinstallation, occupyingabout300MBofspace. ASMClusterFileSystem(ACFS)hasbeenintroducedtostoreapplicationfilesonASMaswell. MajorenhancementshavebeendoneonASMCMDthecommandlineutility,mostimportantly thefeaturestobackupdiskgroupmetadataincaseofheadercorruptionandcopyingoffilesfrom ASMtononASMandviceversa. AnewroleofSYSASMisdefinedwhichownsASMdatabaseseparatefromSYSDBAaccount. IntelligentDataPlacementfeatureintroducedin11gR2enablesfrequentlyaccesseddatatobe placedontheoutermosttracksofthediskforfasteraccess. ThedefaultAllocationUnit(AU)sizeofanASMgranulewhichis1Mcanbechangedupto64M whilecreatingdiskgroups.

Yahoo!WhitePaper

ImageSource:MetalinkNote:1053147.1

INSTALLATION
Highlightedbelowarescreenshotsfromthe11gR2installationwhichwasperformedusingtheGraphical UserInterface(GUI)andSilentmode.Comparingthistoa10gRACInstallation,youwillfindmajor differencesintheClusterwareinstallationsegment. Wewillgothroughtheinstallationscreenshotsanddescribeanynewfeatures/changesaccordingly. Beforeweinitiateinstallation,therearecertainprerequisitestobefulfilledforOracletoinstantiate. RPMS ThelistofO/SspecificrpmsforRHEL5.3Linuxx86_64wereinstalledandconfigured.Additionalrpmsfor 11gR2installationareunixODBCrelatedwhichneedtobeconfiguredonlyifODBCsetupisbeingutilized.

Yahoo!WhitePaper

ASM ASMLib2.0relatedrpmsoneachRAChostwhichareO/SandKernelversionspecific,needtobeinstalled asrootuser #rpmUvhoracleasmsupport2.1.31.el5.x86_64.rpm #rpmUvhoracleasm2.6.18128.el52.0.51.el5.x86_64.rpm #rpmUvhoracleasmlib2.0.41.el5.x86_64.rpm SincePowerpathsoftwareisinstalledforEMCSAN,wewillformatthedisksandcreateASMdisksusingthe belowprocedure: #/etc/init.d/oracleasmconfigure WefollowLUN>DiskmappinginanorderwhereeveryLUNnumbershouldcorrespondtotheDisk numbere.g.LUN1>DISK1etc. Usingfdiskwewillformatalltheemcpower*deviceslistedunder/proc/partitions Onceformatted,wewillcreateASMdisksforalltheLUNsusingthebelowprocedure: #/etc/init.d/oracleasmcreatediskDISK1/dev/emcpoweraj1 Wewillscanthedisksonallotherclusternodessothattheyarevisibleacrosstheentirecluster #/etc/init.d/oracleasmscandisks #/etc/init.d/oracleasmlistdisks USERS&GROUPS CreateOracleuseroracleandprimarygroupdbawhichwillowntheOraclesoftwareoneachserver. Theuidandgidoftheusershouldbethesameacrossallthenodesinacluster. CreatehomedirectoriesfororacleuserandalsoaseparatedirectorywhereOracleGridInfrastructurewill beinstalled.TheconceptofOracleBaseisextremelyimportantstartingwith11gR2. ItismandatorytocreatethisOracleBaseasaseparatedirectoryfromOracleGridInfrastructure Postinstallation,OraclechangespermissionsofGridHometorootandhenceGridHomeandDatabase Homecannotresideunderthesameparentdirectory #groupaddg500dba #useraddu500d/home/oraclegdbaGdbas/usr/local/bin/zshoracle #lns/home/oracle/oracle #mkdir/home/oragrid #lns/home/oragrid/oragrid #lsl/oracle/oragrid #chownRoracle:dba/home/oracle #chownRoracle:dba/home/oragrid

Yahoo!WhitePaper

SCAN(SingleClientAccessName)isthevirtualhostnametoprovideforallclientsconnectingtothe cluster.SCANisadomainnameregisteredtoatleastoneanduptothreeIPaddresses,eitherinthe domainnameservice(DNS)ortheGridNamingService(GNS). Bydefault,thenameusedastheSCANisalsothenameoftheclusterandmustbegloballyunique throughoutyourenterprise.SCANVIPaddressesmustbeonthesamesubnetasvirtualIPaddressesand publicIPaddresses. Forinstallationtosucceed,theSCANNAMEandIPmustbedefinedandmustresolvetoatleastone address. BecausetheSCANisassociatedwiththeclusterasawhole,ratherthantoaparticularnode,theSCAN makesitpossibletoaddorremovenodesfromtheclusterwithoutneedingtoreconfigureclients.Italso addslocationindependenceforthedatabases,sothatclientconfigurationdoesnothavetodependon whichnodesarerunningaparticulardatabase. NTP SinceweareusingNetworkTimeProtocol(NTP)forsynchronizationoftimeacrossalltheserversinthe cluster,amandatoryrequirementwith11gR2istoenabletheslewingoptionbyaddingxargumentin thentpconfigurationfileasseenbelow: #cat/etc/sysconfig/ntpd #Droproottoid'ntp:ntp'bydefault. OPTIONS="xuntp:ntpp/var/run/ntpd.pid" #Setto'yes'tosynchwclockaftersuccessfulntpdate SYNC_HWCLOCK=no #Additionaloptionsforntpdate NTPDATE_OPTIONS="" CLUVFY ToverifyiftheinstallationfulfillstheUserequivalence,SSHsetup,User&Groupinformationaswellas Public,VIPandInterconnectinformationweruntheclusterverificationutilityprovidedbyOraclefromthe Gridsoftwarelocation/DVD #./runcluvfy.shstageprecrsinstnnode1,node2verbose LetuswalkthroughthescreenshotsforOracleGridInfrastructure(Clusterware)installation:

Yahoo!WhitePaper

AfterchoosingInstallandConfigureGridInfrastructureforaCluster,weproceedwithAdvanced InstallationandchoosetheLanguageswewanttheinstallationtoreachtotheSCANNamescreen.

WithoutSCANName,theinstallationwillnotproceedincaseofanew11gR2Installationorevenupgrade ofa10gR2installation.PleaseworkwithSATeamtogetaSCANnamewhichisuniqueforthecluster.

Yahoo!WhitePaper

WeprovidethelistofserversandVirtualIPsinthisscreenwhichwillcompriseourRACcluster.

Thepublicnetworkisconfiguredoneth0andprivateinterconnectforclustercommunicationisconfigured onib0(Infiniband).

Yahoo!WhitePaper

OCRandVotingdiskswillbestoredwithinASMunderdiskgroup+DG_ARCHwhichwillalsobea placeholderforOraclesarchivelogfilesandsincewehaveimplementedRAID5,wewillbeusingexternal redundancyforboththesefiles.

IPMI(IntelligentPlatformManagementInterface)isanewfeaturewherehardwarebasedcontrollersareconfigured formemberkillescalationandnodefencingoperationswithoutcooperationfromClusterware.

Yahoo!WhitePaper

OracleBaseandGridHomeareunderseparatedirectorieswhichismandatoryfor11gR2Installation.

Thisisanewscreenfortheinstaller,wherealltheprerequisitesareverifiedandacolumnFixableensuresthata runtimescriptrunfixup.shisgeneratedandwhenrun,fixesallparameterswhichcanbemodifiedonline.

Yahoo!WhitePaper

Theabovescreenensuresthatinstallationwascompletedandroot.shshouldberunonallclusternodestoconfigure andstarttheGridInfrastructuredaemons.
#shroot.sh RunningOracle11groot.shscript... Thefollowingenvironmentvariablesaresetas: ORACLE_OWNER=oracle ORACLE_HOME=/oragrid/product/11.2 Enterthefullpathnameofthelocalbindirectory:[/usr/local/bin]: Copyingdbhometo/usr/local/bin... Copyingoraenvto/usr/local/bin... Copyingcoraenvto/usr/local/bin... Creating/etc/oratabfile... Entrieswillbeaddedtothe/etc/oratabfileasneededby DatabaseConfigurationAssistantwhenadatabaseiscreated Finishedrunninggenericpartofroot.shscript. Nowproductspecificrootactionswillbeperformed. 2010042813:48:29:Parsingthehostname 2010042813:48:29:Checkingforsuperuserprivileges 2010042813:48:29:Userhassuperuserprivileges Usingconfigurationparameterfile:/oragrid/product/11.2/crs/install/crsconfig_params Creatingtracedirectory LOCALADDMODE CreatingOCRkeysforuser'root',privgrp'root'.. Operationsuccessful. rootwallet rootwalletcert rootcertexport peerwallet profilereaderwallet pawallet peerwalletkeys pawalletkeys peercertrequest pacertrequest peercert pacert peerrootcertTP profilereaderrootcertTP

Yahoo!WhitePaper
parootcertTP peerpacertTP papeercertTP profilereaderpacertTP profilereaderpeercertTP peerusercert pausercert Addingdaemontoinittab CRS4123:OracleHighAvailabilityServiceshasbeenstarted. ohasdisstarting CRS2672:Attemptingtostart'ora.gipcd'on'gq1dnadb001' CRS2672:Attemptingtostart'ora.mdnsd'on'gq1dnadb001' CRS2676:Startof'ora.gipcd'on'gq1dnadb001'succeeded CRS2676:Startof'ora.mdnsd'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.gpnpd'on'gq1dnadb001' CRS2676:Startof'ora.gpnpd'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.cssdmonitor'on'gq1dnadb001' CRS2676:Startof'ora.cssdmonitor'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.cssd'on'gq1dnadb001' CRS2672:Attemptingtostart'ora.diskmon'on'gq1dnadb001' CRS2676:Startof'ora.diskmon'on'gq1dnadb001'succeeded CRS2676:Startof'ora.cssd'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.ctssd'on'gq1dnadb001' CRS2676:Startof'ora.ctssd'on'gq1dnadb001'succeeded ASMcreatedandstartedsuccessfully. DiskGroupDG_ARCHcreatedsuccessfully. clscfg:installmodespecified SuccessfullyaccumulatednecessaryOCRkeys. CreatingOCRkeysforuser'root',privgrp'root'.. Operationsuccessful. CRS2672:Attemptingtostart'ora.crsd'on'gq1dnadb001' CRS2676:Startof'ora.crsd'on'gq1dnadb001'succeeded CRS4256:Updatingtheprofile Successfuladditionofvotingdisk57c48a165f8d4fcbbff98bf735ff0784. Successfullyreplacedvotingdiskgroupwith+DG_ARCH. CRS4256:Updatingtheprofile CRS4266:Votingfile(s)successfullyreplaced ##STATEFileUniversalIdFileNameDiskgroup 1.ONLINE57c48a165f8d4fcbbff98bf735ff0784(ORCL:DISK1)[DG_ARCH] Located1votingdisk(s). CRS2673:Attemptingtostop'ora.crsd'on'gq1dnadb001' CRS2677:Stopof'ora.crsd'on'gq1dnadb001'succeeded CRS2673:Attemptingtostop'ora.asm'on'gq1dnadb001' CRS2677:Stopof'ora.asm'on'gq1dnadb001'succeeded CRS2673:Attemptingtostop'ora.ctssd'on'gq1dnadb001' CRS2677:Stopof'ora.ctssd'on'gq1dnadb001'succeeded CRS2673:Attemptingtostop'ora.cssdmonitor'on'gq1dnadb001' CRS2677:Stopof'ora.cssdmonitor'on'gq1dnadb001'succeeded CRS2673:Attemptingtostop'ora.cssd'on'gq1dnadb001' CRS2677:Stopof'ora.cssd'on'gq1dnadb001'succeeded CRS2673:Attemptingtostop'ora.gpnpd'on'gq1dnadb001' CRS2677:Stopof'ora.gpnpd'on'gq1dnadb001'succeeded CRS2673:Attemptingtostop'ora.gipcd'on'gq1dnadb001' CRS2677:Stopof'ora.gipcd'on'gq1dnadb001'succeeded CRS2673:Attemptingtostop'ora.mdnsd'on'gq1dnadb001' CRS2677:Stopof'ora.mdnsd'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.mdnsd'on'gq1dnadb001' CRS2676:Startof'ora.mdnsd'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.gipcd'on'gq1dnadb001' CRS2676:Startof'ora.gipcd'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.gpnpd'on'gq1dnadb001' CRS2676:Startof'ora.gpnpd'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.cssdmonitor'on'gq1dnadb001' CRS2676:Startof'ora.cssdmonitor'on'gq1dnadb001'succeeded

Yahoo!WhitePaper
CRS2672:Attemptingtostart'ora.cssd'on'gq1dnadb001' CRS2672:Attemptingtostart'ora.diskmon'on'gq1dnadb001' CRS2676:Startof'ora.diskmon'on'gq1dnadb001'succeeded CRS2676:Startof'ora.cssd'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.ctssd'on'gq1dnadb001' CRS2676:Startof'ora.ctssd'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.asm'on'gq1dnadb001' CRS2676:Startof'ora.asm'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.crsd'on'gq1dnadb001' CRS2676:Startof'ora.crsd'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.evmd'on'gq1dnadb001' CRS2676:Startof'ora.evmd'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.asm'on'gq1dnadb001' CRS2676:Startof'ora.asm'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.DG_ARCH.dg'on'gq1dnadb001' CRS2676:Startof'ora.DG_ARCH.dg'on'gq1dnadb001'succeeded CRS2672:Attemptingtostart'ora.registry.acfs'on'gq1dnadb001' CRS2676:Startof'ora.registry.acfs'on'gq1dnadb001'succeeded gq1dnadb0012010/04/2813:52:44/oragrid/product/11.2/cdata/gq1dnadb001/backup_20100428_135244.olr Preparingpackagesforinstallation... cvuqdisk1.0.71 ConfigureOracleGridInfrastructureforaCluster...succeeded Updatinginventorypropertiesforclusterware StartingOracleUniversalInstaller... Checkingswapspace:mustbegreaterthan500MB.Actual12287MBPassed Theinventorypointerislocatedat/etc/oraInst.loc Theinventoryislocatedat/oracle/oraInventory 'UpdateNodeList'wassuccessful.

Afterrunningthisscriptonallnodes,OracleGridInfrastructureshouldbeupandrunningonallthenodesandcan beverifiedusingthebelowcommand: #<GRID_HOME>/bin/crsctlstatusresourcet Once,GridInfrastructureInstallationiscompletedsuccessfullywewillproceedwithinstallingOracleDatabase softwareonalltheclusternodes.

Yahoo!WhitePaper

ChooseallnodesoftheclustertoinstallOracleRACDatabasesoftware

OracleHomeisconfiguredunder/oracle/product/11.2

Yahoo!WhitePaper

ThenewprerequisitecheckfeatureisavailableforDatabasesoftwareinstallationtoo.

Finally,aftercompletingtheDatabasesoftwareinstallationandrunningroot.shonallclusternodeswecanconclude theinstallationofOracle11gR2RealApplicationClustersandAutomaticStorageManagementinstallation.

CONCLUSION
TherearemajordifferencesintheClusterwarestackfor11gR2ascomparedto10gR2andeveryDatabase Administratorshouldgothroughthe11gR2technicalstacktounderstandthechangesOraclehasmade.