Virtualization, Oracle RAC 11g and more..

Vijay Ragunathan
Guidewire Software Inc
vragunathan@guidewire.com

Agenda

Introduction Set up VM and RAC Take a look at a few 11g RAC/ASM features Introduce Cache fusion and trace Cache Fusion

Guidewire Software Inc

Virtualization

Abstraction layer between applications and hardware Methodology to divide the resources of a computer hardware into multiple operating environments Partition at various levels like Storage, Network, Server, Application

Guidewire Software Inc

Why Virtualize

Consolidate IT infrastructure Improve QoS Reduce cost Provide more reliable working environment for development and testing Ease replication of environments Partition applications to get security, reliability and flexibility
Guidewire Software Inc

Where can we Virtualize

Development and Test Virtual Desktops PoC deployments Disaster Recovery and Business continuity Technology refresh

Guidewire Software Inc

Evolution

Virtualization in place for three decades Mainframes RISC based systems And now commodity hardware

Guidewire Software Inc

Commodity Virtualization Software

VmWare Xen Oracle VM (Xen) Microsoft Virtual PC

And probably many more

Guidewire Software Inc

Oracle Real Application Cluster

Cluster is two or more nodes sharing some common resources They communicate with each other to check health and guarantee data integrity For Oracle, it is two or more Oracle instances sharing a physical database Each instance use their own SGA and background processes Database components like Datafiles, control files and redo log files are shared
Guidewire Software Inc

Real World Oracle RAC Implementations

RISC or CISC based processers on two or more servers A SAN or NAS for storing database Interconnect mechanisms like Infiniband, Gigabit Ethernet, Memory channel

Guidewire Software Inc

Real Application Clusters

Node1

Node2

Storage

Guidewire Software Inc

VM and Oracle RAC

Hardware for clustering too costly Difficult to always build dev and test environments with clustering Typically Applications built on single instance and tested and deployed on RAC VM is Beneficial for small, medium and large companies and cost effective
Guidewire Software Inc

Real Application Clusters

Virtual Node1

Virtual Node2

Same Disks in Host

Physical Machine
Guidewire Software Inc

Setup Xen

Xen Architecture

Xen hypervisor has management and virtual hardware API Dom0 hosts the management code and has control interfaces to manage other domains All guest Virtual machines are referred as domU

Guidewire Software Inc

Packages

xen-3.0.3-41.el5 kernel-xen-2.6.18-53.el5 xen-libs-3.0.3-41.el5 kernel-xen-devel-2.6.18-53.el5

Guidewire Software Inc

Dom0 Host

Default Centos 5.1 install Virtual Machine Manager GUI Command line tool virt-install

Guidewire Software Inc

Virtual Machine Manager

The GUI VMM displays all the domains and their resource usage

Guidewire Software Inc

Create a VM

File -> New Machine

Guidewire Software Inc

Name the VM

We name the first VM as dtrac41

Guidewire Software Inc

Virtualization Method

Guidewire Software Inc

ParaVirtualization

ParaVirtualization uses a modified kernel Not very portable Not all hardware is simulated Best performing for Network and IO Unique address map for each VM Easy to implement

Guidewire Software Inc

Fully Virtualized

Complete Hardware simulation Any supported guest OS can be used Not supported by all hardware architecture VT support may need to be enabled in BIOS for some hardware Use techniques to trap and emulate instructions
Guidewire Software Inc

Install using PXE

PXE server enables storing images in central location and install large number of machines Using centos 5.1 for guestVM Kickstart Template file instructs what to install

Guidewire Software Inc

OS and Swap for GuestVM

Using simple files Default path /var/lib/xen/images

Guidewire Software Inc

Network for the GuestVM

Guidewire Software Inc

Memory and CPU

Do not over allocate as we need resources for the dom0 Host and other GuestVMs

Guidewire Software Inc

Summary of the Options and Resources

Guidewire Software Inc

Vm Installation operations

VMM will start allotting the disk storage Create virtual network Map memory to the Dom0 Virtual memory map Will start installing the Guest OS

Guidewire Software Inc

PXE install looks at dhcp server

Guidewire Software Inc

Xen Utilities

xm utility - can be used to create domains - useful for cloning virsh - can be used to change CPU, memory - define config, dump config in xml form for cloning - start, shutdown and reboot domains
Guidewire Software Inc

Xen Utilities

xentop - similar to top but customized to list all virtual resources and domains

xentop - 21:55:32 Xen 3.1.0-53.el5 3 domains: 2 running, 1 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown Mem: 15723456k total, 15721284k used, 2172k free CPUs: 8 @ 2992MHz NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR SSID Domain-0 -----r 9576 2.9 13498460 85.8 no limit n/a 8 4 398061 38880 0 0 0 0 0 dtrac41 -----r 2755 2.1 1048152 6.7 1048576 6.7 1 2 883749 857104 0 0 0 0 0 dtrac42 --b--1302 8.1 1048080 6.7 1048576 6.7 1 2 850349 813954 0 0 0 0 0

Guidewire Software Inc

XenStore

Configuration database for all domain Multiple commands to manage this database xenstore-ls - can be used to list details of all hardware components in each domain

Guidewire Software Inc

Add Virtual Hardware

Add more storage for Oracle clusterware

Guidewire Software Inc

Add Virtual Storage

Specify Simple file or disk partition (if available) Min size of 256Mb for Clusterware

Guidewire Software Inc

Add virtual network interface

For Interconnect use

Guidewire Software Inc

Add virtual network interface

Choose virtual network as this interconnect does not need to talk to the external world

Guidewire Software Inc

Virtual hardware of one guestVM

Guidewire Software Inc

Oracle Install Preparation

Update sysctl.conf with the new kernel parameters rmem_default and wmem_default should be 4MB min Update limits.conf Install any additional RPMs needed as listed in Oracle Install documentation Edit /etc/hosts and add localhost and 6 other addresses to indicate public, private and virtual networks for both guestVMs

Guidewire Software Inc

Create users and groups

groupadd g 10000 oinstall groupadd g 10001 dba groupadd g 10002 asmdba useradd g oinstall Goinstall,dba,asmdba d /home/oracle u 10000 oracle passwd oracle

Guidewire Software Inc

Mapping raw devices

Create something like /etc/rc3.d/S90raw and/or /etc/rc5.d/S90raw based on boot run level with contents like:
raw /dev/raw/raw1 /dev/xvdb raw /dev/raw/raw2 /dev/xvdc raw /dev/raw/raw3 /dev/xvdd raw /dev/raw/raw4 /dev/xvde chown root:dba /dev/raw/raw2 chmod 660 /dev/raw/raw2 chown oracle:dba /dev/raw/raw1 chmod 660 /dev/raw/raw1 chown oracle /dev/raw/raw3 chown oracle /dev/raw/raw4 chmod 660 /dev/raw/raw3 chmod 660 /dev/raw/raw4 Guidewire Software Inc

Cloning GuestVM

In dom0, /etc/xen/dtrac41 has list of devices created for this guest VM Copy /etc/xen/dtrac41 to /etc/xen/dtrac42 Change the mac address of the two virtual network interfaces in dtrac42 and edit the primary disk Modify the name and uuid for the new VM Ensure the order of the network is right in the config files Copy the virtual disk of dtrac41(dtrac41.img) to /var/lib/xen/images/dtrac42.img
Guidewire Software Inc

Config file for guestVM

name = "dtrac41" uuid = "3656ca7c24e378bd2c63d380206a5f8d" maxmem = 1024 memory = 1024 vcpus = 1 bootloader = "/usr/bin/pygrub" kernel = "/var/lib/xen/boot_kernel.qFE_y1" ramdisk = "/var/lib/xen/boot_ramdisk.HlhabB" extra = "ro root=/dev/vgSys/lvRoot rhgb quiet console=tty0 console=ttyS1,57600n8" on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" vfb = [ "type=vnc,vncdisplay=0,keymap=en-us" ] disk = [ "tap:aio:/var/lib/xen/images/dtrac41.img,xvda,w", "tap:aio:/var/lib/xen/images/dtrac4x-ocr11g-shared.img,xvdb,w", "tap:aio:/var/lib/xen/images/dtrac4x-vote-shared.img,xvdc,w", "tap:aio:/var/lib/xen/images/dtrac4x-asm1-shared.img,xvdd,w", "tap:aio:/var/lib/xen/images/dtrac4x-asm2-shared.img,xvde,w" ] vif = [ "mac=00:16:3e:0c:81:f5,bridge=xenbr0,script=vif-bridge", "mac=00:16:3e:75:41:84,bridge=virbr0,script=vif-bridge" ]

Guidewire Software Inc

Cloning GuestVM

Use xm to create the domain xm create dtrac42 Start the new domain xm start dtrac42 You should see both the GuestVMs in the Virtual Machine Manager

Guidewire Software Inc

Setup GuestVMs

Reboot both the virtual machines Ensure fdisk l is showing all the disks Edit the network configuration files to setup the IP addresses and hostname and start the network Ping the IP addresses of the other node and confirm that the network works fine
Guidewire Software Inc

User equivalence Setup

Use ssh-keygen and generate keys Concatenate the public keys for both machines and store as ~/.ssh/authorized_keys Check you can login to all combinations without prompts. For e.g.,
ssh ssh ssh ssh dtrac41 date dtrac42 date dtrac41p date dtrac42p date

Run cluster verify tool and do pre CRS inst check

Guidewire Software Inc

Vmware

Install Vmware server Create two virtual machines using Vmware Server Console For the shared disks, edit the config files (suffixed vmx) and add:
disk.locking = scsi0:1.shared scsi0:2.shared scsi0:3.shared scsi0:4.shared "false = "true" = "true" = "true" = "true"

Guidewire Software Inc

Oracle install media

Download the clusterware and database software Unzip the files

unzip linux_11gR1_clusterware.zip unzip linux_11gR1_database.zip

If using cdrom/dvd, mount the clusterware media first Start the Installer
cd clusterware ./runInstaller Guidewire Software Inc

Clusterware Software Home

Guidewire Software Inc

Cluster Configuration

Cluster name is dtrac4_cluster Edit details of dtrac41 Add details of dtrac42

Guidewire Software Inc

Network Interface Usage

Select the right Interface for use as Public and Private network Here eth0 is chosen as Public and eth1 as private

Guidewire Software Inc

OCR and Voting disk

Both OCR in the previous installer screen and voting disk here should have min of 256MB space Specify mirror copies here we use single voting disk location

Guidewire Software Inc

Clusterware Install Summary

Ensure all cluster nodes are seen

Guidewire Software Inc

Root Scripts

Run orainstRoot.sh and root.sh on all nodes (one at a time)

Guidewire Software Inc

Root.sh on node1

Configures OCR and Voting Disk and starts CRS stack on node1

Guidewire Software Inc

Root.sh on node2

Starts CRS stack on node2 and creates cluster resources

Guidewire Software Inc

Config Assistants for Clusterware

Guidewire Software Inc

Clusterware Install Completion

That completes the clusterware install OCR and Voting disk are configured Resources like ONS and VIP should be created by now and can be seen using crs_stat Processes like evmd, crsd, ocssd and oprocd should be running now Ready to Install RAC software
cd database ./runInstaller

Guidewire Software Inc

Installation Type

Guidewire Software Inc

Node Selection

Choose all nodes in the cluster both VMs

Guidewire Software Inc

OS Authentication

Guidewire Software Inc

RAC Install Summary

Check for selection of all Cluster nodes

Guidewire Software Inc

Create RAC database

Guidewire Software Inc

Create ASM Disk Group

Change disk discovery path and check permissions if disks are not seen

Guidewire Software Inc

Starts Cluster database

Database Creation is complete

Guidewire Software Inc

CRS status
crs_stat -t Name Type Target State Host -----------------------------------------------------------ora....SM1.asm application ONLINE ONLINE dtrac41 ora....41.lsnr application ONLINE ONLINE dtrac41 ora....c41.gsd application ONLINE ONLINE dtrac41 ora....c41.ons application ONLINE ONLINE dtrac41 ora....c41.vip application ONLINE ONLINE dtrac41 ora....SM2.asm application ONLINE ONLINE dtrac42 ora....42.lsnr application ONLINE ONLINE dtrac42 ora....c42.gsd application ONLINE ONLINE dtrac42 ora....c42.ons application ONLINE ONLINE dtrac42 ora....c42.vip application ONLINE ONLINE dtrac42 ora.xenrac.db application ONLINE ONLINE dtrac42 ora....c1.inst application ONLINE ONLINE dtrac41 ora....c2.inst application ONLINE ONLINE dtrac42 Guidewire Software Inc

RAC and ASM

11g New Features

11g New Features

Automatic Diagnostic Repository should be placed in shared disk if using shared oracle home Kill sessions from anywhere in the cluster ADDM for RAC dbms_addm.analyze_db procedure and dbms_addm.get_report function Inter-node parallelism can use services to determine the instances that it can use Grid provisioning and EM changes Clone ASM and RAC homes clone.pl

Guidewire Software Inc

11g ASM New Features

asm_preferred_read_failure_groups parameter ASM rolling migration alter system start rolling migration to <pnumber>; Convert single instance ASM to cluster using rconfig/EM ASM compatibility attributes Variable size extents Higher AU values for Coarse striping 1M 64M
Guidewire Software Inc

11g ASM New features

Ability to drop ASM instances using dbca

dbca silent deleteasm nodelist dtrac41,dtrac42

ASM fast mirror resync SYSASM and OSASM ASMcmd command additions cp, md_backup, lsdsk, remap Mount/drop diskgroup with Force option
Guidewire Software Inc

Cache Fusion

Cache Fusion

Shared cache architecture Cache coherency preserve integrity of data across different caches for the same data Employs high-speed Interconnects Created in Oracle OPS 8i to reduce pings 8i handled read/write contention 9i RAC handles other contentions like writewrite
Guidewire Software Inc

Cache Fusion

Locks mode, role, past image E.g lock : SG1 Modes: Null, Shared, Exclusive Role: Local, Global Past Image component: 0 or 1 Maximum 3-way protocol (master, holder and requestor)
Guidewire Software Inc

Events

Event 10432 for diagnosing unexpected GCS events and lock requests Event 10708 shows block request and grants (level 7 or 103 is common) Event 10046 common sql trace and combines well with event 10708

Guidewire Software Inc

Trace commands
SQL> alter system set events '10432 trace name context forever, level 15:10708 trace name context forever, level 103';

System altered.
SQL> conn scott/tiger Connected. SQL> alter session set events '10046 trace name context forever, level 8'; Session altered.

Guidewire Software Inc

First Read session

Select on node 1

*** 2008-05-15 02:40:36.228 kclscrs: req=0 block=5/36 *** 2008-05-15 02:40:36.228 kclscrs: req=0 typ=immediate mode=S WAIT #3: nam='db file sequential read' ela= 786 file#=5 block#=36 blocks=1 obj#=16215 tim=1210844436231630 *** 2008-05-15 02:40:36.231 KCL: P02: cr prewarm *** 2008-05-15 02:40:36.231 kclscrs: req=0 block=5/28 *** 2008-05-15 02:40:36.231 kclscrs: req=0 typ=immediate mode=S *** 2008-05-15 02:40:36.231 kclscrs: req=1 block=5/29 *** 2008-05-15 02:40:36.231 kclscrs: req=1 typ=immediate mode=S *** 2008-05-15 02:40:36.232 kclscrs: req=7 block=5/25 *** 2008-05-15 02:40:36.232 kclscrs: req=7 typ=immediate mode=S Guidewire Software WAIT #3: nam='db file scattered read' ela= 7342 file#=5 block#=25 blocks=8 obj#=16214 tim=1210844436239474

Inc

Second read from same node

PARSE #9:c=0,e=1478,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,tim=12108 47692139409 EXEC #9:c=0,e=37,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,tim=1210847 692139492 WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1650815232 #bytes=1 p3=0 obj#=523 tim=1210847692139514 FETCH #9:c=0,e=29,p=0,cr=2,cu=0,mis=0,r=1,dep=0,og=1,tim=1210847 692139568

No communication to the other node

Guidewire Software Inc

Write Operation Tracing..

update emp set sal=sal+100 where empno = 7654 END OF STMT PARSE #3:c=20001,e=44395,p=1,cr=55,cu=0,mis=1,r=0,dep=0,og=1,tim =1210848457075954 WAIT #3: nam='enq: TM - contention' ela= 226 name|mode=1414332419 object #=16214 table/partition=0 obj#=-1 tim=1210848457076262 *** 2008-05-15 03:47:37.076 kclscrs: req=0 block=5/36 *** 2008-05-15 03:47:37.076 kclscrs: bid=1:3:2:0:2:c:0:0:2:0:0..

Guidewire Software Inc

Tracing Contd..
Lms trace of remote node
*** 2008-05-15 03:47:37.077 kcl_snd_cur: block=5/36 time=54

Back to update session trace

*** 2008-05-15 03:47:37.077 kclwcrs: got 1 blocks from ksxprcvimd WAIT #3: nam='gc current block 2-way' ela= 413 p1=5 p2=36 p3=1 obj#=16215 tim=1210848457077694 *** 2008-05-15 03:47:37.077 kclwcrs: req=0 typ=cur wtyp=2hop tm=489

Guidewire Software Inc

Tracing Contd..
*** 2008-05-15 03:47:37.140 kcllkopb: req=7 block=5/25 *** 2008-05-15 03:47:37.140 kcllkopb: bid=1:3:2:0:2:14:0.. WAIT #3: nam='gc current multi block request' ela= 27265 file#=5 block#=25 id#=33554446 obj#=16214 tim=1210848457226586 *** 2008-05-15 03:47:37.254 kcllkopb: req=0 typ=cur wtyp=2hop tm=174615 WAIT #3: nam='db file scattered read' ela= 83230 file#=5 block#=25 blocks=3 obj#=16214 tim=1210848457337661

Nothing in the LMS trace Back to sessions trace

Guidewire Software Inc

Causing Write-Write Contention

update emp set sal=sal+100 where empno != 7654 PARSE #5:c=4000,e=49748,p=1,cr=9,cu=0,mis=1,r=0,dep=0,og=1 ,tim=1210848625816852 *** 2008-05-15 03:50:25.817 kclscrs: req=0 block=5/28

*** 2008-05-15 03:50:25.817 kclscrs: bid=1:3:1:0:2:5:0:0:4:0:..

Guidewire Software Inc

Write-Write Contention Tracing..

Remote LMS

*** 2008-05-15 03:50:25.816 kclgcr: req=231 block=5/28 *** 2008-05-15 03:50:25.816 kclgcr: bid=1:3:1:0:2:5:0:0.. *** 2008-05-15 03:50:25.825 kclqsnd: req=231 status=2 fscn=0:b49c2 time=8523 *** 2008-05-15 03:50:25.826 kclpdc: le=1c/50000 rls=49

Guidewire Software Inc

Write-Write Contention Tracing..

Back to local session

*** 2008-05-15 03:50:25.825 kclwcrs: got 1 blocks from ksxprcvimd WAIT #5: nam='gc cr block busy' ela= 8846 p1=5 p2=28 p3=1 obj#=16214 tim=1210848625825992 *** 2008-05-15 03:50:25.825 kclwcrs: got 1 blocks from ksxprcvimd WAIT #5: nam='gc cr block busy' ela= 8846 p1=5 p2=28 p3=1 obj#=16214 tim=1210848625825992 *** 2008-05-15 03:50:25.826 kclwcrs: req=0 typ=cr wtyp=busy tm=9178
Guidewire Software Inc

Write - Write contention tracing

*** 2008-05-15 03:50:25.826 kcllkopb: req=0 block=5/28 *** 2008-05-15 03:50:25.826 kcllkopb: bid=1:3:1:0:2:6:0:0:

LMS trace
*** 2008-05-15 03:50:25.826 kcl_snd_cur: block=5/28 time=64

Local Session Trace

*** 2008-05-15 03:50:25.827 kcllkopb: got 1 blocks from ksxprcvimd WAIT #5: nam='gc current block 2-way' ela= 270 p1=5 p2=28 p3=33554433 obj#=16214 tim=1210848625827163 *** 2008-05-15 03:50:25.827 kcllkopb: req=0 typ=cur wtyp=2hop tm=417

Guidewire Software Inc

Q& A