A Short And Getting More Knowledge with Oracle RAC

Oracle 10g and 11g R1 RAC Architecture ...................................................................... 2

Oracle 11g R2 RAC Architecture .................................................................................... 3
IP Addresses in Oracle RAC ............................................................................................ 4
RAC Using NFS ................................................................................................................ 4
RAC Using SAN And ASM 11gR2 .................................................................................... 5
Storage ........................................................................................................................... 5
SAN (Storage Area Network).......................................................................................... 5
What is SAN ? ................................................................................................................. 5
To Check The LUN On The Unix/Linux Server Hosts: ..................................................... 5
LUN and RAID implementation (General Recommendations) ...................................... 6
Oracle RAC ..................................................................................................................... 6
The main differences between 11g R1 and 11g R2 RAC are ......................................... 6
IPMI ................................................................................................................................ 6
BMC ................................................................................................................................ 6
RACCheck ....................................................................................................................... 7
ASMCA from 11gR2 ........................................................................................................ 7
ASMCMD ........................................................................................................................ 7
What are the new features with 11g ASM?.................................................................. 7
ACFS ............................................................................................................................... 7
SRVCTL & CRSCTL Commands ........................................................................................ 8
Conclusion ...................................................................................................................... 8
Links ............................................................................................................................... 8

This article is about getting more (theoretical) knowledge with Oracle RAC
I wrote the first part of this article i.e. about Oracle 10g RAC in the year 2010
to do the knowledge transfer for my two good friends who did not do training
with Oracle RAC
In the last year it ended up with some more input from the dear Tim Hall
(oracle-base.com)
I'm indeed very much thankful to him in taking time to correct and add his
some more valuable information in this article !!! His comments are as it is
marked in red color

Oracle 10g and 11g R1 RAC Architecture

Disk Array controller

Disk arrays located here
SAN

Fabric channel switch

RAC 1
RDBMS_Instance
RDBMS_Home
ASM_Instance
ASM_Home
CRS_Home

RAC 2

Ethernet
Gigabyte

CRS s/w loaded

o/s

RDBMS_Instance
RDBMS_Home
ASM_Instance
ASM_Home
CRS_Home
CRS s/w loaded
o/s

Public
Network

Oracle 11g R2 RAC Architecture

Disk Array controller

Disk arrays located here
SAN

Fabric channel switch

RAC 1
rdbms_instance
rdbms_home
asm_instance
grid_Home

RAC 2

Ethernet
Gigabyte

CRS s/w loaded

O/S

rdbms_instance
rdbms_home
asm_instance
grid_home
CRS s/w loaded
O/S

Public
Network

By what technology Oracle RAC works with?

Oracle RAC is about Cache Fusion Technology
Good Link
http://www.dba-oracle.com/t_gupta_oracle_rac_cache_fusion.htm

IP Addresses in Oracle RAC

The three types of IP Addresses used in Oracle RAC are:
Public IP, Private IP and Virtual IP
If these three IP addresses confuses you again and again here is my explanation that should
make you comfortable. So, let's begin with Virtual IP
Virtual IP : In short call it VIP IP. Used for RAC Failover. Failover here is if RAC
Instance XX is failed then RAC Instance YY should be available. Who is connecting
these IPs frequently? RAC Instances are connecting. Will you allow Applications or
Users to connect these IP ? Absolutely NO
Private IP : In short call it Priv IP. This takes us to the next thing that is Private IP. For
example, If Oracle RAC is setup in a Production Environment then it is mostly on SAN
for Storage or any other Storage using RAID or so.
How should the Storage System and RAC communicate? Will you allow this
communication/data transfer on a busy network? Absolutely NO. I hope you got the
answer :)
Public IP : As it is called! The public IP address is of the Server on which the Oracle
RAC is created / installed
What are the benefits of implementation of RAC using NFS and RAC using SAN and ASM
11gR2?

RAC Using NFS

* It is low-cost RAC Implementation but not fully HA solution. (Tim: If your NFS
storage is resilient and your storage network is resilient, this has exactly the same
resilient as any ASM solution!)
* During ASM installation, it asks how the ASM disks Redundancy should be
configured : High or Normal or External. (Tim: All three modes are available in ASM,
regardless of storage layer. You should almost always pick externally redundant and
use hardware redundancy (RAID))
* ASM disks comes from NFS. (Tim: Only if you choose to use ASM and NFS together.
You dont need to use ASM if you are using NFS.)
* So, if NFS has problem then complete RAC and ASM Disks have got PROBLEM. (Tim:
If your storage network has a problem, RAC on a SAN has a problem. No different.
Not a reason to choose between them.)

RAC Using SAN And ASM 11gR2

* 10g ASM was not so good. (Tim: Was fine after the initial bugs were fixed)
* 11g R1 ASM was good
* 11g R2 ASM is powerful
* Maintaining Files are of less headache. But, ASMCMD has to be understood and
little bit practiced. ASMCMD is very easy. (Tim: Possibly not as easy as using NFS,
which just feels like using a file system. Very easy.)

Storage
SAN (Storage Area Network)
What is the difference between NAS & SAN ?
* NAS provides shared storage capacity at file level
* SAN provides shared storage raw capacity/raw disk capacity at block level
(Tim: So? What does this mean for Oracle? Pretty much nothing! Oracle plays well
with both)

What is SAN ?
* SAN is platform independent and SAN has no IP address assigned for connectivity
* SAN has wwn (World Wide Number) for connectivity
* SAN is usually with FC & Fiber Optic. Fabric Channel (FC) uses Switch based
Topology
* In SAN we have Disk Array Controller, which is very intelligent and has built in
Firmware
* With the SAN Management Software, we create disk arrays and assign RAID level
to it. It is important to understand which RAID level is implemented and best suited
for the requirement. Then a LUN Number (Logical Unit Number) is generated and
attached to RAC Nodes
To Check The LUN On The Unix/Linux Server Hosts:
#] cat /proc/scsi/scsi
The same LUN can be assigned to the other Nodes

LUN and RAID implementation (General Recommendations)

* To get good performance, it is best not to do more partitions on the LUN. For
example, 2 LUN's are provided by SAN Admin
* 1st LUN with RAID 1+0 for Database Files that is for ASM (Tim: Equal size and
performance of all LUNs within a disk group. Minimum of 4 LUNs per disk group.)
* 2nd LUN with RAID 0 for Control files, Online Redolog Files and Archived Logs. (Tim:
Depends. If you are multiplexing all those file types, then that is fine, but most
people RAID 10 everything)
* FRA (Flash Recovery Area) can be stored on Local File System or on 2nd LUN (Tim:
Ouch! All storage for RAC needs to be shared, otherwise other nodes cant perform
instance recovery for a failed node.)

Oracle RAC
* RAC uses Cache Fusion Technology
* File System types :
1. Local File System (ext2, ext3, ext4 and NTFS etc.)
2. Shared / Global File System
* Shared / Global File System is CFS (Cluster File System)
* Examples of CFS are Veritas, ASM and Oracle OCFS2 (Tim: NFS)
The main differences between 11g R1 and 11g R2 RAC are
* In 11g R1 RAC, we needed to create 3 homes, that were :
1. ASM home, 2. CRS home and 3. RDBMS home
* In 11g R2 RAC, only 2 homes exists that is Grid home (asm_home + crs_home) and
RDBMS home

IPMI
* This is a software part of 11gR2 RAC
* With IPMI Commands we can communicate with Nodes directly

BMC
* Baseboard Management Card helps us from Node-Fencing problems
* Node-Fencing means RAC Node coming out from Cluster
* Node-Fencing is also called as Node-Affinity

RACCheck
* Oracle provide the RACcheck tool (MOS [ID 1268927.1]) to audit the configuration
of RAC, CRS, ASM, GI etc.
* It supports database versions from 10.2-11.2, making it a useful starting point for
most analysis
* The MOS note includes the download and setup details. If you are using 11.2.0.4 or
later you will have RACcheck by default
ASMCA from 11gR2
* This is a new utility called ASM Configuration Assistant

ASMCMD
* SYSASM is the new Privilege from 11g R1ASM version
* SYSASM is activated when ASM is installed and the Disks and Disk Groups can be
managed from operating system using " asmcmd "

What are the new features with 11g ASM?

* In 10g ASM the backup of ASM Metadata was not possible
* From 11g ASM, the backup of ASM Metadata is possible
* If any ASM Disks or Disk Groups are corrupted then Backup of Metadata helps to
get the data back

ACFS
* ASM Cluster File Systems
http://docs.oracle.com/cd/E11882_01/server.112/e16102/asmfilesystem.htm
* Oracle ASM Cluster File System (ACFS) is a general purpose cluster file system
implemented as part of ASM
* The only things that should not be stored in ACFS are the Grid Infrastructure home
and any Oracle f iles that can be directly stored in Oracle ASM Disks. (Tim: Recent
versions allow Oracle datafiles to be stored in ACFS, but you probably shouldnt.)
* Oracle ACFS is the preferred file manager for non-database files
* Other supported files are video, audio, text, images, engineering drawings, and
other general-purpose application file data.
* Starting with Oracle ASM 11g R 2, Oracle ACFS supports RMAN backups
(BACKUPSET file type), archive logs (ARCHIVELOG file type), and Data Pump
dumpsets (DUMPSET file type)

SRVCTL & CRSCTL Commands

* This is the way to manage Oracle RAC. The "CRS_STAT" kind of Commands are used with
10g Oracle RAC and deprecated
* So, SRVCTL and CRSCTL Commands to be learned and pretty cool !

Conclusion
* To think RAC is one of the Oracle High Availability Solution is a good think. But, to
think it is the only solution is not a good thinking
* Oracle provides many other HA solutions so exploring all of them is also a primary
thing for us to do
Links

RAC Installations

http://oracle-base.com/articles/rac/articles-rac.php

Real Application Clusters Administration and Deployment Guide (SRVCTL

Usage)

http://docs.oracle.com/database/121/RACAD/srvctladmin.htm

Troubleshooting Grid Infrastructure startup (Very interesting one)

https://martincarstenbach.wordpress.com/2011/03/17/troubleshooting-gridinfrastructure-startup/