Diagnosing Oracle Clusterware and RAC Components

10
Copyright 2008, Oracle. All rights reserved.

Objectives

After completing this lesson, you should be able to: Collect Oracle Clusterware diagnostic files Use Cluster Verify

10 - 2

Copyright 2008, Oracle. All rights reserved.

The One Golden Rule in RAC Debugging

Always make sure that your nodes have exactly the same system time to:
Facilitate log information analysis Ensure accurate results when reading GV$ views Avoid untimely instance evictions

The best recommendation is to synchronize nodes using Network Time Protocol.

10 - 3

Copyright 2008, Oracle. All rights reserved.

Oracle Clusterware Main Log Files

ORA_CRS_HOME ORACLE_HOME

log

log

<hostname>

<hostname>

crsd

cssd

evmd

racg

client

client

racg

racgeut alert<nodename>.log

racgevtf

racgmain

racgeut

racgimon racgmain

racgmdb

10 - 4

Copyright 2008, Oracle. All rights reserved.

Diagnostics Collection Script

Script to collect all important log files:

Must be executed as root Is located in $ORA_CRS_HOME/bin/ Is called diagcollection.pl

Generates the following files in the local directory:

basData_<hostname>.tar.gz crsData _<hostname>. tar.gz ocrData _<hostname>. tar.gz oraData _<hostname>. tar.gz

# # # # #

export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1 export ORA_CRS_HOME=/u01/crs1020 export ORACLE_BASE= =/u01/app/oracle cd $ORA_CRS_HOME/bin ./diagcollection.pl -collect

10 - 5

Copyright 2008, Oracle. All rights reserved.

Cluster Verify: Overview

To verify that you have a well-formed cluster for Oracle Clusterware and RAC:
Installation Configuration Operation

Full stack verification Nonintrusive verification Diagnostic mode seeks to establish a reason for the failure of any verification task. Easy-to-use interface:
Stage commands Component commands
10 - 6 Copyright 2008, Oracle. All rights reserved.

Cluster Verify Stages

$ cluvfy stage -list
-pre dbinst
Installs RAC

-pre dbcfg
Configures RAC DB

-pre crsinst

Installs CRS

-pre cfs
Sets up OCFS (OPT)

-post crsinst

-post cfs
User sets up the hardware, network, & storage

-post hwos

10 - 7

Copyright 2008, Oracle. All rights reserved.

Cluster Verify Components

An individual subsystem or a module of the RAC cluster is known as a component in CVU. The availability and integrity of a cluster component can be verified. Components can be simple like a specific storage device, or complex like the Oracle Clusterware stack:
Space availability Shared storage accessibility Node connectivity Cluster File System integrity Oracle Clusterware integrity Cluster integrity Administrative privileges Peer compatibility System requirements
$ cluvfy comp -list

10 - 8

Copyright 2008, Oracle. All rights reserved.

Cluster Verify Locations

Download it from OTN:

Create a local directory. Copy and extract cvu_<OS>.zip. Set CVU environment variables.

Oracle software DVD:

cluvfy cluvfy directory runcluvfy.sh

Oracle Clusterware home:

$ORA_CRS_HOME/bin/cluvfy

Oracle Home:
$ORACLE_HOME/bin/cluvfy
10 - 9 Copyright 2008, Oracle. All rights reserved.

Environment Variables for Cluster Verify

CV_HOME (epicenter of CVU):

Within CRS home: CV_HOME=$ORA_CRS_HOME Within Oracle Home: CV_HOME=$ORACLE_HOME Standalone mode: CV_HOME=$CV_HOME

CV_DESTLOC:
The workarea on local as well as remote nodes

CV_JDKHOME:
The location for JDK 1.4.2 or later package

10 - 10

Copyright 2008, Oracle. All rights reserved.

Cluster Verify Configuration File

$ cat cvu_config # Configuration file for CVU # Version: 011405 # #CV_NODE_ALL= CV_RAW_CHECK_ENABLED=TRUE CV_ASSUME_DISTID=Taroon #CV_XCHK_FOR_SSH_ENABLED=TRUE #ORACLE_SRVM_REMOTESHELL=/usr/bin/ssh #ORACLE_SRVM_REMOTECOPY=/usr/bin/scp
10 - 11 Copyright 2008, Oracle. All rights reserved.

10 - 12

Copyright 2008, Oracle. All rights reserved.

Cluster Verify: Examples

$ cluvfy comp sys -n node1,node2 -p crs -verbose $ cluvfy comp ssa -n all -s /dev/sda1 $ cluvfy comp space -n all -l /home/product -z 5G $ cluvfy comp nodereach -n node2 -srcnode node1

1 2 3 4

$ cluvfy comp nodecon -n node1,node2 i eth0 -verbose 5 $ cluvfy comp admprv -n all -o user_equiv -verbose $ cluvfy comp nodeapp -n all -verbose $ cluvfy comp peer -n all verbose | more
6 7 8

10 - 13

Copyright 2008, Oracle. All rights reserved.

10 - 14

Copyright 2008, Oracle. All rights reserved.

Cluster Verify Output: Example

$ cluvfy comp crs -n all -verbose Verifying CRS integrity Checking CRS integrity... Checking daemon liveness... Liveness of all the daemons Node Name CRS daemon CSS daemon EVM daemon ------------ ------------- --------------- ---------atlhp9 yes yes yes atlhp8 yes yes yes Checking CRS health... Check: Health of CRS Node Name CRS OK? ---------------------------------- -----------------------atlhp9 yes atlhp8 yes Result: CRS health check passed. CRS integrity check passed. Verification of CRS integrity was successful.
10 - 15 Copyright 2008, Oracle. All rights reserved.

Summary

In this lesson, you should have learned how to: Collect Oracle Clusterware diagnostic files Use Cluster Verify

10 - 16

Copyright 2008, Oracle. All rights reserved.

Practice 10: Overview

This practice covers the following topics: Identifying Oracle Clusterware log files Fixing voting disk corruptions

10 - 17

Copyright 2008, Oracle. All rights reserved.

10 - 18

Copyright 2008, Oracle. All rights reserved.