Ora Clu Hpux

Using Serviceguard Extension
for RAC
Manufacturing Part Number : T1859-90038

May 2006 Update
© Copyright 2006 Hewlett-Packard Development Company, L.P. All rights reserved.

Legal Notices
© Copyright 2003- 2006 Hewlett-Packard Development Company, L.P.
Publication Dates: June 2003, June 2004, February 2005, December 2005, March 2006, May
2006
Confidential computer software. Valid license from HP required for possession, use, or
copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer
Software Documentation, and Technical Data for Commercial Items are licensed to the U.S.
Government under vendor’s standard commercial license.
The information contained herein is subject to change without notice. The only warranties for
HP products and services are set forth in the express warranty statements accompanying
such products and services. Nothing herein should be construed as constituting an additional
warranty. HP shall not be liable for technical or editorial errors or omissions contained
herein.
Oracle ® is a registered trademark of Oracle Corporation.
UNIX® is a registered trademark in the United States and other countries, licensed
exclusively through The Open Group.
VERITAS® is a registered trademark of VERITAS Software Corporation.
VERITAS File System™ is a trademark of VERITAS Software Corporation.
2
Contents
1. Introduction to Serviceguard Extension for RAC
What is a Serviceguard Extension for RAC Cluster? . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Group Membership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Using Packages in a Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Serviceguard Extension for RAC Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Group Membership Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Overview of SGeRAC and Cluster File System (CFS)/
Cluster Volume Manager (CVM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Package Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Storage Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Overview of SGeRAC and Oracle 10g RAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Overview of SGeRAC and Oracle 9i RAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
How Serviceguard Works with Oracle 9i RAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Group Membership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Configuring Packages for Oracle RAC Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Configuring Packages for Oracle Listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Node Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Larger Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Up to Four Nodes with SCSI Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Point to Point Connections to Storage Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Extended Distance Cluster Using Serviceguard Extension for RAC . . . . . . . . . . . . . . 32
2. Serviceguard Configuration for Oracle 10g RAC

Interface Areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Group Membership API (NMAPI2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
SGeRAC Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Cluster Timeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Oracle Cluster Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Shared Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Listener . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Network Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
RAC Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Automated Startup and Shutdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Manual Startup and Shutdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Shared Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Planning Storage for Oracle Cluster Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Planning Storage for Oracle 10g RAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Volume Planning with SLVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3
Contents
Storage Planning with CFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Volume Planning with CVM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Installing Serviceguard Extension for RAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Configuration File Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Creating a Storage Infrastructure with LVM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Building Volume Groups for RAC on Mirrored Disks. . . . . . . . . . . . . . . . . . . . . . . . . 48
Building Mirrored Logical Volumes for RAC with LVM Commands . . . . . . . . . . . . . 50
Creating RAC Volume Groups on Disk Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Creating Logical Volumes for RAC on Disk Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Oracle Demo Database Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Displaying the Logical Volume Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Exporting the Logical Volume Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Installing Oracle Real Application Clusters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Cluster Configuration ASCII File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Creating a Storage Infrastructure with CFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Creating a SGeRAC Cluster with CFS 4.1 for Oracle 10g . . . . . . . . . . . . . . . . . . . . . 65
Initializing the VERITAS Volume Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Deleting CFS from the Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Creating a Storage Infrastructure with CVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Initializing the VERITAS Volume Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Using CVM 4.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Using CVM 3.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Creating Volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Adding Disk Groups to the Cluster Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Prerequisites for Oracle 10g (Sample Installation) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Installing Oracle 10g Cluster Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Installing on Local File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Installing Oracle 10g RAC Binaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Installing RAC Binaries on a Local File System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Installing RAC Binaries on Cluster File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Creating a RAC Demo Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Creating a RAC Demo Database on SLVM or CVM . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Creating a RAC Demo Database on CFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Verify that Oracle Disk Manager is Configured. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Configuring Oracle to Use Oracle Disk Manager Library . . . . . . . . . . . . . . . . . . . . . . . 93
4
Contents
Verify that Oracle Disk Manager is Running. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Configuring Oracle to Stop Using Oracle Disk Manager Library . . . . . . . . . . . . . . . . . 96
Using Serviceguard Packages to Synchronize with Oracle 10g RAC . . . . . . . . . . . . . . 97
Preparing Oracle Cluster Software for Serviceguard Packages. . . . . . . . . . . . . . . . . 97
Configure Serviceguard Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
3. Serviceguard Configuration for Oracle 9i RAC

Planning Database Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Volume Planning with SLVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Storage Planning with CFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Volume Planning with CVM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
Installing Serviceguard Extension for RAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Configuration File Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
Operating System Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Creating a Storage Infrastructure with LVM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
Building Volume Groups for RAC on Mirrored Disks. . . . . . . . . . . . . . . . . . . . . . . . 112
Building Mirrored Logical Volumes for RAC with LVM Commands . . . . . . . . . . . . 114
Creating RAC Volume Groups on Disk Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
Creating Logical Volumes for RAC on Disk Arrays . . . . . . . . . . . . . . . . . . . . . . . . . 118
Oracle Demo Database Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Displaying the Logical Volume Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Exporting the Logical Volume Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Installing Oracle Real Application Clusters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Cluster Configuration ASCII File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Creating a Storage Infrastructure with CFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Creating a SGeRAC Cluster with CFS 4.1 for Oracle 9i . . . . . . . . . . . . . . . . . . . . . 129
Initializing the VERITAS Volume Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Deleting CFS from the Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
Creating a Storage Infrastructure with CVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
Initializing the VERITAS Volume Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
Using CVM 4.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
Using CVM 3.x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
Creating Volumes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Mirror Detachment Policies with CVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Adding Disk Groups to the Cluster Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
Installing Oracle 9i RAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Install Oracle Software into CFS Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
5
Contents
Create Database with Oracle Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
Verify that Oracle Disk Manager is Configured. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
Configure Oracle to use Oracle Disk Manager Library . . . . . . . . . . . . . . . . . . . . . . . . 152
Verify Oracle Disk Manager is Running . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Configuring Oracle to Stop using Oracle Disk Manager Library . . . . . . . . . . . . . . . . 155
Using Packages to Configure Startup and Shutdown of RAC Instances . . . . . . . . . . 156
Starting Oracle Instances. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Creating Packages to Launch Oracle RAC Instances . . . . . . . . . . . . . . . . . . . . . . . . 157
Configuring Packages that Access the Oracle RAC Database . . . . . . . . . . . . . . . . . 158
Adding or Removing Packages on a Running Cluster . . . . . . . . . . . . . . . . . . . . . . . 159
Writing the Package Control Script. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
4. Maintenance and Troubleshooting

Reviewing Cluster and Package States with the cmviewcl Command . . . . . . . . . . . . 168
Types of Cluster and Package States. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Examples of Cluster and Package States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
Online Reconfiguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Online Node Addition and Deletion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Managing the Shared Storage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Single Node Online volume Re-Configuration (SNOR) . . . . . . . . . . . . . . . . . . . . . . 184
Making LVM Volume Groups Shareable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Activating an LVM Volume Group in Shared Mode . . . . . . . . . . . . . . . . . . . . . . . . . 187
Making Offline Changes to Shared Volume Groups . . . . . . . . . . . . . . . . . . . . . . . . . 188
Adding Additional Shared LVM Volume Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
Changing the VxVM or CVM Storage Configuration . . . . . . . . . . . . . . . . . . . . . . . 190
Removing Serviceguard Extension for RAC from a System . . . . . . . . . . . . . . . . . . . . 192
Monitoring Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Using Event Monitoring Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Using EMS Hardware Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Adding Disk Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Replacing Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Replacing a Mechanism in a Disk Array Configured with LVM . . . . . . . . . . . . . . . 195
Replacing a Mechanism in an HA Enclosure Configured with Exclusive LVM . . . 195
Online Replacement of a Mechanism in an HA Enclosure Configured with Shared
LVM (SLVM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
Offline Replacement of a Mechanism in an HA Enclosure Configured with Shared
LVM (SLVM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
6
Contents
Replacing a Lock Disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
On-line Hardware Maintenance with In-line SCSI Terminator . . . . . . . . . . . . . . . 198
Replacement of I/O Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Replacement of LAN Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
Off-Line Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
On-Line Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
After Replacing the Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Monitoring RAC Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
A. Software Upgrades
Rolling Software Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Steps for Rolling Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Example of Rolling Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Limitations of Rolling Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
Non-Rolling Software Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
Steps for Non-Rolling Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Limitations of Non-Rolling Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
Migrating a SGeRAC Cluster with Cold Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
B. Blank Planning Worksheets

LVM Volume Group and Physical Volume Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . 222
VxVM Disk Group and Disk Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
Oracle Logical Volume Worksheet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
7
Contents
8
Printing History
Table 1 Document Edition and Printing Date
Printing Date Part Number Edition
June 2003 T1859-90006 First Edition

Print, CD-ROM (Instant Information), and
Web (http://www.docs.hp.com/)
June 2004 T1859-90017 Second Edition

February 2005 T1859-90017 Second Edition February 2005 Update

October 2005 T1859-90033 Third Edition

December 2005 T1859-90033 Third Edition, First Reprint

March 2006 T1859-90038 Third Edition, Second Reprint

May 2006 T1859-90038 Third Edition May 2006 Update

The last printing date and part number indicate the current edition.
Changes in the May 2006 update version include software upgrade
procedures for SGeRAC clusters.
The last printing date and part number indicate the current edition,
which applies to the 11.14, 11.15, 11.16 and 11.17 version of
Serviceguard Extension for RAC (Oracle Real Application Cluster).
9
The printing date changes when a new edition is printed. (Minor
corrections and updates which are incorporated at reprint do not cause
the date to change.) The part number is revised when extensive technical
changes are incorporated.
New editions of this manual will incorporate all material updated since
the previous edition. To ensure that you receive the new editions, you
should subscribe to the appropriate product support service. See your HP
sales representative for details.
HP Printing Division:
Business Critical Computing
Hewlett-Packard Co.
19111 Pruneridge Ave.
Cupertino, CA 95014
10
Preface
The May 2006 update includes a new appendix on software upgrade
procedures for SGeRAC clusters. Also, this guide describes how to use
the Serviceguard Extension for RAC (Oracle Real Application Cluster) to
configure Serviceguard clusters for use with Oracle Real Application
Cluster software on HP High Availability clusters running the HP-UX
operating system. The contents are as follows:
• Chapter 1, “Introduction,” describes a Serviceguard cluster and

provides a roadmap for using this guide. This chapter should be used
as a supplement to Chapters 1–3 of the Managing Serviceguard
user’s guide.
• Chapter 2, “Serviceguard Configuration for Oracle 10g RAC,”
describes the additional steps you need to take to use Serviceguard
with Real Application Clusters when configuring Oracle 10g RAC.
This chapter should be used as a supplement to Chapters 4–6 of the
Managing Serviceguard user’s guide.
• Chapter 3, “Serviceguard Configuration for Oracle 9i RAC,”
describes the additional steps you need to take to use Serviceguard
with Real Application Clusters when configuring Oracle 9i RAC.
This chapter should be used as a supplement to Chapters 4–6 of the
Managing Serviceguard user’s guide.
• Chapter 4, “Maintenance and Troubleshooting,” describes tools and
techniques necessary for ongoing cluster operation. This chapter
should be used as a supplement to Chapters 7–8 of the Managing
Serviceguard user’s guide.
• Appendix A, “Software Upgrades,” describes rolling, non-rolling and
migration with cold install upgrade procedures for SGeRAC clusters.
• Appendix B, “Blank Planning Worksheets,” contains planning
worksheets for LVM, VxVM, and Oracle Logical Volume.
Related The following documents contain additional useful information:

Publications
• Clusters for High Availability: a Primer of HP Solutions.
Hewlett-Packard Professional Books: Prentice Hall PTR, 2001
(ISBN 0-13-089355-2)
• Managing Serviceguard Twelfth Edition (B3936-90100)
11
• Using High Availability Monitors (B5736-90046)
• Using the Event Monitoring Service (B7612-90015)
• Using Advanced Tape Services (B3936-90032)
• Designing Disaster Tolerant High Availability Clusters
(B7660-90017)
• Managing Serviceguard Extension for SAP (T2803-90002)
• Managing Systems and Workgroups (5990-8172)
• Managing Serviceguard NFS (B5140-90017)
• HP Auto Port Aggregation Release Notes
Before attempting to use VxVM storage with Serviceguard, please refer
to the following:
• VERITAS Volume Manager Administrator’s Guide. This contains a

glossary of VERITAS terminology.
• VERITAS Volume Manager Storage Administrator Administrator’s
Guide
• VERITAS Volume Manager Reference Guide
• VERITAS Volume Manager Migration Guide
• VERITAS Volume Manager for HP-UX Release Notes
If you will be using VERITAS CVM 4.1 or the VERITAS Cluster File
System with Serviceguard, please refer to the HP Serviceguard Storage
Management Suite Version A.01.00 Release Notes (T2771-90028). These
release notes describe suite bundles for the integration of HP
Serviceguard A.11.17 with Symantec’s VERITAS Storage Foundation.
Use the following URL to access HP’s high availability web page:
• http://www.hp.com/go/ha
Use the following URL for access to a wide variety of HP-UX
documentation:
• http://docs.hp.com/hpux
Problem Reporting If you have any problems with the software or documentation, please
contact your local Hewlett-Packard Sales Office or Customer Service
Center.
12
Conventions We use the following typographical conventions.
audit (5) An HP-UX manpage. audit is the name and 5 is the
section in the HP-UX Reference. On the web and on the
Instant Information CD, it may be a hot link to the
manpage itself. From the HP-UX command line, you
can enter “man audit” or “man 5 audit” to view the
manpage. See man (1).
Book Title The title of a book. On the web and on the Instant
Information CD, it may be a hot link to the book itself.
KeyCap The name of a keyboard key. Note that Return and Enter
both refer to the same key.
Emphasis Text that is emphasized.
Emphasis Text that is strongly emphasized.
Term The defined use of an important word or phrase.
ComputerOut Text displayed by the computer.
UserInput Commands and other text that you type.
Command A command name or qualified command phrase.
Variable The name of a variable that you may replace in a
command or function or information in a display that
represents several possible values.
[ ] The contents are optional in formats and command
descriptions. If the contents are a list separated by |,
you must choose one of the items.
{ } The contents are required in formats and command
descriptions. If the contents are a list separated by |,
you must choose one of the items.
... The preceding element may be repeated an arbitrary
number of times.
| Separates items in a list of choices.
13
14
Introduction to Serviceguard Extension for RAC
1 Introduction to Serviceguard
Extension for RAC
Serviceguard Extension for RAC (SGeRAC) enables the Oracle Real

Application Cluster (RAC), formerly known as Oracle Parallel Server
RDBMS, to run on HP high availability clusters under the HP-UX
operating system. This chapter introduces Serviceguard Extension for
RAC and shows where to find different kinds of information in this book.
The following topics are presented:
• What is a Serviceguard Extension for RAC Cluster?

• Serviceguard Extension for RAC Architecture
• Overview of SGeRAC and Cluster File System (CFS)/Cluster Volume
Manager (CVM)
• Overview of SGeRAC and Oracle 10g RAC
• Overview of SGeRAC and Oracle 9i RAC
• Node Failure
• Larger Clusters
• Extended Distance Cluster Using Serviceguard Extension for RAC
Chapter 1 15
What is a Serviceguard Extension for RAC Cluster?
What is a Serviceguard Extension for RAC

Cluster?
A high availability cluster is a grouping of HP servers having
sufficient redundancy of software and hardware components that a
single point of failure will not disrupt the availability of computer
services. High availability clusters configured with Oracle Real
Application Cluster software are known as RAC clusters. Figure 1-1
shows a very simple picture of the basic configuration of a RAC cluster
on HP-UX.
Figure 1-1 Overview of Oracle RAC Configuration on HP-UX
In the figure, two loosely coupled systems (each one known as a node)
are running separate instances of Oracle software that read data from
and write data to a shared set of disks. Clients connect to one node or the
other via LAN.
16 Chapter 1
RAC on HP-UX lets you maintain a single database image that is

accessed by the HP servers in parallel, thereby gaining added processing
power without the need to administer separate databases. Further, when
properly configured, Serviceguard Extension for RAC provides a highly
available database that continues to operate even if one hardware
component should fail.
Group Membership
Oracle RAC systems implement the concept of group membership,
which allows multiple instances of RAC to run on each node. Related
processes are configured into groups. Groups allow processes in
different instances to choose which other processes to interact with. This
allows the support of multiple databases within one RAC cluster.
A Group Membership Service (GMS) component provides a process
monitoring facility to monitor group membership status. GMS is
provided by the cmgmsd daemon, which is an HP component installed
with Serviceguard Extension for RAC.
Figure 1-2 shows how group membership works. Nodes 1 through 4 of
the cluster share the Sales database, but only Nodes 3 and 4 share the
HR database. Consequently, there is one instance of RAC each on Node 1
and Node 2, and there are two instances of RAC each on Node 3 and
Node 4. The RAC processes accessing the Sales database constitute one
group, and the RAC processes accessing the HR database constitute
another group.
Chapter 1 17
Figure 1-2 Group Membership Services
Using Packages in a Cluster

In order to make other important applications highly available (in
addition to the Oracle Real Application Cluster), you can configure your
RAC cluster to use packages. Packages group applications and services
together; in the event of a service, node, or network failure, Serviceguard
Extension for RAC can automatically transfer control of all system
resources in a designated package to another node within the cluster,
allowing your applications to remain available with minimal
interruption.
NOTE In RAC clusters, you create packages to start and stop RAC itself as well
as to run applications that access the database instances. For details on
the use of packages with RAC, refer to section, “Using Packages to
Configure Startup and Shutdown of RAC Instances” on page 156 located
in chapter 3.
18 Chapter 1
Serviceguard Extension for RAC Architecture
Serviceguard Extension for RAC Architecture

This chapter discusses the main software components used by
Serviceguard Extension for RAC in some detail. The components are:
• Oracle Components
— Custom Oracle Database

• Serviceguard Extension for RAC Components
— Group Membership Services (RAC)

• Serviceguard Components
— Package Manager
— Cluster Manager
— Network Manager
• Operating System
— Volume Manager Software

— HP-UX Kernel
Group Membership Daemon

In addition to the Serviceguard daemon processes mentioned in Chapter
3 of Managing Serviceguard user’s guide, there is another daemon that is
used by Oracle to enable communication with Serviceguard Extension
for RAC:
• cmgmsd—Group Membership Daemon for RAC 9i or later

This HP daemon provides group membership services for Oracle Real
Application Cluster 9i or later. Group membership allows multiple
Oracle instances to run on the same cluster node. GMS is illustrated in
Figure 1-2 on page 18.
Chapter 1 19
Overview of SGeRAC and Cluster File System (CFS)/Cluster Volume Manager (CVM)
Overview of SGeRAC and Cluster File System

(CFS)/Cluster Volume Manager (CVM)
SGeRAC supports Cluster File System (CFS) through Serviceguard. For
more detail information on CFS support refer to the Managing
Serviceguard Twelfth Edition user’s guide.
Package Dependencies
When CFS is used as shared storage, the application and software using
the CFS storage should be configured to start and stop using
Serviceguard packages. These application packages should be configured
with a package dependency on the underlying multi-node packages,
which manages the CFS and CVM storage reserves.
Configuring the application to be start/stop through SG package is to
ensure the synchronization of storage activation/deactivation and
application startup/shutdown.
With CVM configurations using multi-node packages, CVM shared
storage should be configured in Serviceguard packages with package
dependencies.
Refer to the Managing Serviceguard Twelfth Edition user’s guide for
detailed information on multi-node package.
Storage Configuration Options

Prior to CFS, the only option in a SGeRAC cluster to provide shared
storage for a RAC cluster was through raw volumes, using either SLVM
or CVM that are used for Oracle data files. The application software is
installed on a local file system. In addition to SLVM and CVM, SGeRAC
supports CFS.
CFS provides SGeRAC with additional options, such as improved
manageability. When planning a RAC cluster, application software could
be installed once and be visible by all cluster nodes. A central location is
available to store runtime logs, for example, RAC alert logs.
20 Chapter 1
Overview of SGeRAC and Cluster File System (CFS)/Cluster Volume Manager (CVM)
Oracle RAC data files can be created on a CFS, allowing the database
administrator or Oracle software to create additional data files without
the need of root system administrator privileges. The archive area can
now be on a CFS. Oracle instances on any cluster node can access the
archive area when database recovery requires the archive logs.
Chapter 1 21
Overview of SGeRAC and Oracle 10g RAC
Overview of SGeRAC and Oracle 10g RAC

Starting with Oracle 10g RAC, Oracle has bundled its own cluster
software. The initial release is called Oracle Cluster Ready Service
(CRS). CRS is used both as a generic term referring to the Oracle cluster
software and as a specific term referring to a component within the
Oracle clusters software. At subsequent release, Oracle generic CRS is
renamed to Oracle Clusterware. Oracle Clusterware is the generic term
referring to the Oracle Cluster Software.
The Oracle Cluster Software includes the following components: Cluster
Synchronization Services (CSS), Cluster Ready Service (CRS), and Event
Management (EVM).
CSS manages the Oracle cluster membership and provides its own group
membership service to RAC instances. When installed on a SGeRAC
cluster, CSS utilizes the group membership service provided by
SGeRAC.
CRS manages Oracle's cluster resources based on configuration,
including start, stop, and monitor, and failover of the resources.
EVM publishes events generated by CRS and may run scripts when
certain events occur.
When installed on a SGeRAC cluster, both the Oracle cluster software
and RAC can continue to rely on the shared storage capability,
networking monitoring, as well as other capabilities provided through
Serviceguard and SGeRAC.
NOTE In this document, the generic terms “CRS” and “Oracle Clusterware” will
subsequently be referred to as “Oracle Cluster Software”. The use of the
term CRS will still be used when referring to a sub-component of Oracle
Cluster Software.
For more detail information on Oracle 10g RAC refer to Chapter 2,

“Serviceguard Configuration for Oracle 10g RAC.”
22 Chapter 1
Overview of SGeRAC and Oracle 9i RAC
Overview of SGeRAC and Oracle 9i RAC
How Serviceguard Works with Oracle 9i RAC

Serviceguard provides the cluster framework for Oracle, a relational
database product in which multiple database instances run on different
cluster nodes. A central component of Real Application Clusters is the
distributed lock manager (DLM), which provides parallel cache
management for database instances. Each node in a RAC cluster starts
an instance of the DLM process when the Oracle instance starts and the
instances then communicate with each other over the network.
Group Membership
The group membership service (GMS) is the means by which Oracle
instances communicate with the Serviceguard cluster software. GMS
runs as a separate daemon process that communicates with the cluster
manager. This daemon is an HP component known as cmgmsd.
The cluster manager starts up, monitors, and shuts down the cmgmsd.
When an Oracle instance starts, the instance registers itself with
cmgmsd; thereafter, if an Oracle instance fails, cmgmsd notifies other
members of the same group to perform recovery. If cmgmsd dies
unexpectedly, Serviceguard will fail the node with a TOC (Transfer of
Control).
Chapter 1 23
Configuring Packages for Oracle RAC Instances
Configuring Packages for Oracle RAC

Instances
Oracle instances can be configured as packages with a single node in
their node list.
NOTE Packages that start and halt Oracle instances (called instance
packages) do not fail over from one node to another; they are
single-node packages. You should include only one NODE_NAME in the
package ASCII configuration file. The AUTO_RUN setting in the package
configuration file will determine whether the RAC instance will start up
as the node joins the cluster. Your cluster may include RAC and
non-RAC packages in the same cluster.
24 Chapter 1
Configuring Packages for Oracle Listeners
Configuring Packages for Oracle Listeners

Oracle listeners can be configured as packages within the cluster (called
listener packages). Each node with a RAC instance can be configured
with a listener package. Listener packages are configured to
automatically fail over from the original node to an adoptive node. When
the original node is restored, the listener package automatically fails
back to the original node.
In the listener package ASCII configuration file, the FAILBACK_POLICY
is set to AUTOMATIC. The SUBNET is a set of monitored subnets. The
package can be set to automatically startup with the AUTO_RUN setting.
Each RAC instance can be configured to be registered with listeners that
are assigned to handle client connections. The listener package script is
configured to add the package IP address and start the listener on the
node.
For example, on a two node cluster with one database, each node can
have one RAC instance and one listener package. Oracle clients can be
configured to connect to either package IP address (or corresponding
hostname) using Oracle Net Services. When a node failure occurs,
existing client connection to the package IP address will be reset after
the listener package fails over and adds the package IP address.
Subsequent connections for clients configured with basic failover, clients
would connect to the next available listener package's IP address and
listener.
Chapter 1 25
Node Failure
Node Failure
RAC cluster configuration is designed so that in the event of a node
failure, another node with a separate instance of Oracle can continue
processing transactions. Figure 1-3 shows a typical cluster with
instances running on both nodes.
Figure 1-3 Before Node Failure
Figure 1-4 shows the condition where Node 1 has failed and Package 1
has been transferred to Node 2. Oracle instance 1 is no longer operating,
but it does not fail over to Node 2. Package 1’s IP address was
transferred to Node 2 along with the package. Package 1 continues to be
26 Chapter 1
Node Failure
available and is now running on Node 2. Also note that Node 2 can now
access both Package 1’s disk and Package 2’s disk. Oracle instance 2 now
handles all database access, since instance 1 has gone down.
Figure 1-4 After Node Failure
In the above figure, pkg1 and pkg2 are not instance packages. They are
shown to illustrate the movement of packages in general.
Chapter 1 27
Larger Clusters
Larger Clusters
Serviceguard Extension for RAC supports clusters of up to 16 nodes. The
actual cluster size is limited by the type of storage and the type of volume
manager used.
Up to Four Nodes with SCSI Storage

You can configure up to four nodes using a shared F/W SCSI bus; for
more than 4 nodes, FibreChannel must be used. An example of a
four-node RAC cluster appears in the following figure.
28 Chapter 1
Larger Clusters
Figure 1-5 Four-Node RAC Cluster
Chapter 1 29
Larger Clusters
In this type of configuration, each node runs a separate instance of RAC

and may run one or more high availability packages as well.
The figure shows a dual Ethernet configuration with all four nodes
connected to a disk array (the details of the connections depend on the
type of disk array). In addition, each node has a mirrored root disk (R
and R'). Nodes may have multiple connections to the same array using
alternate links (PV links) to take advantage of the array's use of RAID
levels for data protection. Alternate links are further described in the
section “Creating RAC Volume Groups on Disk Arrays” on page 116.
Point to Point Connections to Storage Devices

Some storage devices allow point-to-point connection to a large number
of host nodes without using a shared SCSI bus. An example is shown in
Figure 1-6, a cluster consisting of eight nodes with a FibreChannel
interconnect. (Client connection is provided through Ethernet.) The
nodes access shared data on an HP StorageWorks XP series or EMC disk
array configured with 16 I/O ports. Each node is connected to the array
using two separate Fibre channels configured with PV Links. Each
channel is a dedicated bus; there is no daisy-chaining.
30 Chapter 1
Larger Clusters
Figure 1-6 Eight-Node Cluster with XP or EMC Disk Array
FibreChannel switched configurations also are supported using either an

arbitrated loop or fabric login topology. For additional information about
supported cluster configurations, refer to the HP 9000 Servers
Configuration Guide, available through your HP representative.
Chapter 1 31
Extended Distance Cluster Using Serviceguard Extension for RAC
Extended Distance Cluster Using

Serviceguard Extension for RAC
Basic Serviceguard clusters are usually configured in a single data
center, often in a single room, to provide protection against failures in
CPUs, interface cards, and software. Extended Serviceguard clusters are
specialized cluster configurations, which allow a single cluster to extend
across two or three separate data centers for increased disaster
tolerance. Depending on the type of links employed, distances of up to
100 km between data centers can be achieved.
Refer to Chapter 2 of the Designing Disaster Tolerant High Availability
Clusters manual, which discusses several types of extended distance
cluster that use basic Serviceguard technology with software mirroring
(using MirrorDisk/UX or CVM) and Fibre Channel.
32 Chapter 1
Serviceguard Configuration for Oracle 10g RAC
2 Serviceguard Configuration for

Oracle 10g RAC
This chapter shows the additional planning and configuration that is

needed to use Oracle Real Application Clusters 10g with Serviceguard.
• Interface Areas
• Oracle Cluster Software
• Planning Storage for Oracle Cluster Software
• Planning Storage for Oracle 10g RAC
• Installing Serviceguard Extension for RAC
• Installing Oracle Real Application Clusters
• Creating a Storage Infrastructure with CFS
• Creating a Storage Infrastructure with CVM
• Installing Oracle 10g Cluster Software
Chapter 2 33
Interface Areas
Interface Areas
This section documents interface areas where there is expected
interaction between SGeRAC and Oracle 10g Cluster Software and RAC.
Group Membership API (NMAPI2)

The NMAPI2 client links with the SGeRAC provided NMAPI2 library for
group membership service. The group membership is layered on top of
the SGeRAC cluster membership where all the primary group members
are processes within cluster nodes. Cluster membership has node names
and group membership has process names. Upon a SGeRAC group
membership change, SGeRAC delivers the new group membership to
other members of the same group.
SGeRAC Detection
When Oracle 10g Cluster Software is installed on a SGeRAC cluster,
Oracle Cluster Software detects the existence of SGeRAC and CSS uses
SGeRAC group membership.
Cluster Timeouts
SGeRAC uses heartbeat timeouts to determine when any SGeRAC
cluster member has failed or when any cluster member is unable to
communicate with the other cluster members. CSS uses a similar
mechanism for CSS memberships. Each RAC instance group
membership also has a timeout mechanism, which triggers Instance
Membership Recovery (IMR).
NOTE HP and Oracle support SGeRAC to provide group membership to CSS.
Serviceguard Cluster Timeout

The Serviceguard cluster heartbeat timeout is set according to user
requirements for availability. The Serviceguard cluster reconfiguration
time is determined by the cluster timeout, configuration, the
reconfiguration algorithm, and activities during reconfiguration.
34 Chapter 2
Interface Areas
CSS Timeout
When SGeRAC is on the same cluster as Oracle Cluster Software, the
CSS timeout is set to a default value of 600 seconds (10 minutes) at
Oracle software installation.
This timeout is configurable with Oracle tools and should not be changed
without ensuring that the CSS timeout allows enough time for
Serviceguard Extension for RAC (SGeRAC) reconfiguration and to allow
multipath (if configured) reconfiguration to complete.
On a single point of failure, for example, node failure, Serviceguard
reconfigures first and SGeRAC delivers the new group membership to
CSS via NMAPI2. If there is a change in group membership, SGeRAC
updates the members of the new membership. After receiving the new
group membership, CSS in turn initiates its own recovery action as
needed and propagates the new group membership to the RAC instances.
NOTE As a general guideline, the CSS TIMEOUT should be greater of either

180 seconds or 25 times the Serviceguard NODE_TIMEOUT.
RAC IMR Timeout

RAC instance IMR timeout is configurable. RAC IMR expects group
membership changes to occur within this time or IMR will begin evicting
group members. The IMR timeout must be above the SGeRAC
reconfiguration time and adhere to any Oracle-specified relation to CSS
reconfiguration time.
Oracle Cluster Software

Oracle Cluster Software should be started after activating its required
shared storage resources. Shared storage resources can be activated
after SGeRAC completes startup. Oracle Cluster Software should not
activate any shared storage. Similarly, for halting SGeRAC at run level 3
and removing shared storage resources from Oracle Cluster Software,
Oracle Cluster Software must be halted first.
Chapter 2 35
Interface Areas
Automated Oracle Cluster Software Startup and Shutdown

The preferred mechanism that allows Serviceguard to notify Oracle
Cluster Software to start and to request Oracle Cluster Software to
shutdown is the use of Serviceguard packages.
Monitoring
Oracle Cluster Software daemon monitoring is performed through
programs initiated by the HP-UX init process. SGeRAC monitors Oracle
Cluster Software to the extent that CSS is a NMAPI2 group membership
client and group member. SGeRAC provides group membership
notification to the remaining group members when CSS enters and
leaves the group membership.
Shared Storage
SGeRAC supports shared storage using HP Shared Logical Volume
Manager (SLVM), Veritas Cluster File System (CFS) and Veritas Cluster
Volume Manager (CVM). The file /var/opt/oracle/oravg.conf must
not be present so Oracle Cluster Software will not activate or deactivate
any shared storage.
Multipath
Multipath is supported through either SLVM pvlinks or CVM Dynamic
Multipath (DMP). In some configurations, SLVM or CVM does not need
to be configured for multipath as the multipath is provided by the
storage array. Since Oracle Cluster Software checks availability of the
shared device for the vote disk through periodic monitoring, the
multipath detection and failover time must be less than CRS's timeout
specified by the Cluster Synchronization Service (CSS) MISSCOUNT. On
SGeRAC configurations, the CSS MISSCOUNT value is set to 600 seconds.
Multipath failover time is typically between 30 to 120 seconds.
OCR and Vote Device

Shared storage for the OCR and Vote device should be on supported
shared storage volume managers with multipath configured and with
either the correct multipath failover time or CSS timeout.
36 Chapter 2
Interface Areas
Mirroring and Resilvering

On node and cluster wide failures, when SLVM mirroring is used and
Oracle resilvering is available, the recommendation for the logical
volume mirror recovery policy is set to full mirror resynchronization
(NOMWC) for control and redo files and no mirror resynchronization
(NONE) for the datafiles since Oracle would perform resilvering on the
datafiles based on the redo log.
NOTE If Oracle resilvering is not available, the mirror recovery policy should be
set to full mirror resynchronization (NOMWC) of all control, redo, and
datafiles.
Shared Storage Activation

Depending on your version of Oracle Cluster Software, the default
configuration for activation of the shared storage for Oracle Cluster
Software may be controlled by the /var/opt/oracle/oravg.conf file.
For the default configuration where the shared storage is activated by
SGeRAC before starting Oracle Cluster Software or RAC instance, the
oravg.conf file should not be present.
Listener
Automated Startup and Shutdown

CRS can be configured to automatically start, monitor, restart, and halt
listeners.
If CRS is not configured to start the listener automatically at Oracle
Cluster Software startup, the listener startup can be automated with
supported commands, such as srvctl and lsnrctl, through scripts or
SGeRAC packages. If the SGeRAC package is configured to start the
listener, the SGeRAC package would contain the virtual IP address
required by the listener.
Manual Startup and Shutdown

Manual listener startup and shutdown is supported through the
following commands: srvctl and lsnrctl
Chapter 2 37
Interface Areas
Network Monitoring
SGeRAC cluster provides network monitoring. For networks that are
redundant and monitored by Serviceguard cluster, Serviceguard cluster
provides local failover capability between local network interfaces (LAN)
that is transparent to applications utilizing User Datagram Protocol
(UDP) and Transport Control Protocol (TCP).
For virtual IP addresses (floating or package IP address) in
Serviceguard, Serviceguard also provides remote failover capability of
network connection endpoints between cluster nodes and transparent
local failover capability of network connection endpoints between
redundant local network interfaces.
NOTE Serviceguard can not be responsible for networks or connection

endpoints that it is not configured to monitor.
SGeRAC Heartbeat Network

Serviceguard supports multiple heartbeat networks, private or public.
Serviceguard heartbeat network can be configured as a single network
connection with redundant LAN or multiple connections with multiple
LANs (single or redundant).
CSS Heartbeat Network

The CSS IP addresses for peer communications are fixed IP addresses.
When CSS heartbeats are on a single network connection and does not
support multiple heartbeat networks. To protect against a network
single point of failure, the CSS heartbeat network should be configured
with redundant physical networks under SGeRAC monitoring. Since
SGeRAC does not support heartbeat over Hyperfabric (HF) networks,
the preferred configuration is for CSS and Serviceguard to share the
same cluster interconnect.
RAC Cluster Interconnect

Each set of RAC instances maintains peer communications on a single
connection and may not support multiple connections on HP-UX with
SGeRAC. To protect against a network single point of failure, the RAC
cluster interconnect should be configured with redundant networks
under Serviceguard monitoring and for Serviceguard to take action
38 Chapter 2
Interface Areas
(either a local failover or an instance package shutdown, or both) if the

RAC cluster interconnect fails. Serviceguard does not monitor
Hyperfabric networks directly (integration of Serviceguard and HF/EMS
monitor is supported).
Public Client Access

When the client connection endpoint (virtual or floating IP address) is
configured using Serviceguard packages, Serviceguard provides
monitoring, local failover, and remote failover capabilities. When
Serviceguard packages are not used, Serviceguard does not provide
monitor or failover support.
Chapter 2 39
RAC Instances
RAC Instances
Automated Startup and Shutdown

CRS can be configured to automatically start, monitor, restart, and halt
RAC instances.
If CRS is not configured to automatically start the RAC instance at
Oracle Cluster Software startup, the RAC instance startup can be
automated through scripts using supported commands, such as srvctl
or sqlplus, in a SGeRAC package to start and halt RAC instances.
NOTE svrctl and sqlplus are Oracle commands.
Manual Startup and Shutdown

Manual RAC instance startup and shutdown is supported through the
following commands: srvctl or sqlplus
Shared Storage
It is expected the shared storage is available when the RAC instance is
started. Since the RAC instance expects the shared storage to be
available, ensure the shared storage is activated. For SLVM, the shared
volume groups must be activated and for CVM, the disk group must be
activated. For CFS, the cluster file system must be mounted.
40 Chapter 2
Planning Storage for Oracle Cluster Software
Planning Storage for Oracle Cluster Software

Oracle Cluster Software requires shared storage for the Oracle Cluster
Registry (OCR) and a vote device. Automatic Storage Management can
not be used for the OCR and vote device since these files must be
accessible before Oracle Cluster Software starts. The minimum required
size for the OCR is 100MB and for the vote disk is 20 MB.
The Oracle OCR and vote device can be created on supported shared
storage, including SLVM logical volumes, CVM raw volumes, and CFS
file systems.
Chapter 2 41
Planning Storage for Oracle 10g RAC
Volume Planning with SLVM

Storage capacity for the Oracle database must be provided in the form of
logical volumes located in shared volume groups. The Oracle software
requires at least two log files for each Oracle instance, several Oracle
control files and data files for the database itself. For all these files,
Serviceguard Extension for RAC uses HP-UX raw logical volumes, which
are located in volume groups that are shared between the nodes in the
cluster. High availability is achieved by using high availability disk
arrays in RAID modes. The logical units of storage on the arrays are
accessed from each node through multiple physical volume links (PV
links, also known as alternate links), which provide redundant paths to
each unit of storage.
Fill out a Logical Volume worksheet to provide logical volume names for
logical volumes that you will create with the lvcreate command. The
Oracle DBA and the HP-UX system administrator should prepare this
worksheet together. Create entries for shared volumes only. For each
logical volume, enter the full pathname of the raw logical volume device
file. Be sure to include the desired size in MB. Following is a sample
worksheet filled out. However, this sample is only representative. For
different versions of the Oracle database, the size of files are different.
Refer to Appendix B, “Blank Planning Worksheets,” for samples of blank
worksheets. Make as many copies as you need. Fill out the worksheet
and keep it for future reference.
Storage Planning with CFS

With CFS, the database software, database files (control, redo, data
files), and archive logs may reside on a cluster file system visible by all
nodes. Also, the OCR and vote device can reside on CFS directories.
42 Chapter 2
ORACLE LOGICAL VOLUME WORKSHEET FOR LVM Page ___ of ____

===============================================================================
RAW LOGICAL VOLUME NAME SIZE (MB)
Oracle Cluster Registry: _____/dev/vg_ops/rora_ocr_____100___ (once per cluster)
Oracle Cluster Vote Disk: ____/dev/vg_ops/rora_vote_____20___ (once per cluster)
Oracle Control File: _____/dev/vg_ops/ropsctl1.ctl______110______

Oracle Control File 2: ___/dev/vg_ops/ropsctl2.ctl______110______
Instance 1 Redo Log 1: ___/dev/vg_ops/rops1log1.log_____120______
Instance 1 Redo Log 2: ___/dev/vg_ops/rops1log2.log_____120_______
Instance 1 Redo Log: __________________________________________________
Instance 2 Redo Log 1: ___/dev/vg_ops/rops2log1.log____120________
Instance 2 Redo Log: _________________________________________________
Data: System ___/dev/vg_ops/ropssystem.dbf___500__________
Data: Sysaux ___/dev/vg_ops/ropssysaux.dbf___800__________
Data: Temp ___/dev/vg_ops/ropstemp.dbf______250_______
Data: Users ___/dev/vg_ops/ropsusers.dbf_____120_________
Data: User data ___/dev/vg_ops/ropsdata1.dbf_200__________
Data: User data ___/dev/vg_ops/ropsdata2.dbf__200__________
Parameter: spfile1 ___/dev/vg_ops/ropsspfile1.ora __5_____
Password: ______/dev/vg_ops/rpwdfile.ora__5_______
Instance 1 undotbs1: /dev/vg_ops/ropsundotbs1.dbf___500___
Data: example1__/dev/vg_ops/ropsexample1.dbf__________160____
Chapter 2 43
Volume Planning with CVM

volumes located in shared disk groups. The Oracle software requires at
least two log files for each Oracle instance, several Oracle control files
and data files for the database itself. For all these files, Serviceguard
Extension for RAC uses HP-UX raw volumes, which are located in disk
groups that are shared between the nodes in the cluster. High
availability is achieved by using high availability disk arrays in RAID
nodes. The logical units of storage on the arrays are accessed from each
node through multiple physical volume links via DMP (Dynamic
Multi-pathing), which provides redundant paths to each unit of storage.
Fill out the VERITAS Volume worksheet to provide volume names for
volumes that you will create using the VERITAS utilities. The Oracle
DBA and the HP-UX system administrator should prepare this
volume, enter the full pathname of the raw volume device file. Be sure to
include the desired size in MB. Following is a sample worksheet filled
out. Refer to Appendix B, “Blank Planning Worksheets,” for samples of
blank worksheets. Make as many copies as you need. Fill out the
worksheet and keep it for future reference.
44 Chapter 2
ORACLE LOGICAL VOLUME WORKSHEET FOR CVM Page ___ of ____

===============================================================================
RAW VOLUME NAME SIZE (MB)
Oracle Cluster Registry: _____/dev/vx/rdsk/ops_dg/ora_ocr_____100___ (once per
cluster)
Oracle Cluster Vote Disk: ____/dev/vx/rdsk/ops_dg/ora_vote_____20___ (once per
cluster)
Oracle Control File: _____/dev/vx/rdsk/ops_dg/opsctl1.ctl______110______

Oracle Control File 2: ___/dev/vx/rdsk/ops_dg/opsctl2.ctl______110______
Instance 1 Redo Log 1: ___/dev/vx/rdsk/ops_dg/ops1log1.log_____120______
Instance 1 Redo Log 3: ___/dev/vx/rdsk/ops_dg/ops1log3.log_____120_______
Instance 2 Redo Log 1: ___/dev/vx/rdsk/ops_dg/ops2log1.log____120________
Data: System ___/dev/vx/rdsk/ops_dg/opssystem.dbf___500__________
Data: Sysaux ___/dev/vx/rdsk/ops_dg/opssysaux.dbf___800__________
Data: Temp ___/dev/vx/rdsk/ops_dg/opstemp.dbf______250_______
Data: Users ___/dev/vx/rdsk/ops_dg/opsusers.dbf_____120_________
Data: User data ___/dev/vx/rdsk/ops_dg/opsdata1.dbf_200__________
Data: User data ___/dev/vx/rdsk/ops_dg/opsdata2.dbf__200__________
Data: User data ___/dev/vx/rdsk/ops_dg/opsdata3.dbf__200__________
Parameter: spfile1 ___/dev/vx/rdsk/ops_dg/opsspfile1.ora __5_____
Password: ______/dev/vx/rdsk/ops_dg/pwdfile.ora__5_______
Instance 1 undotbs1: /dev/vx/rdsk/ops_dg/opsundotbs1.dbf___500___
Instance 2 undotbs2: /dev/vx/rdsk/ops_dg/opsundotbs2.dbf___500___
Data: example1__/dev/vx/rdsk/ops_dg/opsexample1.dbf__________160____
Chapter 2 45
Installing Serviceguard Extension for RAC

Installing Serviceguard Extension for RAC includes updating the
software and rebuilding the kernel to support high availability cluster
operation for Oracle Real Application Clusters.
Prior to installing Serviceguard Extension for RAC, the following must
be installed:
• Correct version of HP-UX

• Correct version of Serviceguard
To install Serviceguard Extension for RAC, use the following steps for
each node:
NOTE For the up to date version compatibility for Serviceguard and HP-UX, see
the SGeRAC release notes.
1. Mount the distribution media in the tape drive, CD, or DVD reader.
2. Run Software Distributor, using the swinstall command.
3. Specify the correct input device.
4. Choose the following bundle from the displayed list:
5. After choosing the bundle, select OK to install the software.
46 Chapter 2
Configuration File Parameters

You need to code specific entries for all the storage groups that you want
to use in an Oracle RAC configuration. If you are using LVM, the
OPS_VOLUME_GROUP parameter is included in the cluster ASCII file. If you
are using VERITAS CVM, the STORAGE_GROUP parameter is included in
the package ASCII file. Details are as follows:
OPS_VOLUME_GROUP
The name of an LVM volume group whose disks are
attached to at least two nodes in the cluster; the disks
will be accessed by more than one node at a time using
SLVM with concurrency control provided by Oracle
RAC. Such disks are considered cluster aware.
Volume groups listed under this parameter are marked
for activation in shared mode. The entry can contain up
to 40 characters.
STORAGE_GROUP
This parameter is used for CVM disk groups. Enter the
names of all the CVM disk groups the package will use.
In the ASCII package configuration file, this parameter
is called STORAGE_GROUP.
Unlike LVM volume groups, CVM disk groups are not entered in the
cluster configuration file, they are entered in the package configuration
file.
NOTE CVM 4.x with CFS does not use the STORAGE_GROUP parameter because
the disk group activation is performed by the multi-node package. CVM
3.x or 4.x without CFS uses the STORAGE_GROUP parameter in the ASCII
package configuration file in order to activate the disk group.
Do not enter the names of LVM volume groups or VxVM disk groups in
the package ASCII configuration file.
Chapter 2 47
Creating a Storage Infrastructure with LVM

In addition to configuring the cluster, you create the appropriate logical
volume infrastructure to provide access to data from different nodes.
This is done with Logical Volume Manager (LVM), VERITAS Cluster
Volume Manager (CVM), or VERITAS Volume Manager (VxVM). LVM
and VxVM configuration are done before cluster configuration, and CVM
configuration is done after cluster configuration.
This section describes how to create LVM volume groups for use with
Oracle data. Before configuring the cluster, you create the appropriate
logical volume infrastructure to provide access to data from different
nodes. This is done with Logical Volume Manager. Separate procedures
are given for the following:
• Building Volume Groups for RAC on Mirrored Disks

• Building Mirrored Logical Volumes for RAC with LVM Commands
• Creating RAC Volume Groups on Disk Arrays
• Creating Logical Volumes for RAC on Disk Arrays
The Event Monitoring Service HA Disk Monitor provides the capability
to monitor the health of LVM disks. If you intend to use this monitor for
your mirrored disks, you should configure them in physical volume
groups. For more information, refer to the manual Using HA Monitors.
Building Volume Groups for RAC on Mirrored Disks

The procedure described in this section uses physical volume groups
for mirroring of individual disks to ensure that each logical volume is
mirrored to a disk on a different I/O bus. This kind of arrangement is
known as PVG-strict mirroring. It is assumed that your disk hardware
is already configured in such a way that a disk to be used as a mirror
copy is connected to each node on a different bus than the bus that is
used for the other (primary) copy.
For more information on using LVM, refer to the HP-UX Managing
Systems and Workgroups manual.
48 Chapter 2
Creating Volume Groups and Logical Volumes

If your volume groups have not been set up, use the procedure in the next
sections. If you have already done LVM configuration, skip ahead to the
section “Installing Oracle Real Application Clusters.”
Selecting Disks for the Volume Group Obtain a list of the disks on
both nodes and identify which device files are used for the same disk on
both. Use the following command on each node to list available disks as
they are known to each system:
# lssf /dev/dsk/*
In the following examples, we use /dev/rdsk/c1t2d0 and
/dev/rdsk/c0t2d0, which happen to be the device names for the same
disks on both ftsys9 and ftsys10. In the event that the device file
names are different on the different nodes, make a careful note of the
correspondences.
Creating Physical Volumes On the configuration node (ftsys9), use

the pvcreate command to define disks as physical volumes. This only
needs to be done on the configuration node. Use the following commands
to create two physical volumes for the sample configuration:
# pvcreate -f /dev/rdsk/c1t2d0
Creating a Volume Group with PVG-Strict Mirroring Use the

following steps to build a volume group on the configuration node
(ftsys9). Later, the same volume group will be created on other nodes.
1. First, set up the group directory for vgops:

# mkdir /dev/vg_ops
2. Next, create a control file named group in the directory
/dev/vg_ops, as follows:
# mknod /dev/vg_ops/group c 64 0xhh0000
The major number is always 64, and the hexadecimal minor number
has the form
0xhh0000
Chapter 2 49
where hh must be unique to the volume group you are creating. Use
the next hexadecimal number that is available on your system, after
the volume groups that are already configured. Use the following
command to display a list of existing volume groups:
# ls -l /dev/*/group
3. Create the volume group and add physical volumes to it with the
following commands:
# vgcreate -g bus0 /dev/vg_ops /dev/dsk/c1t2d0
# vgextend -g bus1 /dev/vg_ops /dev/dsk/c0t2d0
The first command creates the volume group and adds a physical
volume to it in a physical volume group called bus0. The second
command adds the second drive to the volume group, locating it in a
different physical volume group named bus1. The use of physical
volume groups allows the use of PVG-strict mirroring of disks and
PV links.
4. Repeat this procedure for additional volume groups.
Building Mirrored Logical Volumes for RAC with LVM

Commands
After you create volume groups and define physical volumes for use in
them, you define mirrored logical volumes for data, logs, and control
files. It is recommended that you use a shell script to issue the
commands described in the next sections. The commands you use for
creating logical volumes vary slightly depending on whether you are
creating logical volumes for RAC redo log files or for use with Oracle
data.
Creating Mirrored Logical Volumes for RAC Redo Logs and

Control Files
Create logical volumes for use as redo log and control files by selecting
mirror consistency recovery. Use the same options as in the following
example:
# lvcreate -m 1 -M n -c y -s g -n redo1.log -L 28 /dev/vg_ops
The -m 1 option specifies single mirroring; the -M n option ensures that
mirror write cache recovery is set off; the -c y means that mirror
consistency recovery is enabled; the -s g means that mirroring is
50 Chapter 2
PVG-strict, that is, it occurs between different physical volume groups;

the -n redo1.log option lets you specify the name of the logical volume;
and the -L 28 option allocates 28 megabytes.
NOTE It is important to use the -M n and -c y options for both redo logs and
control files. These options allow the redo log files to be resynchronized
by SLVM following a system crash before Oracle recovery proceeds. If
these options are not set correctly, you may not be able to continue with
database recovery.
If the command is successful, the system will display messages like the
following:
Logical volume “/dev/vg_ops/redo1.log” has been successfully
created
with character device “/dev/vg_ops/rredo1.log”
extended
Note that the character device file name (also called the raw logical
volume name) is used by the Oracle DBA in building the RAC database.
Creating Mirrored Logical Volumes for RAC Data Files

Following a system crash, the mirrored logical volumes need to be
resynchronized, which is known as “resilvering”.
If Oracle does not perform “resilvering” of RAC data files that are
mirrored logical volumes, choose a mirror consistency policy of
“NOMWC”. This is done by disabling mirror write caching and enabling
mirror consistency recovery. With “NOMWC”, SLVM performs the
resynchronization.
Create logical volumes for use as Oracle data files by using the same
options as in the following example:
# lvcreate -m 1 -M n -c y -s g -n system.dbf -L 408 /dev/vg_ops
the -n system.dbf option lets you specify the name of the logical
volume; and the -L 408 option allocates 408 megabytes.
Chapter 2 51
If Oracle performs resilvering of RAC data files that are mirrored logical
volumes, choose a mirror consistency policy of “NONE” by disabling both
mirror write caching and mirror consistency recovery. With a mirror
consistency policy of “NONE”, SLVM does not perform the
resynchronization.
NOTE Contact Oracle to determine if your version of Oracle RAC allows

“resilvering” and to appropriately configure the mirror consistency
recovery policy for your logical volumes.
# lvcreate -m 1 -M n -c n -s g -n system.dbf -L 408 \
/dev/vg_ops
mirror write cache recovery is set off; the -c n means that mirror
consistency recovery is disabled; the -s g means that mirroring is
following:
Logical volume “/dev/vg_ops/system.dbf” has been successfully
created
with character device “/dev/vg_ops/rsystem.dbf”
extended
volume name) is used by the Oracle DBA in building the OPS database.
Creating RAC Volume Groups on Disk Arrays

The procedure described in this section assumes that you are using
RAID-protected disk arrays and LVM’s physical volume links (PV links)
to define redundant data paths from each node in the cluster to every
logical unit on the array.
52 Chapter 2
On your disk arrays, you should use redundant I/O channels from each
node, connecting them to separate controllers on the array. Then you can
define alternate links to the LUNs or logical disks you have defined on
the array. If you are using SAM, choose the type of disk array you wish to
configure, and follow the menus to define alternate links. If you are using
LVM commands, specify the links on the command line.
The following example shows how to configure alternate links using LVM
commands. The following disk configuration is assumed:
8/0.15.0 /dev/dsk/c0t15d0 /* I/O Channel 0 (8/0) SCSI address 15 LUN 0 */

Assume that the disk array has been configured, and that both the
following device files appear for the same LUN (logical disk) when you
run the ioscan command:
/dev/dsk/c0t15d0
/dev/dsk/c1t3d0
Use the following procedure to configure a volume group for this logical
disk:
1. First, set up the group directory for vg_ops:

# mkdir /dev/vg_ops
has the form
0xhh0000
Chapter 2 53
3. Use the pvcreate command on one of the device files associated with
the LUN to define the LUN to LVM as a physical volume.
It is only necessary to do this with one of the device file names for the
LUN. The -f option is only necessary if the physical volume was
previously used in some other volume group.
4. Use the following to create the volume group with the two links:
# vgcreate /dev/vg_ops /dev/dsk/c0t15d0 /dev/dsk/c1t3d0
LVM will now recognize the I/O channel represented by
/dev/dsk/c0t15d0 as the primary link to the disk; if the primary link
fails, LVM will automatically switch to the alternate I/O channel
represented by /dev/dsk/c1t3d0. Use the vgextend command to add
additional disks to the volume group, specifying the appropriate physical
volume name for each PV link.
Repeat the entire procedure for each distinct volume group you wish to
create. For ease of system administration, you may wish to use different
volume groups to separate logs from data and control files.
NOTE The default maximum number of volume groups in HP-UX is 10. If you
intend to create enough new volume groups that the total exceeds ten,
you must increase the maxvgs system parameter and then re-build the
HP-UX kernel. Use SAM and select the Kernel Configuration area,
then choose Configurable Parameters. Maxvgs appears on the list.
Creating Logical Volumes for RAC on Disk Arrays

After you create volume groups and add PV links to them, you define
logical volumes for data, logs, and control files. The following are some
examples:
54 Chapter 2
# lvcreate -n ops1log1.log -L 4 /dev/vg_ops

# lvcreate -n opsctl1.ctl -L 4 /dev/vg_ops
# lvcreate -n system.dbf -L 28 /dev/vg_ops
# lvcreate -n opsdata1.dbf -L 1000 /dev/vg_ops
Oracle Demo Database Files

The following set of files is required for the Oracle demo database which
you can create during the installation process.
Table 2-1 Required Oracle File Names for Demo Database
Oracle
LV File
Size Size
Logical Volume Name (MB) Raw Logical Volume Path Name (MB)*
opsctl1.ctl 118 /dev/vg_ops/ropsctl1.ctl 110
ops1log1.log 128 /dev/vg_ops/rops1log1.log 120
opssystem.dbf 408 /dev/vg_ops/ropssystem.dbf 400
opssysaux.dbf 808 /dev/vg_ops/ropssysaux.dbf 800
opstemp.dbf 258 /dev/vg_ops/ropstemp.dbf 250
opsusers.dbf 128 /dev/vg_ops/ropsusers.dbf 120
opsdata1.dbf 208 /dev/vg_ops/ropsdata1.dbf 200
Chapter 2 55
Table 2-1 Required Oracle File Names for Demo Database (Continued)
Oracle
LV File
Size Size
opsspfile1.ora 5 /dev/vg_ops/ropsspfile1.ora 5
pwdfile.ora 5 /dev/vg_ops/rpwdfile.ora 5
opsundotbs1.dbf 508 /dev/vg_ops/ropsundotbs1.log 500
opsundotbs2.dbf 508 /dev/vg_ops/ropsundotbs2.log 500
example1.dbf 168 /dev/vg_ops/ropsexample1.dbf 160
The size of the logical volume is larger than the Oracle file size because
Oracle needs extra space to allocate a header in addition to the file's
actual data capacity.
Create these files if you wish to build the demo database. The three
logical volumes at the bottom of the table are included as additional data
files, which you can create as needed, supplying the appropriate sizes. If
your naming conventions require, you can include the Oracle SID and/or
the database name to distinguish files for different instances and
different databases. If you are using the ORACLE_BASE directory
structure, create symbolic links to the ORACLE_BASE files from the
appropriate directory. Example:
# ln -s /dev/vg_ops/ropsctl1.ctl \
/u01/ORACLE/db001/ctrl01_1.ctl
After creating these files, set the owner to oracle and the group to dba
with a file mode of 660. The logical volumes are now available on the
primary node, and the raw logical volume names can now be used by the
Oracle DBA.
56 Chapter 2
Displaying the Logical Volume Infrastructure

To display the volume group, use the vgdisplay command:
# vgdisplay -v /dev/vg_ops
Exporting the Logical Volume Infrastructure

Before the Oracle volume groups can be shared, their configuration data
must be exported to other nodes in the cluster. This is done either in
Serviceguard Manager or by using HP-UX commands, as shown in the
following sections.
Exporting with LVM Commands

Use the following commands to set up the same volume group on another
cluster node. In this example, the commands set up a new volume group
on a system known as ftsys10. This volume group holds the same
physical volume that was created on a configuration node known as
ftsys9.
To set up the volume group on ftsys10 (and other nodes), use the
following steps:
1. On ftsys9, copy the mapping of the volume group to a specified file.

# vgexport -s -p -m /tmp/vg_ops.map /dev/vg_ops
2. Still on ftsys9, copy the map file to ftsys10 (and to additional
nodes as necessary.)
# rcp /tmp/vg_ops.map ftsys10:/tmp/vg_ops.map
3. On ftsys10 (and other nodes, as necessary), create the volume group
directory and the control file named group:
# mkdir /dev/vg_ops
For the group file, the major number is always 64, and the
hexadecimal minor number has the form
0xhh0000
Chapter 2 57
where hh must be unique to the volume group you are creating. If

possible, use the same number as on ftsys9. Use the following
4. Import the volume group data using the map file from node ftsys9.
On node ftsys10 (and other nodes, as necessary), enter:
# vgimport -s -m /tmp/vg_ops.map /dev/vg_ops
58 Chapter 2
Installing Oracle Real Application Clusters
NOTE Some versions of Oracle RAC requires installation of additional software.

Refer to your version of Oracle for specific requirements.
Before installing the Oracle Real Application Cluster software, make

sure the storage cluster is running. Login as the oracle user on one node
and then use the Oracle installer to install Oracle software and to build
the correct Oracle runtime executables. When executables are installed
to a local file system on each node, the Oracle installer copies the
executables to the other nodes in the cluster.
For details on Oracle installation, refer to the Oracle installation
documentation. As part of this installation, the Oracle installer installs
the executables and optionally, the Oracle installer can build an Oracle
demo database on the primary node. The demo database files can be the
character (raw) device files names for the logical volumes create earlier.
For a demo database on SLVM or CVM, create logical volumes as shown
in Table 2-1, “Required Oracle File Names for Demo Database.” As the
installer prompts for the database file names, either the pathnames of
the raw logical volumes instead of using the defaults. If you do not wish
to install the demo database, select install software only.
Chapter 2 59
Cluster Configuration ASCII File

The following is an example of an ASCII configuration file generated
with the cmquerycl command using the -w full option on a system
with Serviceguard Extension for RAC. The OPS_VOLUME_GROUP
parameters appear at the end of the file.
# **********************************************************************
# ********* HIGH AVAILABILITY CLUSTER CONFIGURATION FILE ***************
# ***** For complete details about cluster parameters and how to *******
# ***** set them, consult the Serviceguard manual. *********************
# **********************************************************************
# Enter a name for this cluster. This name will be used to identify the
# cluster when viewing or manipulating it.
CLUSTER_NAME cluster 1
# Cluster Lock Parameters

# The cluster lock is used as a tie-breaker for situations
# in which a running cluster fails, and then two equal-sized
# sub-clusters are both trying to form a new cluster. The
# cluster lock may be configured using only one of the
# following alternatives on a cluster:
# the LVM lock disk
# the quorom server
#
#
# Consider the following when configuring a cluster.
# For a two-node cluster, you must use a cluster lock. For
# a cluster of three or four nodes, a cluster lock is strongly
# recommended. For a cluster of more than four nodes, a
# cluster lock is recommended. If you decide to configure
# a lock for a cluster of more than four nodes, it must be
# a quorum server.
# Lock Disk Parameters. Use the FIRST_CLUSTER_LOCK_VG and

# FIRST_CLUSTER_LOCK_PV parameters to define a lock disk.
# The FIRST_CLUSTER_LOCK_VG is the LVM volume group that
# holds the cluster lock. This volume group should not be
# used by any other cluster as a cluster lock device.
# Quorum Server Parameters. Use the QS_HOST, QS_POLLING_INTERVAL,
60 Chapter 2
# and QS_TIMEOUT_EXTENSION parameters to define a quorum server.

# The QS_HOST is the host name or IP address of the system
# that is running the quorum server process. The
# QS_POLLING_INTERVAL (microseconds) is the interval at which
# Serviceguard checks to make sure the quorum server is running.
# The optional QS_TIMEOUT_EXTENSION (microseconds) is used to increase
# the time interval after which the quorum server is marked DOWN.
#
# The default quorum server timeout is calculated from the
# Serviceguard cluster parameters, including NODE_TIMEOUT and
# HEARTBEAT_INTERVAL. If you are experiencing quorum server
# timeouts, you can adjust these parameters, or you can include
# the QS_TIMEOUT_EXTENSION parameter.
#
# The value of QS_TIMEOUT_EXTENSION will directly effect the amount
# of time it takes for cluster reformation in the event of failure.
# For example, if QS_TIMEOUT_EXTENSION is set to 10 seconds, the cluster
# reformation will take 10 seconds longer than if the QS_TIMEOUT_EXTENSION
# was set to 0. This delay applies even if there is no delay in
# contacting the Quorum Server. The recommended value for
# QS_TIMEOUT_EXTENSION is 0, which is used as the default
# and the maximum supported value is 30000000 (5 minutes).
#
# For example, to configure a quorum server running on node
# “qshost” with 120 seconds for the QS_POLLING_INTERVAL and to
# add 2 seconds to the system assigned value for the quorum server
# timeout, enter:
#
# QS_HOST qshost
# QS_POLLING_INTERVAL 120000000
# QS_TIMEOUT_EXTENSION 2000000
# Definition of nodes in the cluster.

# Repeat node definitions as necessary for additional nodes.
# NODE_NAME is the specified nodename in the cluster.
# It must match the hostname and both cannot contain full domain name.
# Each NETWORK_INTERFACE, if configured with IPv4 address,
# must have ONLY one IPv4 address entry with it which could
# be either HEARTBEAT_IP or STATIONARY_IP.
# Each NETWORK_INTERFACE, if configured with IPv6 address(es)
# can have multiple IPv6 address entries(up to a maximum of 2,
# only one IPv6 address entry belonging to site-local scope
# and only one belonging to global scope) which must be all
# STATIONARY_IP. They cannot be HEARTBEAT_IP.
Chapter 2 61
NODE_NAME ever3a
NETWORK_INTERFACE lan0
STATIONARY_IP15.244.64.140
HEARTBEAT_IP192.77.1.1
# List of serial device file names
# For example:
# SERIAL_DEVICE_FILE /dev/tty0p0
# Primary Network Interfaces on Bridged Net 1: lan0.

# Warning: There are no standby network interfaces on bridged net 1.
# Possible standby Network Interfaces on Bridged Net 2: lan2.
# Cluster Timing Parameters (microseconds).
# The NODE_TIMEOUT parameter defaults to 2000000 (2 seconds).

# This default setting yields the fastest cluster reformations.
# However, the use of the default value increases the potential
# for spurious reformations due to momentary system hangs or
# network load spikes.
# For a significant portion of installations, a setting of
# 5000000 to 8000000 (5 to 8 seconds) is more appropriate.
# The maximum value recommended for NODE_TIMEOUT is 30000000
# (30 seconds).
HEARTBEAT_INTERVAL 1000000
NODE_TIMEOUT 2000000
# Configuration/Reconfiguration Timing Parameters (microseconds).
AUTO_START_TIMEOUT 600000000
NETWORK_POLLING_INTERVAL 2000000
# Network Monitor Configuration Parameters.

# The NETWORK_FAILURE_DETECTION parameter determines how LAN card failures are
detected.
# If set to INONLY_OR_INOUT, a LAN card will be considered down when its inbound
# message count stops increasing or when both inbound and outbound
# message counts stop increasing.
# If set to INOUT, both the inbound and outbound message counts must
62 Chapter 2
# stop increasing before the card is considered down.

NETWORK_FAILURE_DETECTION INOUT
# Package Configuration Parameters.

# Enter the maximum number of packages which will be configured in the cluster.
# You can not add packages beyond this limit.
# This parameter is required.
MAX_CONFIGURED_PACKAGES 150
# Access Control Policy Parameters.

#
# Three entries set the access control policy for the cluster:
# First line must be USER_NAME, second USER_HOST, and third USER_ROLE.
# Enter a value after each.
#
# 1. USER_NAME can either be ANY_USER, or a maximum of
# 8 login names from the /etc/passwd file on user host.
# 2. USER_HOST is where the user can issue Serviceguard commands.
# If using Serviceguard Manager, it is the COM server.
# Choose one of these three values: ANY_SERVICEGUARD_NODE, or
# (any) CLUSTER_MEMBER_NODE, or a specific node. For node,
# use the official hostname from domain name server, and not
# an IP addresses or fully qualified name.
# 3. USER_ROLE must be one of these three values:
# * MONITOR: read-only capabilities for the cluster and packages
# * PACKAGE_ADMIN: MONITOR, plus administrative commands for packages
# in the cluster
# * FULL_ADMIN: MONITOR and PACKAGE_ADMIN plus the administrative
# commands for the cluster.
#
# Access control policy does not set a role for configuration
# capability. To configure, a user must log on to one of the
# cluster’s nodes as root (UID=0). Access control
# policy cannot limit root users’ access.
#
# MONITOR and FULL_ADMIN can only be set in the cluster configuration file,
# and they apply to the entire cluster. PACKAGE_ADMIN can be set in the
# cluster or a package configuration file. If set in the cluster
# configuration file, PACKAGE_ADMIN applies to all configured packages.
# If set in a package configuration file, PACKAGE_ADMIN applies to that
# package only.
#
# Conflicting or redundant policies will cause an error while applying
# the configuration, and stop the process. The maximum number of access
# policies that can be configured in the cluster is 200.
Chapter 2 63
#
# Example: to configure a role for user john from node noir to
# administer a cluster and all its packages, enter:
# USER_NAME john
# USER_HOST noir
# USER_ROLE FULL_ADMIN
# List of cluster aware LVM Volume Groups. These volume groups will
# be used by package applications via the vgchange -a e command.
# Neither CVM or VxVM Disk Groups should be used here.
# For example:
# VOLUME_GROUP /dev/vgdatabase
# VOLUME_GROUP /dev/vg02
# List of OPS Volume Groups.

# Formerly known as DLM Volume Groups, these volume groups
# will be used by OPS or RAC cluster applications via
# the vgchange -a s command. (Note: the name DLM_VOLUME_GROUP
# is also still supported for compatibility with earlier versions.)
# For example:
# OPS_VOLUME_GROUP /dev/vgdatabase
# OPS_VOLUME_GROUP /dev/vg02
64 Chapter 2
Creating a Storage Infrastructure with CFS
Creating a SGeRAC Cluster with CFS 4.1 for Oracle

10g
With CFS, the database software and database files (control, redo, data
files), and archive logs may reside on a cluster file system, which is
visible by all nodes.
The following software needs to be installed in order to use this
configuration:
• SGeRAC
• CFS
CFS and SGeRAC are available in selected HP Serviceguard Storage
Management Suite bundles.
Refer to the HP Serviceguard Storage Management Suite Version A.01.00
Release Notes.
In the example below, both the Oracle RAC software and datafiles reside
on CFS. There is a single Oracle home. Three CFS file systems are
created for Oracle home, Oracle datafiles, and for the Oracle Cluster
Registry (OCR) and vote device. The Oracle Cluster Software home is on
a local file system.
/cfs/mnt1 – for Oracle Base and Home
/cfs/mnt2 – for Oracle datafiles
/cfs/mnt3 - for OCR and Vote device
Initializing the VERITAS Volume Manager

Use the following steps to create a two node SGeRAC cluster with CFS
and Oracle
1. Initialize the VERITAS Volume Manager

If not already done, install the VxVM license key on all nodes. Use
the following command:
# vxinstall
Chapter 2 65
NOTE CVM 4.1 does not require rootdg
2. Create the Cluster ASCII file:

# cd /etc/cmcluster
# cmquerycl -C clm.asc -n ever3a -n ever3b
Edit Cluster file
3. Create the Cluster
# cmapplyconf -C clm.asc
4. Start the Cluster
# cmruncl
# cmviewcl
The following output will be displayed:
CLUSTER STATUS
ever3_cluster up
NODE STATUS STATE

ever3a up running
ever3b up running
5. Configure the Cluster Volume Manager (CVM)
Configure the system multi-node package, SG-CFS-pkg, to configure
and start the CVM/CFS stack. Unlike VxVM-CVM-pkg, the
SG-CFS-pkg does not restrict heartbeat subnets to a single subnet
and supports multiple subnets.
# cfscluster config -s
CVM is now configured
Starting CVM...
It might take a few minutes to complete
When CVM starts up, it selects a master node, which is the node
from which you must issue the disk group configuration commands.
To determine the master node, issue the following command from
each node in the cluster:
66 Chapter 2
# vxdctl -c mode
mode: enabled: cluster active - SLAVE
master: ever3b
or
mode: enabled: cluster active - MASTER
slave: ever3b
6. Converting Disks from LVM to CVM
You can use the vxvmconvert utility to convert LVM volume groups
into CVM disk groups. Before you can do this, the volume group must
be deactivated, which means that any package that uses the volume
group must be halted. This procedure is described in the Managing
Serviceguard Twelfth Edition user’s guide Appendix G.
7. Initializing Disks for CVM/CFS
You need to initialize the physical disks that will be employed in
CVM disk groups. If a physical disk has been previously used with
LVM, you should use the pvremove command to delete the LVM
header data from all the disks in the volume group (this is not
necessary if you have not previously used the disk with LVM).
To initialize a disk for CVM, log on to the master node, then use the
vxdiskadm program to initialize multiple disks, or use the
vxdisksetup command to initialize one disk at a time, as in the
following example:
# /etc/vx/bin/vxdisksetup -i c4t4d0
8. Create the Disk Group for RAC
Use the vxdg command to create disk groups. Use the -s option to
specify shared mode, as in the following example:
# vxdg -s init cfsdg1 c4t4d0
9. Create the Disk Group Multi-Node package. Use the following
command to add the disk group to the cluster:
# cfsdgadm add cfsdg1 all=sw
Chapter 2 67
Package name “SG-CFS-DG-1” was generated to control the

resource shared disk group “cfsdg1” is associated with
the cluster.
10. Activate the Disk Group
# cfsdgadm activate cfsdg1
11. Creating Volumes and Adding a Cluster Filesystem
# vxassist -g cfsdg1 make vol1 10240m
# newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol1
version 6 layout
10485760 sectors, 10485760 blocks of size 1024, log size
16384 blocks
largefiles supported
version 6 layout
16384 blocks
version 6 layout
307200 sectors, 307200 blocks of size 1024, log size 1024
blocks largefiles supported
12. Configure Mount Point
# cfsmntadm add cfsdg1 vol1 /cfs/mnt1 all=rw
68 Chapter 2
Package name “SG-CFS-MP-1” was generated to control the

resource. Mount point “/cfs/mnt1” was associated with the
cluster.
cluster.
resource. Mount point “/cfs/mnt3” that was associated
with the cluster.
NOTE The diskgroup and mount point multi-node packages

(SG-CFS-DG_ID# and SG-CFS-MP_ID#) do not monitor the health of
the disk group and mount point. They check that the application
packages that depend on them have access to the disk groups and
mount points. If the dependent application package loses access and
cannot read and write to the disk, it will fail; however that will not
cause the DG or MP multi-node package to fail.
13. Mount Cluster Filesystem

# cfsmount /cfs/mnt1
14. Check CFS Mount Points
# bdf | grep cfs
/dev/vx/dsk/cfsdg1/vol1
10485760 19651 9811985 0% /cfs/mnt1
10485760 19651 9811985 0% /cfs/mnt2
307200 1802 286318 1% /cfs/mnt3
Chapter 2 69
15. View the Configuration

# cmviewcl
CLUSTER STATUS
ever3_cluster up
NODE STATUS STATE GMS_STATE

ever3a up running unknown
ever3b up running unknown
MULTI_NODE_PACKAGES
PACKAGE STATUS STATE AUTO_RUN SYSTEM

SG-CFS-pkg up running enabled yes
SG-CFS-DG-1 up running enabled no
SG-CFS-MP-1 up running enabled no
CAUTION Once you create the disk group and mount point packages, it is critical
that you administer the cluster with the cfs commands, including
cfsdgadm, cfsmntadm, cfsmount, and cfsumount. If you use the general
commands such as mount and umount, it could cause serious problems,
such as writing to the local file system instead of the cluster file system.
Any form of the mount command (for example, mount -o cluster,
dbed_chkptmount, or sfrac_chkptmount) other than cfsmount or
cfsumount in a HP Serviceguard Storage Management Suite
environment with CFS should be done with caution. These non-cfs
commands could cause conflicts with subsequent command operations on
the file system or Serviceguard packages. Use of these other forms of
mount will not create an appropriate multi-node package which means
that the cluster packages are not aware of the file system changes
Deleting CFS from the Cluster

Halt the applications that are using CFS file systems.
1. Unmount CFS Mount Points

# cfsumount /cfs/mnt1
70 Chapter 2
2. Delete Mount Point Multi-node Package
# cfsmntadm delete /cfs/mnt1
The following output will be generated:
Mount point “/cfs/mnt1” was disassociated from the
cluster
cluster
cluster Cleaning up resource controlling shared disk
group “cfsdg1” Shared disk group “cfsdg1” was
disassociated from the cluster.
NOTE The disk group package is deleted if there is no dependency.
3. Delete Disk Group Multi-node Package

# cfsdgadm delete cfsdg1
Shared disk group “cfsdg1” was disassociated from the
cluster.
NOTE “cfsmntadm delete” also deletes the disk group if there is no

dependent package. To ensure the disk group deletion is complete,
use the above command to delete the disk group package.
4. De-configure CVM
# cfscluster stop
Chapter 2 71

Stopping CVM...CVM is stopped
# cfscluster unconfig
CVM is now unconfigured
72 Chapter 2
Creating a Storage Infrastructure with CVM

This is done with Logical Volume Manager (LVM), VERITAS Volume
Manager (VxVM), or VERITAS Cluster Volume Manager (CVM). LVM
This section shows how to configure storage using the VERITAS Cluster
Volume Manager (CVM). The examples show how to configure RAC disk
groups, but you can also create CVM disk groups for non-RAC use. For
more information, including details about configuration of plexes
(mirrors), multi-pathing, and RAID, refer to the HP-UX documentation
for the VERITAS Volume Manager.

If you are about to create disk groups for the first time, you need to
initialize the Volume Manager. This is done by creating a disk group
known as rootdg that contains at least one disk. Use the following
command after installing CVM on each node:
# vxinstall
This displays a menu-driven program that steps you through the CVM
initialization sequence. From the main menu, choose the “Custom”
option, and specify the disk you wish to include in rootdg.
IMPORTANT Creating a rootdg disk group is only necessary the first time you use the
Volume Manager. CVM 4.1 does not require a rootdg.
Using CVM 4.x

This section has information on how to prepare the cluster and the
system multi-node package with CVM 4.x only (without the CFS
filesystem).
Chapter 2 73
Preparing the Cluster and the System Multi-node Package for

use with CVM 4.x
1. Create the Cluster file:

# cd /etc/cmcluster
Edit Cluster file
• Start the Cluster

# cmruncl
# cmviewcl
CLUSTER STATUS
ever3_cluster up
NODE STATUS STATE

ever3a up running
ever3b up running
and start the CVM stack. Unlike VxVM-CVM-pkg, the SG-CFS-pkg
does not restrict heartbeat subnets to a single subnet and supports
multiple subnets.
Use the cmapplyconf command:
# cmapplyconf -P /etc/cmcluster/cfs/SG-CFS-pkg.conf
# cmrunpkg SG-CFS-pkg
# vxdctl -c mode
74 Chapter 2

master: ever3b
or
slave: ever3b
• Converting Disks from LVM to CVM

Use the vxvmconvert utility to convert LVM volume groups into
CVM disk groups. Before you can do this, the volume group must
be deactivated, which means that any package that uses the
volume group must be halted. This procedure is described in the
Managing Serviceguard Thirteenth Edition user’s guide
Appendix G.
• Initializing Disks for CVM
It is necessary to initialize the physical disks that will be
employed in CVM disk groups. If a physical disk has been
previously used with LVM, you should use the pvremove
command to delete the LVM header data from all the disks in the
volume group (this is not necessary if you have not previously
used the disk with LVM).
To initialize a disk for CVM, log on to the master node, then use
the vxdiskadm program to initialize multiple disks, or use the
following example:
• Create the Disk Group for RAC
Use the vxdg command to create disk groups. Use the -s option
to specify shared mode, as in the following example:
# vxdg -s init ops_dg c4t4d0
# vxassist -g ops_dg make vol1 10240m
Chapter 2 75
# cmviewcl
CLUSTER STATUS
ever3_cluster up
NODE STATUS STATE

ever3a up running
ever3b up running
MULTI_NODE_PACKAGES

IMPORTANT After creating these files, use the vxedit command to change the
ownership of the raw volume files to oracle and the group membership
to dba, and to change the permissions to 660. Example:
# cd /dev/vx/rdsk/ops_dg
# vxedit -g ops_dg set user=oracle *
# vxedit -g ops_dg set group=dba *
# vxedit -g ops_dg set mode=660 *
The logical volumes are now available on the primary node, and the raw
logical volume names can now be used by the Oracle DBA.
Mirror Detachment Policies with CVM The required CVM disk

mirror detachment policy is ‘global’, which means that as soon as one
node cannot see a specific mirror copy (plex), all nodes cannot see it as
well. The alternate policy is ‘local’, which means that if one node cannot
see a specific mirror copy, then CVM will deactivate access to the volume
for that node only.
This policy can be re-set on a disk group basis by using the vxedit
command, as follows:
# vxedit set diskdetpolicy=global <DiskGroupName>
76 Chapter 2
NOTE The specific commands for creating mirrored and multi-path storage
using CVM are described in the HP-UX documentation for the VERITAS
Volume Manager.
Using CVM 3.x

system multi-node package with CVM 3.x.
Preparing the Cluster for Use with CVM 3.x

In order to use the VERITAS Cluster Volume Manager (CVM) version
3.x, the cluster must be running with a special CVM package. This
means that the cluster must already be configured and running before
you create disk groups.
NOTE Cluster configuration is described in the previous section.
To prepare the cluster for CVM disk group configuration, you need to
ensure that only one heartbeat subnet is configured. Then use the
following command, which creates the special package that
communicates cluster information to CVM:
# cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf
WARNING This file should never be edited.
After the above command completes, start the cluster and create disk
groups for shared use as described in the following sections.
Starting the Cluster and Identifying the Master Node Run the
cluster, which will activate the special CVM package:
# cmruncl
Chapter 2 77
After the cluster is started, it will now run with a special system
multi-node package named VxVM-CVM-pkg, which is on all nodes. This
package is shown in the following output of the cmviewcl -v command:
CLUSTER STATUS
bowls up
NODE STATUS STATE

spare up running
split up running
strike up running
SYSTEM_MULTI_NODE_PACKAGES:
PACKAGE STATUS STATE

VxVM-CVM-pkg up running
When CVM starts up, it selects a master node, and this is the node from
which you must issue the disk group configuration commands. To
determine the master node, issue the following command from each node
in the cluster:
# vxdctl -c mode
One node will identify itself as the master. Create disk groups from this
node.
Converting Disks from LVM to CVM You can use the vxvmconvert
utility to convert LVM volume groups into CVM disk groups. Before you
can do this, the volume group must be deactivated, which means that
any package that uses the volume group must be halted. This procedure
is described in the Managing Serviceguard Thirteenth Edition user’s
guide Appendix G.
Initializing Disks for CVM You need to initialize the physical disks
that will be employed in CVM disk groups. If a physical disk has been
previously used with LVM, you should use the pvremove command to
delete the LVM header data from all the disks in the volume group (this
is not necessary if you have not previously used the disk with LVM).
vxdiskadm program to initialize multiple disks, or use the vxdisksetup
command to initialize one disk at a time, as in the following example:
# /usr/lib/vxvm/bin/vxdisksetup -i /dev/dsk/c0t3d2
78 Chapter 2
Creating Disk Groups for RAC Use the vxdg command to create disk
groups. Use the -s option to specify shared mode, as in the following
example:
Verify the configuration with the following command:
# vxdg list
NAME STATE ID
rootdg enabled 971995699.1025.node1

ops_dg enabled,shared 972078742.1084.node2
Creating Volumes
Use the vxassist command to create logical volumes. The following is
an example:
# vxassist -g ops_dg make log_files 1024m
This command creates a 1024 MB volume named log_files in a disk
group named ops_dg. The volume can be referenced with the block device
file /dev/vx/dsk/ops_dg/log_files or the raw (character) device file
/dev/vx/rdsk/ops_dg/log_files.
# vxdg list
Chapter 2 79
Mirror Detachment Policies with CVM

The required CVM disk mirror detachment policy is ‘global’, which
means that as soon as one node cannot see a specific mirror copy (plex),
all nodes cannot see it as well. The alternate policy is ‘local’, which
means that if one node cannot see a specific mirror copy, then CVM will
deactivate access to the volume for that node only.
Volume Manager.

The following set of volumes is required for the Oracle demo database
which you can create during the installation process.
Oracle
Size File Size
Volume Name (MB) Raw Device File Name (MB)
opsctl1.ctl 118 /dev/vx/rdsk/ops_dg/opsctl1.ctl 110
ops1log1.log 128 /dev/vx/rdsk/ops_dg/ops1log1.log 120
80 Chapter 2
Oracle
Size File Size
opssystem.dbf 508 /dev/vx/rdsk/ops_dg/opssystem.dbf 500
opssysaux.dbf 808 /dev/vx/rdsk/ops_dg/opssysaux.dbf 800
opstemp.dbf 258 /dev/vx/rdsk/ops_dg/opstemp.dbf 250
opsusers.dbf 128 /dev/vx/rdsk/ops_dg/opsusers.dbf 120
opsdata1.dbf 208 /dev/vx/rdsk/ops_dg/opsdata1.dbf 200
opsspfile1.ora 508 /dev/vx/rdsk/ops_dg/opsspfile1.ora 500
opspwdfile.ora 508 /dev/vx/rdsk/ops_dg/opspwdfile.ora 500
opsundotbs1.dbf 508 /dev/vx/rdsk/ops_dg/opsundotbs1.dbf 500
opsexmple1.dbf 168 /dev/vx/rdsk/ops_dg/opsexample1.dbf 160
appropriate directory.
Example:
# ln -s /dev/vx/rdsk/ops_dg/opsctl1.ctl \
Example:
Chapter 2 81
1. Create an ASCII file, and define the path for each database object.
control1=/u01/ORACLE/db001/ctrl01_1.ctl
2. Set the following environment variable where filename is the name
of the ASCII file created.
# export DBCA_RAW_CONFIG=<full path>/filename
Adding Disk Groups to the Cluster Configuration

For CVM 4.x, if the multi-node package was configured for disk group
activation, the application package should be configured with package
dependency to ensure the CVM disk group is active.
For CVM 3.5 and CVM 4.x (without using multi-node package) after
creating units of CVM storage with VxVM commands, you need to
specify the disk groups in each package configuration ASCII file. Use one
STORAGE_GROUP parameter for each CVM disk group the package will
use. You also need to identify the CVM disk groups, file systems, logical
volumes, and mount options in the package control script.
For more detailed information on the package configuration process refer
to the Managing Serviceguard Thirteenth Edition user’s guide.
82 Chapter 2
Prerequisites for Oracle 10g (Sample Installation)
Prerequisites for Oracle 10g (Sample

Installation)
The following sample steps prepare a SGeRAC cluster for Oracle 10g.
Refer to the Oracle documentation for Oracle installation details.
1. Create Inventory Groups on each Node

Create the Oracle Inventory group if one does not exist, create the
OSDBA group, and create the Operator Group (optional).
# /usr/sbin/groupadd oinstall
# /usr/sbin/groupadd dba
# /usr/sbin/groupadd oper
2. Create Oracle User on each Node
# /usr/bin/useradd -u 203 -g oinstall -G dba,oper oracle
3. Change the Password on each Node
# passwd oracle
4. Create Symbolic Links
Required if Motif 2.1 Development Environment Package is not
installed.
# ln -s /usr/lib/libX11.3 /usr/lib/libX11.sl
# ln -s /usr/lib/libXIE.2 /usr/lib/libXIE.sl
# ln -s /usr/lib/libXext.3 /usr/lib/libXext.sl
# ln -s /usr/lib/libXhp11.3 /usr/lib/Xhp11.sl
# ln -s /usr/lib/libXi.3 /usr/lib/libXi.sl
# ln -s /usr/lib/libXm.4 /usr/lib/libXm.sl
# ln -s /usr/lib/libXp.2 /usr/lib/libXp.sl
# ln -s /usr/lib/libXt.3 /usr/lib/libXt.sl
# ln -s /usr/lib/libXtst.2 /usr/lib/libXtst.sl
Chapter 2 83
5. Enable Remote Access (ssh and remsh) for Oracle

User on all Nodes
6. Create File System for Oracle Directories
In the following samples, /mnt/app is a mounted file system for
Oracle software. Assume there is a private disk c4t5d0 at 18 GB size
on all nodes. Create the local file system on each node.
# umask 022
# pvcreate /dev/rdsk/c4t5d0
# mkdir /dev/vg01
# mknod /dev/vg01/group c 64 0x010000
# vgcreate /dev/vg01 /dev/dsk/c4t5d0
# lvcreate -L 16000 /dev/vg01
# newfs -F vxfs /dev/vg01/rlvol1
# mkdir -p /mnt/app
# mount /dev/vg01/lvol1 /mnt/app
# chmod 775 /mnt/app
7. Create Oracle Cluster Software Home Directory
For installing Oracle Cluster Software on local file system, create the
directories on each node.
# mkdir -p /mnt/app/crs/oracle/product/10.2.0/crs
# chown -R oracle:oinstall \
/mnt/app/crs/oracle/product/10.2.0/crs
# chmod -R 775 /mnt/app/crs/oracle/product/10.2.0/crs
8. Create Oracle Base Directory (for RAC Binaries on
Local File System)
If installing RAC binaries on local file system, create the oracle base
directory on each node.
# mkdir -p /mnt/app/oracle
# chown -R oracle:oinstall /mnt/app/oracle
# chmod -R 775 /mnt/app/oracle
84 Chapter 2
# usermod -d /mnt/app/oracle oracle

9. Create Oracle Base Directory (For RAC Binaries on
Cluster File System)
If installing RAC binaries on Cluster File System, create the oracle
base directory once since this is CFS directory visible by all nodes.
The CFS file system used is /cfs/mnt1.
# mkdir -p /cfs/mnt1/oracle
# chown -R oracle:oinstall /cfs/mnt1/oracle
# chmod -R 775 /cfs/mnt1/oracle
# chmod 775 /cfs
# chmod 775 /cfs/mnt1
Modify oracle user to use new home directory on each node.
# usermod -d /cfs/mnt1/oracle oracle
10. Prepare Shared Storage on SLVM
This section assumes the OCR, Vote device, and database files are
created on SLVM volume group vg_ops.
a. Change Permission of Shared Logical Volume Group

This section assumes the OCR, Vote device, and database files
are created on SLVM volume group vg_ops.
# chmod 755 /dev/vg_ops
b. Change Permission and Ownership of Oracle Cluster
Software Vote Device and Database Files
# chown oracle:oinstall /dev/vg_ops/r*
# chmod 660 /dev/vg_ops/r*
c. Change Permission of OCR Device
# chown root:oinstall /dev/vg_ops/rora_ocr
# chmod 640 /dev/vg_ops/rora_ocr
d. Create Raw Device Mapping File for Oracle Database
Configuration Assistant
In this example, the database name is “ver10”
Chapter 2 85
# ORACLE_BASE=/mnt/app/oracle; export ORACLE_BASE

# mkdir -p $ORACLE_BASE/oradata/ver10
# chown -R oracle:oinstall $ORACLE_BASE/oradata
# chmod -R 755 $ORACLE_BASE/oradata
The following is a sample of the mapping file for DBCA:
system=/dev/vg_ops/ropssystem.dbf
sysaux=/dev/vg_ops/ropssysaux.dbf
undotbs1=/dev/vg_ops/ropsundotbs01.dbf
undotbs2=/dev/vg_ops/ropsundotbs02.dbf
example=/dev/vg_ops/ropsexample1.dbf
users=/dev/vg_ops/ropsusers.dbf
redo1_1=/dev/vg_ops/rops1log1.log
control1=/dev/vg_ops/ropsctl1.ctl
temp=/dev/vg_ops/ropstmp.dbf
spfile=/dev/vg_ops/ropsspfile1.ora
In this sample, create the DBCA mapping file and place at:
/mnt/app/oracle/oradata/ver10/ver10_raw.conf.
11. Prepare Shared Storage on CFS
This section assumes the OCR, Vote device, and database files are
created on CFS directories. The OCR and vote device reside on
/cfs/mnt3 and the demo database files reside on /cfs/mnt2.
a. Create OCR and Vote Device Directories on CFS

Create OCR and vote device directories on Cluster File System.
Run commands only on one node.
# chmod 775 /cfs
# cd /cfs/mnt3
# mkdir OCR
# chmod 755 OCR
# mkdir VOTE
86 Chapter 2
# chmod 755 VOTE

# chown -R oracle:oinstall /cfs/mnt3
b. Create Directory for Oracle Demo Database on CFS
Create the CFS directory to store Oracle database files. Run
commands only on one node.
# chmod 775 /cfs
# cd /cfs/mnt2
# mkdir oradata
# chown oracle:oinstall oradata
# chmod 775 oradata
Chapter 2 87
Installing Oracle 10g Cluster Software
Installing Oracle 10g Cluster Software

The following sample steps for a SGeRAC cluster for Oracle 10g. Refer to
the Oracle documentation for Oracle installation details.
Installing on Local File System

Logon as a “oracle” user:
$ export DISPLAY={display}:0.0
$ cd <10g Cluster Software disk directory>
$ ./runInstaller
Use following guidelines when installing on a local file system:
1. Specify CRS HOME as

/mnt/app/crs/oracle/product/10.2.0/crs. This is a local file
system.
2. Specify OCR Location as /dev/vg_ops/rora_ocr if using SLVM for
OCR or as /cfs/mnt3/OCR/ocr_file if using CFS for OCR.
3. Specify Vote Disk Location as /dev/vg_ops/rora_vote if using
SLVM vote device or as /cfs/mnt3/VOTE/vote_file if using CFS
for vote device.
4. When prompted, run orainstRoot.sh on each node.
5. When prompted, run root.sh on each node.
88 Chapter 2
Installing Oracle 10g RAC Binaries
Installing Oracle 10g RAC Binaries

The following sample steps for a SGeRAC cluster for Oracle 10g. Refer to
Installing RAC Binaries on a Local File System

$ export ORACLE BASE=/mnt/app/oracle
$ cd <10g RAC installation disk directory>
$ ./runInstaller
1. In this example, the path to ORACLE_HOME is on a local file

system /mnt/app/oracle/product/10.2.0/db_1.
2. Select installation for database software only.
Installing RAC Binaries on Cluster File System

$ export ORACLE BASE=/cfs/mnt1/oracle
$ cd <10g RAC installation disk directory>
$ ./runInstaller
1. In this example, the path to ORACLE_HOME is locate on a CFS

directory /cfs/mnt1/oracle/product/10.2.0/db_1.
2. Select installation for database software only.
Chapter 2 89
Creating a RAC Demo Database

This section demonstrates the steps for creating a demo database with
datafiles on raw volumes with SLVM or CVM, or with Cluster File
System.
Creating a RAC Demo Database on SLVM or CVM

Export environment variables for “oracle” user:
export ORACLE_BASE=/mnt/app/oracle
export \
DBCA_RAW_CONFIG=/mnt/app/oracle/oradata/ver10/ver10_raw.conf
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=/mnt/app/crs/oracle/product/10.2.0/crs
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:$ORACLE_HOME/rd
bms/lib
SHLIB_PATH=$ORACLE_HOME/lib32:$ORACLE_HOME/rdbms/lib32
export LD_LIBRARY_PATH SHLIB_PATH
export \
PATH=$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:/usr/local/bin:
CLASSPATH=$ORACLE_HOME/jre:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbm
s/jlib:$ORACLE_HOME/network/jlib
export CLASSPATH
export DISPLAY={display}:0.0
1. Setting up Listeners with Oracle Network Configuration

Assistant
Use the Oracle network configuration assistant to configure the
listeners with the following command:
$ netca
2. Create Database with Database Configuration Assistant
Use the Oracle database configuration assistant to create the
database with the following command:
$ dbca
90 Chapter 2
a. In this sample, the database name and SID prefix are ver10.
b. Select the storage option for raw devices.
Creating a RAC Demo Database on CFS

Export environment variables for “oracle” user:
export ORACLE_BASE=/cfs/mnt1/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=/mnt/app/crs/oracle/product/10.2.0/crs
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:$ORACLE_HOME/rd
bms/lib
export \
PATH=$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:/usr/local/bin:
CLASSPATH=$ORACLE_HOME/jre:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbm
s/jlib:$ORACLE_HOME/network/jlib
export CLASSPATH
1. Setting up Listeners with Oracle Network Configuration

Assistant
Use the Oracle network configuration assistant to configure the
listeners with the following command:
$ netca
2. Create Database with Database Configuration Assistant
Use the Oracle database configuration assistant to create the
database with the following command:
$ dbca
a. In this sample, the database name and SID prefix are ver10.
b. Select the storage option for Cluster File System.
c. Enter /cfs/mnt2/oradata as the common directory.
Chapter 2 91
Verify that Oracle Disk Manager is Configured
NOTE The following steps are specific to CFS.
1. Check the license

# /opt/VRTS/bin/vxlictest -n “VERITAS Storage Foundation
for Oracle” -f “ODM”
output: Using VERITAS License Manager API Version 3.00,
Build 2 ODM feature is licensed
2. Check that the VRTSodm package is installed:
# swlist VRTSodm
output:
VRTSodm 4.1m VERITAS Oracle Disk Manager
VRTSodm.ODM-KRN 4.1m VERITAS ODM kernel files
VRTSodm.ODM-MAN 4.1m VERITAS ODM manual pages
VRTSodm.ODM-RUN 4.1m VERITAS ODM commands
3. Check that libodm.sl is present
# ll -L /opt/VRTSodm/lib/libodm.sl
output: -rw-r--r-- 1 root sys 14336 Apr 25 18:42
/opt/VRTSodm/lib/libodm.sl
92 Chapter 2
Configuring Oracle to Use Oracle Disk Manager Library
Configuring Oracle to Use Oracle Disk

Manager Library
1. Login as Oracle user

2. Shutdown database
3. Link the Oracle Disk Manager library into Oracle home for Oracle
10g
For HP 9000 Systems:
$ rm ${ORACLE_HOME}/lib/libodm10.sl
$ ln -s /opt/VRTSodm/lib/libodm.sl \
${ORACLE_HOME}/lib/libodm10.sl
For Integrity Systems:
$ rm ${ORACLE_HOME}/lib/libodm10.so
${ORACLE_HOME}/lib/libodm10.so
4. Start Oracle database
Chapter 2 93
Verify that Oracle Disk Manager is Running
1. Start the cluster and Oracle database (if not already started)
2. Check that the Oracle instance is using the Oracle Disk Manager
function:
# cat /dev/odm/stats
abort: 0
cancel: 0
commit: 18
create: 18
delete: 0
identify: 349
io: 12350590
reidentify: 78
resize: 0
unidentify: 203
mname: 0
vxctl: 0
vxvers: 10
io req: 9102431
io calls: 6911030
comp req: 73480659
comp calls: 5439560
io mor cmp: 461063
io zro cmp: 2330
cl receive: 66145
cl ident: 18
cl reserve: 8
cl delete: 1
cl resize: 0
cl same op: 0
cl opt idn: 0
cl opt rsv: 332
**********: 17
3. Verify that the Oracle Disk Manager is loaded:
94 Chapter 2
# kcmodule -P state odm

Output: state loaded
4. In the alert log, verify the Oracle instance is running. The log should
contain output similar to the following:
Oracle instance running with ODM: VERITAS 4.1 ODM
Library, Version 1.1
Chapter 2 95
Configuring Oracle to Stop Using Oracle Disk Manager Library
Configuring Oracle to Stop Using Oracle Disk

Manager Library

3. Change directories:
$ cd ${ORACLE_HOME}/lib
4. Remove the file linked to the ODM library
For HP 9000 Systems:
$ rm libodm10.sl
$ ln -s ${ORACLE_HOME}/lib/libodmd10.sl \
For Integrity Systems:
$ rm libodm10.so
$ ln -s ${ORACLE_HOME}/lib/libodmd10.so \
5. Restart the database
96 Chapter 2
Using Serviceguard Packages to Synchronize with Oracle 10g RAC
Using Serviceguard Packages to Synchronize

with Oracle 10g RAC
It is recommended to start and stop Oracle Cluster Software in a
Serviceguard package, as that will ensure that Oracle Cluster Software
will start after SGeRAC is started and will stop before SGeRAC is
halted. Serviceguard packages should also be used to synchronize
storage activation and deactivation with Oracle Cluster Software and
RAC instances.
Preparing Oracle Cluster Software for Serviceguard

Packages
• Stopping the Oracle Cluster Software on each Node
For 10g 10.1.0.04 or later:
# /sbin/init.d/init.crs stop
# <CRS HOME>/bin/crsctl stop crs
Wait until Oracle Cluster Software completely stops. (Check CRS
logs or check for Oracle processes, ps -ef | grep ocssd.bin)
• Change Oracle Cluster Software from Starting at Boot Time
on each Node
# /sbin/init.d/init.crs disable
# <CRS HOME>/bin/crsctl disable crs
Configure Serviceguard Packages

A Serviceguard package is needed for each node to start and stop Oracle
Cluster Software.
• Storage Activation (SLVM)
Chapter 2 97
When the Oracle Cluster Software required storage is configured on

SLVM volume groups or CVM disk groups, the Serviceguard package
should be configured to activate and deactivate the required storage
in the package control script.
As an example, modify the control script to activate the volume group
in shared mode and set VG in the package control script for SLVM
volume groups.
VG[0]=”vg_ops”
• Storage Activation (CVM)
CVM disk groups, the Serviceguard package should be configured to
activate and deactivate the required storage in the package
configuration file and control script.
In the package configuration file, specify the disk group with the
STORAGE_GROUP key word. In the package controls script, specify the
disk group with the CVM_DG variable.
As an example, in the package configuration file should have the
following:
STORAGE_GROUP ops_dg
Modify the package control script to set the CVM disk group to
“activate” for shared write and to specify the disk group.
CVM_DG[0]=”ops_dg”
• Storage Activation (CFS)
a Cluster File System (CFS), the Serviceguard package should be
configured to depend on the CFS multi-node package through
package dependency. With package dependency, the Serviceguard
package that starts Oracle Cluster Software will not run until its
dependent CFS multi-node package is up and will halt before the
CFS multi-node package is halted.
Setup dependency conditions in Serviceguard package configuration
file (SAMPLE).
DEPENDENCY_NAME mp1
DEPENDENCY_CONDITION SG-CFS-MP-1=UP
DEPENDENCY_LOCATION SAME_NODE
98 Chapter 2
DEPENDENCY_NAME mp2
DEPENDENCY_NAME mp3
• Starting and Stopping Oracle Cluster Software
In the Serviceguard package control script, configure the Oracle
Cluster Software start in the customer_defined_run_cmds function
/sbin/init.d/init.crs start
<CRS HOME>/bin/crsctl start crs
In the Serviceguard package control script, configure the Oracle
Cluster Software stop in the customer_defined_halt_cmds function.
/sbin/init.d/init.crs stop
<CRS HOME>/bin/crsctl stop crs
When stopping Oracle Cluster Software in a Serviceguard package, it
may be necessary to verify that the Oracle processes have stopped
and exited before deactivating storage or halting CFS multi-node
package. The verification can be done with a script that loops and
checks for the successful stop message in the Oracle Cluster Software
logs or the existences of Oracle processes that needed to be stopped,
specifically the CSS daemon (ocssd.bin). For example, this script
could be called by the Serviceguard package control script after the
command to halt Oracle Cluster Software and before storage
deactivation.
Chapter 2 99
100 Chapter 2
Serviceguard Configuration for Oracle 9i RAC
3 Serviceguard Configuration for

Oracle 9i RAC
This chapter shows the additional planning and configuration that is

needed to use Oracle Real Application Clusters 9i with Serviceguard.
• Planning Database Storage

• Installing Serviceguard Extension for RAC
• Creating a Storage Infrastructure with LVM
• Installing Oracle Real Application Clusters
• Cluster Configuration ASCII File
• Creating a Storage Infrastructure with CFS
• Creating a Storage Infrastructure with CVM
• Configuring Packages that Access the Oracle RAC Database
• Using Packages to Configure Startup and Shutdown of RAC
Instances
Chapter 3 101
Planning Database Storage

The files needed by the Oracle database must be placed on shared
storage that is accessible to all RAC cluster nodes. This section shows
how to plan the storage using SLVM, VERITAS CFS, or VERITAS CVM.
Volume Planning with SLVM

logical volumes located in shared volume groups. The Oracle software
requires at least two log files (and one undo tablespace for Oracle9) for
each Oracle instance, several Oracle control files and data files for the
database itself. For all these files, Serviceguard Extension for RAC uses
HP-UX raw logical volumes, which are located in volume groups that are
shared between the nodes in the cluster. High availability is achieved by
using high availability disk arrays in RAID modes. The logical units of
storage on the arrays are accessed from each node through multiple
physical volume links (PV links, also known as alternate links), which
provide redundant paths to each unit of storage.
Fill out a Logical Volume worksheet to provide logical volume names for
logical volumes that you will create with the lvcreate command. The
Oracle DBA and the HP-UX system administrator should prepare this
logical volume, enter the full pathname of the raw logical volume device
file. Be sure to include the desired size in MB. Following is a sample
worksheet filled out. However, this sample is only representative. For
different versions of the Oracle database, the size of files are different.
Refer to Appendix B, “Blank Planning Worksheets,” for samples of blank
worksheets. Make as many copies as you need. Fill out the worksheet
and keep it for future reference.
102 Chapter 3
ORACLE LOGICAL VOLUME WORKSHEET FOR LVM Page ___ of ____

===============================================================================
Oracle Control File _____/dev/vg_ops/ropsctl1.ctl_______108______

Instance 1 Redo Log 1: ___/dev/vg_ops/rops1log1.log_____20______
Instance 2 Redo Log 3: ___/dev/vg_ops/rops2log3.log____20_________
Data: System ___/dev/vg_ops/ropssystem.dbf___400__________
Data: Temp ___/dev/vg_ops/ropstemp.dbf______100_______
Data: Users ___/dev/vg_ops/ropsusers.dbf_____120_________
Data: Tools ___/dev/vg_ops/ropstools.dbf____15___________
Data: User data ___/dev/vg_ops/ropsdata1.dbf_200__________
Data: Rollback ___/dev/vg_ops/ropsrollback.dbf__300_________
Parameter: spfile1 /dev/vg_ops/ropsspfile1.ora __5_____
Data: example1__/dev/vg_ops/ropsexample1.dbf__________160____
data: cwmlite1__/dev/vg_ops/ropscwmlite1.dbf__100____
Data: indx1__/dev/vg_ops/ropsindx1.dbf____70___
Data: drsys1__/dev/vg_ops/ropsdrsys1.dbf___90___
Chapter 3 103
Storage Planning with CFS

With CFS, the database software and database files (control, redo, data
files), and archive logs may reside on a cluster file system, which is
visible by all nodes.
The following software needs to be installed in order to use this
configuration:
• SGeRAC
• CFS
NOTE CFS and SGeRAC is available in the following HP Serviceguard Storage

Management Suite bundle:
• HP Serviceguard Cluster File System for RAC (SGCFSRAC)

(T2777BA).
For specific bundle information, refer to the HP Serviceguard Storage
Management Suite Version A.01.00 Release Notes.
Configuration Combinations with Oracle 9i RAC

The following configuration combinations show a sample of the
configuration choices (See Table 3-1):
• Configuration 1 is for Oracle software and database files to reside on

CFS.
• Configuration 2 is for Oracle software and archives to reside on CFS,
while the database files are on raw volumes either SLVM or CVM.
• Configuration 3 is for Oracle software and archives to reside on local
file system, while database files reside on CFS.
104 Chapter 3
• Configuration 4 is for Oracle software and archive to reside on local

file system, while the database files are on raw volumes either SLVM
or CVM.
Table 3-1 RAC Software, Archive, Datafiles, SRVM
RAC Software, RAC Datafiles,

Configuration
Archive SRVM
1 CFS CFS
2 CFS Raw (SLVM or CVM)
3 Local FS CFS
4 Local FS Raw (SLVM or CVM)
NOTE Mixing files between CFS database files and raw volumes is allowable,
but not recommended. RAC datafiles on CFS requires Oracle Disk
Manager (ODM).
Using Single CFS Home or Local Home

With a single CFS home, the software installs once and all the files are
visible on all nodes. Serviceguard cluster services needs to be up for the
CFS home to be accessible. For Oracle RAC, some files are still local
(/etc/oratab, /var/opt/oracle/, /usr/local/bin/).
With a local file system home, the software is installed and copied to
every node’s local file system. Serviceguard cluster services does not
need to be up for the local file system to be accessible. There would be
multiple homes for Oracle.
Considerations on using CFS for RAC datafiles and Server

Management Storage (SRVM)
Use the following list when considering to use CFS for database storage:
• Single file system view
— Simpler setup for archive recovery since archive area is visible by

all nodes
Chapter 3 105
• Oracle create database files

• Online changes (OMF - Oracle Managed Files) within CFS
• Better manageability
• Manual intervention when modifying volumes, DGs, disks
• Requires the SGeRAC and CFS software
CFS and SGeRAC is available in selected HP Serviceguard Storage
Management Suite bundles.
Refer to the HP Serviceguard Storage Management Suite Version
A.01.00 Release Notes.
Considerations on using Raw Volumes for RAC datafiles and

Server Management storage (SRVM)
Use the following list when considering to use Raw Volumes for database
storage:
• SLVM is part of SGeRAC.

• Create volumes for each datafile.
• CVM 4.x
— Disk group activation performed by disk group multi-node

package.
— Disk group activation performed by application package (without
the HP Serviceguard Storage Management Suite bundle-see
bundle information above).
• CVM 3.x
— Disk group activation is performed by application package.
Volume Planning with CVM

volumes located in shared disk groups. The Oracle software requires at
least two log files (and one undo tablespace for Oracle9) for each Oracle
instance, several Oracle control files and data files for the database itself.
For all these files, Serviceguard Extension for RAC uses HP-UX raw
volumes, which are located in disk groups that are shared between the
nodes in the cluster. High availability is achieved by using high
106 Chapter 3
availability disk arrays in RAID nodes. The logical units of storage on

the arrays are accessed from each node through multiple physical
volume links via DMP (Dynamic Multi-pathing), which provides
redundant paths to each unit of storage.
Fill out the VERITAS Volume worksheet to provide volume names for
volumes that you will create using the VERITAS utilities. The Oracle
DBA and the HP-UX system administrator should prepare this
volume, enter the full pathname of the raw volume device file. Be sure to
include the desired size in MB. Following is a sample worksheet filled
out. Refer to Appendix B, “Blank Planning Worksheets,” for samples of
blank worksheets. Make as many copies as you need. Fill out the
worksheet and keep it for future reference.
Chapter 3 107
ORACLE LOGICAL VOLUME WORKSHEET FOR CVM Page ___ of ____

===============================================================================

Instance 1 Redo Log: ___________________________________________________
Instance 1 Redo Log: ___________________________________________________
Instance 2 Redo Log 3: ___/dev/vx/rdsk/ops_dg/ops2log3.log____20_________
Data: System ___/dev/vx/rdsk/ops_dg/system.dbf___400__________
Data: Temp ___/dev/vx/rdsk/ops_dg/temp.dbf______100_______
Data: Users ___/dev/vx/rdsk/ops_dg/users.dbf_____120_________
Data: Tools ___/dev/vx/rdsk/ops_dg/tools.dbf____15___________
Data: User data ___/dev/vx/rdsk/ops_dg/data1.dbf_200__________
Data: User data ___/dev/vx/rdsk/ops_dg/data2.dbf__200__________
Data: User data ___/dev/vx/rdsk/ops_dg/data3.dbf__200__________
Data: Rollback ___/dev/vx/rdsk/ops_dg/rollback.dbf__300_________
Parameter: spfile1 /dev/vx/rdsk/ops_dg/spfile1.ora __5_____
Instance 1 undotbs1: /dev/vx/rdsk/ops_dg/undotbs1.dbf___312___
Instance 2 undotbs2: /dev/vx/rdsk/ops_dg/undotbs2.dbf___312___
Data: example1__/dev/vx/rdsk/ops_dg/example1.dbf__________160____
data: cwmlite1__/dev/vx/rdsk/ops_dg/cwmlite1.dbf__100____
Data: indx1__/dev/vx/rdsk/ops_dg/indx1.dbf____70___
Data: drsys1__/dev/vx/rdsk/ops_dg/drsys1.dbf___90___
108 Chapter 3

Installing Serviceguard Extension for RAC includes updating the
software and rebuilding the kernel to support high availability cluster
operation for Oracle Real Application Clusters.
Prior to installing Serviceguard Extension for RAC, the following must
be installed:
• Correct version of HP-UX

• Correct version of Serviceguard
NOTE For the most current version compatibility for Serviceguard and HP-UX,
see the SGeRAC release notes.
To install Serviceguard Extension for RAC, use the following steps for
each node:
1. Mount the distribution media in the tape drive, CD, or DVD reader.
2. Run Software Distributor, using the swinstall command.
3. Specify the correct input device.
4. Choose the following bundle from the displayed list:
5. After choosing the bundle, select OK. The software is loaded.
Chapter 3 109

You need to code specific entries for all the storage groups that you want
to use in an Oracle RAC configuration. If you are using LVM, the
OPS_VOLUME_GROUP parameter is included in the cluster ASCII file. If
you are using VERITAS CVM, the STORAGE_GROUP parameter is included
in the package ASCII file. Details are as follows:
OPS_VOLUME_GROUP
The name of an LVM volume group whose disks are
attached to at least two nodes in the cluster; the disks
will be accessed by more than one node at a time using
SLVM with concurrency control provided by Oracle
RAC. Such disks are considered cluster aware.
Volume groups listed under this parameter are marked
for activation in shared mode. The entry can contain up
to 40 characters.
STORAGE_GROUP
This parameter is used for CVM disk groups. Enter the
names of all the CVM disk groups the package will use.
In the ASCII package configuration file, this parameter
is called STORAGE_GROUP.
NOTE CVM 4.x with CFS does not use the STORAGE_GROUP parameter because
the disk group activation is performed by the multi-node package. CVM
3.x or 4.x without CFS uses the STORAGE_GROUP parameter in the ASCII
package configuration file in order to activate the disk group.
Do not enter the names of LVM volume groups or VxVM disk groups in
the package ASCII configuration file.
110 Chapter 3
Operating System Parameters
Operating System Parameters

The maximum number of Oracle server processes cmgmsd can handle is
8192. When there are more than 8192 server processes connected to
cmgmsd, then cmgmsd will start to reject new requests.
Oracle foreground server processes are needed to handle the requests of
the DB client connected to the DB instance.
Each foreground server process can either be a “dedicated” or a “shared”
server process. In the case of a dedicated process, there is a one-to-one
correspondence between the DB client and the foreground server process
it invokes. The shared server processes can handle multiple DB clients.
In the case where a DB instance is configured to support a large number
of DB clients, it is necessary to adjust the maxfiles parameter. This to
make sure there are enough file descriptors to support the necessary
number of Oracle foreground and background server processes.
Chapter 3 111

This is done with Logical Volume Manager (LVM), VERITAS Cluster
Volume Manager (CVM), or VERITAS Volume Manager (VxVM). LVM
This section describes how to create LVM volume groups for use with
Oracle data. Before configuring the cluster, you create the appropriate
logical volume infrastructure to provide access to data from different
nodes. This is done with Logical Volume Manager. Separate procedures
are given for the following:
• Building Volume Groups for RAC on Mirrored Disks

• Building Mirrored Logical Volumes for RAC with LVM Commands
• Creating RAC Volume Groups on Disk Arrays
• Creating Logical Volumes for RAC on Disk Arrays
The Event Monitoring Service HA Disk Monitor provides the capability
to monitor the health of LVM disks. If you intend to use this monitor for
your mirrored disks, you should configure them in physical volume
groups. For more information, refer to the manual Using HA Monitors.
Building Volume Groups for RAC on Mirrored Disks

The procedure described in this section uses physical volume groups
for mirroring of individual disks to ensure that each logical volume is
mirrored to a disk on a different I/O bus. This kind of arrangement is
known as PVG-strict mirroring. It is assumed that your disk
hardware is already configured in such a way that a disk to be used as a
mirror copy is connected to each node on a different bus than the bus
that is used for the other (primary) copy.
For more information on using LVM, refer to the HP-UX Managing
Systems and Workgroups manual.
112 Chapter 3
Creating Volume Groups and Logical Volumes

If your volume groups have not been set up, use the procedure in the
next sections. If you have already done LVM configuration, skip ahead to
the section “Installing Oracle Real Application Clusters.”
Selecting Disks for the Volume Group Obtain a list of the disks on
both nodes and identify which device files are used for the same disk on
both. Use the following command on each node to list available disks as
they are known to each system:
# lssf /dev/dsk/*
In the following examples, we use /dev/rdsk/c1t2d0 and
/dev/rdsk/c0t2d0, which happen to be the device names for the same
disks on both ftsys9 and ftsys10. In the event that the device file
names are different on the different nodes, make a careful note of the
correspondences.
Creating Physical Volumes On the configuration node (ftsys9), use

the pvcreate command to define disks as physical volumes. This only
needs to be done on the configuration node. Use the following commands
to create two physical volumes for the sample configuration:
Creating a Volume Group with PVG-Strict Mirroring Use the

following steps to build a volume group on the configuration node
(ftsys9). Later, the same volume group will be created on other nodes.
1. First, set up the group directory for vgops:

# mkdir /dev/vg_ops
has the form
0xhh0000
Chapter 3 113
3. Create the volume group and add physical volumes to it with the
following commands:
# vgcreate -g bus0 /dev/vg_ops /dev/dsk/c1t2d0
# vgextend -g bus1 /dev/vg_ops /dev/dsk/c0t2d0
The first command creates the volume group and adds a physical
volume to it in a physical volume group called bus0. The second
command adds the second drive to the volume group, locating it in a
different physical volume group named bus1. The use of physical
volume groups allows the use of PVG-strict mirroring of disks and
PV links.
4. Repeat this procedure for additional volume groups.
Building Mirrored Logical Volumes for RAC with LVM

Commands
After you create volume groups and define physical volumes for use in
them, you define mirrored logical volumes for data, logs, and control
files. It is recommended that you use a shell script to issue the
commands described in the next sections. The commands you use for
creating logical volumes vary slightly depending on whether you are
creating logical volumes for RAC redo log files or for use with Oracle
data.
Creating Mirrored Logical Volumes for RAC Redo Logs and

Control Files
Create logical volumes for use as redo log and control files by selecting
mirror consistency recovery. Use the same options as in the following
example:
# lvcreate -m 1 -M n -c y -s g -n redo1.log -L 28 /dev/vg_ops
114 Chapter 3

the -n redo1.log option lets you specify the name of the logical volume;
and the -L 28 option allocates 28 megabytes.
NOTE It is important to use the -M n and -c y options for both redo logs and
control files. These options allow the redo log files to be resynchronized
by SLVM following a system crash before Oracle recovery proceeds. If
these options are not set correctly, you may not be able to continue with
database recovery.
following:
created
with character device “/dev/vg_ops/rredo1.log”
extended
volume name) is used by the Oracle DBA in building the RAC database.
Creating Mirrored Logical Volumes for RAC Data Files

Following a system crash, the mirrored logical volumes need to be
resynchronized, which is known as “resilvering”.
If Oracle does not perform “resilvering” of RAC data files that are
mirrored logical volumes, choose a mirror consistency policy of
“NOMWC”. This is done by disabling mirror write caching and enabling
mirror consistency recovery. With “NOMWC”, SLVM performs the
resynchronization.
# lvcreate -m 1 -M n -c y -s g -n system.dbf -L 408 /dev/vg_ops
Chapter 3 115
If Oracle performs the resilvering of RAC data files that are mirrored
logical volumes, choose a mirror consistency policy of “NONE” by
disabling both mirror write caching and mirror consistency recovery.
With a mirror consistency policy of “NONE”, SLVM does not
perform the resynchronization.
NOTE Contact Oracle to determine if your version of Oracle RAC allows

“resilvering” and to appropriately configure the mirror consistency
recovery policy for your logical volumes.
# lvcreate -m 1 -M n -c n -s g -n system.dbf -L 408 /dev/vg_ops
mirror write cache recovery is set off; the -c n means that mirror
consistency recovery is disabled; the -s g means that mirroring is
following:
created
with character device “/dev/vg_ops/rsystem.dbf”
extended
volume name) is used by the Oracle DBA in building the OPS database.
Creating RAC Volume Groups on Disk Arrays

The procedure described in this section assumes that you are using
RAID-protected disk arrays and LVM’s physical volume links (PV links)
to define redundant data paths from each node in the cluster to every
logical unit on the array.
On your disk arrays, you should use redundant I/O channels from each
node, connecting them to separate controllers on the array. Then you can
define alternate links to the LUNs or logical disks you have defined on
116 Chapter 3
the array. If you are using SAM, choose the type of disk array you wish to
configure, and follow the menus to define alternate links. If you are using
LVM commands, specify the links on the command line.
The following example shows how to configure alternate links using
LVM commands. The following disk configuration is assumed:

Assume that the disk array has been configured, and that both the
following device files appear for the same LUN (logical disk) when you
run the ioscan command:
/dev/dsk/c0t15d0
/dev/dsk/c1t3d0
Use the following procedure to configure a volume group for this logical
disk:
1. First, set up the group directory for vg_ops:

# mkdir /dev/vg_ops
has the format:
0xhh0000
Chapter 3 117
3. Use the pvcreate command on one of the device files associated with
the LUN to define the LUN to LVM as a physical volume.
It is only necessary to do this with one of the device file names for the
LUN. The -f option is only necessary if the physical volume was
previously used in some other volume group.
4. Use the following to create the volume group with the two links:
# vgcreate /dev/vg_ops /dev/dsk/c0t15d0 /dev/dsk/c1t3d0
LVM will now recognize the I/O channel represented by
/dev/dsk/c0t15d0 as the primary link to the disk; if the primary link
fails, LVM will automatically switch to the alternate I/O channel
represented by /dev/dsk/c1t3d0. Use the vgextend command to add
additional disks to the volume group, specifying the appropriate physical
volume name for each PV link.
Repeat the entire procedure for each distinct volume group you wish to
create. For ease of system administration, you may wish to use different
volume groups to separate logs from data and control files.
NOTE The default maximum number of volume groups in HP-UX is 10. If you
intend to create enough new volume groups that the total exceeds ten,
you must increase the maxvgs system parameter and then re-build the
HP-UX kernel. Use SAM and select the Kernel Configuration area,
then choose Configurable Parameters. Maxvgs appears on the list.
Creating Logical Volumes for RAC on Disk Arrays

After you create volume groups and add PV links to them, you define
logical volumes for data, logs, and control files. The following are some
examples:
# lvcreate -n ops1log1.log -L 28 /dev/vg_ops
# lvcreate -n opsctl1.ctl -L 108 /dev/vg_ops
# lvcreate -n system.dbf -L 408 /dev/vg_ops
# lvcreate -n opsdata1.dbf -L 208 /dev/vg_ops
118 Chapter 3

The following set of files is required for the Oracle demo database which
you can create during the installation process.
Oracle
LV File
Size Size
opssystem.dbf 408 /dev/vg_ops/ropssystem.dbf 400
opstemp.dbf 108 /dev/vg_ops/ropstemp.dbf 100
opsusers.dbf 128 /dev/vg_ops/ropsusers.dbf 120
opstools.dbf 24 /dev/vg_ops/ropstools.dbf 15
opsrollback.dbf 308 /dev/vg_ops/ropsroolback.dbf 300
opsspfile1.ora 5 /dev/vg_ops/ropsspfile1.ora 5
opsundotbs1.dbf 320 /dev/vg_ops/ropsundotbs1.dbf 312
Chapter 3 119
Oracle
LV File
Size Size
opsundotbs2.dbf 320 /dev/vg_ops/ropsundotbs2.dbf 312
opsexample1.dbf 168 /dev/vg_ops/ropsexample1.dbf 160
opscwmlite1.dbf 108 /dev/vg_ops/ropscwmlite1.dbf 100
opsindx1.dbf 78 /dev/vg_ops/ropsindx1.dbf 70
opsdrsys1.dbf 98 /dev/vg_ops/ropsdrsys1.dbf 90
*
The size of the logical volume is larger than the Oracle file size because
Oracle needs extra space to allocate a header in addition to the file's
actual data capacity.
appropriate directory. Example:
# ln -s /dev/vg_ops/ropsctl1.ctl \
After creating these files, set the owner to oracle and the group to dba
with a file mode of 660. The logical volumes are now available on the
primary node, and the raw logical volume names can now be used by the
Oracle DBA.
120 Chapter 3

To display the volume group, use the vgdisplay command:
# vgdisplay -v /dev/vg_ops
Exporting the Logical Volume Infrastructure

Before the Oracle volume groups can be shared, their configuration data
must be exported to other nodes in the cluster. This is done either in
Serviceguard Manager or by using HP-UX commands, as shown in the
following sections.
Exporting with LVM Commands

Use the following commands to set up the same volume group on another
cluster node. In this example, the commands set up a new volume group
on a system known as ftsys10. This volume group holds the same
physical volume that was created on a configuration node known as
ftsys9.
To set up the volume group on ftsys10 (and other nodes), use the
following steps:
1. On ftsys9, copy the mapping of the volume group to a specified file.

# vgexport -s -p -m /tmp/vg_ops.map /dev/vg_ops
2. Still on ftsys9, copy the map file to ftsys10 (and to additional
nodes as necessary.)
# rcp /tmp/vg_ops.map ftsys10:/tmp/vg_ops.map
3. On ftsys10 (and other nodes, as necessary), create the volume group
directory and the control file named group:
# mkdir /dev/vg_ops
For the group file, the major number is always 64, and the
hexadecimal minor number has the format:
0xhh0000
Chapter 3 121
where hh must be unique to the volume group you are creating. If

possible, use the same number as on ftsys9. Use the following
4. Import the volume group data using the map file from node ftsys9.
On node ftsys10 (and other nodes, as necessary), enter:
# vgimport -s -m /tmp/vg_ops.map /dev/vg_ops
122 Chapter 3
NOTE Some versions of Oracle RAC requires installation of additional software.

Refer to your version of Oracle for specific requirements.
Before installing the Oracle Real Application Cluster software, make

sure the cluster is running. Login as the oracle user on one node and
then use the Oracle installer to install Oracle software and to build the
correct Oracle runtime executables. When the executables are installed
to a cluster file system, the Oracle installer has an option to install the
executables once. When executables are installed to a local file system on
each node, the Oracle installer copies the executables to the other nodes
in the cluster.
For details on Oracle installation, refer to the Oracle installation
documentation. As part of this installation, the Oracle installer installs
the executables and optionally, the Oracle installer can build an Oracle
demo database on the primary node. The demo database files can be
either the character (raw) device files names for the logical volumes
created earlier, or the database can reside on a cluster file system.
For a demo database on SLVM or CVM, create logical volumes as shown
in Table 3-2, “Required Oracle File Names for Demo Database,” earlier in
this chapter. As the installer prompts for the database file names, use
the pathnames of the raw logical volumes instead of using the defaults.
NOTE If you do not wish to install the demo database, select install software
only.
Chapter 3 123

The following is an example of an ASCII configuration file generated
with the cmquerycl command using the -w full option on a system
with Serviceguard Extension for RAC. The OPS_VOLUME_GROUP
parameters appear at the end of the file.
# **********************************************************************
# ********* HIGH AVAILABILITY CLUSTER CONFIGURATION FILE ***************
# ***** For complete details about cluster parameters and how to *******
# ***** set them, consult the Serviceguard manual. *********************
# **********************************************************************
# Enter a name for this cluster. This name will be used to identify the
# cluster when viewing or manipulating it.
CLUSTER_NAME cluster 1
# Cluster Lock Parameters

# The cluster lock is used as a tie-breaker for situations
# in which a running cluster fails, and then two equal-sized
# sub-clusters are both trying to form a new cluster. The
# cluster lock may be configured using only one of the
# following alternatives on a cluster:
# the LVM lock disk
# the quorom server
#
#
# Consider the following when configuring a cluster.
# For a two-node cluster, you must use a cluster lock. For
# a cluster of three or four nodes, a cluster lock is strongly
# recommended. For a cluster of more than four nodes, a
# cluster lock is recommended. If you decide to configure
# a lock for a cluster of more than four nodes, it must be
# a quorum server.
# Lock Disk Parameters. Use the FIRST_CLUSTER_LOCK_VG and

# FIRST_CLUSTER_LOCK_PV parameters to define a lock disk.
# The FIRST_CLUSTER_LOCK_VG is the LVM volume group that
# holds the cluster lock. This volume group should not be
# used by any other cluster as a cluster lock device.
# Quorum Server Parameters. Use the QS_HOST, QS_POLLING_INTERVAL,
124 Chapter 3
# and QS_TIMEOUT_EXTENSION parameters to define a quorum server.

# The QS_HOST is the host name or IP address of the system
# that is running the quorum server process. The
# QS_POLLING_INTERVAL (microseconds) is the interval at which
# Serviceguard checks to make sure the quorum server is running.
# The optional QS_TIMEOUT_EXTENSION (microseconds) is used to increase
# the time interval after which the quorum server is marked DOWN.
#
# The default quorum server timeout is calculated from the
# Serviceguard cluster parameters, including NODE_TIMEOUT and
# HEARTBEAT_INTERVAL. If you are experiencing quorum server
# timeouts, you can adjust these parameters, or you can include
# the QS_TIMEOUT_EXTENSION parameter.
#
# The value of QS_TIMEOUT_EXTENSION will directly effect the amount
# of time it takes for cluster reformation in the event of failure.
# For example, if QS_TIMEOUT_EXTENSION is set to 10 seconds, the cluster
# reformation will take 10 seconds longer than if the QS_TIMEOUT_EXTENSION
# was set to 0. This delay applies even if there is no delay in
# contacting the Quorum Server. The recommended value for
# QS_TIMEOUT_EXTENSION is 0, which is used as the default
# and the maximum supported value is 30000000 (5 minutes).
#
# For example, to configure a quorum server running on node
# “qshost” with 120 seconds for the QS_POLLING_INTERVAL and to
# add 2 seconds to the system assigned value for the quorum server
# timeout, enter:
#
# QS_HOST qshost
# QS_POLLING_INTERVAL 120000000
# QS_TIMEOUT_EXTENSION 2000000
# Definition of nodes in the cluster.

# Repeat node definitions as necessary for additional nodes.
# NODE_NAME is the specified nodename in the cluster.
# It must match the hostname and both cannot contain full domain name.
# Each NETWORK_INTERFACE, if configured with IPv4 address,
# must have ONLY one IPv4 address entry with it which could
# be either HEARTBEAT_IP or STATIONARY_IP.
# Each NETWORK_INTERFACE, if configured with IPv6 address(es)
# can have multiple IPv6 address entries(up to a maximum of 2,
# only one IPv6 address entry belonging to site-local scope
# and only one belonging to global scope) which must be all
# STATIONARY_IP. They cannot be HEARTBEAT_IP.
Chapter 3 125
NODE_NAME ever3a
STATIONARY_IP15.244.64.140
HEARTBEAT_IP192.77.1.1
# List of serial device file names
# For example:
# SERIAL_DEVICE_FILE /dev/tty0p0

# Warning: There are no standby network interfaces on bridged net 1.
# Possible standby Network Interfaces on Bridged Net 2: lan2.
# Cluster Timing Parameters (microseconds).
# The NODE_TIMEOUT parameter defaults to 2000000 (2 seconds).

# This default setting yields the fastest cluster reformations.
# However, the use of the default value increases the potential
# for spurious reformations due to momentary system hangs or
# network load spikes.
# For a significant portion of installations, a setting of
# 5000000 to 8000000 (5 to 8 seconds) is more appropriate.
# The maximum value recommended for NODE_TIMEOUT is 30000000
# (30 seconds).
HEARTBEAT_INTERVAL 1000000
NODE_TIMEOUT 2000000
# Configuration/Reconfiguration Timing Parameters (microseconds).
AUTO_START_TIMEOUT 600000000
NETWORK_POLLING_INTERVAL 2000000
# Network Monitor Configuration Parameters.

# The NETWORK_FAILURE_DETECTION parameter determines how LAN card failures are
detected.
# If set to INONLY_OR_INOUT, a LAN card will be considered down when its inbound
# message count stops increasing or when both inbound and outbound
# message counts stop increasing.
# If set to INOUT, both the inbound and outbound message counts must
126 Chapter 3
# stop increasing before the card is considered down.

NETWORK_FAILURE_DETECTION INOUT
# Package Configuration Parameters.

# Enter the maximum number of packages which will be configured in the cluster.
# You can not add packages beyond this limit.
# This parameter is required.
MAX_CONFIGURED_PACKAGES 150
# Access Control Policy Parameters.

#
# Three entries set the access control policy for the cluster:
# First line must be USER_NAME, second USER_HOST, and third USER_ROLE.
# Enter a value after each.
#
# 1. USER_NAME can either be ANY_USER, or a maximum of
# 8 login names from the /etc/passwd file on user host.
# 2. USER_HOST is where the user can issue Serviceguard commands.
# If using Serviceguard Manager, it is the COM server.
# Choose one of these three values: ANY_SERVICEGUARD_NODE, or
# (any) CLUSTER_MEMBER_NODE, or a specific node. For node,
# use the official hostname from domain name server, and not
# an IP addresses or fully qualified name.
# 3. USER_ROLE must be one of these three values:
# * MONITOR: read-only capabilities for the cluster and packages
# * PACKAGE_ADMIN: MONITOR, plus administrative commands for packages
# in the cluster
# * FULL_ADMIN: MONITOR and PACKAGE_ADMIN plus the administrative
# commands for the cluster.
#
# Access control policy does not set a role for configuration
# capability. To configure, a user must log on to one of the
# cluster’s nodes as root (UID=0). Access control
# policy cannot limit root users’ access.
#
# MONITOR and FULL_ADMIN can only be set in the cluster configuration file,
# and they apply to the entire cluster. PACKAGE_ADMIN can be set in the
# cluster or a package configuration file. If set in the cluster
# configuration file, PACKAGE_ADMIN applies to all configured packages.
# If set in a package configuration file, PACKAGE_ADMIN applies to that
# package only.
#
# Conflicting or redundant policies will cause an error while applying
# the configuration, and stop the process. The maximum number of access
# policies that can be configured in the cluster is 200.
Chapter 3 127
#
# Example: to configure a role for user john from node noir to
# administer a cluster and all its packages, enter:
# USER_NAME john
# USER_HOST noir
# USER_ROLE FULL_ADMIN
# List of cluster aware LVM Volume Groups. These volume groups will
# be used by package applications via the vgchange -a e command.
# Neither CVM or VxVM Disk Groups should be used here.
# For example:
# VOLUME_GROUP /dev/vgdatabase
# VOLUME_GROUP /dev/vg02
# List of OPS Volume Groups.

# Formerly known as DLM Volume Groups, these volume groups
# will be used by OPS or RAC cluster applications via
# the vgchange -a s command. (Note: the name DLM_VOLUME_GROUP
# is also still supported for compatibility with earlier versions.)
# For example:
# OPS_VOLUME_GROUP /dev/vgdatabase
# OPS_VOLUME_GROUP /dev/vg02
128 Chapter 3

The following are example steps for creating a storage infrastructure for
CFS.
Creating a SGeRAC Cluster with CFS 4.1 for Oracle 9i

The following software needs to be pre-installed in order to use this
configuration:
• SGeRAC and CFS are included with the HP Serviceguard Storage
Management Suite bundle T2777BA or Mission Critical Operating
Environment (MCOE) T2797BA.
• CFS is included with the following bundles: (SGeRAC needs to be
installed separately)
— T2776BA HP Serviceguard Cluster File System for Oracle

(SGCFSO)
— T2775BA HP Serviceguard Cluster File System (SGCFS).
Refer to the HP Serviceguard Storage Management Suite Version A.01.00
Release Notes.
In the following example, both the Oracle software and datafiles reside
on CFS. There is a single Oracle home. The following three CFS file
systems are created for Oracle home, Oracle datafiles, and SRVM data:
/cfs/mnt1 – for Oracle Base and Home
/cfs/mnt2 – for Oracle datafiles
/cfs/cfssrvm - for SRVM data

Use the following steps to create a two node SGeRAC cluster with CFS
and Oracle:
1. Initialize the VERITAS Volume Manager

If not already done, install the VxVM license key on all nodes. Use
the following command:
# vxinstall
Chapter 3 129
NOTE CVM 4.1 does not require rootdg

# cd /etc/cmcluster
Edit Cluster file
4. Start the Cluster
# cmruncl
# cmviewcl
CLUSTER STATUS
ever3_cluster up
NODE STATUS STATE

ever3a up running
ever3b up running
and start the CVM/CFS stack. Unlike VxVM-CVM-pkg, the
SG-CFS-pkg does not restrict heartbeat subnets to a single subnet
and supports multiple subnets.
# cfscluster config -s
CVM is now configured
Starting CVM...
It might take a few minutes to complete
130 Chapter 3
# vxdctl -c mode
master: ever3b
or
slave: ever3b
6. Converting Disks from LVM to CVM
CVM disk groups. Before you can do this, the volume group must be
deactivated, which means that any package that uses the volume
group must be halted. This procedure is described in the Managing
Serviceguard Twelfth Edition user’s guide Appendix G.
7. Initializing Disks for CVM/CFS
CVM disk groups. If a physical disk has been previously used with
LVM, you should use the pvremove command to delete the LVM
header data from all the disks in the volume group (this is not
vxdiskadm program to initialize multiple disks, or use the
following example:
8. Create the Disk Group for RAC
Use the vxdg command to create disk groups. Use the -s option to
specify shared mode, as in the following example:
# vxdg -s init cfsdg1 c4t4d0
9. Create the Disk Group Multi-Node package. Use the following
command to add the disk group to the cluster:
# cfsdgadm add cfsdg1 all=sw
Chapter 3 131
Package name “SG-CFS-DG-1” was generated to control the

resource shared disk group “cfsdg1” is associated with
the cluster.
10. Activate the Disk Group
# cfsdgadm activate cfsdg1
# vxassist -g cfsdg1 make volsrvm 300m
version 6 layout
16384 blocks
version 6 layout
16384 blocks
# newfs -F vxfs /dev/vx/rdsk/cfsdg1/volsrvm
version 6 layout
307200 sectors, 307200 blocks of size 1024, log size 1024
blocks largefiles supported
12. Configure Mount Point
132 Chapter 3

cluster.
cluster.
# cfsmntadm add cfsdg1 volsrvm /cfs/cfssrvm all=rw
resource. Mount point “/cfs/cfssrvm” that was associated
with the cluster.
13. Mount Cluster Filesystem
# cfsmount /cfs/cfssrvm
14. Check CFS Mount Points
# bdf | grep cfs
10485760 19651 9811985 0% /cfs/mnt1
10485760 19651 9811985 0% /cfs/mnt2
/dev/vx/dsk/cfsdg1/volsrvm
307200 1802 286318 1% /cfs/cfssrvm
# cmviewcl
CLUSTER STATUS
ever3_cluster up
NODE STATUS STATE

ever3a up running
ever3b up running
MULTI_NODE_PACKAGES
Chapter 3 133

Deleting CFS from the Cluster

Halt the applications that are using CFS file systems.
1. Unmount CFS Mount Points

# cfsumount /cfs/cfssrvm
2. Delete MP MNP
cluster
cluster
# cfsmntadm delete /cfs/cfssrvm
Mount point “/cfs/cfssrvm” was disassociated from the
cluster Cleaning up resource controlling shared disk
group “cfsdg1” Shared disk group “cfsdg1” was
disassociated from the cluster.
NOTE The disk group is deleted if there is no dependency.
134 Chapter 3
3. Delete DG MNP
# cfsdgadm delete cfsdg1
Shared disk group “cfsdg1” was disassociated from the
cluster.
NOTE “cfsmntadm delete” also deletes the disk group if there is no

dependent package. To ensure the disk group deletion is complete,
use the above command to delete the disk group package.
4. De-configure CVM
# cfscluster stop
Stopping CVM...CVM is stopped
# cfscluster unconfig
CVM is now unconfigured
Chapter 3 135

This is done with Logical Volume Manager (LVM), VERITAS Volume
Manager (VxVM), or VERITAS Cluster Volume Manager (CVM). LVM
This section shows how to configure storage using the VERITAS Cluster
Volume Manager (CVM). The examples show how to configure RAC disk
groups, but you can also create CVM disk groups for non-RAC use. For
more information, including details about configuration of plexes
(mirrors), multi-pathing, and RAID, refer to the HP-UX documentation
for the VERITAS Volume Manager.

If you are about to create disk groups for the first time, you need to
initialize the Volume Manager. This is done by creating a disk group
known as rootdg that contains at least one disk. Use the following
command after installing CVM on each node:
# vxinstall
This displays a menu-driven program that steps you through the CVM
initialization sequence. From the main menu, choose the “Custom”
option, and specify the disk you wish to include in rootdg.
IMPORTANT Creating a rootdg disk group is only necessary the first time you use the
Volume Manager. CVM 4.1 does not require a rootdg.
Using CVM 4.x

system multi-node package with CVM 4.x only (without the CFS
filesystem).
136 Chapter 3
For more detailed information on how to configure CVM 4.x, refer the
Managing Serviceguard Twelfth Edition user’s guide.
Preparing the Cluster and the System Multi-node Package for

use with CVM 4.x

# cd /etc/cmcluster
Edit Cluster file
NOTE To prepare the cluster for CVM configuration, you need to be sure
MAX_CONFIGURED_PACKAGES to minimum of 3 (the default value for
MAX_CONFIGURED_PACKAGES for Serviceguard A.11.17 is 150) cluster
configuration file. In the sample set the value to 10.

• Start the Cluster

# cmruncl
# cmviewcl
CLUSTER STATUS
ever3_cluster up
NODE STATUS STATE

ever3a up running
ever3b up running
and start the CVM stack. Unlike VxVM-CVM-pkg, the SG-CFS-pkg
does not restrict heartbeat subnets to a single subnet and supports
multiple subnets.
# cmapplyconf -P /etc/cmcluster/cfs/SG-CFS-pkg.conf
Chapter 3 137
# cmrunpkg SG-CFS-pkg
# vxdctl -c mode
master: ever3b
• Converting Disks from LVM to CVM

CVM disk groups. Before you can do this, the volume group must
be deactivated, which means that any package that uses the
volume group must be halted. This procedure is described in the
Managing Serviceguard Thirteenth Edition user’s guide
Appendix G.
• Initializing Disks for CVM
CVM disk groups. If a physical disk has been previously used
with LVM, you should use the pvremove command to delete the
LVM header data from all the disks in the volume group (this is
not necessary if you have not previously used the disk with
LVM).
To initialize a disk for CVM, log on to the master node, then use
the vxdiskadm program to initialize multiple disks, or use the
following example:
• Create the Disk Group for RAC
Use the vxdg command to create disk groups. Use the -s option
to specify shared mode, as in the following example:
138 Chapter 3

# vxassist -g ops_dg make volsrvm 300m
# cmviewcl
CLUSTER STATUS
ever3_cluster up
NODE STATUS STATE

ever3a up running
ever3b up running
MULTI_NODE_PACKAGES

Mirror Detachment Policies with CVM The required CVM disk

mirror detachment policy is ‘global’, which means that as soon as one
node cannot see a specific mirror copy (plex), all nodes cannot see it as
well. The alternate policy is ‘local’, which means that if one node cannot
see a specific mirror copy, then CVM will deactivate access to the volume
for that node only.
Chapter 3 139
Volume Manager.
Using CVM 3.x

This section has information on how to prepare the cluster with CVM
3.x.
Preparing the Cluster for Use with CVM 3.x

In order to use the VERITAS Cluster Volume Manager (CVM) version
3.5, the cluster must be running with a special CVM package. This
means that the cluster must already be configured and running before
you create disk groups.
NOTE Cluster configuration is described in the previous section.
To prepare the cluster for CVM disk group configuration, you need to
ensure that only one heartbeat subnet is configured. Then use the
following command, which creates the special package that
communicates cluster information to CVM:
# cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf
WARNING This file should never be edited.
After the above command completes, start the cluster and create disk
groups for shared use as described in the following sections.
Starting the Cluster and Identifying the Master Node Run the
cluster, which will activate the special CVM package:
# cmruncl
140 Chapter 3
After the cluster is started, it will now run with a special system
multi-node package named VxVM-CVM-pkg, which is on all nodes. This
package is shown in the following output of the cmviewcl -v command:
CLUSTER STATUS
bowls up
NODE STATUS STATE

spare up running
split up running
strike up running

When CVM starts up, it selects a master node, and this is the node from
which you must issue the disk group configuration commands. To
determine the master node, issue the following command from each node
in the cluster:
# vxdctl -c mode
node.
Converting Disks from LVM to CVM Use the vxvmconvert utility to

convert LVM volume groups into CVM disk groups. Before you can do
this, the volume group must be deactivated, which means that any
package that uses the volume group must be halted. This procedure is
described in the Managing Serviceguard Twelfth Edition user’s guide
Appendix G.
Initializing Disks for CVM Initialize the physical disks that will be
employed in CVM disk groups. If a physical disk has been previously
used with LVM, you should use the pvremove command to delete the
LVM header data from all the disks in the volume group (this is not
vxdiskadm program to initialize multiple disks, or use the vxdisksetup
command to initialize one disk at a time, as in the following example:
# /usr/lib/vxvm/bin/vxdisksetup -i /dev/dsk/c0t3d2
Chapter 3 141
Creating Disk Groups for RAC Use the vxdg command to create disk
groups. Use the -s option to specify shared mode, as in the following
example:
# vxdg list
NAME STATE ID
rootdg enabled 971995699.1025.node1

ops_dg enabled,shared 972078742.1084.node2
142 Chapter 3
Creating Volumes
Creating Volumes
Use the vxassist command to create logical volumes. The following is
an example:
# vxassist -g log_files make ops_dg 1024m
This command creates a 1024 MB volume named log_files in a disk
group named ops_dg. The volume can be referenced with the block device
file /dev/vx/dsk/ops_dg/log_files or the raw (character) device file
/dev/vx/rdsk/ops_dg/log_files.
# vxdg list
Mirror Detachment Policies with CVM

The required CVM disk mirror detachment policy is ‘global’, which
means that as soon as one node cannot see a specific mirror copy (plex),
all nodes cannot see it as well. The alternate policy is ‘local’, which
means that if one node cannot see a specific mirror copy, then CVM will
deactivate access to the volume for that node only.
Chapter 3 143
Creating Volumes
Volume Manager.
144 Chapter 3

The following set of volumes is required for the Oracle demo database
which you can create during the installation process.
Oracle
Size File Size
ops1log1.log 28 /dev/vx/rdsk/ops_dg/oops1log1.log 20
opssystem.dbf 408 /dev/vx/rdsk/ops_dg/opssystem.dbf 400
opstemp.dbf 108 /dev/vx/rdsk/ops_dg/opstemp.dbf 100
opsusers.dbf 128 /dev/vx/rdsk/ops_dg/opsusers.dbf 120
opstools.dbf 24 /dev/vx/rdsk/ops_dg/opstools.dbf 15
opsrollback.dbf 308 /dev/vx/rdsk/ops_dg/opsroolback.dbf 300
opsspfile1.ora 5 /dev/vx/rdsk/ops_dg/opsspfile1.ora 5
Chapter 3 145
Oracle
Size File Size
opsexample1.dbf 168 /dev/vx/rdsk/ops_dg/opsexample1.dbf 160
opscwmlite1.dbf 108 /dev/vx/rdsk/ops_dg/opscwmlite1.dbf 100
opsindx1.dbf 78 /dev/vx/rdsk/ops_dg/opsindx1.dbf 70
appropriate directory.
Example:
# ln -s /dev/vx/rdsk/ops_dg/opsctl1.ctl \
Example, Oracle9:
1. Create an ASCII file, and define the path for each database object.
control1=/dev/vx/rdsk/ops_dg/opsctl1.ctl
or
control1=/u01/ORACLE/db001/ctrl01_1.ctl
2. Set the following environment variable where filename is the name
of the ASCII file created.
# export DBCA_RAW_CONFIG=<full path>/filename
146 Chapter 3
Adding Disk Groups to the Cluster Configuration
Adding Disk Groups to the Cluster

Configuration
For CVM 4.x, if the multi-node package was configured for disk group
activation, the application package should be configured with package
dependency to ensure the CVM disk group is active.
For CVM 3.5 and CVM 4.x (without using multi-node package) after
creating units of CVM storage with VxVM commands, you need to
specify the disk groups in each package configuration ASCII file. Use one
STORAGE_GROUP parameter for each CVM disk group the package will
use. It is necessary to identify the CVM disk groups, file systems, logical
volumes, and mount options in the package control script.
For more detailed information on the package configuration process refer
to the Managing Serviceguard Thirteenth Edition user’s guide.
Chapter 3 147
Installing Oracle 9i RAC

The following sample steps for a SGeRAC cluster for Oracle 9i. Refer to
Install Oracle Software into CFS Home

Oracle RAC software is installed using the Oracle Universal Installer.
This section describes installation of Oracle RAC software onto a CFS
home.
1. Oracle Pre-installation Steps
a. Create user accounts.

Create user and group for Oracle accounts on all nodes using the
following commands:
# groupadd -g 99 dba
# useradd -g dba -u 999 -d /cfs/mnt1/oracle oracle
Create Oracle home directory on CFS
# cd /cfs/mnt1
# mkdir /cfs/mnt1/oracle
# chown oracle:dba oracle
Change password for Oracle account on all nodes.
# passwd oracle
b. Set up for remote commands
Setup user equivalence for all nodes by adding node name entries
to /etc/hosts.equiv or add entries to the .rhosts of oracle
account.
c. Set up CFS directory for Oracle datafiles.
# cd /cfs/mnt2
# mkdir oradata
# chown oracle:dba oradata
# chmod 755 oradata
148 Chapter 3
# ll
total 0
drwxr-xr-x 2 root root 96 Jun 3 11:43
lost+found
drwxr-xr-x 2 oracle dba 96 Jun 3 13:45
oradat
d. Set up CFS directory for Server Management.
Preallocate space for srvm (200MB)
# prealloc /cfs/cfssrvm/ora_srvm 209715200
# chown oracle:dba /cfs/cfssrvm/ora_srvm
2. Install Oracle RAC Software
a. Install Oracle (software only) with Oracle Universal Installer as

oracle user
# su - oracle
When using CFS for SRVM, set SRVM_SHARED_CONFIG
$ export SRVM_SHARED_CONFIG=/cfs/cfssrvm/ora_srvm
b. Set DISPLAY
$ export DISPLAY=${display}:0.0
c. Run Oracle Universal Installer and follow installation steps
$ cd <Oracle installation disk directory>
$ ./runInstaller
Create Database with Oracle Tools

Refer to Oracle documentation for more detailed information on creating
Oracle database.
1. Set up Environment Variables

Use the following as an example:
export ORACLE_BASE=/cfs/mnt1/oracle
export ORACLE_HOME=$ORACLE_BASE/product/9.2.0.2
export PATH=$ORACLE_HOME/bin:$PATH
Chapter 3 149
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:$ORACLE_HOM
E/rdbms/lib
export CLASSPATH=/opt/java1.3/lib
CLASSPATH=$CLASSPATH:$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$O
RACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib
export CLASSPATH
2. Set up Listeners with Oracle Network Configuration
Assistant
$ netca
3. Start GSD on all Nodes
$ gsdctl start
Output: Successfully started GSD on local node
4. Run Database Configuration Assistant to Create the

Database on CFS File System
$ dbca -datafileDestination /cfs/mnt2/oradata
150 Chapter 3
Verify that Oracle Disk Manager is

Configured
1. Check the license:

# /opt/VRTS/bin/vxlictest -n “VERITAS Storage Foundation
for Oracle” -f “ODM”
Using VERITAS License Manager API Version 3.00, Build 2
ODM feature is licensed
2. Check that the VRTSodm package is installed:
# swlist VRTSodm
VRTSodm 4.1m VERITAS Oracle Disk Manager
VRTSodm.ODM-KRN 4.1m VERITAS ODM kernel files
VRTSodm.ODM-MAN 4.1m VERITAS ODM manual pages
VRTSodm.ODM-RUN 4.1m VERITAS ODM commands
3. Check that libodm.sl is present:
# ll -L /opt/VRTSodm/lib/libodm.sl
-rw-r--r-- 1 root sys 14336 Apr 25 18:42
/opt/VRTSodm/lib/libodm.sl
Chapter 3 151
Configure Oracle to use Oracle Disk Manager Library
Configure Oracle to use Oracle Disk Manager

Library
1. Logon as Oracle user

3. Link the Oracle Disk Manager library into Oracle home using the
following commands:
For HP 9000 systems:
$ rm ${ORACLE_HOME}/lib/libodm9.sl
For Integrity systems:
$ rm ${ORACLE_HOME}/lib/libodm9.so
4. Start the Oracle database
152 Chapter 3
Verify Oracle Disk Manager is Running
1. Start the cluster and Oracle database (if not already started)
2. Check that the Oracle instance is using the Oracle Disk Manager
function with following command:
# cat /dev/odm/stats
abort: 0
cancel: 0
commit: 18
create: 18
delete: 0
identify: 349
io: 12350590
reidentify: 78
resize: 0
unidentify: 203
mname: 0
vxctl: 0
vxvers: 10
io req: 9102431
io calls: 6911030
comp req: 73480659
comp calls: 5439560
io mor cmp: 461063
io zro cmp: 2330
cl receive: 66145
cl ident: 18
cl reserve: 8
cl delete: 1
cl resize: 0
cl same op: 0
cl opt idn: 0
cl opt rsv: 332
**********: 17
Chapter 3 153
3. Verify that Oracle Disk Manager is loaded with the following

command:
# kcmodule -P state odm
state loaded
4. In the alert log, verify the Oracle instance is running. The log should
contain output similar to the following:
Oracle instance running with ODM: VERITAS 4.1 ODM
Library, Version 1.1
154 Chapter 3
Configuring Oracle to Stop using Oracle Disk Manager Library
Configuring Oracle to Stop using Oracle Disk

Manager Library

2. Shutdown the database
3. Change directories:
$ cd ${ORACE_HOME}/lib
4. Remove the file linked to the ODM library:
For HP 9000 systems:
$ rm libodm9.sl
$ ln -s ${ORACLE_HOME}/lib/libodmd9.sl \
For Integrity systems:
$ rm libodm9.so
$ ln -s ${ORACLE_HOME}/lib/libodmd9.so \
5. Restart the database
Chapter 3 155
Using Packages to Configure Startup and Shutdown of RAC Instances
Using Packages to Configure Startup and

Shutdown of RAC Instances
To automate the startup and shutdown of RAC instances on the nodes of
the cluster, you can create packages which activate the appropriate
volume groups and then run RAC. Refer to the section “Creating
Packages to Launch Oracle RAC Instances”
NOTE The maximum number of RAC instances for Oracle 9i is 127 per cluster.
For Oracle 10g refer to Oracle’s requirements.
Starting Oracle Instances

Once the Oracle installation is complete, ensure that all package control
scripts are in place on each node and that each
/etc/rc.config.d/cmcluster script contains the entry
AUTOSTART_CMCLD=1. Then reboot each node. Within a couple of minutes
following reboot, the cluster will reform, and the package control scripts
will bring up the database instances and application programs.
When Oracle has been started, you can use the SAM process
management area or the ps -ef command on both nodes to verify that
all RAC daemons and Oracle processes are running.
Starting Up and Shutting Down Manually

To start up and shut down RAC instances without using packages, you
can perform the following steps.
• Starting up involves the following sequence:
1. Start up the cluster (cmrunnode or cmruncl)

2. Activate the database volume groups or disk groups in shared
mode.
3. Bring up Oracle in shared mode.
4. Bring up the Oracle applications, if any.
• Shutting down involves the following sequence:
156 Chapter 3
1. Shut down the Oracle applications, if any.

2. Shut down Oracle.
3. Deactivate the database volume groups or disk groups.
4. Shut down the cluster (cmhaltnode or cmhaltcl).
If the shutdown sequence described above is not followed, cmhaltcl or
cmhaltnode may fail with a message that GMS clients (RAC 9i) are
active or that shared volume groups are active.
Creating Packages to Launch Oracle RAC Instances

To coordinate the startup and shutdown of RAC instances with cluster
node startup and shutdown, you create a one-node package for each node
that runs an RAC instance. In the package configuration file, you should
specify only the single node on which the instance will run and specify
the control script that is to be executed every time the instance node or
the entire RAC cluster starts up or shuts down.
NOTE You must create the RAC instance package with a PACKAGE_TYPE of
FAILOVER, but the fact that you are entering only one node ensures that
the instance will only run on that node.
To simplify the creation of RAC instance packages, you can use the
Oracle template provided with the separately purchasable ECM Toolkits
product (T1909BA). Use the special toolkit scripts that are provided, and
follow the instructions that appear in the README file. Also refer to the
section “Customizing the Control Script for RAC Instances” below for
more information.
To create the package with Serviceguard Manager select the cluster. Go
to the actions menu and choose configure package. To modify a package,
select the package. For an instance package, create one package for each
instance. On each node, supply the SID name for the package name.
To create a package on the command line, use the cmmakepkg command
to get an editable configuration file.
Set the AUTO_RUN parameter to YES, if you want the instance to start up
as soon as the node joins the cluster. In addition, you should set the
NODE_FAILFAST_ENABLED parameter to NO.
Chapter 3 157
If you are using CVM disk groups for the RAC database, be sure to
include the name of each disk group on a separate STORAGE_GROUP line in
the configuration file.
If you are using CFS or CVM for RAC shared storage with multi-node
packages, the package containing the RAC instance should be configured
with package dependency to depend on the multi-node packages.
The following is a sample of the setup dependency conditions in
application package configuration file:
DEPENDENCY_NAME mp1
DEPENDENCY_NAME mp2
DEPENDENCY_NAME mp3
Configuring Packages that Access the Oracle RAC

Database
You can also use packages to start up applications that access the RAC
instances. If an application is intended to fail over among cluster nodes,
then you must set it up as a distinct package, separate from the package
that starts and stops the RAC instance. Use the following procedures for
packages that contain applications which access the RAC database:
1. In the ASCII package configuration file, set the AUTO_RUN parameter

to NO, or if you are using Serviceguard Manager to configure
packages, set Automatic Switching to Disabled. This keeps the
package from starting up immediately when the node joins the
cluster, and before RAC is running.
2. You can then manually start the package using the cmmodpkg -e
packagename command after RAC is started. Alternatively, you can
choose to automate the process of package activation by writing your
158 Chapter 3
own script, and copying it to all nodes that can run the package. This
script should contain the cmmodpkg -e command and activate the
package after RAC and the cluster manager have started.
Adding or Removing Packages on a Running Cluster

You can add or remove packages while the cluster is running, subject to
the limit of MAX_CONFIGURED_PACKAGES. For more detailed information
on adding or removing packages online, refer to section “Cluster and
Package Maintenance” in the Managing Serviceguard Thirteenth Edition
user’s guide.
Writing the Package Control Script

The package control script contains all the information necessary to run
all the services in the package, monitor them during operation, react to a
failure, and halt the package when necessary. You can use either
Serviceguard Manager or HP-UX commands to create or modify the
package control script. For security reasons, the control script must
reside in a directory with the string cmcluster in the path.
Using Serviceguard Manager to Write the Package Control

Script
As you complete the tabs for the configuration, the control script can be
generated automatically. When asked to supply the pathname of the
package run and halt scripts, use the filenames from the ECM toolkit.
For more information, use the Help key.
When you create a package control script this way, you do not need to do
any further editing, but you may customize the script if you wish.
Using Commands to Write the Package Control Script

Each package must have a separate control script, which must be
executable. The control script is placed in the package directory and is
given the same name as specified in the RUN_SCRIPT and HALT_SCRIPT
parameters in the package ASCII configuration file. The package control
script template contains both the run instructions and the halt
instructions for the package. You can use a single script for both run and
halt operations, or, if you wish, you can create separate scripts.
Use the following procedure to create a control scripts for the sample
package pkg1.
Chapter 3 159
First, generate a control script template:

# cmmakepkg -s /etc/cmcluster/pkg1/control.sh
You may customize the script, as described in the section, “Customizing
the Package Control Script.”
Customizing the Package Control Script

Check the definitions and declarations at the beginning of the control
script using the information in the Package Configuration worksheet.
You need to customize as follows:
• Update the PATH statement to reflect any required paths needed to

start your services.
• If you are using LVM, enter the names of volume groups to be
activated using the VG[] array parameters, and select the
appropriate options for the storage activation command, including
options for mounting and unmounting filesystems, if desired. Do not
use the VXVM_DG[] or CVM_DG[] parameters for LVM volume groups.
• If you are using CVM, enter the names of disk groups to be activated
using the CVM_DG[] array parameters, and select the appropriate
storage activation command, CVM_ACTIVATION_CMD. Do not use the
VG[] or VXVM_DG[] parameters for CVM disk groups.
• If you are using VxVM disk groups without CVM, enter the names of
VxVM disk groups that will be imported using the VXVM_DG[] array
parameters. Enter one disk group per array element. Do not use the
CVM_DG[] or VG[] parameters for VxVM disk groups without CVM.
Also, do not specify an activation command.
• Add the names of logical volumes and file systems that will be
mounted on them.
• If you are using mirrored VxVM disks, specify the mirror recovery
option VXVOL.
• Select the appropriate options for the storage activation command
(not applicable for basic VxVM disk groups), and also include options
for mounting and unmounting filesystems, if desired.
• Specify the filesystem mount retry and unmount count options.
• Define IP subnet and IP address pairs for your package.
• Add service name(s).
160 Chapter 3
• Add service command(s)

• Add a service restart parameter, if desired.
NOTE Use care in defining service run commands. Each run command is
executed by the control script in the following way:
• The cmrunserv command executes each run command and then

monitors the process id of the process created by the run command.
• When the command started by cmrunserv exits, Serviceguard
determines that a failure has occurred and takes appropriate action,
which may include transferring the package to an adoptive node.
• If a run command is a shell script that runs some other command
and then exits, Serviceguard will consider this normal exit as a
failure.
To avoid problems in the execution of control scripts, ensure that each
run command is the name of an actual service and that its process
remains alive until the actual service stops.
If you need to define a set of run and halt operations in addition to the
defaults, create functions for them in the sections under the heading
CUSTOMER DEFINED FUNCTIONS.
Optimizing for Large Numbers of Storage Units

A set of four variables is provided to allow performance improvement
when employing a large number of filesystems or storage groups. For
more detail, see the comments in the control script template. They are:
• CONCURRENT_VGCHANGE_OPERATIONS—defines a number of parallel

LVM volume group activations during package startup as well and
deactivations during package shutdown.
• CONCURRENT_FSCK_OPERATIONS—defines a number of parallel fsck
operations that will be carried out at package startup.
• CONCURRENT_MOUNT_AND_UMOUNT_OPERATIONS—defines a number of
parallel mount operations during package startup and unmount
operations during package shutdown.
Chapter 3 161
Customizing the Control Script for RAC Instances

Use the package control script to perform the following:
• Activation and deactivation of RAC volume groups.

• Startup and shutdown of the RAC instance.
• Monitoring of the RAC instance.
Set RAC environment variables in the package control script to define
the correct execution environment for RAC.
Enter the names of the LVM volume groups you wish to activate in
shared mode in the VG[] array. Use a different array element for each
RAC volume group. (Remember that RAC volume groups must also be
coded in the cluster configuration file using OPS_VOLUME_GROUP
parameters.) Be sure to specify shared activation with the vgchange
command by setting the VGCHANGE parameter as follows:
VGCHANGE="vgchange -a s”
If your disks are mirrored with LVM mirroring on separate physical
paths and you want to override quorum, use the following setting:
VGCHANGE="vgchange -a s -q n”
Enter the names of the CVM disk groups you wish to activate in shared
mode in the CVM_DG[] array. Use a different array element for each RAC
disk group. (Remember that CVM disk groups must also be coded in the
package ASCII configuration file using STORAGE_GROUP parameters.) Be
sure to an appropriate type of shared activation with the CVM activation
command. For example:
CVM_ACTIVATION_CMD="vxdg -g \$DiskGroup set
activation=sharedwrite"
Do not define the RAC instance as a package service. Instead, include the
commands that start up an RAC instance in the
customer_defined_run_commands section of the package control script.
Similarly, you should include the commands that halt an RAC instance
in the customer_defined_halt_commands section of the package control
script.
Define the Oracle monitoring command as a service command, or else
use the special Oracle script provided with the ECM Toolkit.
162 Chapter 3
Using the Command Line to Configure an Oracle RAC Instance

Package
Serviceguard Manager provides a template to configure package
behavior that is specific to an Oracle RAC Instance package. The RAC
Instance package starts the Oracle RAC instance, monitors the Oracle
processes, and stops the RAC instance.
The configuration of the RAC Instance Package make use of the
Enterprise Cluster Master Toolkit (ECMT) to start, monitor, and stop the
Oracle database instance. For details on the use of ECMT, reference
ECMT documentation.
Each Oracle RAC database can have database instance running on all
nodes of a SGeRAC cluster. Therefore, it is not necessary to failover the
database instance to a different SGeRAC node. This is the main
difference between an Oracle RAC Instance Package and a single
instance Oracle package.
Information for Creating the Oracle RAC Instance Package on a

SGeRAC Node Use the following steps to set up the pre-package
configuration on a SGeRAC node:
1. Gather the RAC Instance SID_NAME. If you are using Serviceguard

Manager, this is in the cluster Properties.
Example: SID_NAME=ORACLE_TEST0
For an ORACLE RAC Instance for a two-node cluster, each node
would have an SID_NAME.
2. Gather the RAC Instance package name for each node, which should
be the same as the SID_NAME for each node
Example: ORACLE_TEST0
3. Gather the shared volume group name for the RAC database. In
Serviceguard Manager, see cluster Properties.
Example: /dev/vgora92db
4. Create the Oracle RAC Instance Package directory
/etc/cmcluster/pkg/${SID_NAME}
Example: /etc/cmcluster/pkg/ORACLE_TEST0
Chapter 3 163
5. Copy the Oracle shell script templates from the ECMT default source
directory to the package directory.
# cd /etc/cmcluster/pkg/${SID_NAME}
# cp -p /opt/cmcluster/toolkit/oracle/*
Example:
# cd /etc/cmcluster/pkg/ORACLE_TEST0
# cp -p /opt/cmcluster/toolkit/oracle/*
Edit haoracle.conf as per README
6. Gather the package service name for monitoring Oracle instance
processes. In Serviceguard Manager, this information can be found
under the “Services” tab.
SERVICE_NAME[0]=${SID_NAME}
SERVICE_CMD[0]=”etc/cmcluster/pkg/${SID_NAME}/toolkit.sh
”
SERVICE_RESTART[0]=”-r 2”
Example:
SERVICE_NAME[0]=ORACLE_TEST0
SERVICE_CMD[0]=”/etc/cmcluster/pkg/ORACLE_TEST0/toolkit.
sh”
SERVICE_RESTART[0]=”-r 2”
7. Gather how to start the database using an ECMT script. In
Serviceguard Manager, enter this filename for the control script
start command.
/etc/cmcluster/pkg/${SID_NAME}/toolkit.sh start
Example: /etc/cmcluster/pkg/ORACLE_TEST0/toolkit.sh start
8. Gather how to stop the database using an ECMT script. In
Serviceguard Manager, enter this filename for the control script stop
command.
/etc/cmcluster/pkg/${SID_NAME}/toolkit.sh stop
Example: /etc/cmcluster/pkg/ORACLE_TEST0/toolkit.sh stop
164 Chapter 3
Using Serviceguard Manager to Configure Oracle RAC Instance

Package
The following steps use the information from the example in section 2.2.
It is assumed that the SGeRAC cluster environment is configured and
the ECMT can be used to start the Oracle RAC database instance.
1. Start Serviceguard Manager and Connect to the cluster. Figure 3-1

shows a RAC Instance package for node sg21. The package name is
ORACLE_TEST0
Figure 3-1 Serviceguard Manager display for a RAC Instance package
2. Create the Package.

3. Select the Parameters and select the parameters to edit.
Chapter 3 165
Next select the check box “Enable template(x)” to enable Package

Template for Oracle RAC. The template defaults can be reset with
the “Reset template defaults” push button.
When enabling the template for Oracle RAC, the package can only be
run on one node.
4. Select the Node tab and select the node to run this package.
5. Select Networks tab and add monitored subnet for package.
6. Select Services tab and configure services.
7. Select the Control Script tab and configure parameters.
Configure volume groups and customer defined run/halt functions.
8. Apply the package configuration after filling in the specified
parameters.
Enabling DB Provider Monitoring

To monitor a remote Serviceguard RAC cluster, the entry of the GUI
user name and server name must be in /etc/cmcluster/cmclnodelist
file on all nodes in the cluster to be viewed.
DB Provider Monitoring Using Control Access Policy (CAP) To

monitor a local SGeRAC cluster as a non-root user, the GUI user name,
server name and user role (at least monitor role) must be configured
through CAP.
For a remote cluster, any GUI user name (non-root or root), server name
and user role (at least monitor role) should be configured through CAP.
Please refer Control Access Policy for Serviceguard Commands and API
clients External Specification for details.
166 Chapter 3
Maintenance and Troubleshooting
4 Maintenance and
Troubleshooting
This chapter includes information about carrying out routine

maintenance on an Real Application Cluster configuration. As presented
here, these tasks differ in some details from the similar tasks described
in the Managing Serviceguard user’s guide.
Tasks include:
• Reviewing Cluster and Package States with the cmviewcl Command

• Online Reconfiguration
• Managing the Shared Storage
• Removing Serviceguard Extension for RAC from a System
• Monitoring Hardware
• Adding Disk Hardware
• Replacing Disks
• Replacement of I/O Cards
• Replacement of LAN Cards
Chapter 4 167
Reviewing Cluster and Package States with the cmviewcl Command
Reviewing Cluster and Package States with

the cmviewcl Command
A cluster or its component nodes may be in several different states at
different points in time. Status information for clusters, packages and
other cluster elements is shown in the output of the cmviewcl command
and in some displays in Serviceguard Manager. This section explains the
meaning of many of the common conditions the cluster or package may
be in.
Information about cluster status is stored in the status database, which
is maintained on each individual node in the cluster. You can display
information contained in this database by issuing the cmviewcl
command:
# cmviewcl -v
The command when issued with the -v option displays information
about the whole cluster. See the man page for a detailed description of
other cmviewcl options.
TIP Some commands take longer to complete in large configurations. In

particular, you can expect Serviceguard’s CPU utilization to increase
during cmviewcl -v as the number of packages and services increases.
You can also specify that the output should be formatted as it was in a
specific earlier release by using the -r option indicating the release
format you wish. Example:
# cmviewcl -r A.11.16
See the man page for a detailed description of other cmviewcl options.
Types of Cluster and Package States

different points in time. The following sections describe many of the
common conditions the cluster or package may be in.
168 Chapter 4
Examples of Cluster and Package States

The following is an example of the output generated shown for the
cmviewcl command:
CLUSTER STATUS
cluster_mo up
NODE STATUS STATE

minie up running
Quorum_Server_Status:
NAME STATUS STATE
white up running
Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/0/0/0 lan0
PRIMARY up 0/8/0/0/4/0 lan1
STANDBY up 0/8/0/0/6/0 lan3
NODE STATUS STATE

mo up running
Quorum_Server_Status:
NAME STATUS STATE
white up running
Network_Parameters:
PRIMARY up 0/0/0/0 lan0
PRIMARY up 0/8/0/0/4/0 lan1
STANDBY up 0/8/0/0/6/0 lan3
MULTI_NODE_PACKAGES

NODE_NAME STATUS SWITCHING

minie up enabled
Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Service up 0 0 SG-CFS-vxconfigd
Service up 5 0 SG-CFS-sgcvmd
Service up 5 0 SG-CFS-vxfsckd
Service up 0 0 SG-CFS-cmvxd
Service up 0 0 SG-CFS-cmvxpingd
NODE_NAME STATUS SWITCHING
Chapter 4 169
mo up enabled
Script_Parameters:
Service up 0 0 SG-CFS-vxconfigd
Service up 5 0 SG-CFS-sgcvmd
Service up 5 0 SG-CFS-vxfsckd
Service up 0 0 SG-CFS-cmvxd
Service up 0 0 SG-CFS-cmvxpingd

NODE_NAME STATUS STATE SWITCHING

minie up running enabled
Dependency_Parameters:
DEPENDENCY_NAME SATISFIED
SG-CFS-pkg yes

mo up running enabled
SG-CFS-pkg yes


SG-CFS-DG-1 yes

SG-CFS-DG-1 yes


170 Chapter 4
SG-CFS-DG-1 yes

SG-CFS-DG-1 yes


SG-CFS-DG-1 yes

SG-CFS-DG-1 yes
Types of Cluster and Package States

different points in time. The following sections describe many of the
common conditions the cluster or package may be in.
Cluster Status
The status of a cluster may be one of the following:
• Up. At least one node has a running cluster daemon, and

reconfiguration is not taking place.
• Down. No cluster daemons are running on any cluster node.
• Starting. The cluster is in the process of determining its active
membership. At least one cluster daemon is running.
• Unknown. The node on which the cmviewcl command is issued cannot
communicate with other nodes in the cluster.
Chapter 4 171
Node Status and State

The status of a node is either up (active as a member of the cluster) or
down (inactive in the cluster), depending on whether its cluster daemon
is running or not. Note that a node might be down from the cluster
perspective, but still up and running HP-UX.
A node may also be in one of the following states:
• Failed. A node never sees itself in this state. Other active members
of the cluster will see a node in this state if that node was in an
active cluster, but is no longer, and is not halted.
• Reforming. A node is in this state when the cluster is re-forming.
The node is currently running the protocols which ensure that all
nodes agree to the new membership of an active cluster. If agreement
is reached, the status database is updated to reflect the new cluster
membership.
• Running. A node in this state has completed all required activity for
the last re-formation and is operating normally.
• Halted. A node never sees itself in this state. Other nodes will see it
in this state after the node has gracefully left the active cluster, for
instance with a cmhaltnode command.
• Unknown. A node never sees itself in this state. Other nodes assign a
node this state if it has never been an active cluster member.
Package Status and State

The status of a package can be one of the following:
• Up. The package control script is active.

• Down. The package control script is not active.
• Unknown.
A system multi-node package is up when it is running on all the
activecluster nodes. A multi-node package is up if it is running on any of
itsconfigured nodes.
The state of the package can be one of the following:
• Starting. The start instructions in the control script are being run.
• Running. Services are active and being monitored.
• Halting. The halt instructions in the control script are being run.
172 Chapter 4
Package Switching Attributes

Packages also have the following switching attributes:
• Package Switching. Enabled means that the package can switch to

another node in the event of failure.
• Switching Enabled for a Node. Enabled means that the package
can switch to the referenced node. Disabled means that the package
cannot switch to the specified node until the node is enabled for the
package using the cmmodpkg command.
Every package is marked Enabled or Disabled for each node that is
either a primary or adoptive node for the package.
For multi-node packages, node switching Disabled means the
package cannot start on that node.
Status of Group Membership

The state of the cluster for Oracle RAC is one of the following:
• Up. Services are active and being monitored. The membership

appears in the output of cmviewcl -l group.
• Down. The cluster is halted and GMS services have been stopped. The
membership does not appear in the output of the cmviewcl -l
group.
The following is an example of the group membership output shown in
the cmviewcl command:
# cmviewcl -l group
GROUP MEMBER PID MEMBER_NODE

DGop 1 10394 comanche
0 10499 chinook
DBOP 1 10501 comanche
0 10396 chinook
DAALL_DB 0 10396 comanche
1 10501 chinook
IGOPALL 2 10423 comanche
1 10528 chinook
where the cmviewcl output values are:

GROUP the name of a configured group
Chapter 4 173
MEMBER the ID number of a member of a group

PID the Process ID of the group member
MEMBER_NODE the Node on which the group member is running
Service Status
Services have only status, as follows:
• Up. The service is being monitored.

• Down. The service is not running. It may have halted or failed.
• Uninitialized. The service is included in the package
configuration, but it was not started with a run command in the
control script.
• Unknown.
Network Status
The network interfaces have only status, as follows:
• Up.
• Down.
• Unknown. We cannot determine whether the interface is up or down.
This can happen when the cluster is down. A standby interface has
this status.
Serial Line Status

The serial line has only status, as follows:
• Up. Heartbeats are received over the serial line.

• Down. Heartbeat has not been received over the serial line within 2
times the NODE_TIMEOUT value.
• Recovering. A corrupt message was received on the serial line, and
the line is in the process of resynchronizing.
• Unknown. We cannot determine whether the serial line is up or down.
This can happen when the remote node is down.
174 Chapter 4
Failover and Failback Policies

Packages can be configured with one of two values for the
FAILOVER_POLICY parameter:
• CONFIGURED_NODE. The package fails over to the next node in the

node list in the package configuration file.
• MIN_PACKAGE_NODE. The package fails over to the node in the cluster
with the fewest running packages on it.
Packages can also be configured with one of two values for the
FAILBACK_POLICY parameter:
• AUTOMATIC. With this setting, a package, following a failover, returns

to its primary node when the primary node becomes available again.
• MANUAL. With this setting, a package, following a failover, must be
moved back to its original node by a system administrator.
Failover and failback policies are displayed in the output of the
cmviewcl -v command.
Examples of Cluster and Package States

The following sample output from the cmviewcl -v command shows
status for the cluster in the sample configuration.
Normal Running Status

Everything is running normally; both nodes in a two-node cluster are
running, and each Oracle RAC instance package is running as well. The
only packages running are Oracle RAC instance packages.
CLUSTER STATUS
example up
NODE STATUS STATE
ftsys9 up running
Network_Parameters:
PRIMARY up 56/36.1 lan0
STANDBY up 60/6 lan1
PACKAGE STATUS STATE AUTO_RUN NODE

ops_pkg1 up running disabled ftsys9
Chapter 4 175
Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Start configured_node
Failback manual
Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up enabled ftsys9 (cur
rent)
NODE STATUS STATE

ftsys10 up running
Network_Parameters:
PRIMARY up 28.1 lan0
STANDBY up 32.1 lan1

ops_pkg2 up running disabled ftsys1
0
Policy_Parameters:
Start configured_node
Failback manual
Primary up enabled ftsys10 (cur
rent)
Alternate up enabled ftsys9
Quorum Server Status

If the cluster is using a quorum server for tie-breaking services, the
display shows the server name, state and status following the entry for
each node, as in the following excerpt from the output of cmviewcl -v:
CLUSTER STATUS
example up
NODE STATUS STATE

ftsys9 up running
176 Chapter 4
Quorum Server Status:

NAME STATUS STATE
lp-qs up running
...
NODE STATUS STATE

ftsys10 up running
Quorum Server Status:

NAME STATUS STATE
lp-qs up running
CVM Package Status

If the cluster is using the VERITAS Cluster Volume Manager for disk
storage, the system multi-node package CVM-VxVM-pkg must be running
on all active nodes for applications to be able to access CVM disk groups.
This package is shown in the following output of the cmviewcl command:
CLUSTER STATUS
example up
NODE STATUS STATE

ftsys8 down halted
ftsys9 up running

When you use the -v option, the display shows the system multi-node
package associated with each active node in the cluster, as in the
following:

NODE STATUS STATE

ftsys8 down halted
NODE STATUS STATE

ftsys9 up running
Script_Parameters:
Chapter 4 177

Service up 0 0
VxVM-CVM-pkg.srv
Status After Moving the Package to Another Node

After issuing the following command:
# cmrunpkg -n ftsys9 pkg2
the output of the cmviewcl -v command is as follows:
CLUSTER STATUS
example up
NODE STATUS STATE

ftsys9 up running
Network_Parameters:
STANDBY up 60/6 lan1

pkg1 up running enabled ftsys9
Policy_Parameters:
Failover min_package_node
Failback manual
Script_Parameters:
Service up 0 0 service1
Subnet up 0 0 15.13.168.0
Resource up /example/float
Primary up enabled ftsys9 (current)
Alternate up enabled ftsys10

pkg2 up running disabled ftsys9
178 Chapter 4
Policy_Parameters:
Failback manual
Script_Parameters:
ITEM STATUS NAME MAX_RESTARTS RESTARTS
Service up service2.1 0 0
Subnet up 15.13.168.0 0 0
Primary up enabled ftsys10
Alternate up enabled ftsys9 (current)
NODE STATUS STATE

ftsys10 up running
Network_Parameters:
STANDBY up 32.1 lan1
Now pkg2 is running on node ftsys9. Note that it is still disabled from
switching.
Status After Package Switching is Enabled

The following command changes package status back to Package
Switching Enabled:
# cmmodpkg -e pkg2
The output of the cmviewcl command is now as follows:
CLUSTER STATUS
example up
NODE STATUS STATE

ftsys9 up running

NODE STATUS STATE

ftsys10 up running
Chapter 4 179
Both packages are now running on ftsys9 and pkg2 is enabled for
switching. Ftsys10 is running the daemon and no packages are running
on ftsys10.
Status After Halting a Node

After halting ftsys10, with the following command:
# cmhaltnode ftsys10
the output of cmviewcl is as follows on ftsys9:
CLUSTER STATUS
example up
NODE STATUS STATE

ftsys9 up running

NODE STATUS STATE

ftsys10 down halted
This output is seen on both ftsys9 and ftsys10.
Viewing RS232 Status

If you are using a serial (RS232) line as a heartbeat connection, you will
see a list of configured RS232 device files in the output of the cmviewcl
-v command. The following shows normal running status:
CLUSTER STATUS
example up
NODE STATUS STATE
ftsys9 up running
Network_Parameters:
Serial_Heartbeat:
DEVICE_FILE_NAME STATUS CONNECTED_TO:
/dev/tty0p0 up ftsys10 /dev/tty0p0
180 Chapter 4
NODE STATUS STATE

ftsys10 up running
Network_Parameters:
Serial_Heartbeat:
/dev/tty0p0 up ftsys9 /dev/tty0p0
The following shows status when the serial line is not working:
CLUSTER STATUS
example up
NODE STATUS STATE
ftsys9 up running
Network_Parameters:
Serial_Heartbeat:
/dev/tty0p0 down ftsys10 /dev/tty0p0
NODE STATUS STATE

ftsys10 up running
Network_Parameters:
Serial_Heartbeat:
/dev/tty0p0 down ftsys9 /dev/tty0p0
Viewing Data on Unowned Packages

The following example shows packages that are currently unowned, that
is, not running on any configured node. Information on monitored
resources is provided for each node on which package can run this
information allows you to identify the cause of a failure and decide where
to start the package up again.
Chapter 4 181
UNOWNED_PACKAGES

PKG3 down halted enabled unowned
Policy_Parameters:
Failback automatic
Script_Parameters:
ITEM STATUS NODE_NAME NAME
Resource up manx /resource/random
Subnet up manx 192.8.15.0
Resource up burmese /resource/random
Subnet up burmese 192.8.15.0
Resource up tabby /resource/random
Subnet up tabby 192.8.15.0
Resource up persian /resource/random
Subnet up persian 192.8.15.0
Primary up enabled manx
Alternate up enabled burmese
Alternate up enabled tabby
Alternate up enabled persian
182 Chapter 4
Online Reconfiguration
Online Reconfiguration
The online reconfiguration feature provides a method to make
configuration changes online to a Serviceguard Extension for RAC
(SGeRAC) cluster. Specifically, this provides the ability to add or/and
delete nodes from a running SGeRAC Cluster, and to reconfigure SLVM
VG while it is being accessed by only one node.
Online Node Addition and Deletion

Online node addition enables the addition or deletion of nodes in a
SGeRAC cluster to or from another running cluster.
Node(s) can be added and/or deleted by changing the cluster
configuration. This is done by editing the cluster specification file and
re-applying the configuration to the already running cluster. For
deleting online node(s), the node(s) needs to be halted before deleting
them from the cluster.
Use the following steps for adding a node using online node
reconfiguration:
1. Export the mapfile for the volume groups that needs to be visible in
the new node (vgexport -s -m mapfile -p <sharedvg>).
2. Copy the mapfile to the new node.
3. Import the volume groups into the new node. (vgimport -s -m
mapfile <sharedvg>).
4. Add node to the cluster online- edit the cluster configuration file to
add the node details and run cmapplyconf.
5. Make the new node join the cluster (cmrunnode) and run the
services.
Use the following steps for deleting a node using online node
reconfiguration:
1. Halt the node in the cluster by running cmhaltnode.

2. Edit the cluster configuration file to delete a node(s).
3. Run cmapplyconf.
Chapter 4 183
Managing the Shared Storage
Single Node Online volume Re-Configuration (SNOR)

The SLVM Single Node Online volume Re-configuration (SNOR) feature
provides a method for changing the configuration for an active shared
volume group (VG) in a SGeRAC cluster.
SLVM SNOR allows the reconfiguration of a shared volume group and of
logical and physical volumes in the VG. This is done while keeping the
VG active on a single node in exclusive mode. After the reconfiguration,
the VG can be reactivated in shared mode and later can be activated on
other nodes.
In addition, as the VG is activated on a single node, in exclusive mode ,
the application can be up and running. The volume group changes need
to be performed on the node when the vgchange command is executed.
The command line option to the vgchange command (vgchange -a <e
or s> -x) allows activation mode switches between “shared” and
“exclusive” on a shared VG active in a single node.
Use the following procedures to set up an on-line re-configuration:
1. Identify the shared volume group on which a configuration change is

required, for example vg_shared.
2. Identify one node, for example node1, of the cluster that is running
an application, such as RAC using the shared volume group. The
applications using the volume group, vg_shared, on this node will
remain unaffected during the procedure. The cluster application
needs to be scaled down to a single cluster node, using the volume
group, vg_shared.
3. On all other nodes of the cluster, except node1, deactivate the volume
group using the -n option to vgchange(1M). For example,
# vgchange -a n vg_shared
Ensure the volume group, vg_shared, is now active only on a single
cluster node, node1.
4. Change the activation mode to exclusive on the node1 where the
volume group is active, use the vgchange(1M) command option. For
example,
184 Chapter 4
# vgchange -a e -x vg_shared
NOTE Ensure that none of the mirrored logical volumes in this volume
group have Consistency Recovery set to MWC (refer lvdisplay(1M)).
Changing the mode back to “shared” will not be allowed in that case,
since Mirror Write Cache consistency recovery (MWC) is not valid in
volume groups activated in shared mode.
5. Make the desired configuration change for the volume group on the
node where the volume group is active, run the required command to
change the configuration. For example, to add a mirror copy, use the
following command:
# lvextend -m 2 /dev/vg_shared/lvol1
6. Export the changes to other cluster nodes if required.
If the configuration change required the creation or deletion of a new
logical or physical volume (i.e., any of the following commands were
used - lvcreate(1M), lvreduce(1M), vgextend(1M),
vgreduce(1M), lvsplit(1M), lvmerge(1M) then the following
sequence of steps is required.
a. From the same node, export the mapfile for vg_shared. For
example
# vgexport -s -p -m /tmp/vg_shared.map vg_shared
b. Copy the mapfile thus obtained to all the other nodes of the
cluster.
c. On the other cluster nodes, export vg_shared and re-import it
using the new map file. For example,
# vgexport vg_shared
# mkdir /dev/vg_shared
# mknod /dev/vg_shared/group c 64 0xhh0000
# vgimport -s -m /tmp/vg_shared.map vg_shared
CAUTION If Business Copies, or Business Continuity Volumes (BCs or BCVs)

are in use, then run vgchgid(1M) before starting the procedure.
Chapter 4 185
The vgimport(1M)/vgexport(1M) sequence will not preserve the

order of physical volumes in the /etc/lvmtab file. If the ordering is
significant due to the presence of active-passive devices, or if the
volume group has been configured to maximize throughput by
ordering the paths accordingly, the ordering would need to be
repeated.
7. Change the activation mode back to “shared” on the node in the

cluster where the volume group vg_shared is active. Change the
mode back to shared.
# vgchange -a s -x vg_shared
On the other cluster nodes, activate vg_shared in shared mode
# vgchange -a s vg_shared
8. Backup the changes made to the volume group using vgcfgbackup
on all nodes.
# vgcfgbackup vg_shared
Making LVM Volume Groups Shareable

Normally, volume groups are marked to be activated in “shared” mode
when they are listed with the OPS_VOLUME_GROUP parameter in the
cluster configuration file or in Serviceguard Manager. which occurs when
the configuration is applied. However, in some cases you may want to
manually make a volume group sharable. For example, if you wish to add
a new shared volume group without shutting down the cluster, you can
use the manual method to do it online. However, when convenient, it's a
good practice to bring down the cluster and reconfigure it to include the
new volume group.
1. Use the vgchange command on each node to ensure that the volume
group to be shared is currently inactive on all nodes. Example:
# vgchange -a n /dev/vg_ops
2. On the configuration node, use the vgchange command to make the
volume group shareable by members of the cluster:
# vgchange -S y -c y /dev/vg_ops
186 Chapter 4
This command is issued from the configuration node only, and the
cluster must be running on all nodes for the command to succeed.
Note that both the -S and the -c options are specified. The -S y
option makes the volume group shareable, and the -c y option
causes the cluster id to be written out to all the disks in the volume
group. In effect, this command specifies the cluster to which a node
must belong in order to obtain shared access to the volume group.
Making a Volume Group Unshareable

Use the following steps to unmark a previously marked shared volume
group:
1. Remove the volume group name from the ASCII cluster

configuration file.
2. Enter the following command:
# vgchange -S n -c n /dev/volumegroup
The above example marks the volume group as non-shared and not
associated with a cluster.
Activating an LVM Volume Group in Shared Mode

Activation and deactivation of shared volume groups is normally done
through a control script. If you need to perform activation from the
command line, you can issue the following command from each node to
activate the volume group in shared mode. (The node on which you first
enter the command becomes the server node.)
# vgchange -a s -p /dev/vg_ops
The following message is displayed:
Activated volume group in shared mode.
This node is the Server.
When the same command is entered on the second node, the following
message is displayed:
Activated volume group in shared mode.
This node is a Client.
Chapter 4 187
NOTE Do not share volume groups that are not part of the RAC configuration
unless shared access is controlled.
Deactivating a Shared Volume Group

Issue the following command from each node to deactivate the shared
volume group:
Remember that volume groups remain shareable even when nodes enter
and leave the cluster.
NOTE If you wish to change the capacity of a volume group at a later time, you
must deactivate and unshare the volume group first. If you add disks,
you must specify the appropriate physical volume group name and make
sure the /etc/lvmpvg file is correctly updated on both nodes.
Making Offline Changes to Shared Volume Groups

You may need to change the volume group configuration of RAC shared
logical volumes to add capacity to the data files or to add log files. No
configuration changes are allowed on shared LVM volume groups while
they are activated. The volume group must be deactivated first on all
nodes, and marked as non-shareable. Use the following procedure
(examples assume the volume group is being shared by node 1 and node
2, and they use the volume group vg_ops):
1. Ensure that the Oracle RAC database is not active on either node.
2. From node 2, use the vgchange command to deactivate the volume
group:
3. From node 2, use the vgexport command to export the volume
group:
# vgexport -m /tmp/vg_ops.map.old /dev/vg_ops
This dissociates the volume group from node 2.
188 Chapter 4
4. From node 1, use the vgchange command to deactivate the volume

group:
5. Use the vgchange command to mark the volume group as
unshareable:
# vgchange -S n -c n /dev/vg_ops
6. Prior to making configuration changes, activate the volume group in
normal (non-shared) mode:
# vgchange -a y /dev/vg_ops
7. Use normal LVM commands to make the needed changes. Be sure to
set the raw logical volume device file's owner to oracle and group to
dba, with a mode of 660.
8. Next, still from node 1, deactivate the volume group:
9. Use the vgexport command with the options shown in the example
to create a new map file:
# vgexport -p -m /tmp/vg_ops.map /dev/vg_ops
Make a copy of /etc/lvmpvg in /tmp/lvmpvg, then copy the file to
/tmp/lvmpvg on node 2. Copy the file /tmp/vg_ops.map to node 2.
10. Use the following command to make the volume group shareable by
the entire cluster again:
# vgchange -S y -c y /dev/vg_ops
11. On node 2, issue the following command:
# mkdir /dev/vg_ops
12. Create a control file named group in the directory /dev/vg_ops, as
in the following:
has the format:
0xhh0000
the volume groups that are already configured.
Chapter 4 189
13. Use the vgimport command, specifying the map file you copied from
the configuration node. In the following example, the vgimport
command is issued on the second node for the same volume group
that was modified on the first node:
# vgimport -v -m /tmp/vg_ops.map /dev/vg_ops
/dev/dsk/c0t2d0/dev/dsk/c1t2d0
14. Activate the volume group in shared mode by issuing the following
command on both nodes:
# vgchange -a s -p /dev/vg_ops
Skip this step if you use a package control script to activate and
deactivate the shared volume group as a part of RAC startup and
shutdown.
Adding Additional Shared LVM Volume Groups

To add capacity or to organize your disk resources for ease of
management, you may wish to create additional shared volume groups
for your Oracle RAC databases. If you decide to use additional shared
volume groups, they must conform to the following rules:
• Volume groups should include different PV links to each logical unit

on the disk array.
• Volume group names must be the same on all nodes in the cluster.
• Logical volume names must be the same on all nodes in the cluster.
If you are adding or removing shared LVM volume groups, make sure
that you modify the cluster configuration file and any package control
script that activates and deactiveates the shared LVM volume groups.
Changing the VxVM or CVM Storage Configuration

You can add VxVM disk groups to the cluster configuration while the
cluster is running. To add new CVM disk groups, the cluster must be
running.
If you are creating new CVM disk groups, be sure to determine the
master node on which to do the creation by using the following command:
# vxdctl -c mode
190 Chapter 4
node.
Similarly, you can delete VxVM or CVM disk groups provided they are
not being used by a cluster node at the time.
NOTE For CVM without CFS, if you are adding a disk group to the cluster
configuration, make sure you also modify any package or create the
package control script that imports and deports this disk group. If you
are adding a CVM disk group, be sure to add the STORAGE_GROUP entry
for the disk group to the pacakge ASCII file.
For CVM with CFS, if you are adding a disk group to the cluster
configuration, make sure you also create the corresponding multi-node
package. If you are adding a CVM disk group, be sure to add to the
packages that depend on the CVM disk group the necessary package
dependency.
If you are removing a disk group from the cluster configuration, make
sure that you also modify or delete any package control script that
imports and deports this disk group. If you are removing a CVM disk
group, be sure to remove the STORAGE_GROUP entries for the disk group
from the package ASCII file.
When removig a disk group that is activated and deactivated through a
mulit-node package, make sure to modify or remove any configured
package dependencies to the multi-node package.
Chapter 4 191
Removing Serviceguard Extension for RAC from a System
Removing Serviceguard Extension for RAC

from a System
If you wish to remove a node from Serviceguard Extension for RAC
operation, use the swremove command to delete the software. Note the
following:
• The cluster service should not be running on the node from which
you will be deleting Serviceguard Extension for RAC.
• The node from which you are deleting Serviceguard Extension for
RAC should not be in the cluster configuration.
• If you are removing Serviceguard Extension for RAC from more than
one node, swremove should be issued on one node at a time.
NOTE After removing Serviceguard Extension for RAC, your cluster will still
have Serviceguard installed. For information about removing
Serviceguard, refer to the Managing Serviceguard user’s guide for your
version of the product.
192 Chapter 4
Monitoring Hardware
Monitoring Hardware
Good standard practice in handling a high availability system includes
careful fault monitoring so as to prevent failures if possible or at least to
react to them swiftly when they occur. The following should be monitored
for errors or warnings of all kinds:
• Disks
• CPUs
• Memory
• LAN cards
• Power sources
• All cables
• Disk interface cards
Some monitoring can be done through simple physical inspection, but for
the most comprehensive monitoring, you should examine the system log
file (/var/adm/syslog/syslog.log) periodically for reports on all
configured HA devices. The presence of errors relating to a device will
show the need for maintenance.
Using Event Monitoring Service

Event Monitoring Service (EMS) allows you to configure monitors of
specific devices and system resources. You can direct alerts to an
administrative workstation where operators can be notified of further
action in case of a problem. For example, you could configure a disk
monitor to report when a mirror was lost from a mirrored volume group
being used in a non-RAC package. Refer to the manual Using the Event
Monitoring Service (B7609-90022) for additional information.
Using EMS Hardware Monitors

A set of hardware monitors is available for monitoring and reporting on
memory, CPU, and many other system values. Refer to the EMS
Hardware Monitors User’s Guide (B6191-90020) for additional
information.
Chapter 4 193
Adding Disk Hardware
Adding Disk Hardware

As your system expands, you may need to add disk hardware. This also
means modifying the logical volume structure. Use the following general
procedure:
1. Halt packages.
2. Ensure that the Oracle database is not active on either node.
3. Deactivate and mark as unshareable any shared volume groups.
4. Halt the cluster.
5. Deactivate automatic cluster startup.
6. Shutdown and power off system before installing new hardware.
7. Install the new disk hardware with connections on all nodes.
8. Reboot all nodes.
9. On the configuration node, add the new physical volumes to existing
volume groups, or create new volume groups as needed.
10. Start up the cluster.
11. Make the volume groups shareable, then import each shareable
volume group onto the other nodes in the cluster.
12. Activate the volume groups in shared mode on all nodes.
13. Start up the Oracle RAC instances on all nodes.
14. Activate automatic cluster startup.
NOTE As you add new disks to the system, update the planning worksheets
(described in Appendix B, “Blank Planning Worksheets,” so as to record
the exact configuration you are using.
194 Chapter 4
Replacing Disks
Replacing Disks
The procedure for replacing a faulty disk mechanism depends on the
type of disk configuration you are using and on the type of Volume
Manager software. For a description of replacement procedures using
VERITAS VxVM or CVM, refer to the chapter on “Administering
Hot-Relocation” in the VERITAS Volume Manager Administrator’s
Guide. Additional information is found in the VERITAS Volume
Manager Troubleshooting Guide.
The following paragraphs describe how to replace disks that are
configured with LVM. Separate descriptions are provided for replacing a
disk in an array and replacing a disk in a high availability enclosure.
Replacing a Mechanism in a Disk Array Configured

with LVM
With any HA disk array configured in RAID 1 or RAID 5, refer to the
array’s documentation for instruction on how to replace a faulty
mechanism. After the replacement, the device itself automatically
rebuilds the missing data on the new disk. No LVM activity is needed.
This process is known as hot swapping the disk.
NOTE If your LVM installation requires online replacement of disk

mechanisms, the use of disk arrays may be required, because software
mirroring of JBODs with MirrorDisk/UX does not permit hot swapping
for disks that are activated in shared mode.
Replacing a Mechanism in an HA Enclosure

Configured with Exclusive LVM
Non-Oracle data that is used by packages may be configured in volume
groups that use exclusive (one-node-at-a-time) activation. If you are
using exclusive activation and software mirroring with MirrorDisk/UX
and the mirrored disks are mounted in a high availability disk enclosure,
you can use the following steps to hot plug a disk mechanism:
Chapter 4 195
Replacing Disks
1. Identify the physical volume name of the failed disk and the name of
the volume group in which it was configured. In the following
examples, the volume group name is shown as /dev/vg_sg01 and
the physical volume name is shown as /dev/c2t3d0. Substitute the
volume group and physical volume names that are correct for your
system.
2. Identify the names of any logical volumes that have extents defined
on the failed physical volume.
3. On the node on which the volume group is currently activated, use
the following command for each logical volume that has extents on the
failed physical volume:
# lvreduce -m 0 /dev/vg_sg01/lvolname /dev/dsk/c2t3d0
4. At this point, remove the failed disk and insert a new one. The new
disk will have the same HP-UX device name as the old one.
5. On the node from which you issued the lvreduce command, issue
the following command to restore the volume group configuration
data to the newly inserted disk:
# vgcfgrestore /dev/vg_sg01 /dev/dsk/c2t3d0
6. Issue the following command to extend the logical volume to the
newly inserted disk:
# lvextend -m 1 /dev/vg_sg01 /dev/dsk/c2t3d0
7. Finally, use the lvsync command for each logical volume that has
extents on the failed physical volume. This synchronizes the extents
of the new disk with the extents of the other mirror.
# lvsync /dev/vg_sg01/lvolname
Online Replacement of a Mechanism in an HA

Enclosure Configured with Shared LVM (SLVM)
If you are using software mirroring for shared concurrent activation of
Oracle RAC data with MirrorDisk/UX and the mirrored disks are
mounted in a high availability disk enclosure, use the following steps to
carry out online replacement:
1. Make a note of the physical volume name of the failed mechanism

(for example, /dev/dsk/c2t3d0).
196 Chapter 4
Replacing Disks
2. Halt all the applications using the SLVM VG on all the nodes but
one.
3. Re-activate the volume group in exclusive mode on all nodes of the
cluster:
# vgchange -a e -x <slvm vg>
4. Reconfigure the volume:
vgextend, lvextend, disk addition, etc
5. Activate the volume group to shared mode:
# vgchange -a s -x <slvm vg>
Offline Replacement of a Mechanism in an HA

Enclosure Configured with Shared LVM (SLVM)
Hot plugging of disks is not supported for Oracle RAC data, which is
configured in volume groups with Shared LVM (SLVM). If you need this
capability, you should use disk arrays for your Oracle RAC data.
If you are using software mirroring for shared concurrent activation of
Oracle RAC data with MirrorDisk/UX and the mirrored disks are
mounted in a high availability disk enclosure, use the following steps to
carry out offline replacement:
1. Make a note of the physical volume name of the failed mechanism

(for example, /dev/dsk/c2t3d0).
2. Deactivate the volume group on all nodes of the cluster:
# vgchange -a n vg_ops
3. Replace the bad disk mechanism with a good one.
4. From one node, initialize the volume group information on the good
mechanism using vgcfgrestore(1M), specifying the name of the
volume group and the name of the physical volume that is being
replaced:
# vgcfgrestore /dev/vg_ops /dev/dsk/c2t3d0
5. Activate the volume group on one node in exclusive mode then
deactivate the volume group:
# vgchange -a e vg_ops
Chapter 4 197
Replacing Disks
This will synchronize the stale logical volume mirrors. This step can
be time-consuming, depending on hardware characteristics and the
amount of data.
6. Deactivate the volume group:
# vgchange -a n vg_ops
7. Activate the volume group on all the nodes in shared mode using
vgchange - a s:
# vgchange -a s vg_ops
Replacing a Lock Disk

Replacing a failed lock disk mechanism is the same as replacing a data
disk. If you are using a dedicated lock disk (one with no user data on it),
then you need to issue only one LVM command:
# vgcfgrestore /dev/vg_lock /dev/dsk/c2t1d0
After doing this, wait at least an hour, then review the syslog file for a
message showing that the lock disk is healthy again.
On-line Hardware Maintenance with In-line SCSI

Terminator
Serviceguard allows on-line SCSI disk controller hardware repairs to all
cluster nodes if you use HP’s in-line terminator (C2980A) on nodes
connected to the end of the shared FW/SCSI bus. The in-line terminator
cable is a 0.5 meter extension cable with the terminator on the male end,
which connects to the controller card for an external bus. The in-line
terminator is used instead of the termination pack that is attached to the
controller card and makes it possible to physically disconnect the node
from the end of the F/W SCSI bus without breaking the bus's
termination. (Nodes attached to the middle of a bus using a Y cable also
can be detached from the bus without harm.) When using in-line
terminators and Y cables, ensure that all orange-socketed termination
packs are removed from the controller cards.
198 Chapter 4
Replacing Disks
NOTE You cannot use inline terminators with internal FW/SCSI buses on D
and K series systems, and you cannot use the inline terminator with
single-ended SCSI buses. You must not use an inline terminator to
connect a node to a Y cable.
Figure 4-1 shows a three-node cluster with two F/W SCSI buses. The
solid line and the dotted line represent different buses, both of which
have inline terminators attached to nodes 1 and 3. Y cables are also
shown attached to node 2.
Figure 4-1 F/W SCSI Buses with In-line Terminators
The use of in-line SCSI terminators allows you to do hardware

maintenance on a given node by temporarily moving its packages to
another node and then halting the original node while its hardware is
serviced. Following the replacement, the packages can be moved back to
the original node.
Chapter 4 199
Replacing Disks
Use the following procedure to disconnect a node that is attached to the

bus with an in-line SCSI terminator or with a Y cable:
1. Move any packages on the node that requires maintenance to a

different node.
2. Halt the node that requires maintenance. The cluster will re-form,
and activity will continue on other nodes. Packages on the halted
node will switch to other available nodes if they are configured to
switch.
3. Disconnect the power to the node.
4. Disconnect the node from the in-line terminator cable or Y cable if
necessary. The other nodes accessing the bus will encounter no
problems as long as the in-line terminator or Y cable remains
connected to the bus.
5. Replace or upgrade hardware on the node, as needed.
6. Reconnect the node to the in-line terminator cable or Y cable if
necessary.
7. Reconnect power and reboot the node. If AUTOSTART_CMCLD is set to 1
in the /etc/rc.config.d/cmcluster file, the node will rejoin the
cluster.
8. If necessary, move packages back to the node from their alternate
locations and restart them.
200 Chapter 4
Replacement of I/O Cards
Replacement of I/O Cards

After an I/O card failure, you can replace the card using the following
steps. It is not necessary to bring the cluster down to do this if you are
using SCSI inline terminators or Y cables at each node.
1. Halt the node by using Serviceguard Manager or the cmhaltnode

command. Packages should fail over normally to other nodes.
2. Remove the I/O cable from the card. With SCSI inline terminators,
this can be done without affecting the disks or other nodes on the
bus.
3. Using SAM, select the option to do an on-line replacement of an I/O
card.
4. Remove the defective I/O card.
5. Install the new card. The new card must be exactly the same card
type, and it must be installed in the same slot as the card you
removed.
6. In SAM, select the option to attach the new I/O card.
7. Add the node back into the cluster by using Serviceguard Manager or
the cmrunnode command.
Chapter 4 201
Replacement of LAN Cards

If you have a LAN card failure, which requires the LAN card to be
replaced, you can replace it on-line or off-line depending on the type of
hardware and operating system you are running. It is not necessary to
bring the cluster down to do this.
Off-Line Replacement
The following steps show how to replace a LAN card off-line. These steps
apply to both HP-UX 11.0 and 11i:
1. Halt the node by using the cmhaltnode command.

2. Shut down the system using /etc/shutdown, then power down the
system.
3. Remove the defective LAN card.
4. Install the new LAN card. The new card must be exactly the same
card type, and it must be installed in the same slot as the card you
removed.
5. Power up the system.
6. If necessary, add the node back into the cluster by using the
cmrunnode command. (You can omit this step if the node is
configured to join the cluster automatically.)
On-Line Replacement
If your system hardware supports hotswap I/O cards, and if the system is
running HP-UX 11i (B.11.11 or later), you have the option of replacing
the defective LAN card on-line. This will significantly improve the
overall availability of the system. To do this, follow the steps provided in
the section “How to On-line Replace (OLR) a PCI Card Using SAM” in
the document Configuring HP-UX for Peripherals. The OLR procedure
also requires that the new card must be exactly the same card type as
the card you removed to avoid improper operation of the network driver.
Serviceguard will automatically recover the LAN card once it has been
replaced and reconnected to the network.
202 Chapter 4
After Replacing the Card

After the on-line or off-line replacement of LAN cards has been done,
Serviceguard will detect that the MAC address (LLA) of the card has
changed from the value stored in the cluster binary configuration file,
and it will notify the other nodes in the cluster of the new MAC address.
The cluster will operate normally after this.
It is also recommended that you update the new MAC address in the
cluster binary configuration file by re-applying the cluster configuration.
Use the following steps for on-line reconfiguration:
1. Use the cmgetconf command to obtain a fresh ASCII configuration

file, as follows:
# cmgetconf config.ascii
2. Use the cmapplyconf command to apply the configuration and copy
the new binary file to all cluster nodes:
# cmapplyconf -C config.ascii
This procedure updates the binary file with the new MAC address and
thus avoids data inconsistency between the outputs of the cmviewconcl
and lanscan commands.
Chapter 4 203
Monitoring RAC Instances
Monitoring RAC Instances

The DB Provider provides the capability to monitor RAC databases. RBA
(Role Based Access) enables a non-root user to have the capability to
monitor RAC instances using Serviceguard Manager.
204 Chapter 4
Software Upgrades
A Software Upgrades
Serviceguard Extension for RAC (SGeRAC) software upgrades can be

done in the two following ways:
• rolling upgrade
• non-rolling upgrade
Instead of an upgrade, moving to a new version can be done with:
• migration with cold install

Rolling upgrade is a feature of SGeRAC that allows you to perform a
software upgrade on a given node without bringing down the entire
cluster. SGeRAC supports rolling upgrades on version A.11.15 and later,
and requires all nodes to be running on the same operating system
revision and architecture.
Non-rolling upgrade allows you to perform a software upgrade from
any previous revision to any higher revision or between operating system
versions but requires halting the entire cluster.
The rolling and non-rolling upgrade processes can also be used any time
one system needs to be taken offline for hardware maintenance or patch
installations. Until the upgrade process is complete on all nodes, you
cannot change the cluster configuration files, and you will not be able to
use any of the new features of the Serviceguard/SGeRAC release.
There may be circumstances when, instead of doing an upgrade, you
prefer to do a migration with cold install. The cold install process
erases the pre-existing operating system and data and then installs the
new operating system and software; you must then restore the data. The
advantage of migrating with cold install is that the software can be
installed without regard for the software currently on the system or
concern for cleaning up old software.
A significant factor when deciding to either do an upgrade or cold install
is overall system downtime. A rolling upgrade will cause the least
downtime. This is because only one node in the cluster is down at any
one time. A non-rolling upgrade may require more down time, because
the entire cluster has to be brought down during the upgrade process.
Appendix A 205
Software Upgrades
One advantage of both rolling and non-rolling upgrades versus cold

install is that upgrades retain the pre-existing operating system,
software and data. Conversely, the cold install process erases the
pre-existing system; you must re-install the operating system, software
and data. For these reasons, a cold install may require more downtime.
The sections in this appendix are as follows:
• Rolling Software Upgrades
— Steps for Rolling Upgrades

— Example of Rolling Upgrade
— Limitations of Rolling Upgrades
• Non-Rolling Software Upgrades
— Steps for Non-Rolling Upgrades

— Limitations of Non-Rolling Upgrades
• Migrating a SGeRAC Cluster with Cold Install
206 Appendix A
Software Upgrades
Rolling Software Upgrades

SGeRAC version A.11.15 and later allow you to roll forward to any
higher revision provided all of the following conditions are met:
• The upgrade must be done on systems of the same architecture (HP

9000 or Integrity Servers).
• All nodes in the cluster must be running on the same version of
HP-UX.
• Each node must be running a version of HP-UX that supports the
new SGeRAC version.
• Each node must be running a version of Serviceguard that supports
the new SGeRAC version.
For more information on support, compatibility, and features for
SGeRAC, refer to the Serviceguard Compatibility and Feature Matrix,
located at http://docs.hp.com -> High Availability -> Serviceguard
Extension for RAC.
Steps for Rolling Upgrades

Use the following steps when performing a rolling SGeRAC software
upgrade:
1. Halt Oracle (RAC, CRS, Clusterware, OPS) software on the local

node (if running).
2. Halt Serviceguard/SGeRAC on the local node by issuing the
Serviceguard cmhaltnode command.
3. Edit the /etc/rc.config.d/cmcluster file to include the following
line:
AUTOSTART_CMCLD = 0
4. Upgrade the node to the new Serviceguard and SGeRAC release.
(SGeRAC requires the compatible version of Serviceguard.)
5. Edit the /etc/rc.config.d/cmcluster file, on the local node, to
include the following line:
AUTOSTART_CMCLD = 1
Appendix A 207
Software Upgrades
NOTE It is optional to set this parameter to “1”. If you want the node to join
the cluster at boot time, set this parameter to “1”, otherwise set it to
“0”.
6. Restart the cluster on the upgraded node (if desired). You can do this
in Serviceguard Manager, or from the command line, issue the
Serviceguard cmrunnode command.
7. Restart Oracle (RAC, CRS, Clusterware, OPS) software on the local
node.
8. Repeat steps 1-7 on the other nodes, one node at a time until all
nodes have been upgraded.
NOTE Be sure to plan sufficient system capacity to allow moving the

packages from node to node during the upgrade process to maintain
optimum performance.
If a cluster fails before the rolling upgrade is complete (perhaps because

of a catastrophic power failure), the cluster could be restarted by
entering the cmruncl command from a node which has been upgraded to
the latest revision of the software.
Keeping Kernels Consistent

If you change kernel parameters or perform network tuning with ndd as
part of doing a rolling upgrade, be sure to change the parameters to the
same values on all nodes that can run the same packages in a failover
scenario. The ndd command allows the examination and modification of
several tunable parameters that affect networking operation and
behavior.
208 Appendix A
Software Upgrades
Example of Rolling Upgrade

The following example shows a simple rolling upgrade on two nodes,
each running standard Serviceguard and RAC instance packages, as
shown in Figure A-1. (This and the following figures show the starting
point of the upgrade as SGeRAC A.11.15 for illustration only. A roll to
SGeRAC version A.11.16 is shown.)
SGeRAC rolling upgrade requires the same operating system version on
all nodes. The example assumes all nodes are running HP-UX 11i v2. For
your systems, substitute the actual release numbers of your rolling
upgrade path.
NOTE While you are performing a rolling upgrade, warning messages may
appear while the node is determining what version of software is
running. This is a normal occurrence and not a cause for concern.
Figure A-1 Running Cluster Before Rolling Upgrade
Appendix A 209
Software Upgrades
Step 1.
1. Halt Oracle (RAC, CRS, Clusterware, OPS) software on node 1.

2. Halt node 1. This will cause the node’s packages to start up on an
adoptive node. You can do this in Serviceguard Manager, or from the
command line issue the following:
# cmhaltnode -f node1
This will cause the failover package to be halted cleanly and moved to
node 2. The Serviceguard daemon on node 1 is halted, and the result is
shown in Figure A-2.
Figure A-2 Running Cluster with Packages Moved to Node 2
210 Appendix A
Software Upgrades
Step 2.
Upgrade node 1 and install the new version of Serviceguard and
SGeRAC (A.11.16), as shown in Figure A-3.
NOTE If you install Serviceguard and SGeRAC separately, Serviceguard must

be installed before installing SGeRAC.
Figure A-3 Node 1 Upgraded to SG/SGeRAC 11.16
Appendix A 211
Software Upgrades
Step 3.
1. Restart the cluster on the upgraded node (node 1) (if desired). You
can do this in Serviceguard Manager, or from the command line issue
the following:
# cmrunnode node1
2. At this point, different versions of the Serviceguard daemon (cmcld)
are running on the two nodes, as shown in Figure A-4.
3. Start Oracle (RAC, CRS, Clusterware, OPS) software on node 1.
Figure A-4 Node 1 Rejoining the Cluster
212 Appendix A
Software Upgrades
Step 4.
1. Halt Oracle (RAC, CRS, Clusterware, OPS) software on node 2.

2. Halt node 2. You can do this in Serviceguard Manager, or from the
command line issue the following:
# cmhaltnode -f node2
This causes both packages to move to node 1; see Figure A-5.
3. Upgrade node 2 to Serviceguard and SGeRAC (A.11.16) as shown in
Figure A-5.
4. When upgrading is finished, enter the following command on node 2
to restart the cluster on node 2:
# cmrunnode node2
5. Start Oracle (RAC, CRS, Clusterware, OPS) software on node 2.
Figure A-5 Running Cluster with Packages Moved to Node 1
Appendix A 213
Software Upgrades
Step 5.
Move PKG2 back to its original node. Use the following commands:
# cmhaltpkg pkg2
# cmrunpkg -n node2 pkg2
# cmmodpkg -e pkg2
The cmmodpkg command re-enables switching of the package, which is
disabled by the cmhaltpkg command. The final running cluster is shown
in Figure A-6.
Figure A-6 Running Cluster After Upgrades
214 Appendix A
Software Upgrades
Limitations of Rolling Upgrades

The following limitations apply to rolling upgrades:
• During a rolling upgrade, you should issue Serviceguard/SGeRAC

commands (other than cmrunnode and cmhaltnode) only on a node
containing the latest revision of the software. Performing tasks on a
node containing an earlier revision of the software will not work or
will cause inconsistent results.
• You cannot modify the cluster or package configuration until the
upgrade is complete. Also, you cannot modify the hardware
configuration—including the cluster’s network
configuration—during a rolling upgrade. This means that you must
upgrade all nodes to the new release before you can modify the
configuration file and copy it to all nodes.
• The new features of the Serviceguard/SGeRAC release may not work
until all nodes have been upgraded.
• Binary configuration files may be incompatible between releases of
Serviceguard/SGeRAC. Do not manually copy configuration files
between nodes.
• Within a Serviceguard/SGeRAC cluster, no more than two versions of
Serviceguard and SGeRAC can be running while the rolling upgrade
is in progress.
• You can perform a rolling upgrade only on a configuration that has
not been modified since the last time the cluster was started.
• Rolling upgrades are not intended as a means of using mixed
releases of Serviceguard and SGeRAC within the same cluster.
SGeRAC requires the compatible version of Serviceguard. Upgrade
all cluster nodes as quickly as possible to the new release level.
For more information on support, compatibility, and features for
SGeRAC, refer to the Serviceguard Compatibility and Feature
Matrix, located at http://docs.hp.com -> High Availability ->
• You cannot delete Serviceguard/SGeRAC software (via swremove)
from a node while the cluster is in the process of a rolling upgrade.
Appendix A 215
Software Upgrades
Non-Rolling Software Upgrades

A non-rolling upgrade allows you to perform a software upgrade from
any previous revision to any higher revision or between operating system
versions. For example, you may do a non-rolling upgrade from SGeRAC
A.11.14 on HP-UX 11i v1 to A.11.16 on HP-UX 11i v2, given both are
running the same architecture.
The cluster cannot be running during a non-rolling upgrade, therefore it
is necessary to halt the entire cluster in order to perform the upgrade.
The next section describes the steps for doing a non-rolling software
upgrade.
216 Appendix A
Software Upgrades
Steps for Non-Rolling Upgrades

Use the following steps for a non-rolling software upgrade:
1. Halt Oracle (RAC, CRS, Clusterware, OPS) software on all nodes in

the cluster.
2. Halt all nodes in the cluster.
# cmhaltcl -f
3. If necessary, upgrade all the nodes in the cluster to the new HP-UX
release.
4. Upgrade all the nodes in the cluster to the new
Serviceguard/SGeRAC release.
5. Restart the cluster. Use the following command:
# cmruncl
6. If necessary, upgrade all the nodes in the cluster to the new Oracle
(RAC, CRS, Clusterware, OPS) software release.
7. Restart Oracle (RAC, CRS, Clusterware, OPS) software on all nodes
in the cluster and configure the Serviceguard/SGeRAC packages and
Oracle as needed.
Appendix A 217
Software Upgrades
Limitations of Non-Rolling Upgrades

The following limitations apply to non-rolling upgrades:
• Binary configuration files may be incompatible between releases of

Serviceguard. Do not manually copy configuration files between
nodes.
• It is necessary to halt the entire cluster when performing a
non-rolling upgrade.
218 Appendix A
Software Upgrades
Migrating a SGeRAC Cluster with Cold Install

There may be circumstances when you prefer a cold install of the HP-UX
operating system rather than an upgrade. The cold install process erases
the pre-existing operating system and data and then installs the new
operating system and software; you must then restore the data.
CAUTION The cold install process erases the pre-existing software, operating
system, and data. If you want to retain any existing software, make sure
to back up that software before migrating.
Use the following process as a checklist to prepare the migration:
1. Back up the required data, including databases, user and application

data, volume group configurations, etc.
2. Halt cluster applications, including RAC and then halt the cluster.
3. Do a cold install of the HP-UX operating system. For more
information on the cold install process, see the HP-UX Installation
and Update Guide located at http://docs.hp.com -> By OS Release
-> Installing and Updating
4. Install additional required software that did not come with your
version of HP-UX OE.
5. Install a Serviceguard/SGeRAC version that is compatible with the
new HP-UX operating system version. For more information on
support, compatibility, and features for SGeRAC, refer to the
Serviceguard Compatibility and Feature Matrix, located at
http://docs.hp.com -> High Availability -> Serviceguard
Extension for RAC
6. Recreate any user accounts needed for the cluster applications.
7. Recreate the network and storage configurations (that is, set up
stationary IP addresses and create LVM volume groups and/or CVM
disk groups required for the cluster).
8. Recreate the SGeRAC cluster.
9. Restart the cluster.
10. Reinstall the cluster applications, such as RAC.
11. Restore the data.
Appendix A 219
Software Upgrades
220 Appendix A
Blank Planning Worksheets
B Blank Planning Worksheets
This appendix reprints blank planning worksheets used in preparing the

RAC cluster. You can duplicate any of these worksheets that you find
useful and fill them in as a part of the planning process.
Appendix B 221
LVM Volume Group and Physical Volume Worksheet
LVM Volume Group and Physical Volume

Worksheet
VG and PHYSICAL VOLUME WORKSHEET Page ___ of ____
==========================================================================
Volume Group Name: ______________________________________________________

PV Link 1 PV Link2
Physical Volume Name:_____________________________________________________
Physical Volume Name: ____________________________________________________
Volume Group Name: _______________________________________________________
PV Link 1 PV Link2
Physical Volume Name: _____________________________________________________
Physical Volume Name:______________________________________________________
222 Appendix B
VxVM Disk Group and Disk Worksheet
VxVM Disk Group and Disk Worksheet

DISK GROUP WORKSHEET Page ___ of ____
===========================================================================
Disk Group Name: __________________________________________________________
Disk Group Name: __________________________________________________________
Appendix B 223
Oracle Logical Volume Worksheet
Oracle Logical Volume Worksheet

NAME SIZE
Oracle Control File 1:

_____________________________________________________

_____________________________________________________

_____________________________________________________
Instance 1 Redo Log 1: _____________________________________________________
Instance 1 Redo Log: _____________________________________________________
Data: System _____________________________________________________
Data: Rollback _____________________________________________________
Data: Temp _____________________________________________________
Data: Users _____________________________________________________
Data: Tools _____________________________________________________
224 Appendix B
Index
A D
activation of volume groups deactivation of volume groups, 188
in shared mode, 187 deciding when and where to run packages, 24
adding packages on a running cluster, 159 deleting from the cluster, 70
administration deleting nodes while the cluster is running,
cluster and package states, 168 190
array demo database
replacing a faulty mechanism, 195, 196, 197 files, 55, 80, 119, 145
AUTO_RUN parameter, 157 disk
AUTO_START_TIMEOUT choosing for volume groups, 49, 113
in sample configuration file, 60, 124 disk arrays
creating logical volumes, 53, 117
B disk storage
creating the infrastructure with CVM, 73,
building a cluster 136
CVM infrastructure, 73, 136
disks
building an RAC cluster replacing, 195
displaying the logical volume
infrastructure, 57, 121
E
logical volume infrastructure, 48, 112
building logical volumes eight-node cluster with disk array
for RAC, 54, 118 figure, 31
EMS
C for preventive monitoring, 193
enclosure for disks
CFS, 65, 70 replacing a faulty mechanism, 195
creating storage infrastructure, 129 Event Monitoring Service
deleting from the cluster, 134 in troubleshooting, 193
cluster exporting
state, 175 shared volume group data, 57, 121
status options, 171 exporting files
cluster configuration file, 60, 124 LVM commands, 57, 121
cluster node extended distance cluster
startup and shutdown OPS instances, 157 building, 32
cluster volume group
creating physical volumes, 49, 113
F
CLUSTER_NAME (cluster name)
in sample configuration file, 60, 124 figures
control script eight-node cluster with EMC disk array, 31
creating with commands, 159 node 1 rejoining the cluster, 212
creating with SAM, 159 node 1 upgraded to HP-UX 111.00, 211
in package configuration, 159 running cluster after upgrades, 214
starting OPS instances, 162 running cluster before rolling upgrade, 209
creating a SGeRAC Cluster, 65 running cluster with packages moved to
creating a storage infrastructure, 65 node 1, 213
CVM running cluster with packages moved to
creating a storage infrastructure, 73, 136 node 2, 210
use of the VxVM-CVM-pkg, 77, 140 FIRST_CLUSTER_LOCK_PV
CVM_ACTIVATION_CMD in sample configuration file, 60, 124
in package control script, 161 FIRST_CLUSTER_LOCK_VG
CVM_DG in sample configuration file, 60, 124
in package control script, 161 FS
225
Index
in sample package control script, 161 using to obtain a list of disks, 49, 113
FS_MOUNT_OPT LV
in sample package control script, 161 in sample package control script, 161
LVM
G creating on disk arrays, 53, 117
LVM commands
GMS
group membership services, 23 exporting files, 57, 121
group membership services
define, 23 M
maintaining a RAC cluster, 167
H maintenance
adding disk hardware, 194
hardware
making changes to shared volume groups,
adding disks, 194
188
monitoring, 193
monitoring hardware, 193
heartbeat subnet address
parameter in cluster manager
configuration, 47, 110 N
HEARTBEAT_INTERVAL network
in sample configuration file, 60, 124 status, 174
HEARTBEAT_IP NETWORK_INTERFACE
in sample configuration file, 60, 124 in sample configuration file, 60, 124
high availability cluster NETWORK_POLLING_INTERVAL
defined, 16 (network polling interval)
in sample configuration file, 60, 124
I node
halting status, 180
in-line terminator in an RAC cluster, 16
permitting online hardware maintenance,
status and state, 172
198
NODE_FAILFAST_ENABLED parameter,
installing 157
Oracle RAC, 59, 123 NODE_TIMEOUT (heartbeat timeout)
installing software in sample configuration file, 60, 124
Serviceguard Extension for RAC, 46, 109
IP
in sample package control script, 161 O
IP address online hardware maintenance
switching, 27 by means of in-line SCSI terminators, 198
Online node addition and deletion, 183
L Online reconfiguration, 183
OPS
lock disk control scripts for starting instances, 162
replacing a faulty mechanism, 198 packages to access database, 158
logical volumes startup and shutdown instances, 157
blank planning worksheet, 223, 224 startup and shutdown volume groups, 156
creating, 54, 118 OPS cluster
creating for a cluster, 50, 79, 114, 142, 143 starting up with scripts, 156
creating the infrastructure, 48, 112 opsctl.ctl
disk arrays, 53, 117 Oracle demo database files, 55, 80, 119, 145
filled in planning worksheet, 42, 44, 102, opslog.log
106 Oracle demo database files, 55, 80, 119, 145
lssf
226
Index
optimizing packages for large numbers of worksheets for physical volume planning,
storage units, 161 222
Oracle planning worksheets
demo database files, 55, 80, 119, 145 blanks, 221
Oracle 10 RAC point to point connections to storage devices,
installing binaries, 89 30
Oracle 10g RAC PVG-strict mirroring
introducing, 33 creating volume groups with, 49, 113
Oracle 9i RAC
installing, 148 Q
introducing, 101
quorum server
Oracle Disk Manager status and state, 176
configuring, 92
Oracle Parallel Server
starting up instances, 156 R
Oracle RAC RAC
installing, 59, 123 group membership services, 23
Oracle10g overview of configuration, 16
installing, 88 status, 173
RAC cluster
P defined, 16
package removing packages on a running cluster, 159
basic concepts, 17, 18 removing Serviceguard Extension for RAC
from a system,, 192
moving status, 178
replacing disks, 195
state, 175 rollback.dbf
status and state, 172 Oracle demo database files, 55, 56, 81, 119,
switching status, 179 145
package configuration rolling software upgrades
service name parameter, 47, 110 example, 209
writing the package control script, 159 steps, 207, 217
package control script rolling upgrade
generating with commands, 159 limitations, 215, 218
packages RS232 status, viewing, 180
accessing OPS database, 158 running cluster
deciding where and when to run, 24 adding or removing packages, 159
launching OPS instances, 157
startup and shutdown volume groups, 156 S
parameter
AUTO_RUN, 157 serial line
status, 174
NODE_FAILFAST_ENABLED, 157
service
performance
status, 174
optimizing packages for large numbers of
storage units, 161 service name
parameter in package configuration, 47, 110
physical volumes SERVICE_CMD
creating for clusters, 49, 113
in sample package control script, 161
filled in planning worksheet, 222 SERVICE_NAME
planning in sample package control script, 161
worksheets for logical volume planning, 42,
parameter in package configuration, 47, 110
44, 102, 106
SERVICE_RESTART
227
Index
Serviceguard Extension for RAC V
installing, 46, 109 VG
introducing, 15 in sample package control script, 161
shared mode VGCHANGE
activation of volume groups, 187 in package control script, 161
deactivation of volume groups, 188 viewing RS232 status, 180
shared volume groups volume group
making volume groups shareable, 186 creating for a cluster, 49, 113
sharing volume groups, 57, 121 creating physical volumes for clusters, 49,
SLVM 113
making volume groups shareable, 186 volume groups
SNOR adding shared volume groups, 190
configuration, 184 displaying for RAC, 57, 121
software upgrades, 205 exporting to other nodes, 57, 121
state making changes to shared volume groups,
cluster, 175
188
node, 172
making shareable, 186
of cluster and package, 168
making unshareable, 187
package, 172, 175
status OPS startup and shutdown, 156
cluster, 171 VOLUME_GROUP
in sample configuration file, 60, 124
halting node, 180
VXVM_DG
moving package, 178
in package control script, 161
network, 174 VxVM-CVM-pkg, 77, 140
node, 172
normal running RAC, 176
W
of cluster and package, 168
package, 172 worksheet
RAC, 173 logical volume planning, 42, 44, 102, 106
serial line, 174 worksheets
physical volume planning, 222
service, 174
worksheets for planning
switching package, 179
blanks, 221
SUBNET
switching IP addresses, 27
system multi-node package
used with CVM, 77, 140
system.dbf
Oracle demo database files, 55, 81
T
temp.dbf
Oracle demo database files, 55, 81, 119, 145
tools.dbf
Oracle demo database files, 119, 145
troubleshooting
monitoring hardware, 193
replacing disks, 195
228

Ora Clu Hpux

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Ora Clu Hpux

Загружено:

Авторское право:

Доступные форматы

Using Serviceguard Extension

Manufacturing Part Number : T1859-90038

© Copyright 2006 Hewlett-Packard Development Company, L.P. All rights reserved.

2. Serviceguard Configuration for Oracle 10g RAC

3. Serviceguard Configuration for Oracle 9i RAC

4. Maintenance and Troubleshooting

B. Blank Planning Worksheets

Printing Date Part Number Edition

June 2003 T1859-90006 First Edition

June 2004 T1859-90017 Second Edition

February 2005 T1859-90017 Second Edition February 2005 Update

October 2005 T1859-90033 Third Edition

December 2005 T1859-90033 Third Edition, First Reprint

March 2006 T1859-90038 Third Edition, Second Reprint

May 2006 T1859-90038 Third Edition May 2006 Update

• Chapter 1, “Introduction,” describes a Serviceguard cluster and

Related The following documents contain additional useful information:

• VERITAS Volume Manager Administrator’s Guide. This contains a

Serviceguard Extension for RAC (SGeRAC) enables the Oracle Real

• What is a Serviceguard Extension for RAC Cluster?

What is a Serviceguard Extension for RAC

Figure 1-1 Overview of Oracle RAC Configuration on HP-UX

RAC on HP-UX lets you maintain a single database image that is

Figure 1-2 Group Membership Services

Using Packages in a Cluster

Serviceguard Extension for RAC Architecture

— Custom Oracle Database

— Group Membership Services (RAC)

— Volume Manager Software

Group Membership Daemon

• cmgmsd—Group Membership Daemon for RAC 9i or later

Overview of SGeRAC and Cluster File System

Storage Configuration Options

Overview of SGeRAC and Oracle 10g RAC

For more detail information on Oracle 10g RAC refer to Chapter 2,

Overview of SGeRAC and Oracle 9i RAC

How Serviceguard Works with Oracle 9i RAC

Configuring Packages for Oracle RAC

Configuring Packages for Oracle Listeners

Figure 1-3 Before Node Failure

Figure 1-4 After Node Failure

Up to Four Nodes with SCSI Storage

Figure 1-5 Four-Node RAC Cluster

In this type of configuration, each node runs a separate instance of RAC

Point to Point Connections to Storage Devices

Figure 1-6 Eight-Node Cluster with XP or EMC Disk Array

FibreChannel switched configurations also are supported using either an

Extended Distance Cluster Using

2 Serviceguard Configuration for

This chapter shows the additional planning and configuration that is

Group Membership API (NMAPI2)

NOTE HP and Oracle support SGeRAC to provide group membership to CSS.

Serviceguard Cluster Timeout

NOTE As a general guideline, the CSS TIMEOUT should be greater of either

RAC IMR Timeout

Oracle Cluster Software

Automated Oracle Cluster Software Startup and Shutdown

OCR and Vote Device

Mirroring and Resilvering

Shared Storage Activation

Automated Startup and Shutdown

Manual Startup and Shutdown

NOTE Serviceguard can not be responsible for networks or connection

ORACLE LOGICAL VOLUME WORKSHEET FOR LVM Page _ of __

Oracle Control File: _____/dev/vg_ops/ropsctl1.ctl110

ORACLE LOGICAL VOLUME WORKSHEET FOR CVM Page _ of __

Oracle Control File: _____/dev/vx/rdsk/ops_dg/opsctl1.ctl110