Академический Документы
Профессиональный Документы
Культура Документы
Rac11gR1OnHPUX............................................................................................................................................1 1. *Introduction.......................................................................................................................................1 1.1. *What you need to know....................................................................................................1 1.1.1. Software required for install (Itanium or PA-RISC)..........................................1 1.1.2. Processor Model.................................................................................................1 1.2. *Installation steps ..........................................................................................................................................1 1.3. *Schematic....................................................................................................................................................2 1.3.1. Hardware/software configuration BEFORE Oracle software install............................................2 1.3.2. Hardware/software configuration AFTER Oracle software install..............................................2 1.4. *Installation Method.....................................................................................................................................3 2. *Prepare the cluster nodes for Oracle RAC.....................................................................................................3 2.1. *User Accounts and Software Directories.......................................................................................4 2.1.1. *Creating the OSDBA (DBA) Group..............................................................................4 2.1.2. *Creating an OSOPER Group (Optional) ........................................................................4 2.1.3. *Creating the Oracle Software Owner User....................................................................4 2.1.4. *Creating an Oracle Software Owner User.....................................................................5 2.1.5. *Modifying an Oracle Software Owner User..................................................................5 2.1.6. *Creating Software Directories.......................................................................................5 2.2. *SSH Setup...................................................................................................................................................6 2.2.1. *Checking Existing SSH Configuration on the System...............................................................6 2.2.2. Configuring SSH ...........................................................................................................................6 2.3. *Hardware Requirements and Preparation...................................................................................................7 2.3.1. *Hardware Requirements ..............................................................................................................7 2.4. *Kernel Parameters.......................................................................................................................................8 2.5. *Required HPUX Patches.............................................................................................................................8 2.5.1. *HPUX 11.23 Patches..................................................................................................................9 2.5.2. *HPUX 11.31 Patches..................................................................................................................9 2.6. *Network Requirements and Preparation.....................................................................................................9 2.6.1. *Network Ping Tests...................................................................................................................10 2.6.2. *Network Adapter configuration................................................................................................10 2.7. *Stage the Oracle Software.........................................................................................................................10 2.8. *Check Preparations...................................................................................................................................11 3. *Prepare the Shared Storage for Oracle RAC This section describes how to prepare the shared storage for Oracle RAC..................................................................................................................................................14 3.1. Configuring Raw Disk for Storage To configure raw disks for database file storage, follow the following steps:...............................................................................................................................14 4. Oracle Clusterware Installation and Configuration.......................................................................................15 4.1. Oracle Clusterware Install ..............................................................................................................16 5. Oracle Clusterware patching..........................................................................................................................24 5.1. Oracle Clusterware patching complete..........................................................................................31 6. Oracle ASM Home Software Install..............................................................................................................31 6.1. CVU check.....................................................................................................................................31 6.2. Oracle ASM Home Software Install..............................................................................................31 6.2.1. Oracle ASM software home has been installed.............................................................38 7. Oracle ASM Software Home Patching..........................................................................................................38 7.1. Start the OUI for Oracle ASM Software Home Patching..............................................................38 8. Oracle RAC Database Listener Creation.......................................................................................................45 8.1. Create Node specific network listeners ..........................................................................................45 8.2. Completed Node specific network listeners creation .....................................................................49 9. Oracle ASM Instance and diskgroup Creation..............................................................................................49 9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups..........................................49 9.2. Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups............58 10. Oracle RAC Database Home Software Install.............................................................................................58 10.1. CVU check - Pre Database Install...............................................................................................58 10.2. Oracle RAC Database Home Software Install.............................................................................58 i
Table of Contents
Rac11gR1OnHPUX 10.2.1. Oracle RAC Database software home has been installed............................................66 11. Oracle RAC Software Home Patching........................................................................................................66 11.1. Using the installer to patch the Oracle RAC Software Home.....................................................66 12. Oracle RAC Database Creation...................................................................................................................73 12.1. use dbca to create the RAC database...........................................................................................73
ii
Rac11gR1OnHPUX
1. *Introduction
1.1. *What you need to know
For those of you starting an Oracle RAC install for the first time, all this will be new. For those of you with Oracle 10g RAC on HPUX experience, there are just a few changes. Disk requirements for Oracle Clusterware devices The storage requirement for the OCR & vote devices used by Oracle Clusterware has been increased to 280 MB (256 MB, plus an additional buffer to allow for fdisk variation in space allocation). Better support for IP address The previous release had some restrictions that could be easily worked around, with regard to IP addresses from the non-routable range. These restrictions have been removed in the Oracle Clusterware 11g version and the workarounds are no longer required.
Patch the ASM Home to the latest patchset Create the Listeners Create the ASM Instances and initial Disk Group Establish RAC Database Install an Oracle Software Home for RAC Database Patch the RAC Database Home to the latest patchset Create the RAC Database Instances
1.3. *Schematic
The following is a schematic of the software & hardware layout of a 2 node RAC cluster. As explained in this document the actual number of LUNs required will vary depending on your mirroring requirements.
Notice that there are a number of LUNs present to both machines from the shared storage. 4 of 500MB LUNs
2 are used for the OCR devices (/dev/rdsk/c4t0d1,/dev/rdsk/c6t0d1) 2 are used for the vote devices (/dev/c4t0d2,/dev/c6t0d2) 12 of 15890MB LUNs 10 are used for the +DATA ASM diskgroup (/dev/rdsk/c4t0d[3-7],/dev/rdsk/c6t0d[3-7]) 2 are used for the +FLASH diskgroup. (/dev/rdsk/c4t0d10,/dev/rdsk/c6t0d10) Also notice there is a dedicated switch on the cluster interconnect.
If the OSDBA group does not exist or if you require a new OSDBA group, then create it as follows. In the following command, use the group name dba unless a group with that name already exists. The default OSDBA group name is dba.
# /usr/sbin/groupadd -g 201 dba
Check too whether the oinstall group exists. If it does not, then add this group too.
# /usr/sbin/groupadd -g 200 oinstall
If the oracle user exists, then the output from this command is similar to the following:
uid=440(oracle) gid=200(oinstall) groups=201(dba),202(oper)
If the user exists, then determine whether you want to use the existing user or create another oracle user. If you want to use the existing user, then ensure that the user's primary group is the Oracle Inventory group and that it is a member of the appropriate OSDBA and OSOPER groups.
In this command: The -g option specifies the primary group, which must be the Oracle Inventory group, for example oinstall The -G option specifies the secondary groups, which must include the OSDBA group and if required, the OSOPER group 2.Set the password of the oracle user:
# passwd oracle
Make sure that the user environment variables are set correctly:
# export ORACLE_BASE=/opt/oracle/product # export ORACLE_HOME=$ORACLE_BASE/RDBMS # export ORA_CRS_HOME=$ORACLE_BASE/CRS
If SSH is running, then the response to this command is one or more process ID numbers. In the home directory of the software owner that you want to use for the installation (crs, oracle), use the command ls -al to ensure that the .ssh directory is owned and writable only by the user. You need either an RSA or a DSA key for the SSH protocol. RSA is used with the SSH 1.5 protocol, while DSA is the default for the SSH 2.0 protocol. With OpenSSH, you can use either RSA or DSA. The instructions that follow are for SSH1. If you have an SSH2 installation, and you cannot use SSH1, then refer to your SSH distribution documentation to configure SSH1 compatibility or configure SSH2 with DSA.
Note that in this case, we will leave the passphrase empty. The identification information has been saved in /home/oracle/.ssh/id_rsa. The public key has been saved in /home/oracle/.ssh/id_rsa.pub. The contents of the id_rsa.pub file of both nodes need to be put into a file called /home/oracle/.ssh/authorized_keys on both nodes. Basically, the files are identical, so create the file on node 1 and copy it to the same location on node 2.
# # # # cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys ssh oracle@node2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 644 ~/.ssh/authorized_keys scp ~/.ssh/authorized_keys node2:~/.ssh/authorized_keys
Now, check connectivity in each direction, from each of the cluster servers, first from node1 and then repeat from node2.
# ssh node1 date # ssh node2 date
At the end of this process, the public hostname for each member node should be registered in the known_hosts file for all other cluster member nodes.
swap space equivalent to a multiple of the physical memory, according to the following situations. If RAM is less than 2GB, swap space should be double the size of RAM. If RAM is between 2GB and 8GB, then the recommended swap space should be equal to the RAM. Otherwise, if RAM is over 8GB, then allocate a swap space .75 times the size of the RAM. Use the following command to confirm the size of swap:
# /usr/sbin/swapinfo -a
ensure that there is at least 400MB free space in the /tmp directory. To verify the free space available in /tmp, issue the following command:
# bdf /tmp
the Oracle Clusterware home must have at least 650MB of available disk space. the Oracle database home should have at least 4GB of available space confirm that the OS-version is correct, and 64-bit, issue the follow commands:
# uname -a # /bin/getconf KERNEL_BITS
confirm that Async I/O is enabled (default for HPUX 11.31 and 11.23):
# ll /dev/async
in order to successfully relink the Oracle executables (if required) after installation, ensure that the following symbolic links have been created:
# # # # # # # # # # cd ln ln ln ln ln ln ln ln ln /usr/lib -s /usr/lib/libX11.3 libX11.sl -s /usr/lib/libXIE.2 libXIE.sl -s /usr/lib/libXext.3 libXext.sl -s /usr/lib/libXhp11.3 libXhp11.sl -s /usr/lib/libXi.3 libXi.sl -s /usr/lib/libXm.4 libXm.sl -s /usr/lib/libXp.2 libXp.sl -s /usr/lib/libXt.3 libXt.sl -s /usr/lib/libXtst.2 libXtst.sl
make sure that each of the cluster nodes is set to the same date and time. Oracle strongly recommends using the Network Time Protocol feature of most operating systems for this purpose, with all nodes using the same reference Network Time Protocol server.
To determine if a specific operating system patch has been installed, enter the following command:
# /usr/sbin/swlist -l patch <patch_number>
To determine which operating system bundles are installed, enter the following command:
# /usr/sbin/swlist -l bundle
The public IP address, which should be recorded in hosts file on each node and, if available, DNS. This IP Address should be bound to the public adapter before starting the install. It should be a static, not DHCP, address. The private IP address, which should be from a different subnet than the public IP address. This address does not require registering in DNS but you should place an entry in the hosts file on each node. This IP Address should be bound to the private adapter before starting the install. It should be a static, not DHCP, address. Oracle recommends that you use a subnet reserved for private networks, such as 10.0.0.0 or 192.168.0.0 A VIP address, which should be from the same subnet as the public IP address and should be recorded in DNS and the hosts file on each node. This IP Address should NOT be bound to the public adapter before starting the install. Oracle Clusterware is responsible for binding this address. It should be a static, not DHCP, address. Refer to Metalink Note 296874.1 "Configuring the HPUX Operating System for the Oracle 10g VIP". If you do not have a DNS server then make sure both the public and the VIP addresses are entered into all the relevant hosts files that are normally found in the /etc/hosts file.
This will produce a listing of pre-installation tests and success or failure states for each test, much like the following: Show CVU output Hide CVU output
runcluvfy.sh stage -pre crsinst -n atlrp441,atlrp442 -r 11gR1 -verbose Performing pre-checks for cluster services setup Checking node reachability... Check: Node reachability from node "atlrp441" Destination Node Reachable? ------------------------------------ -----------------------atlrp441 yes atlrp442 yes Result: Node reachability check passed from node "atlrp441".
Checking user equivalence... Check: User equivalence for user "oracle" Node Name Comment ------------------------------------ -----------------------atlrp442 passed atlrp441 passed Result: User equivalence check passed for user "oracle". Checking administrative privileges... Check: Existence of user "oracle" Node Name User Exists Comment ------------ ------------------------ -----------------------atlrp442 yes passed atlrp441 yes passed Result: User existence check passed for "oracle". Check: Existence of group "oinstall" Node Name Status Group ID ------------ ------------------------ -----------------------atlrp442 exists 402 atlrp441 exists 402 Result: Group existence check passed for "oinstall". Check: Membership of user "oracle" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary ---------------- ------------ ------------ ------------ -----------atlrp442 yes yes yes yes atlrp441 yes yes yes yes Result: Membership check for user "oracle" in group "oinstall" [as Primary] Administrative privileges check passed. Checking node connectivity... Interface information for node "atlrp442" Interface Name IP Address Subnet ---------------- ----------------------eth0 10.143.181.34 10.143.181.0 eth2 10.143.207.17 10.143.207.0
Interface information for node "atlrp441" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.143.181.33 10.143.181.0 eth2 10.143.207.16 10.143.207.0
Connected?
11
------------------------------ ------------------------------ ---------------atlrp442:eth0 atlrp441:eth0 yes Result: Node connectivity check passed for subnet "10.143.181.0" with node(s) atlrp442,atlrp441. Check: Node connectivity of subnet "10.143.207.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------atlrp442:eth2 atlrp441:eth2 yes Result: Node connectivity check passed for subnet "10.143.207.0" with node(s) atlrp442,atlrp441. Interfaces found on subnet "10.143.181.0" that are likely candidates for a private interconnect: atlrp442 eth0:10.143.181.34 atlrp441 eth0:10.143.181.33 Interfaces found on subnet "10.143.207.0" that are likely candidates for a private interconnect: atlrp442 eth2:10.143.207.17 atlrp441 eth2:10.143.207.16 WARNING: Could not find a suitable set of interfaces for VIPs. Result: Node connectivity check passed. Checking system requirements for 'crs'... Check: Total memory Node Name Available ------------ -----------------------atlrp442 7.98GB (8371808KB) atlrp441 7.98GB (8371808KB) Result: Total memory check passed. Check: Free disk space in "/tmp" dir Node Name Available ------------ -----------------------atlrp442 5.02GB (5262320KB) atlrp441 5.02GB (5262320KB) Result: Free disk space check passed. Check: Swap space Node Name Available ------------ -----------------------atlrp442 15.98GB (16760416KB) atlrp441 15.98GB (16760416KB) Result: Swap space check passed. Check: System architecture Node Name Available ------------ -----------------------atlrp442 PA-RISC atlrp441 PA-RISC Result: System architecture check passed. Check: Kernel version Node Name Available ------------ -----------------------atlrp442 HP-UX B.11.31 atlrp441 HP-UX B.11.31 Result: Kernel version check passed.
Check: Package existence for "OS-Core:B.11.31" Node Name Status ------------------------------ -----------------------------atlrp441 OS-Core:B.11.31 atlrp442 OS-Core:B.11.31 Result: Package existence check passed for "OS-Core:B.11.31".
Check: Operating system patch for "PHKL_36248" => Superceded by PHKL_37454 Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------atlrp441 missing PHKL_36248 failed atlrp442 missing PHKL_36248 failed Result: Operating system patch check failed for "PHKL_36248". Check: Operating system patch for "PHKL_36249" => Superceded by Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 missing PHKL_36249 atlrp442 missing PHKL_36249 Result: Operating system patch check failed for "PHKL_36249". Check: Operating system patch for "PHKL_35900" => Superceded by Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 missing PHKL_35900 atlrp442 missing PHKL_35900 Result: Operating system patch check failed for "PHKL_35900". PHKL_37453 Comment ---------failed failed
Check: Operating system patch for "PHKL_35936" => Superceded by PHSS_37452 Node Name Applied Required Comment
12
------------ ------------------------ -----------------------atlrp441 missing PHKL_35936 atlrp442 missing PHKL_35936 Result: Operating system patch check failed for "PHKL_35936". Check: Operating system patch for "PHSS_37948" Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 PHSS_37948 PHSS_37948 atlrp442 PHSS_37948 PHSS_37948 Result: Operating system patch check passed for "PHSS_37948". Check: Package existence for "ACXX:C.06.10" Node Name Status ------------------------------ -----------------------------atlrp441 ACXX:C.06.10 atlrp442 ACXX:C.06.10 Result: Package existence check passed for "ACXX:C.06.10". Check: Package existence for "C-ANSI-C:C.06.10" Node Name Status ------------------------------ -----------------------------atlrp441 C-ANSI-C:C.06.10 atlrp442 C-ANSI-C:C.06.10 Result: Package existence check passed for "C-ANSI-C:C.06.10". Check: Operating system patch for "PHKL_37296" Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 PHKL_37296 PHKL_37296 atlrp442 PHKL_37296 PHKL_37296 Result: Operating system patch check passed for "PHKL_37296". Check: Operating system patch for "PHCO_37476" Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 PHKL_34941 PHCO_37476 atlrp442 PHKL_34941 PHCO_37476 Result: Operating system patch check passed for "PHCO_37476". Check: Operating system patch for "PHCO_37807" Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 missing PHCO_37807 atlrp442 missing PHCO_37807 Result: Operating system patch check passed for "PHCO_37807". Check: Operating system patch for "PHSS_37948" Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 PHSS_37948 PHSS_37948 atlrp442 PHSS_37948 PHSS_37948 Result: Operating system patch check passed for "PHCO_37807". Check: Package existence for "Perl5-64:D.5.8.8" Node Name Status ------------------------------ -----------------------------atlrp441 Perl5-64:D.5.8.8.B atlrp442 Perl5-64:D.5.8.8.B Result: Package existence check passed for "Perl5-64:D.5.8.8". Check: Group existence for "dba" Node Name Status Comment ------------ ------------------------ -----------------------atlrp442 exists passed atlrp441 exists passed Result: Group existence check passed for "dba". Check: Group existence for "oinstall" Node Name Status Comment ------------ ------------------------ -----------------------atlrp442 exists passed atlrp441 exists passed Result: Group existence check passed for "oinstall". Check: User existence for "nobody" Node Name Status Comment ------------ ------------------------ -----------------------atlrp442 exists passed atlrp441 exists passed Result: User existence check passed for "nobody". System requirement failed for 'crs'
---------failed failed
Pre-check for cluster services setup was unsuccessful on all the nodes.
13
Show all
Hide all
Note that in this case the CVU failed, but only because some of the patches it was checking for had been superceded with more recent versions.
3. *Prepare the Shared Storage for Oracle RAC This section describes how to prepare the shared storage for Oracle RAC
For all installations, you must choose the storage option that you want to use for Oracle Clusterware files, Automatic Storage Manager (ASM) and Oracle Real Application Clusters databases (Oracle RAC). Oracle Clusterware files include: Voting disks, used to monitor cluster node status, and Oracle Cluster Registry (OCR) which contains configuration information about the cluster. There are two ways of storing Oracle Clusterware files: Raw Devices: Oracle Clusterware files can be placed on RAW devices based on shared disk partitions. A supported shared file system - Supported file systems include the following: A supported cluster file system Network File System (NFS): A file-level protocol that enables access and sharing of files In the example given in this document, Raw Devices are used.
3.1. Configuring Raw Disk for Storage To configure raw disks for database file storage, follow the following steps:
Make sure that the disks are available, enter the following command on every node:
# ioscan -fnNkCdisk
If the ioscan command does not display device name information for a device that you want to use, enter the following command to install the special device files for any new devices:
# insf -e
The disk names for the same disk can be different on each node. A disk can be identified as the same one via the WWID. The WWID of a disk can be checked via the following command:
# scsimgr lun_map -D /dev/rdsk/disk25 | grep WWID
The System Management Homepage also shows the WWID for each disk:
3. *Prepare the Shared Storage for Oracle RAC This section describes how to prepare the shared storage for14 Oracle RA
For each disk that you want to use, enter the following command on any node to verify that it is not already part of an LVM volume group:
# pvdisplay /dev/rdsk/ora_data1
If this command displays volume group information, the disk is already part of a volume group. The disks that you choose must not be part of an LVM volume group. We recommend to create a special Oracle device directory, and to use mknod to create device paths in this special Oracle folder. This has the advantage, that you get same names for OCR and Voting files across all nodes in the cluster. In addition, it ensures that the permissions of these Oracle device files remain untouched of 'insf e'.
# mkdir /dev/oracle # ll /dev/rdsk/disk25 crw-r----- 1 bin sys 23 0x000019 Jan 16 12:16 /dev/rdisk/disk25 # mknod /dev/oracle/asmdisk1 c 23 0x000019
Later during set-up, for the ASM instance, set the ASM_DISKSTRING parameter to /dev/oracle/* Now when 'insf -e' is run, it only touches 'standard' DSF, rather than this special ones. Modify the owner, group, and permissions on the character raw device files (ASM and Voting Disks) on all nodes:
# chown oracle:dba /dev/oracle/* # chmod 660 /dev/oracle/*
Modify the owner, group, and permissions on the character raw device files (OCR files) on all nodes:
# chown root:dba /dev/oracle/OCR # chmod 640 /dev/oracle/OCR
3.1. Configuring Raw Disk for Storage To configure raw disks fordatabase file storage, follow the following15 steps:
Notes Notice that the Oracle 11g Installer now combines the Oracle Database, Client and Clusterware components Action Select the Oracle Clusterware radio button Click Next
Notes The OUI will name the Oracle Clusterware Home'OraCrs11g_home'. If you change this you should make sure that the name you use is unique Actions Specify a location for the Oracle Clusterware Home Click Next 4.1. Oracle Clusterware Install 16
Notes The installer will validate the state of the cluster before continuing. If there are issues you should rectify them before continuing Actions Click Next
Notes Each Cluster requires a name, this should be unique within your organisation, The default is a substring of the node name followed by _cluster This is where you specify details of all the nodes in the cluster. The installer will default names for the node it is running on. You must add other nodes manually Oracle defaults the names to 'nodename', 'nodename-priv', 'nodename-vip' Actions Confirm the Cluster Name selected is acceptable Confirm the Details for the current node are OK. The defaults are: Public Node Name : must resolve via hosts and or DNS to the public IP address and must be live 4.1. Oracle Clusterware Install 17
Private Node Name : must resolve via hosts to the interconnect IP address and must be live Virtual Host Name : must resolve via hosts and or DNS to a new IP address and must not be live If these are not correct select the node entry and click Edit... to modify OR Click Add... to add more nodes to the cluster
Notes If you have more nodes repeast the Add... cycle Actions Click Next
Notes Here you can see we have successfully configured the network adapter usage. Ideally you will have only 1 adapter set as public and 1 adapter set as private. Other adapters, if available, set to do not use. If you have multiple public or multiple private adapters it is better to team them at the OS adapter driver level before commencing the install.Type : Private Actions Click Next 4.1. Oracle Clusterware Install 18
Notes Here we specify the shared storage devices that will be used by Oracle Clusterware. Ideally you will have 2 devices for the OCR, Oracle will mirror to these devices to protect you from a single OCR device failure. Also you will have an additional 3 vote devices to protect your cluster from the failure of a single vote device. Actions Select the Normal radio button Enter the device to be used for the First OCR Enter the device to be used for the Second OCR Click Next
Notes Next we specify the devices to be used for the Oracle Clusterware vote disks Actions Select the Normal radio button Enter the device to be used for the First vote disk Enter the device to be used for the Second vote disk Enter the device to be used for the Third vote disk 4.1. Oracle Clusterware Install 19
Click Next
Notes the installer lists a summary of the planned actions Actions Click Install
Notes the OUI first copies software to the local node and then copies the software to the remote nodes Actions none required
20
Notes The installer requires commands to be run as root on each of the nodes Actions On the first node open a root shell window & run the root.sh script Show root.sh output Hide root.sh output
# /opt/oracle/product/CRS/root.sh WARNING: directory '/opt/oracle/product' is not owned by root WARNING: directory '/opt/oracle' is not owned by root WARNING: directory '/opt' is not owned by root Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up Network socket directories Oracle Cluster Registry configuration upgraded successfully The directory '/opt/oracle/product' is not owned by root. Changing owner to root The directory '/opt/oracle' is not owned by root. Changing owner to root The directory '/opt' is not owned by root. Changing owner to root Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: atlrp441 atlrp441-priv atlrp441 node 2: atlrp442 atlrp442-priv atlrp442 Creating OCR keys for user 'root', privgrp 'sys'.. Operation successful. Now formatting voting device: /dev/oracle/vote1 Now formatting voting device: /dev/oracle/vote2 Now formatting voting device: /votedisk/vote3 Format of 3 voting devices complete. Startup will be queued to init within 30 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. Cluster Synchronization Services is active on these nodes. atlrp441 Cluster Synchronization Services is inactive on these nodes. atlrp442 Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.
Show all
Hide all
Once this has completed successfully on the first node, open a root shell window on the next node & run the root.sh script
21
# /opt/oracle/product/CRS/root.sh WARNING: directory '/opt/oracle/product' is not owned by root WARNING: directory '/opt' is not owned by root Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up Network socket directories Oracle Cluster Registry configuration upgraded successfully The directory '/opt/oracle/product' is not owned by root. Changing owner to root The directory '/opt' is not owned by root. Changing owner to root clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: atlrp441 atlrp441-priv atlrp441 node 2: atlrp442 atlrp442-priv atlrp442 clscfg: Arguments check out successfully. NO KEYS WERE WRITTEN. Supply -force parameter to override. -force is destructive and will destroy any previous cluster configuration. Oracle Cluster Registry for cluster has already been initialized Startup will be queued to init within 30 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. Cluster Synchronization Services is active on these nodes. atlrp441 atlrp442 Cluster Synchronization Services is active on all the nodes. Oracle CRS stack installed and running under init(1M) Running vipca(silent) for configuring nodeapps Creating Creating Creating Starting Starting Starting Done. VIP GSD ONS VIP GSD ONS application application application application application application resource resource resource resource resource resource on on on on on on (2) (2) (2) (2) (2) (2) nodes... nodes... nodes... nodes... nodes... nodes...
Show all
Hide all
Notes a series of configuration assistants are run at the end of the install Actions 4.1. Oracle Clusterware Install 22
none required
Notes The installer has completed installation and configuration of Oracle Clusterware Actions Click Exit
Verify your CRS installation by executing the olsnodes command from the /opt/oracle/product/CRS/bin directory:
# olsnodes -n # atlrp441 1 # atlrp442 2
Now you should see the following processes running: oprocd -- Process monitor for the cluster. evmd -- Event manager daemon that starts the racgevt process to manage callouts. ocssd -- Manages cluster node membership and runs as oracle user; failure of this process results in cluster restart. crsd -- Performs high availability recovery and management operations such as maintaining the OCR. Also manages application resources and runs as root user and restarts automatically upon failure. You can check whether the Oracle processes evmd, occsd, and crsd are running by issuing the following command.
# ps -ef | grep d.bin
At this point, you have completed phase one, the installation of Cluster Ready Services Please note that Oracle added the following three lines to the automatic startup file /etc/inittab h1:3:respawn:/sbin/init.d/init.evmd run >/dev/null 2>&1 </dev/null h2:3:respawn:/sbin/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null h3:3:respawn:/sbin/init.d/init.crsd run >/dev/null 2>&1 </dev/null
23
Oracle Support recommends NEVER modifying these entries in the inittab or modifying the init scripts unless you use this method to stop a reboot loop or are given explicit instructions from Oracle support. To ensure that the Oracle Clusterware install on all the nodes is valid, the following should be checked on all the nodes:
# /opt/oracle/product/CRS/bin/crsctl check crs
softwareversion activeversion
Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directory where you unpacked the patch set software:
$ cd patchset_directory/Disk1 $ ./runInstaller
24
Notes You should ensure that the Clusterware home is selected in the first drop down list box The installer should default the directory to the correct location Actions Ensure the correct home is selected Click Next
25
Notes the installer detects that this is a clustered home and automatically selects all the nodes in the cluster Actions Click Next
26
Notes Some parameters are validated by the installer Actions Click Next
Notes This is a summary of the actions the installer will complete Actions Click Install
27
Notes the installer stages the patch on all the nodes in the cluster Actions No action required
At the end the installer lists the mandatory steps that must be completed to apply this patch Actions Log in as the root user and enter the following command to shut down the Oracle Clusterware: # CRS_home/bin/crsctl stop crs Run the root111.sh script. It will automatically start the Oracle Clusterware on the patched node:
# CRS_home/install/root111.sh
# /opt/oracle/product/CRS/install/root111.sh Creating pre-patch directory for saving pre-patch clusterware files Completed patching clusterware files to /opt/crs Relinking some shared libraries. Relinking of patched files is complete. Preparing to recopy patched init and RC scripts. Recopying init and RC scripts. Startup will be queued to init within 30 seconds. Starting up the CRS daemons. Waiting for the patched CRS daemons to start. This may take a while on some systems. . 11107 patch successfully applied. clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: atlrp441 atlrp441-priv atlrp441 node 2: atlrp442 atlrp442-priv atlrp442 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully Creating '/opt/oracle/product/CRS/install/paramfile.crs' with data used for CRS configuration Setting CRS configuration values in /opt/oracle/product/CRS/install/paramfile.crs Setting cluster unique identifier Restarting Oracle clusterware Stopping Oracle clusterware Stopping resources. This could take several minutes. Successfully stopped Oracle Clusterware resources Stopping Cluster Synchronization Services. Shutting down the Cluster Synchronization Services daemon. Shutdown request successfully issued. Waiting for Cluster Synchronization Services daemon to stop Cluster Synchronization Services daemon has stopped Starting Oracle clusterware Attempting to start Oracle Clusterware stack Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Cluster Synchronization Services daemon has started Event Manager daemon has started Cluster Ready Services daemon has started
# /opt/oracle/product/CRS/install/root111.sh Creating pre-patch directory for saving pre-patch clusterware files Completed patching clusterware files to /opt/crs Relinking some shared libraries. Relinking of patched files is complete. Preparing to recopy patched init and RC scripts. Recopying init and RC scripts. Startup will be queued to init within 30 seconds.
29
Starting up the CRS daemons. Waiting for the patched CRS daemons to start. This may take a while on some systems. . 11107 patch successfully applied. clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: atlrp441 atlrp441-priv atlrp441 node 2: atlrp442 atlrp442-priv atlrp442 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully Creating '/opt/oracle/product/CRS/install/paramfile.crs' with data used for CRS configuration Setting CRS configuration values in /opt/oracle/product/CRS/install/paramfile.crs
Continue with these two steps on all nodes, one by one, this will achieve a rolling upgrade of the Oracle Clusterware. When done, verify the Oracle Clusterware is running on all nodes before exiting the installer. $ CRS_home/bin/crsctl check crs Cluster Synchronization Services appears healthy Cluster Ready Services appears healthy Event Manager appears healthy
# /opt/oracle/product/CRS/bin/crsctl query crs Oracle Clusterware version on node [atlrp441] is [11.1.0.7.0] # /opt/oracle/product/CRS/bin/crsctl query crs Oracle Clusterware active version on the cluster is [11.1.0.7.0]
softwareversion activeversion
Notes Once the Oracle Clusterware is running on all nodes, exit the installer Actions
30
Click Yes
In this section we establish the Oracle software for the ASM home.
Notes Here we will create a ASM Software home on all the nodes in the cluster Actions Select the Oracle Database 11g radio button Click Next
31
Notes Here we can choose the type of database software to be installed Actions Select the Enterprise Edition radio button Click Next
32
Notes Here we specify the location of various components. An 11g install makes more use of the ORACLE_BASE. Most logs will be stored in sub directories under the oracle base. The oracle base will be common to all installs. Modify as required. Also this is where you specify the location of the ASM software home. I usually modify the Home name and Home Path to include the word ASM. This makes it easier to identify later on. If you change the path you should ensure that you do not use the exact same path as the Oracle Clusterware home Actions Confirm entries are OK Click Next
Notes The installer will detect the presence of Oracle Clusterware and uses this to populate the dialog box. To build a cluster which includes all nodes you must ensure that there are check-boxes next to the node names Actions Click Select All Click Next
33
Notes The installer will then complete some Product-Specific Prerequisite checks. These should all pass OK as you have already run the CVU check Actions Click Next
34
Notes We are going to install a Software only home and then subsequently configure the software Actions Select the Install Software Only radio button Click Next
Notes We must assign the Unix group name of the priveleged users Action Choose oinstall (default) from the drop down lists Click Next
35
Notes A summary of the install will be displayed In the first step, the installer will copy the software to all of the identified cluster nodes Actions Click Install
36
Notes At various steps, the installer will pause, with instructions to run scrips (such as root.sh) Action open a shell window on each node and follow instructions provided by installer
# /opt/oracle/product/asm/root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /opt/oracle/product/asm Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only. No files will be copied.
Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions.
Notes Run the root scripts - they should only take a few seconds to run on each node, one at a time Action run the scripts indicated in the previous screen on both nodes then return to the installer and Click OK
37
Notes After the software install completes you will see this End of Installation dialog Actions Click Exit
7.1. Start the OUI for Oracle ASM Software Home Patching
Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directory where you unpacked the patch set software:
$ cd patchset_directory/Disk1 $ ./runInstaller
Notes the Universal Installer screen appears Action On the Welcome screen, click Next.
38
Notes Specify the name and the location of the asm home Action Check that the name and location are correct
Notes: 7.1. Start the OUI for Oracle ASM Software Home Patching 39
here you can specify your metalik credentials for this install. If you leave both fields blank you can opt out of notifications (see next screen). Action Enter your email address Enter your Metalink password Click Next
Notes:* If you leave both fields blank, you will opt out of notifications and get the above screen. Action Click Yes
7.1. Start the OUI for Oracle ASM Software Home Patching
40
Notes the installer detects that this is a clustered home and automatically selects all the nodes in the cluster Actions Click Next
7.1. Start the OUI for Oracle ASM Software Home Patching
41
Notes Some parameters are validated by the installer Action Click Next
Notes This is a summary of the actions the installer will complete Actions Click Install
7.1. Start the OUI for Oracle ASM Software Home Patching
42
Notes The installer copies the patch to all the nodes in the cluster Actions No action required
Notes 7.1. Start the OUI for Oracle ASM Software Home Patching 43
The installer pauses, root.sh need to be run as root on both nodes of the cluster Action open a shell window on each node and run root.sh (one after the other) then return to the installer and Click OK
#/opt/oracle/product/asm/root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /opt/oracle/product/asm Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only. No files will be copied.
Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions.
Notes The output from the root.sh script should be similar to this it should only take a few seconds to run on each node Action run the scripts on all nodes then return to the installer and Click OK
Notes 7.1. Start the OUI for Oracle ASM Software Home Patching 44
After the software install completes you will see this End of Installation dialog, exit the installer Actions Click Yes
Notes Ensure the ORACLE_HOME environement variable isse tto the recently installed ASM home location Action change to the ASM home bin directory run ./netca
Notes Netca detects that the Oracle Clusterware layer is running and offers Cluster or Single Node configuration Actions Select the Cluster configuration radio button Click Next
45
Notes Netca uses Oracle Clusterware to determine all the nodes in the cluster Actions Click Select all nodes Click Next
Notes You get various options we need to configure listeners Actions Select the Listener configuration radio button Click Next
Notes
46
We need to add a listener Actions Select the Add radio button Click Next
Notes Here you get the opportunity to name the listener Do not change this. The listeners will eventually be called LISTENER_nodename1 & LISTENER_nodename2. This is important for RAC Actions Click Next
Notes Oracle Net supports various network protocols, although TCP is the most common. 8.1. Create Node specific network listeners 47
Actions Ensure the Selected Protocols list includes TCP Click Next
Notes It is possible to choose a non-default Port I do not recommend this Actions Ensure the Use the Standard port number of 1521 radio button is set Click Next Notes After configuring the node listeners you get the opportunity to configure more network components Actions Select the No radio button Click Next
48
Notes You get the opportunity to configre other networking components Action Click the Finish button to exit the tool Notes You can see the listener resources inside Oracle Clusterware Action run the ./crs_stat -t comand from the Oracle Clusterware home bin directory
49
Notes dbca detects the Oracle Clusterware layer is running and offers to create either cluster or single instance database Actions Select the Oracle Real Application Clusters database radio button Click Next
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
50
Notes We use dbca to create RAC databases and ASM instances Actions Select the Configure Automatic Storage Management radio button Click Next Notes You need to make sure you create ASM instances on all the cluster nodes Actions Click Select All Click Next
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
51
Notes Here we specify the password for the ASM Oracle SYS user Actions Enter the SYS password Enter the same password for the Confirm SYS password Click Next
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
52
Notes dbca will create and start ASM instances Actions Click OK
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
53
Notes ASM requires disks to be group together into diskgroups. This section will be used to create 2 disk groups +DATA and +FLASH Actions Click Create New
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
54
Notes At the moment no disks are visible Actions Click Change Discovery Path
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
55
Notes Now we will assign disks to specific disk groups and create the DATA diskgroup Use a discovery string to enable dbca to find the available disks Actions In the Disk Group Name enter DATA Select the External Redundancy radio button Select the Show All radio button Select the 6 disks to be used for the DATA diskgroup Click OK Notes A progress message will be displayed - this should not take very long Actions none
Notes Here we can see the DATA diskgroup has been created and is mounted on 2/2 instances. We now need to create the FLASH diskgroup Actions Click Create New Notes Repeat the above steps, this time allocating the disks for the FLASH disk group 9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups 56
Notes Here we can see the DATA and FLASH diskgroups have been created and are mounted on 2/2 instances. This completed the ASM configuration. Actions Click Finish
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
57
Notes A confirmation dialog box Actions Click No run ./crs_stat -t from the Oracle Clusterware bin directory
9.2. Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups
Congratulations you have installed ASM, Network listeners and created 2 ASM diskgroups.
In this section we establish the Oracle software for the RAC Database home. 9.2. Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups 58
Notes We use the same installer for Database homes as Clusterware homes Actions Select the Oracle Database 11g radio button Click Next
59
Notes Here we can choose the type of database software to be installed Actions Select the Enterprise Edition radio button Click Next
Notes Here we specify the location of various components. If you change the path you should ensure that you do not use the exact same path as the Oracle Clusterware or ASM homes Actions Confirm entries are OK Click Next
60
Notes The installer will detect the presence of Oracle Clusterware and uses this to populate a dialog box. To build a cluster which includes all nodes you must ensure that there are check-boxes next to the node names Actions Click Select All Click Next
61
Notes The installer will then complete some Product-Specific Prerequisite checks. These should all pass OK - as you have already run the CVU check Actions Click Next
62
Notes We are going to install a Software only home and then subsequently configure the software Actions Select the Install Software Only radio button Click Next
Notes Assign the Unix groups for the privileged db users (default is oinstall) Action Click Next
63
Notes Here we can see a summary of the install. Actions Click Install
The installer will copy the software to all nodes in the cluster The installer will pause and display instructions for running scripts, such as root.sh Be sure to follow directions carefully, running scripts one at a time, on each node Action open a shell and follow instructions closely then return to the installer and Click OK
Notes After the software install completes you will see this End of Installation dialog Actions Click Exit
65
11.1. Using the installer to patch the Oracle RAC Software Home
Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directory where you unpacked the patch set software:
$ cd patchset_directory/Disk1 $ ./runInstaller
66
Notes the Universal Installer screen appears Actions On the Welcome screen, click Next.
Notes 11.1. Using the installer to patch the Oracle RAC Software Home 67
Specify the name and the location of the Database home Actions Check that the name and location are correct, Click Next
Notes here you can specify your metalik credentials for this install. If you leave both fields blank you can opt out of notifications (see next screen). Actions Enter your email address Enter your Metalink password Click Next
11.1. Using the installer to patch the Oracle RAC Software Home
68
Notes If you leave both fields blank, you will opt out of notifications and get the above screen. Actions Click Yes
Notes 11.1. Using the installer to patch the Oracle RAC Software Home 69
the installer detects that this is a clustered home and automatically selects all the nodes in the cluster Actions Click Next
Notes Some parameters are validated by the installer Actions Click Next
11.1. Using the installer to patch the Oracle RAC Software Home
70
Notes This is a summary of the actions the installer will complete Actions Click Install
Notes 11.1. Using the installer to patch the Oracle RAC Software Home 71
The installer copies the patch to all the nodes in the cluster Actions No action required
Notes The installer pauses, root.sh need to be run as root on both nodes of the cluster Actions open a shell window on each node and run root.sh (one after the other), output is shown below then return to the installer and Click OK
# /opt/oracle/product/RAC11/root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /opt/oracle/product/RAC11 Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only. No files will be copied. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions.
11.1. Using the installer to patch the Oracle RAC Software Home
72
73
Notes dbca detects the Oracle Clusterware layer is running and offers to create either cluster or single instance database Actions Select the Oracle Real Application Clusters database radio button Click Next
74
* Notes We use dbca to create RAC databases Actions Select the Create a Database radio button Click Next Notes You need to make sure you create RAC database instances on all the cluster nodes Actions Click Select All Click Next
75
Notes Here you get to select the type of database Actions Select the General Purpose... radio button Click Next
76
Notes Now you specify the preface for the SIDs for the RAC database instances recommendation is to keep it simple Actions Enter APX in the Global Database name, The SID should autofill Click Next
77
Notes Here you get to configure Oracle Enterprise Manager. If you have no Grid Control server then the best method is to use dbcontrol as detailed here. Actions Check the Configure Enterprise manager checkbox Select the Configure Database Control for local management radio button Click Next
78
Notes Here we specify the password for the Oracle Database users. In this example we are setting the same password for all users. You should set a password scheme that meets your requirements. Actions Select the Use the Same Administrative Password radio button Enter the password Enter the same password for the Confirm password Click Next
79
Notes Here you specify where you would like your database datafiles stored. We are going to use the ASM diskgroups we created earlier. Actions Select the Automatic Storage Management (ASM) radio button Click Next
80
Notes dbca displays the diskgroups we previously created. We are going to use the DATA diskgroup Actions Check the box on the DATA diskgroup line Click Next
81
Notes Here we can adjust where various files are placed Actions Select the Use Oracle-managed Files radio button Ensure +DATA is in the Database Area field Click Next
82
Notes Here we specify recovery configuration information. We are going to use a flash recovery area. Actions Check the Specify Flash Recovery Area checkbox In the Flash recovery area enter +FLASH Set the flash recovery size remember the size of the +FASH diskgroup Check the Enable Archiving checkbox Click Next
83
Notes Here you can choose to add the Oracle Sample schemas Actions Check the Sample schemas checkbox Click Next
84
Notes Here you can modify the default initialisation parameters. Actions Click Next
85
Notes 11g introduces enhanced security settings Actions Select the Keep the enhanced radio button Click Next
Notes 11g will set up some automatic maintenance tasks Actions Check the Enable automatic maintenance tasks checkbox Click Next
86
Notes Here you can review the placement of various database files Actions Click Next
87
Notes You can decide to generate database creation scripts Actions Check the Create Database checkbox Click Next Notes A summary screen will be displayed Actions Click Next
88
Notes After the database is created the summary screen is displayed note the URL for the Database Control Actions Click Exit Notes The Database instances will be started on all nodes Actions none Notes You can see that cluster has started the database instances on each n node Action check the status of the db instances run ./crs_stat -t
89