Oracle10gRAC SystemX-06302007

Oracle 10g R2 RAC implementation with IBM System x, BladeCenter, System Storage and Linux
June 30th, 2007
COPYRIGHT IBM CORPORATION, 2007
Gregory Verstraeten IBM ATS Americas gregvers@us.ibm.com
June 2007
Oracle 10g R2 RAC Implementation with System x
Table of Contents
Notices.................................................................................................................................................. .................3 Introduction........................................................................................................................................ ...................4
About About About About Oracle Real Application Clusters........................................................................................................................ ...4 IBM System x............................................................................................................................................. ...........5 IBM BladeCenter......................................................................................................................... .........................5 IBM System Storage....................................................................................................................... ......................5 Special Notice................................................................................................................................................ .................3 Acknowledgments...................................................................................................................................... .....................3
Hardware Requirements................................................................................................................... ....................7

Oracle RAC requirements............................................................................................................................................. ...7 System x and BladeCenter Servers....................................................................................................... ..........................9 Configuration examples............................................................................................................................................. ....14 Operating System.................................................................................................................................................... ......20 Storage related software................................................................................................................................... ............20 Oracle 10g................................................................................................................................................................ .....21 Optional Software Products........................................................................................................................................ ...21 BIOS...................................................................................................................................................................... ........23 Remote System Management.......................................................................................................................... ..............23 Installing Linux Operating System........................................................................................................... .....................24 Set up a Linux kernel crash utility............................................................................................................................ .....24 Network.................................................................................................................................................. ......................25 Multipath IO driver............................................................................................................................... ........................25 LUN creation............................................................................................................................................................ .....27 Shared storage configuration.......................................................................................................................... ..............29 Pre-Installation Tasks......................................................................................................................................... ...........32 Oracle 10g R2 clusterware installation................................................................................................ .........................34 Oracle 10g R2 database installation................................................................................................................... ...........41
Software requirements................................................................................................................................. .......20
Set Up Server Environment .......................................................................................................... .....................23
Shared Storage setup............................................................................................................................. .............27
Installing Oracle 10g R2 RAC..................................................................................................... ........................32
Database creation......................................................................................................................................... .......44 Administration and tuning.................................................................................................................................. .55

Asynchronous IO....................................................................................................................................... ....................55 Direct IO............................................................................................................................................. ..........................55 Swappiness.......................................................................................................................................... .........................55 Allocating the SGA in large pages............................................................................................................... ..................56 CRS administration.................................................................................................................................. .....................56 OCR administration............................................................................................................................................. ..........57 ASM administration.................................................................................................................................................... ...59 Adding/removing nodes in a RAC cluster............................................................................................. .........................60 Performance report......................................................................................................................................... ..............61 Database Files...................................................................................................................................................... .........44 ASM configuration............................................................................................................................... .........................44
Appendix A: ip Bonding........................................................................................................................ ...............62 Appendix B: Kernel crash utilities...................................................................................................................... .66 Appendix C: Sample Configurations................................................................................................ ...................68
LKCD for SLES..................................................................................................................................................... .........66 netdump for RHEL3 and RHEL4............................................................................................................ .......................66 BladeCenter and DS4500............................................................................................................................ ..................68 x366 and DS6800......................................................................................................................................... .................71 Enabling Async IO in Oracle 9i and 10gR1......................................................................................................... ...........73 LD_ASSUME_KERNEL Environmental Variable................................................................................................ ............73 OCFS..................................................................................................................................................................... ........73
Appendix D: Miscellaneous Notes.................................................................................................................... ...73
References....................................................................................................................................... ....................75 Trademarks................................................................................................................................................. .........76
COPYRIGHT IBM CORPORATION, 2007. All rights reserved.
2/82
June 2007
Notices
This document reflects the authors' understanding about Oracle products running on IBM hardware servers and storage. It was produced and reviewed by the members of the IBM's Advanced Technical Support, the IBM / Oracle International Competency Center and others. This document is presented As-Is and IBM does not assume responsibility for the statements expressed herein. It reflects the experiences of the IBM Advanced Technical Support and IBM / Oracle International Competency Center. If you have questions about the contents of this document, please direct them to authors. Information in this document concerning non-IBM products was obtained from the suppliers of these products, published announcement material or other publicly available sources. IBM has not tested these products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of a specific Statement of General Direction. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements quoted in this document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally available systems. Some measurements quoted in this document may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment.
Special Notice
The material in this document was produced by IBM Advanced Technical Support and the IBM/Oracle International Competency Center based on Oracle, Red Hat and Novell products and documentation. The material herein is copyrighted by IBM, Oracle, Red Hat, and Novell. The first version of the document was co-written with Stephen Poon from IBM Advanced Technical Support.
Acknowledgments
Thanks to the following for their support and assistance on this project:

John McAbel, WW Solutions Marketing Manager, IBM System x IBM / Oracle International Competency, San Mateo, California IBM / Oracle Joint Solutions Center, Montpellier, France Dale Martin, Executive I/T Specialist, Advanced Technical Support (ATS) Americas
3/82
June 2007
Introduction
The purpose of this paper is to assist those who are implementing Oracle Real Application Clusters on Linux running on IBM System x and BladeCenter servers and IBM System storage systems. The information provided herein is based on experiences gained working on customer implementations, creating test environments at the IBM / Oracle International Competency Center, and available documentation from IBM, Oracle, Red Hat and Novell. In the sections that follow, we describe activities needed to prepare for and successfully install Linux and Oracle 10g RAC on IBM System x and BladeCenter servers and IBM System Storage. The operating system environments described will be the 2.6 kernel-based Novell SuSE Linux Enterprise Server 10 (SLES10) and Red Hat Enterprise Linux 4.0 (RHEL4). An implementation of Oracle RAC consists of 3 steps: 1. Planning the hardware for Oracle RAC implementation 2. Configuring the servers and storage 3. Installing and configuring Oracle clusterware and database In this paper, we will focus mainly on the first two steps. We will discuss various hardware choices to be made, describe server and storage configurations for RAC and document best practices for high availability and performance. Since the third step is well documented in Oracle documentation and existing white papers, we will not provide as much detail, but outline the required activities. Though this paper will focus on Oracle 10g R2 RAC, many of concepts and recommendations apply regardless of the release.
About Oracle Real Application Clusters

Oracle Real Application Clusters (RAC) is an option of Oracle Database that brings together two or more computers to form a clustered database, cooperating as a single system. In a RAC database, Oracle processes running in each node (instances) access the same data residing on shared disk storage. First introduced in Oracle9i, RAC provides high availability and flexible scalability. If one of the clustered nodes fails, Oracle continues processing on the other nodes. If additional capacity is needed, nodes and instances can be added without taking down the cluster. In Oracle10g, Oracle provides Oracle Clusterware, which is designed specifically for Oracle RAC. You do not need a third party clusterware product to implement Oracle RAC. Since storage is shared, the file system and volume management must be cluster-aware. Oracle 10g includes Automated Storage Management (ASM), which provides automated I/O optimization and simplified storage management. For Intel/AMD-based Linux, RAC may also be implemented with raw devices or a supported clustered file system, such as Oracle Clustered File System (OCFS). For further introduction to Oracle RAC, refer to the Oracle white paper entitled, Oracle Real Application Clusters 10g, An Oracle Technical White Paper, by Barb Lundhild and Peter Sechser, May 2005. This paper can be found at: http://www.oracle.com/technology/products/database/clustering/index.html
4/82
June 2007
About IBM System x

IBM System x solutions are designed to deliver exceptional availability, simplified manageability, outstanding performance and revolutionary scalability to help you build a cost-effective, flexible IT infrastructure. System x servers feature Intel Xeon processors or AMD Opteron, which enable them to run 32-bit or 64-bit operating systems. They can run 32-bit applications under the 32-bit or 64-bit operating system. The System x x3850 and x3950, feature IBM X3 Architecture, the third-generation of IBM Enterprise X-Architecture. The XA-64e chipset provides an integrated processor/memory controller that delivers exceptional latency reductions (almost 3 times) over previous generations and sophisticated error recovery. For more information on IBM System x, visit: http://www.ibm.com/systems/x/
About IBM BladeCenter

The unique BladeCenter design addresses today's customers most serious issues: space constraints, efficient manageability, resiliency, and the physical environment, which includes cooling and power. The BladeCenter servers take less time to install, fewer people to manage and maintain, and cost less than traditional server solutions. These blade servers are so compact and easy to use that customers can increase the system capacity by simply sliding an additional BladeCenter server into the integrated chassis. Since BladeCenter servers share a common, integrated infrastructure, basic components such as power, system ports and fans, they also lower power consumption and reduce system complexity. Highly scalable and modular in design, the BladeCenter allows customers to XpandonDemand - add capacity only as needed. For more information on the IBM BladeCenter, visit: http://www.ibm.com/systems/bladecenter/
About IBM System Storage

DS3000 series
The IBM System Storage DS3200 and DS3400 Express models are designed to provide solutions to meet midrange and departmental storage requirements for advanced function, high availability, modular and scalable storage capacity, Serial Attached SCSI (SAS) or Fiber Channel (FC) connectivity, and support for RAID levels 0, 1, 3, 5, and 10. The DS3200 Express Models support up to three hosts in a redundant direct-attached SAS configuration. The DS3400 Express Models support direct-attach and Fiber Channel attachment.
5/82
June 2007
DS4000 series
The IBM System Storage DS4000 (formerly FAStT) Mid-range Disk Systems have been enhanced with these new features:

DS4000 Storage Manager V9.10 for the DS4500, DS4400, and DS4300 with Turbo Disk Systems Global Copy, Global Mirroring, and Metro Mirroring with Enhanced Remote Mirroring DS4100 EXP100 Attachment feature Support for attachment of the DS4000 EXP100 Serial ATA Expansion Unit to the DS4400 Disk System DS4000 Intermix Attachment of FC/SATA Storage Expansion Units
For more information about the IBM System Storage DS4000 series, visit: http://www-03.ibm.com/servers/eserver/bladecenter/switch/switch_ethernet_overview.html
DS6000 series
The IBM System Storage DS6000 series is designed to deliver the resiliency, performance and many of the key features of the IBM System Storage DS8000 series and the Enterprise Storage Server (ESS) in a small, modular package. The DS6000 features:
Easy installation and management using IBM System Storage DS Storage Manager, an intuitive Web-based GUI designed to enhance management efficiency with straight forward configuration, management and maintenance functions Extensive autonomic and management functions including Predictive Failure Analysis capabilities, Light Path Diagnostics and controls, and the IBM System Storage Productivity Center for Disk (TPC) is designed to provide a single interface through which administrators can monitor and manage multiple and different members of the IBM System Storage DS Family, as well as non-IBM disk arrays Flexible, open standard-based architecture designed to help facilitate the flow of information across the enterprise, even in complex, heterogeneous server environments. The DS6000 series supports IBM z/OS, IBM OS/400, IBM AIX, as well as Linux, UNIX, Microsoft Windows, HP-UX and SUN Solaris environments Designed to provide disaster tolerance with advanced copy services. The DS6000 series is equipped with four paths to each HDD and uses storage expansion enclosures with redundant Fibre Channel switches providing switch access to disks
For more information about the DS6000, visit: http://www-03.ibm.com/servers/storage/index.html
6/82
June 2007
Hardware Requirements
Oracle RAC requirements
An Oracle RAC cluster consists of the following (see Figure 1): 1. Cluster nodes 2 to n nodes or hosts, running Oracle database server 2. Network Interconnect a private network used for cluster communications and cache fusion 3. Shared Storage used to hold database system and data files and accessed by the cluster nodes 4. Production network used by clients and application servers to access the database Refer to Oracle's RAC Technologies Matrix for Linux Clusters for details regarding technology supported by Oracle. This matrix can be accessed through the Oracle product certification website: http://www.oracle.com/technology/support/metalink/index.html
Highspeed interconnect
Application Servers
Users
SharedcachewithOracleCacheFusion
Production Network
StorageArea Network
SAN Fabric
Shared storage
Figure 1: RAC Overview
Although most of the System x and BladeCenter server models can be used for an Oracle RAC implementation, other requirements need to be considered in selecting a server model. In the following sections, we discuss some of these requirements and look at a selection of System x and BladeCenter models, covering a wide range of processor selections, memory capacity, and features. We discuss what is needed to configure redundancy for missioncritical database applications. Finally, we will position the server models based on a set of requirements and assumptions.
Server CPU
Since the Intel Xeon and AMD Opteron processors can run 32-bit and 64-bit operating
COPYRIGHT IBM CORPORATION, 2007. All rights reserved. 7/82
June 2007
systems, customers can run existing 32-bit operating systems and applications and move to a 64-bit environment with the same System x servers. All AMD Opteron processors are both 32-bit and 64-bit capable. The Intel Xeon processors are 64-bit capable when they have the EM64T or Intel 64 feature. Most of the IBM System x Intel Xeon based servers are equipped with Intel chips that have the EM64T or Intel 64 option, but for example the HS20 with Intel Dual-Core Intel Xeon Processor LV is not 64-bit capable. Please, check this capability with your IBM representative or visit http://www.intel.com/products/server/processors/index.htm?iid=process+server The main advantage of 64-bit is the significantly larger addressability afforded. In an Oracle database environment, one of the key constraints in a 32-bit environment is the size of the Oracle System Global Area (SGA). In 32-bit Linux environment, the SGA is limited by default to 1.5GB, but can be increased, using various techniques, to 3.5GB. The database buffer cache can also be placed outside the SGA and accessed by indirect addressing, using facilities such as ramfs in Red Hat Enterprise Level 3. In 64-bit environment, the size of the SGA is limited by the amount of physical memory available. While 64-bit Intel and AMD processor hardware have become more common, many applications are still running on 32-bit. Oracle database is available as separate products for 32-bit and 64-bit Linux. Oracle does not support running the 32-bit database under a 64-bit Linux operating system. If the application architecture includes a middle or application tier, the middle tier needs to be available in 64-bit or supported with a 64-bit database back-end. For example, Oracle E-Business Suite's application tier is 32-bit and is certified to run with either 32-bit or 64-bit database. The latter is called a Split Configuration Database Tier. For latest information on Oracle product certification, refer to: http://www.oracle.com/technology/support/metalink/index.html Regardless of the environment, there needs to be enough server CPU capacity, in terms of speed and number of CPU's, to handle the workload. As a rule of thumb, there should be enough CPU capacity to have an average CPU utilization of 65%. This will allow the server absorb peak activity more easily.
Server Memory
An Oracle database may require a lot of memory. This depends on the activity of the users and the nature of the workload. The only rule of thumb is that the server should have more memory than it actually uses, because performance degradation is particularly severe when memory is insufficient and disk swapping occurs. It is important to select servers that are available with the amount of memory required, with some room for growth. In production, we recommend that memory utilization be at a maximum of 75-85% of the physical memory. This reduces the chance of doing disk swapping, which dramatically degrades the performance of the server.
Network
Servers in the a RAC cluster need at least two separate networks, a public network and a private network. The public network is used for the communication between the clients or applications servers and the RAC cluster. The private network, referred to as the
June 2007
interconnect network, is used for cluster node communication; for heartbeat and maintaining RAC Cache Fusion. The private network can be implemented with Gigabit Ethernet or Infiniband technologies. UDP/IP over Gigabit Ethernet is the most common choice for private network at the time this paper is written. It offers good performance for clusters with up to 4 nodes for most of the workloads. Infiniband (IB) is supported with Oracle 10g R2 and Linux with two protocols, IP over IB and RDS (Reliable Datagram Sockets). IP over IB does not offer a real advantage over UDP/IP over Gigabit Ethernet because of the latency of the IP protocol. RDS is a low latency protocol designed by Silverstorm and Oracle for Oracle RAC. Silverstorm has merged with Qlogic in 2006. Proprietary RDS drivers for Linux are available from Silverstorm/Qlogic. The Open source RDS driver is still in development at the time this paper is written. RDS with Infiniband for the Interconnect network is recommended for RAC clusters of more than 4 nodes and/or for workload with an heavy interconnect traffic such as a datawarehouse. The 10 Gigabit Ethernet technology is not supported by Oracle at the time this paper is written. As for Infiniband, a 10 Gigabit Ethernet network would not bring much advantage in terms of latency with the UDP/IP protocol. The Open RDS protocol is planned to run on Infiniband hardware as well as 1 Gigabit and 10 Gigabit Ethernet.
Shared Storage
Shared storage for Oracle RAC devices may be logical drives or LUNs from a Storage Area Network (SAN) controller or an NFS file from a supported Network Attached Storage (NAS) device. NAS has some advantages, but a SAN is recommended for higher performance. We will not discuss NAS in this paper; but for more information on IBM's NAS offering, such as the N5000, refer to: http://www-03.ibm.com/servers/storage/nas/ For SAN products, the IBM System Storage DS3400, DS4000, DS6000 and DS8000 series are supported for Oracle RAC implementations. Third party storage subsystem can also be used with System x and BladeCenter servers. Refer to third party documentation or contact a third party representative for product certification information. The IBM System Storage DS3200 is being tested for Oracle RAC at the time this paper is written. The DS3200 has SAS connectivity to servers. With RAC, Oracle database files may be located on shared storage using the following options:

Raw devices: disk partitions that are not formatted by the operating system. Cluster File System: a file system that may be accessed (read and write) by all members in a cluster at the same time, with all cluster members having the same view of the file system. Oracle Cluster File System (OCFS) is an Oracle-provided Cluster File System that is available on Intel/AMD-based Linux RAC environment.
9/82
June 2007
IBM GPFS is not supported for Oracle RAC.
Oracle Automated Storage Management (ASM): a simplified database storage management and provisioning system that provides file system and volume management capabilities in Oracle. ASM is included in 10g and may be used for Oracle files only. Note that ASM may not be used for Oracle Clusterware's voting disks and OCR devices.
We will cover Oracle Cluster File System (OCFS) and ASM later in this paper
System x and BladeCenter Servers

Table 1 below shows selected System x servers and BladeCenter blades models that are commonly implemented for Oracle RAC. The table shows the maximum number of CPU sockets, cores, and maximum amount of memory they support. Note: The information are the feature available at the time this paper is written. For update, visit www.ibm.com. Model HS21 HS21 XM LS21 LS41 x3650 x3655 x3755 x3850 x3950 Blades CPU technology Intel Xeon Intel Xeon AMD Opteron AMD Opteron Intel Xeon AMD Opteron AMD Opteron Intel Xeon Intel Xeon Max # of Max # of Max memory sockets cores (GB) 2 8 16 2 8 32 2 4 16 4 8 32 2 8 48 2 4 64 4 8 128 4 8 64 8 16 128 Onboard NICs 2 2 2 4 2 2 2 2 2
Rack mount
Table 1: Selected System x and BladeCenter servers
For detailed information on these and other System x and BladeCenter blades, refer to the IBM System x Configuration and Options guide: http://www.ibm.com/systems/x/configtools.html
NUMA optimization
The x3950 with more than 4 sockets is NUMA (Non Uniform Memory Architecture). The AMD Opteron servers, such as the x3655 and x3755 can also be considered NUMA. The access time to all the memory available on the server is not the same. The Linux kernel from RHEL4 and SLES10 is NUMA aware. The kernel can detect whether it is running on a NUMA system. Libraries and a control interface are available to optimize the execution of an application for NUMA. Oracle 10g for x86_64 platform does not use NUMA optimization. Oracle 10g runs on every system with Linux x86_64 as on an SMP system. Thanks to the IBM X3 Architecture, the x3950 can run applications that are not NUMA aware, with very good performance. The Oracle workloads generally benefit from up to 8
June 2007
socket dual core x3950. IBM recommends to get the advice from IBM Advanced Technical Support for Oracle implementations with servers of more than 8 sockets. The AMD Opteron servers of less than 4 sockets, like the x3655 and x3755, run Oracle workloads with very good performance even though Oracle does not use NUMA optimizations. The AMD HyperTransport technology reduces the difference of memory access time to a minimum, and enables Oracle RDBMS to perform very well.
Oracle licensing
Intel and AMD have made available dual core CPU. At the time this paper is written, only Intel has made available quad core CPU. The Oracle database Enterprise Edition (EE) licensing is based on the Licensing Unit. See the definition and update on the Oracle licensing at the Oracle Store: http://oraclestore.oracle.com An AMD or Intel single core CPU is considered as 1 licensing unit. One core of an AMD or Intel dual core or quad core CPU is considered as 0.50 of a licensing unit. Therefore, a x3850 with 4 dual core CPU is licensed with 4 Licensing units. An x3650 with 2 quad core CPU is licensed with 4 licensing units. Table 2 shows the number of Oracle licensing units for the IBM Blades and System x servers commonly implemented with RAC. Model HS21 HS21 LS21 LS41 x3650 x3650 x3655 x3755 x3850 x3950 Blades CPU technology Intel Xeon dual core Intel Xeon quad core AMD Opteron dual core AMD Opteron dual core Intel Xeon dual core Intel Xeon quad core AMD Opteron dual core AMD Opteron dual core Intel Xeon dual core Intel Xeon dual core # of sockets 2 2 2 4 2 2 2 4 4 8 # of cores 4 8 4 8 4 8 4 8 8 16 Oracle EE Licensing Units 2 4 2 4 2 4 2 4 4 8
For a RAC implementation, the RAC license is added on top of the Oracle database Enterprise Edition (EE) license. The Oracle Standard Edition is licensed by server CPU socket. It is limited to 4 socket as part of a RAC cluster or single instance. There is no additional license cost for Oracle RAC (Note that Oracle ASM must be used). Therefore you can run Oracle Standard edition on a one x3850 4 sockets dual core or on 2 x3650 with RAC each with 2 sockets of quad core CPU.
Scalability considerations
For an Oracle RAC implementation, the node internal scalability and the RAC cluster scalability have to be considered. Implementing a RAC cluster with few nodes, each with many processor cores is a scale-up
Rack mount
Table 2: Selected System x and BladeCenter servers with Oracle Licensing Units
June 2007
approach. Implementing a RAC cluster with many nodes, each with few processors cores is a scale-out approach. The IBM x3950 can have up to 16 sockets and 32 cores with Linux. Although for Oracle with Linux, IBM recommends to consider the x3950 with 8 sockets and 16 cores. For an implementation with a x3950 with more CPU, IBM recommends to test the workload with the IBM Advanced Technical Support (ATS). The scalability of the x3950 with more than 8 sockets/16 cores is workload dependent. Does Oracle RAC scale well? This question cannot be answered. The scalability of Oracle database workloads depends on the nature of the workload and the design of the database. A read-only database workload usually scales very well with RAC. The read-write database workload scalability depends on different elements:

The design of the database/workload The user type per instance affinity or action type per instance affinity
A database workload with 1000 users intensively updating a small table would likely not scale well with RAC. This is a database design problem, a heavy concurrency on database blocks should be avoided. They are called hot blocks. Reducing the total number of requests for blocks held by a remote database instance is usually a key to improve RAC scalability. In a RAC cluster, the time taken for a node to access a database block is either the read time to the storage subsystem or the time taken for shipment the block across the RAC interconnect network. To update the block, some negotiation between the requesting node and remote node has to be done. Therefore it is ideal when a RAC node accesses only its own database blocks. This is the notion of user type per instance affinity or action type per instance affinity. For example, in an ERP implementation with RAC with financial, manufacturing, sales and human resource users, one node would be dedicated to the financial users, another node for manufacturing, another for sales, and a fourth node for HR. This would lead to fewer remote block requests than mixing the different users on all the nodes in the cluster. This type of configuration is achieved by the definition of services for the different type of users. The services would have a preferred instance and available backup instance in case of failover. IBM recommends to involve IBM Advanced Technical Support (ATS) for Oracle RAC projects with more than 4 nodes managing a single database. Note that more than one database can be managed in an Oracle RAC cluster. For example, an 8 node RAC cluster can manage 4 databases, each of the databases would be managed by 2 nodes. This does not increase the interconnect traffic.
Infiniband
The IBM System x server and BladeCenter can be configured with Infiniband hardware. As discussed previously, the RDS protocol supported for Oracle RAC offers an advantage for RAC interconnect traffic. Oracle internal interconnect test tool shows 50% less CPU utilization with RDS over Infiniband than UDP/IP over Infiniband, the latency of UDP with RDS (no user-modeacks), and a 50% faster cache to cache Oracle block throughput. Note that all workloads may not benefit directly from a low latency RAC interconnect such as RDS with Infiniband. A RAC database application for which the performance bottleneck is
June 2007
on the cache fusion, will benefit from a low latency interconnect. But for example, if the application response time is due 70% to I/O, 20% to CPU, 5% to interconnect activity and 5% to other, the interconnect performance may only provide a few percent improvement at the application level. At the time this paper is written, the open RDS driver is not available. The Qlogic/Silverstorm proprietary RDS driver is available as part as SilverStorm's QuickSilver package (version 3.2 or later). The Qlogic/Silverstorm RDS driver only works with Qlogic/Silverstorm Infiniband hardware.
High Availability Considerations

High availability is usually a key requirement for RAC implementations. From a hardware configuration standpoint, this means eliminating single points of failure. IBM products are designed for high availability, with such standard features as redundant power supplies and cooling fans, hot-swappable components, and so on. For high availability environments, the following recommendations should also be taken into consideration when selecting the server: 1. Configure additional network interfaces and use ip bonding to combine at least two network interfaces for each of the two RAC networks. This reduces the downtime due to a NIC or network component failure. Along with redundant adapters, it is a good idea to have different physical paths for the network. This means that the NICs used for bonding should be on separate physical network cards and connected to different network switches. 2. There should be two access path to the storage from the nodes. For SAN storage implementations, there should be at least two fibre channel host bus adapters (HBA) on each node to provide redundant IO paths to the storage subsystem. Along with multiple fibre channel ports, the SAN must also be configured with redundant components, such as SAN switches, cabling, and so on. For NAS storage implementations, there should be at least two bonded network interfaces to access the NAS. 3. We recommend a kernel crash utility configured for every node in the cluster. With proper configuration, if a server crashes and throws a kernel panic, the kernel dump is saved. This core dump can then be analyzed and the problem fixed, dramatically reducing problem resolution time. The kernel dump can be saved on a separate server (dump over the network) in the datacenter or locally on the nodes (dump to local disk). Linux Kernel Crash Dump (LKCD) is the default kernel crash utility for SLES10. It supports dump over the network and dump to local disk. Netdump and diskdump are the default kernel crash utilities for RHEL4. Netdump supports dump over the network. Diskdump supports dump to local disk. On RAC implementations with a cluster file system, system architects sometimes like to have a separate network dedicated to cluster file system communication. This requires an additional network interface; two if using ip bonding. This may be more of a performance issue and may be dependent on the cluster file system product. For Oracle Cluster File System (OCFS), Oracle states that using the network interconnect is adequate. For this discussion, we will not include this as an additional network requirement. To summarize, for high availability, a server for an Oracle RAC implementation requires a
June 2007
minimum of 4 network interfaces and 2 Fiber Channel HBA with SAN storage, or 6 network interfaces with NAS storage.
14/82
June 2007
Configuration examples
BladeCenter H with Blade servers HS21/LS21 attached to an IBM DS4800
Each blades server used for Oracle RAC need two network interfaces for the public network, two other for the private network and two Fiber Channel HBA ports. The Blades also needs a single or redundant internal hard drive or an IBM 4GB Modular Flash Drive for the operating system. With these requirements, there are two way to configure the BladeCenter and the Blade servers. The first type of configuration is a high density configuration, there can be up to 14 HS21 or LS21 blades in the BladeCenter H used for Oracle RAC. The second type of configuration is a lower density configuration, the blades servers would take 2 slots in the BladeCenter, therefore the BladeCenter can host up to 7 blade servers used for Oracle RAC. See Figure 2 for the front view of the BladeCenter H with the 14 Blades slots.
Figure 2: Front view of the BladeCenter H
High density BladeCenter H configuration: The two on board network ports from the HS21 or LS21 will be used for the public network. A Qlogic Gigabit Ethernet and Fiber Channel CFFh card (part number 39Y9306) will be used for the the private network and the shared storage access. The Blades needs an single or redundant internal hard drive or an IBM 4GB Modular Flash drive for the operating system. The BladeCenter Chassis H needs six IO modules, two vertical layer 2/3 network switch modules for the public network in the switch module bay 1 and 2, and two horizontal MultiSwitch Interconnect Module (MSIM, part number 39Y9314) in high speed switch module bay 7 and 9, each of the horizontal modules are contains a layer 2/3 network switch module for the private network and a Fiber channel SAN fabric module for the shared storage access. See Figure 3.
June 2007
Optionally, for high IO throughput workload a small form factor Qlogic Fiber channel card (part number 41Y8527) can be added to each blades, two Fiber channel SAN fabric modules would be added in switch module bay 3 and 4. This raises the FC IO theoretical throughput of each blades from 8Gbits/sec to 16Gbits/sec. For the whole BladeCenter H, the total FC IO throughput is raised from 12Gbits/sec to 24GBits/sec (with 20 ports SAN switch modules). Lower density BladeCenter H configuration: The HS21 or LS21 can be configured with a memory and IO expansion unit (part number 42C1600). Four on board network ports are available for the public network and the private network. It is important to configure for bonding the first 2 network ports together for the public network, and the 2 network ports from the expansion unit together for the private network. This is for a high availability point of view. Each Blades must have a 4Gbps FC card for shared storage access. The Blades needs an single or redundant internal hard drive or an IBM 4GB Modular Flash drive for the operating system. The BladeCenter Chassis H needs four IO modules, two vertical layer 2/3 network switch modules for the public and private network in the switch module bay 1 and 2, two Fiber channel SAN fabric modules for the storage subsystem attachment in switch module bay 3 and 4. See Figure 3. It is a good practice to configure VLANs on the network switch modules to separate the public network from the private network traffic.
Figure 3: Rear view of the BladeCenter H
Shared storage In this configuration example, we use a Fiber Channel storage, the IBM System Storage DS4800, for the shared storage. Any other System Storage model from the DS4000 or
June 2007
DS8000 families could be implemented. The System Storage models from the DS3000 family are not tested and supported for RAC at the time this paper is written. A Network Attached Storage (NAS) can be implemented instead of a Fiber Channel SAN storage. The IBM System Storage N series models are certified for Oracle RAC. For an implementation with NAS, the 4GBps FC SAN switch modules from the MSIM have to be replaced with with two Gigabit Ethernet switch modules. Each MSIM will contain 2 Gigabit Ethernet switch modules Figure 4 shows the DS4800 with disk enclosures EXP810. There can be a maximum of 4 disk enclosures per channels. The FC switch modules from the MSIM in the BladeCenter bay 7 and 9 and from bay 3 and 4 are connected to Host channels from Controllers A and B.
Figure 4: DS4800 with enclosures EXP810
Cluster of x3650/x3655/x3755/x3850/x3950 attached to a DS4800

Each servers, x3650, x3655, x3755 or x3850 used for Oracle RAC need two network interfaces for the public network, two other for the private network and two Fiber Channel HBA ports. The servers also needs a single or redundant internal hard drive for the operating system. The two on board network ports from the servers will be used for the public network. A dual port Gigabit Ethernet card will be used for the the private network. A dual port 4GBps Fiber Channel card will be used for the shared storage access.
June 2007
The servers can be connected to the storage subsystem directly or through SAN fabrics. Figure 5 and 6 have been extracted from the IBM Redbook DS4000 Best Practices and Performance Tuning Guide, SG24-6363-02. For comprehensive information of the DS4000 series, refer to this Redbook. Figure 5 shows a configuration of two System x servers with a redundant SAN fabric and a DS4800. One of the SAN fabric is connected to the controller A of the the DS4800, the second SAN fabric is connected to the controller B of the SAN fabric. The first FC HBA port from each server is attached to the first SAN fabric, and the second FC HBA port is attached to the second SAN fabric. This configuration guaranties the redundancy of the server to storage attachment.
Figure 5: Two node cluster with two SAN fabrics and the DS4800
Figure 6 shows a configuration of four System x servers directly attached to a DS4800. The first FC HBA port from each server is attached to the controller A of the DS4800, and the second FC HBA port is attached to the controller B of the DS4800. This configuration guaranties the redundancy of the server to storage attachment. The nodes of a same Oracle RAC cluster need to access both controllers of a DS4000 and DS3400. For an implementation with SAN fabrics, the maximum number of nodes in a RAC cluster is determined by the number of ports available on the SAN fabric. For an implementation with nodes directly attached to the storage subsystem, the following applies:
June 2007
the DS4800 has 4 host ports on each controller, a maximum of 4 nodes can be directly attached to the storage the DS4700 model 70 has 2 host ports on each controller, a maximum of 2 nodes can be directly attached to the storage the DS4700 model 72 has 4 host ports on each controller, a maximum of 4 nodes can be directly attached to the storage the DS4200 has 2 host ports on each controller, a maximum of 2 nodes can be directly attached to the storage the DS3400 has 2 host ports on each controller, a maximum of 2 nodes can be directly attached to the storage
Figure 6: Four node cluster directly attached to the DS4800
Three node x3650/x3655 attached to a DS3200 At the time this paper is written, the IBM System Storage DS3200 is being tested for Oracle RAC clusters. Contact the IBM/Oracle International Competency Center or IBM ATS for an update on the support of Oracle RAC with the DS3200. The IBM System Storage DS3200 communication technology is 3 Gbps Serial Attached SCSI (SAS). The IBM System Storage DS4000 series and DS3400 (discussed in the previous configurations) communication technology is 4 Gbps Fiber Channel. With the Fiber Channel technology, the limit is of servers that can be attached to the storage subsystem is fairly high thanks to the availability of Fiber Channel SAN fabrics switches. At the time this paper is written, the SAS switches are not available. An implementation of Oracle RAC with DS3200 is only possible with servers directly attached to the DS3200. The DS3200 has a maximum of 3 host interface per controllers, therefore a maximum of 3
June 2007
Oracle RAC nodes is possible. At the time this paper is written, the x3755, x3850 and x3950 are not supported with the DS3200. The DS3200 is designed for relatively small IO workloads. Figure 7 shows the cabling of three servers each with 2 SAS HBA.

S is the IBM System Storage DS3200 D is the DS3200 SAS 2 port daughter card, necessary to expand the number of host connections from 1 to 3 per controllers. H is the IBM SAS HBA controller C is the IBM SAS cable (1 or 3 meters)
Figure 7: Three RAC nodes attached to a DS3200
20/82
June 2007
Software requirements
Operating System
In this paper, we will focus on the following Linux operating systems:
Novell SuSE Novell SuSE Linux Enterprise Server 10 for AMD64 and IntelEM64T Novell SuSE Linux Enterprise Server 10 for x86 Red Hat Red Hat Enterprise Linux 4 for AMD64 and IntelEM64T Red Hat Enterprise Linux 4 for x86
For the latest information on supported operating systems for IBM BladeCenter and System x, visit http://www-03.ibm.com/servers/eserver/serverproven/compat/us/ For the latest information regarding IBM hardware certification by Red Hat and Novell SuSE, refer to:
Red Hat https://hardware.redhat.com/hwcert/index.cgi#list Novell SuSEs web site: http://developer.novell.com/yessearch/Search.jsp
Storage related software

The multipath IO (MPIO) driver and the software that manages the configuration of the storage (for example: IBM DS3000 or DS4000 Storage Manager) has to be downloaded and installed. The Device Mapper (DM) driver, included in the Linux distributions, can do multipath IO. Device mapper is not supported with any storage subsystem from the DS3000 and DS4000 series. Device mapper is supported with the storage subsystem from the DS6000 and DS8000 series with Red Hat EL 5 and SuSE SLES 10. It can be supported on special request with RHEL4. The request has to be addressed to IBM Storage Advanced Technical Support (ATS). Downloads for IBM System Storage DS3200: http://www-304.ibm.com/jct01004c/systems/support/storage/disk/ds3200/firmware1.html Downloads for IBM System Storage DS3400: http://www-304.ibm.com/jct01004c/systems/support/storage/disk/ds3400/firmware1.html Downloads for IBM System Storage DS4200: http://www-304.ibm.com/jct01004c/systems/support/storage/disk/ds4200/firmware1.html Downloads for IBM System Storage DS4700: http://www-304.ibm.com/jct01004c/systems/support/storage/disk/ds4700/firmware1.html Downloads for IBM System Storage DS4800: http://www-304.ibm.com/jct01004c/systems/support/storage/disk/ds4800/firmware1.html Downloads for the Multipath Subsystem Device Driver (SDD) or Device Mapper instructions
June 2007
for IBM System Storage DS6000 and DS8000: http://www1.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D430&uid=ssg1S4000107& loc=en_US&cs=utf-8&lang=en For EMC storage subsystems, you will need EMC PowerPath. Refer to EMC documentation or contact an EMC representative. VERITAS DMP can be implemented for some storage subsystems. Refer to VERITAS/Symantec documentation or contact an VERITAS/Symantec representative.
Oracle 10g
Oracle10g Release 2 is the current release of Oracles database product and is available on 32 and 64-bit Linux/Intel platforms. Version 10.2.0.1 is available by CD pack or by download from Oracle Technology Network (OTN). The 32-bit and 64-bit Oracle database products are differentiated by how they are named. For 32-bit Linux, Linux x86 is part of the product name, for 64-bit EM64T/AMD64, it's Linux x86-64. As mentioned earlier, Oracle does not support running the 32-bit product under 64-bit Linux (x86-64). Note that the product for 64-bit Intel Itanium-based server is a separate product and is known by the Linux Itanium designation in its name. For the latest information on Oracle product certification, refer to: http://www.oracle.com/technology/support/metalink/index.html RAC is a separately licensed option of Oracle Enterprise Edition. Oracle Standard Edition includes RAC and is limited to a maximum configuration of four sockets, single or clustered servers. RAC with Standard Edition requires the use of Automatic Storage Management (ASM). For the latest information about Oracle licensing, including the licensing policy for multi-core Intel/AMD processors, refer to http://oraclestore.oracle.com.
Optional Software Products

In the Oracle RAC cluster, there are 3 types of files to store:

the Oracle database binaries the RAC voting disk and Oracle Cluster Registry (OCR) the databases files
The Oracle binaries files can either be stored locally on a filesystem (ext2/3 or reiserfs) on each nodes or shared by multiple nodes on a Cluster filesystem (like OCFS, GFS...). A local filesystem does not require extra software products. A Cluster filesystem require the installation of the chosen cluster filesystem software package, see the following section on OCFS. The RAC voting disk and OCR can be stored in raw devices or on a Cluster filesystem (like OCFS, GFS...). Raw devices does not require extra software products. A Cluster filesystem require the installation of the chosen cluster filesystem software package, see the following section on OCFS.
June 2007
The choice of raw devices for the voting disk and OCR may be done when the Oracle datafiles are stored in an ASM instance, because the voting disk and OCR are 2 files that are created at the installation of the cluster and will stay the same all through the life of the RAC cluster. The choice of a Cluster filesystem may be done when the Oracle datafiles are stored in a Cluster filesystem. The installation and configuration of the filesystem is done so it does not require more work or software to store to the voting disk and OCR on this Cluster filesystem. The database files can be stored in raw devices, on a Cluster filesystem (like OCFS, GFS...) or in an Automatic Storage Management (ASM) instance. Few modern RAC implementation are done with raw devices for datafiles. A Cluster filesystem choice require the installation and configuration of the Cluster filesystem software package, see the following section on OCFS. The ASM choice require the installation of the ASMlib (see next section) or the configuration of raw devices to be used with ASM. We recommend the installation of ASMlib.
Automatic Storage Management Library (ASMlib)

Automatic Storage Management (ASM) is a new feature in Oracle 10g that provides file system and volume management capabilities for Oracle. The storage devices can be presented to the ASM instance in one of the following ways:

As raw devices (ie. /dev/raw/raw1) As block devices (ie. /dev/sda1) using Oracle's Automatic Storage Management Library (ASMlib)
Oracle provides a separate Automatic Storage Management library driver (ASMlib) that simplifies the configuration and management of the disk devices that you use with ASM. A disk that is configured for use with Automatic Storage Management is known as a candidate disk. For the latest packages of the ASMlib drivers, libraries and tools, visit: http://www.oracle.com/technology/tech/linux/asmlib/index.html These packages are specific to the Linux distributions (RHEL, SLES), the update level (U6 for RHEL or SP2 for SLES) and the server architecture (x86 or x86_64). Three packages have to be downloaded, Library and Tools

oracleasm-support-<version>.rpm oracleasmlib-<version>.rpm
Drivers for specific kernels oracleasm-<kernel version>.rpm
OCFS
OCFS Version 2 (OCFS2) is the latest generation of the Oracle Cluster File System for Linux. It is a POSIX compliant, general-purpose file system that can be used for shared Oracle home installations, making management of Oracle Real Application Cluster (RAC) installations even easier. OCFS2 is supported with the 2.6 kernel, therefore you can run OCFS v2 with RHEL4 and
June 2007
SLES9, and SLES10 but not with the 2.4 kernel-based distributions, such as RHEL3, RHEL2.1, United Linux 1.0. OCFS2 is packaged with Novell SLES9 SP2+. For RHEL4, download the OCFS2 packages from http://oss.oracle.com/projects/ocfs2/.
24/82
June 2007
Set Up Server Environment

BIOS
Upgrade the system BIOS and adapters BIOS to the latest level. Look for your System x model on http://www.ibm.com/support/us/ or at http://www304.ibm.com/jct01004c/systems/support/supportsite.wss/docdisplay?lndocid=MIGR4JTS2T&brandind=5000008 For Intel Xeon based System x servers, such as the x3650, x3850, x3950 and IBM blade HS21, for the best performance, the following BIOS options have to be set for Oracle workloads:

High Performance Memory Array: Enabled HW Prefetch to: Disabled Adjacent Sector prefetch: Enabled
The Intel Xeon based System x servers have high performance memory mapping features. This feature is enabled in BIOS and increases performance by up to 20%. This feature is automatically enabled on the x3650 whenever EXACTLY 4 dual-ranked DIMMs or 8 singleranked DIMMS are installed. For AMD Opteron based System x servers, such as the x3655, x3755 and IBM blade LS21, for the best performance, the following BIOS options have to be set for Oracle workloads:

Memory Node Interleave: Disabled System Cache Type: Write Back High Precision Event Timer: Enabled MMIO Alignment for Linux Numa: Disabled
Remote System Management

On the newer System x servers, the optional Remote Supervisor Adapter provides a number of system management functions, including:

Automatic notification and alerts Continuous health monitoring and control Event log Remote power control Server console redirection
On the BladeCenter, the Management Module functions as a system-management processor and a keyboard/video/mouse-multiplexing switch for the blade servers. It provides keyboard, video, and mouse ports for a local console and a 10/100 Ethernet port which provides access to the system management processor. The system management processor communicates with other BladeCenter components, providing functions such as:
June 2007
Status monitoring of the blade servers, switch modules, power modules, blower modules Blade server management and control, e.g. power/restart, upgrading firmware, switching the keyboard/video/mouse, etc., in conjunction with the blade server service processors Switch module configuration, such as enabling/disabling external ports Remote console
Set up the Ethernet ports on the RSA II or BladeCenter Management Module and connect them to your management LAN. For information and instructions, refer to the IBM Redbook, IBM eServer System x and BladeCenter Server Management, SG24-6495-00. IBM Redbooks are available at: http://www.redbooks.ibm.com/
Installing Linux Operating System

We will not describe installation of the operating systems in detail here, but rather, refer you to the operating system vendor documentation. If you are installing the default operating system (vs installing everything), a number of steps are important: 1. Download drivers that may not be included in the Linux distribution at http://www304.ibm.com/jct01004c/systems/support/supportsite.wss/docdisplay?lndocid=MIGR4JTS2T&brandind=5000008 2. Be sure to create sufficient swap space appropriate for the amount of physical memory on your servers. The general recommendation is:
Physical RAM
1 2GB 2 8GB
Swap Space
1.5 times size of RAM Equal size of RAM
> 8GB .75 times size of RAM Table 3: Swap space requirement
3. We recommend not to start XWindows automatically in the production server. The server should run in runlevel 3. 4. During the installation process, in the software package selection section,

Select the development packages, e.g. C/C++ Compiler and Tools If you are using IBM System Storage DS4000, install kernel-source and kernel-sym It is also useful to install Linux tools such as ftp server (vsftpd), telnet server, and others.
For the list of packages to be installed and other OS requirements, check Metalink Note:169706.1, Oracle RDBMS on AIX,HP-UX,Solaris, Tru64, Linux, MacOSX: Versions, Sizes, Requirements Quick Reference.
Set up a Linux kernel crash utility

As mentioned earlier, if a server crashes and throws a kernel panic, the kernel dump can be sent to a network attached server or to a partition on the local disk. This is done using one of the Linux kernel crash utilities, LKCD for SLES9, netdump or diskdump for RHEL3/4.
June 2007
The kernel crash dump can then be analyzed and the problem fixed, dramatically reducing problem resolution time. Refer to Appendix B for details on how to set up these kernel crash utilities.
Network
The following network addresses are required for each node: Public network address Private network address Virtual IP network address Set up the interfaces and IP addresses for the public and private networks. You can do this by modifying the system configuration files or using the GUI utilities - yast2 for SLES, redhat-config-network for Red Hat. For instructions on how to set up ip bonding, see Appendix A.

Add the host names and IP addresses to /etc/hosts as shown in the example below. If the public host names and IP addresses are registered to the Domain Name Server (DNS), you may exclude them. Interconnect (private) host names and IP addresses should always be placed in /etc/hosts. /etc/hosts
127.0.0.1 100.100.100.101 100.100.100.102 100.100.100.111 100.100.100.112 1.1.1.1 1.1.1.1 localhost.localdomain localhost x1.domain.com x2.domain.com x1-vip.domain.com x2-vip.domain.com x1-priv.domain.com x2-priv.domain.com x1 x2 x1-vip x2-vip x1-priv x2-priv
We recommend setting up the network from the console. If you do this with a network connection, you could lose the network connection to the server.
Multipath IO driver
In this section, we outline the installation of:

IBM RDAC for IBM DS4x00 storage subsystems IBM SDD for IBM DS6x00 and DS8x00 storage subsystems
For the full installation instructions, please refer to the readme files accompanying the driver packages. For third party Multipath IO drivers, refer to the third party documentation.
IBM RDAC Driver

If you are using IBM System Storage DS4000 series (FAStT) you will need to install the RDAC driver. The RDAC driver works with the QLogic driver or Emulex driver depending of the Host Bus Adapter present on the server. You may also download the driver from IBM at the URL for the storage subsystem, see Storage related downloads on page 21. If you download the driver from IBM, be sure to install the non-failover version.
27/82
June 2007
For full details on the RDAC installation, refer to the RDAC readme. We are just giving the main steps of the installation. 1. Disable Automatic Volume Transfer (AVT) on the DS4000 storage subsystem for the nodes in the cluster. 2. Reminder: You will also need the kernel-source and kernel-syms packages listed in the section entitled Installing Linux Operating System. If you did not install them earlier, be sure to do so now. 3. On SLES, compile the kernel modules with a kernel name identical to the current one. 4. Untar the RDAC tarball. This creates the directory linuxrdac-xx.xx.xx.xx, this name includes the version of RDAC. 5. Go to this directory. Build and install the loadable module for your kernel with the following commands:
make clean make make install
6. Add an entry to the GRUB boot loader configuration to boot the same kernel with the new ramdisk file, edit, add the following entry to /etc/grub/grub.conf (on RHEL) or /boot/grub/menu.lst (SLES):
title Red Hat Enterprise Linux AS (2.6.9-22.ELsmp) with MPP support root (hd0,0) kernel /vmlinuz-2.6.9-22.ELsmp ro root=LABEL=/ rhgb quiet initrd /mpp-2.6.9-22.ELsmp.img
7. Reboot the server. Now, you can access the devices from the DS4000 subsystem with multipath through the usual device names: /dev/sda, /dev/sdb, /dev/sdc, and so on. 8. Failover testing. It is a good practice to test the storage paths failover in order to check the installation of RDAC and to calculate the failover completion time. We can observe the following failover completion time in our environment:
Between 30 and 40 seconds of failover time for a failure between the servers and the SAN fabric. Between 70 and 80 seconds of failover time for a failure between the SAN fabric and the storage subsystem.
Knowing this failover time is necessary for the OCFS configuration.
IBM SDD
If you are using DS6800, DS8000 or ESS, Install the following IBM SDD packages for the appropriate operating system.
28/82
June 2007
Use the following command:

rpm -ihv <IBM SDD package>
Set up SDD to load automatically during system startup. Use the command:
chkconfig --level 35 sdd on
29/82
June 2007
Shared Storage setup

LUN creation
The shared storage from SAN will be logical drives (LUNs) configured to be accessed from HBAs from all nodes. LUN creation is different from one storage subsystem model to the next. For the DS4000 series, IBM DS4000 Storage Manager is used, for the DS6000 series, IBM DS6000 Storage Manager is used. Refer to the appropriate documentation for details on how to use these utilities. The directory /proc/scsi on the server nodes will contain a subdirectory named qla2xxx or qla2300, depending on the driver installed. In this subdirectory, there will be one or more files named 0, 1, 2 or 3 containing information about the HBAs installed on the system. To configure access to the logical drives, you have to map the LUNs to the adapter port addresses (World Wide Names) of the nodes' HBAs. Obtain the adapter port address from the HBAs on each node with the following command:
x1:~ # grep port /proc/scsi/qla2xxx/* /proc/scsi/qla2xxx/2:scsi-qla0-adapter-port=210000e08b0a4d67; /proc/scsi/qla2xxx/3:scsi-qla1-adapter-port=210000e08b0a8d2b; x2:~ #
If the SAN is set up properly, Storage Manager will detect and list these port addresses. All you need to do is pick from Storage Manager's list when mapping the LUNs. Several LUNs have to be created for the Oracle RAC cluster:
One or more LUNs for the Clusterware voting disk and Oracle Cluster Registry (OCR) The Oracle Clusterware can be configured with 1 or 3 voting disks. OCR can be configured with 1 or 2 devices. The number of devices depends on whether you chose to let the storage subsystem or Oracle Clusterware manage mirroring. For a Storage Subsystem which has RAID 1 or 5 arrays and hot spare disks, one Voting disk and one OCR should be sufficient. For a Storage subsystem with no RAID 1 or 5 capability, or if you are implementing more than one Storage Subsystem, Oracle Clusterware can be configured with 3 voting disks and 2 OCR devices. One or more LUNs for the Oracle database files. These LUNs can be RAID 1, 5 or 10 arrays depending on the availability and price/performance requirements. For Oracle database files on a cluster file system like OCFS, you need one LUN per file system. For Oracle data files in an ASM instance, we recommend configuring multiple LUNs with the same size in arrays of the same RAID level and same number of disks.
Once the new LUNs have been created on the storage subsystem, the HBA driver needs to be reloaded in order to use the devices on the nodes. You may see the devices listed twice under different names if you have not installed a multipath IO driver.
DS4000
To rescan the storage LUNs from the nodes, it is recommended that you reboot the nodes.
June 2007
Check that the devices are created, give read and write permission to oracle user (as member of disk group):
x1:/home/oracle # ls -l brw------- 1 root root brw------- 1 root root x1:/home/oracle # chown x1:/home/oracle # ls -l brw------- 1 root disk brw------- 1 root disk x1:/home/oracle # chmod x1:/home/oracle # ls -l brw-rw---- 1 root disk brw-rw---- 1 root disk /dev/sda* 252, 0 Nov 2 11:18 252, 64 Nov 2 11:18 root:disk /dev/sd* /dev/vpath* 252, 0 Nov 2 11:18 252, 64 Nov 2 11:18 g+rw /dev/sd* /dev/sd* 252, 0 Nov 2 11:18 252, 64 Nov 2 11:18 /dev/sda /dev/sdb
/dev/sda /dev/sdb
/dev/sda /dev/sdb
To list the discovered storage logical drives (LUNs), you can also run:
fdisk -l
Now, you can access the LUNs on the DS4000 subsystem with multipath through the sdX devices, /dev/sda, /dev/sdb, and so on.
DS6000
To rescan the storage LUNs from the nodes, run (on each node) as root user:
modprobe -r qla2300 modprobe qla2300
Run the following command to start SDD:

x1:~ # sdd start
Run the following command to view the virtual path configuration:

x1:~ # lsvpcfg 000 vpatha ( 252, /dev/sdi 001 vpathb ( 252, /dev/sdj x1:~ # 0) 13ACZBA1008 = 600507630efe01310000000000001008 = /dev/sdc 64) 13ACZBA1009 = 600507630efe01310000000000001009 = /dev/sdd
Check that the devices are created, give read and write permission to oracle user (as member of disk group):
x1:/home/oracle # ls -l brw------- 1 root root brw------- 1 root root x1:/home/oracle # chown x1:/home/oracle # ls -l brw------- 1 root disk brw------- 1 root disk x1:/home/oracle # chmod x1:/home/oracle # ls -l brw-rw---- 1 root disk brw-rw---- 1 root disk /dev/vpath* 252, 0 Nov 2 11:18 252, 64 Nov 2 11:18 root:disk /dev/vpath* /dev/vpath* 252, 0 Nov 2 11:18 252, 64 Nov 2 11:18 g+rw /dev/vpath* /dev/vpath* 252, 0 Nov 2 11:18 252, 64 Nov 2 11:18 /dev/vpatha /dev/vpathb
/dev/vpatha /dev/vpathb
/dev/vpatha /dev/vpathb
To list the discovered storage logical drives (LUNs), you can also run:
fdisk -l
Now, you can access the LUNs on the DS6000/DS8000 subsystem with multipath through the vpath devices, /dev/vpatha, /dev/vpathb, and so on.
June 2007
Shared storage configuration

As discussed earlier, the Oracle Clusterware voting disk and OCR cannot be stored in ASM. They have to be stored on a cluster file system, such as OCFS, or on raw devices. The voting disks and OCR are vital resources for Oracle Clusterware and we prefer using raw disks rather than a cluster file system because:

it does not require additional software we feel this is a more straightforward solution, given that the voting disks and OCR are not changed very often
The shared storage for the Oracle datafiles can be stored in an ASM instance or on OCFS filesystems.
Raw devices
Identify the shared storage block devices you have reserved for the voting disks and OCR. Create partitions on each block device, and configure them as raw devices. The following disk space is required:

25MB (minimum) of shared storage for each voting disk 120MB (minimum) for each (OCR) device
As root user, run the fdisk utility, specify as the argument the storage block device you want to partition. For example fdisk /dev/sdc. Note that if you are using DS6800 and SDD, you will have to create partitions on the vpath devices, e.g. /dev/vpatha. On the fdisk command prompt, type 'm' for help, 'p' to list the current partitions. Type 'c' to create a new partition, then 'p' for a primary partition, enter the partition number '1', the first cylinder number or just Enter for the first available one, the last cylinder number or the size of the partition (+21M for example). Type 'w' to save the new partition table and exit. Reboot all the nodes to make them discover the new disk partitions. The next step is to configure the raw devices. Associating (binding) raw devices to block devices (i.e. /dev/sdc1) will make the IO direct to disk instead of going to the buffer cache. Any IO in a RAC cluster has to be direct to disk, this is true for the voting disk and the OCR. In SLES, the association is made in /etc/raw. Edit /etc/raw as follows:
raw1:sdc1 raw2:sdc2
In RHEL3 and 4, the association is made in /etc/sysconfig/rawdevices as follows:

/dev/raw/raw1 /dev/sdc1 /dev/raw/raw2 /dev/sdc2
In these two previous examples, the block devices are /dev/sdc1 and /dev/sdc2. These names would have been, for example, vpathc1 and vpathc2, if the IBM SDD multipath IO driver had been implemented.
June 2007
In the above example, two raw devices were created, one for voting disk and one for the OCR. This is satisfactory if you are using external redundancy. If you are going to let Oracle manage redundancy, you will need to create 2 OCRs and 3 voting disks. In order to apply the raw to block device association, on SLES, run
rcraw start
on RHEL, run
service rawdevices start
To make these bindings persistent through reboot,

chkconfig --level 35 raw on
Set permissions to allow oracle user (member of the disk group) access to the raw devices:
chmod 660 /dev/raw/raw*
Note that the permissions of the raw devices will be changed again during installation of Clusterware. Initialize all the voting disks and OCR devices using the dd command as shown. Do this for all the raw devices.
dd if=/dev/zero of=/dev/raw/raw1 bs=1024k
An error will occur when the dd command reaches the end of the device. This is an expected result as long as some blocks are written.
OCFS
The Voting disk file, the OCR file and the Oracle datafiles can be located on one or multiple OCFS filesystems. Check Oracle Metalink note 391771.1 OCFS2 Frequently Asked Questions for the configuration of OCFS. Use the tool ocfs2console to configure the cluster nodes and the filesystems. Run /etc/init.d/o2cb configure as root on each node to make OCFS started at each boot. The parameter O2CB_HEARTBEAT_THRESHOLD in the configuration file etc/sysconfig/o2cb has to be updated to half of the maximum storage path failover completion time plus 1. In our installation, with a DS4000 series, the failover time between the HBA and the SAN switch is 35 seconds and 70 seconds between the SAN switch and the DS4000 controllers, therefore we set O2CB_HEARTBEAT_THRESHOLDto36(70/2+1). When the filesystems are created, edit /etc/fstab on each node and add the mount information on the filesystems,forexample: /dev/sda1 /u01 ocfs2 datavolume,nointr00
Filesystems labels can be used instead of block device names. The Voting disk file and the OCR file have to be created manually on the OCFS filesystem. The Voting disk file has to be at least 20MB and the OCR has to be at least 110MB. Use the
June 2007
following commands:
dd if=/dev/zero of=/u01/VotingDisk.dbf bs=1024k count=20 dd if=/dev/zero of=/u01/OCR.dbf bs=1024k count=110
ASM
The configuration of the shared storage for ASM is shown in the Chapter Database creation on page 47. ASM can only be configured when the Oracle Database binaries are installed.
34/82
June 2007
Installing Oracle 10g R2 RAC

Pre-Installation Tasks
Perform the following tasks to prepare for 10g R2 installation. 1. Kernel parameters Oracle database requires that some kernel parameters be changed. Edit /etc/sysctl.conf, and apply the changes with the command:
sysctl -p
Set the value of SHMMAX to the largest SGA size on the system, SHMALL to the sum of all the SGAs on the server. /etc/sysctl.conf
net.core.rmem_max = 262144 net.core.rmem_default = 262144 net.core.wmem_max = 262144 net.core.wmem_default = 262144 net.ipv4.ip_local_port_range = 1024 65000 kernel.shmall = 2097152 kernel.shmmax = 2147483648 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 65536
2. Create oracle user as follows:

groupadd oinstall groupadd dba useradd -g oinstall -G dba,disk -m oracle
3. Edit oracle user's .bash_profile:

# .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs export PATH=$PATH:$HOME/bin unset USERNAME export export export export export export export ORACLE_BASE=/oracle ORA_CRS_BASE=/oracle_crs ORACLE_HOME=$ORACLE_BASE/product/10.2.0/DB ORA_CRS_HOME=$ORA_CRS_BASE/product/10.2.0/CRS PATH=$PATH:$ORACLE_HOME/bin:$ORA_CRS_HOME/bin ORACLE_SID=TEST LD_LIBRARY_PATH=$ORACLE_HOME/lib
4. Create the home directories for the Oracle binaries, ORACLE_HOME and ORA_CRS_HOME. The minimum disk space requirements for each node are: 2GB for ORACLE_HOME 600MB for ORA_CRS_HOME Oracle recommends that the mount point for ORA_CRS_HOME be owned by root and the

June 2007
mount point for ORACLE_HOME be owned by the oracle user. After Clusterware installation, Oracle recommends that write access to ORA_CRS_HOME be given only to root. Therefore, ORA_CRS_HOME and ORACLE_HOME should be created under separate mount points. Use the following commands:
mkdir mkdir chown chown chmod -p -p -R -R -R /oracle/product/10.2.0/DB /oracle_crs/product/10.2.0/CRS oracle:oinstall /oracle root:oinstall /oracle_crs 775 /oracle_crs
5. Set oracle user's limits Edit /etc/security/limits.conf, and adjust the limits for the Oracle user. The value MEMLOCK is the size of the largest SGA on the system. /etc/security/limits.conf
oracle oracle oracle oracle oracle oracle soft hard soft hard soft hard nproc nproc nofile nofile memlock memlock 16384 16384 65536 65536 4000000 4000000
6. Set up ssh host equivalence for the oracle user. As oracle user, run ssh-keygen to generate RSA and DSA keys on all nodes. We have chosen not to use a passphrase.
oracle@x1:~> ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/oracle/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/oracle/.ssh/id_rsa. Your public key has been saved in /home/oracle/.ssh/id_rsa.pub. The key fingerprint is: 21:fb:88:8e:4a:07:df:02:ef:76:4f:6e:d6:88:4f:be oracle@x1 oracle@x1:~> ssh-keygen -t dsa Generating public/private dsa key pair. Enter file in which to save the key (/home/oracle/.ssh/id_dsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/oracle/.ssh/id_dsa. Your public key has been saved in /home/oracle/.ssh/id_dsa.pub. The key fingerprint is: fe:6f:81:a0:ba:fe:b3:8a:a0:ed:f5:00:0e:a2:51:b0 oracle@x1 oracle@x1:~> cd .ssh oracle@x1:~/.ssh> ls id_dsa id_dsa.pub id_rsa id_rsa.pub oracle@x1:~/.ssh>
The public keys will have to be known by all nodes in the cluster. Put the content of id_rsa.pub and id_dsa.pub into a file named authorized_keys.
oracle@x1:~/.ssh> cat id_rsa.pub > authorized_keys oracle@x1:~/.ssh> cat id_dsa.pub >> authorized_keys
Repeat these steps on all nodes, merge the content of authorized_keys file from all nodes into one single authorized_keys file, and place it in the /home/oracle/.ssh directory on all nodes.
June 2007
Do ssh connections between all the nodes in the cluster, and ssh connections on the nodes themselves (on node2, ssh node2). This will create an entry in the known_hosts file for the node, so you will not have to answer this kind of question again:
oracle@x1:~> ssh x1 The authenticity of host 'x1 (x.xx.xxx.xxx)' can't be established. RSA key fingerprint is 02:7f:d0:1c:d2:89:d8:bb:fc:d5:f1:ed:a9:42:da:85. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'x1,x.xx.xxx.xxx' (RSA) to the list of known hosts. Last login: Mon Dec 12 10:06:46 2005 from x2.sanmateo.ibm.com
Do this for all short host names and fully qualified host names. You should not be prompted for a password. If you are, check for the authorized_key files in /home/oracle/.ssh. If you chose to use a passphrase, you will need to run ssh-agent and ssh-add. Refer to Oracle Clusterware and Oracle Real Application Clusters Installation Guide for further details. 7. IO elevator Four algorithms are available for storage IO with Linux 2.6 kernel, the Noop, Deadline, Anticipatory (AS) and Complete Fair Queuing (CFQ) schedulers. The AS scheduler is designed for IO on individual disks, it is not good for storage subsystems like the IBM System storage DS family. The Deadline or Complete Fair Queuing (CFQ) can both be considered. CFQ is the default scheduler on RHEL and SLES. We would recommend to do some testing with both the Deadline and CFQ schedulers as the performance varies with the workload. For more informations on the IO schedulers, check http://www.redhat.com/magazine/008jun05/features/schedulers/ To change the scheduler at boot time, edit grub.conf or menu.lst and add the elevator parameter as a kernel argument with the value deadline or cfq:
title Red Hat Enterprise Linux AS (2.6.9-22.ELsmp) with MPP support root (hd0,0) kernel /vmlinuz-2.6.9-22.ELsmp ro root=LABEL=/ elevator=deadline initrd /mpp-2.6.9-22.ELsmp.img
8. Hangcheck-timer vi /etc/rc.d/rc.local, and add the following line at the end of the file. This will load the hangcheck-timer module at server startup. /etc/rc.d/rc.local
/sbin/insmod hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
Oracle 10g R2 clusterware installation

Oracle 10g Release 2 Database for Linux comes on one DVD in the CD pack, or you can download the product from: http://www.oracle.com/technology/software/products/database/oracle10g/index.html To install Oracle10g RAC, run the Oracle Universal Installer (OUI) from one node. For the most part, OUI handles installation to the other cluster nodes. There are a number of steps that need to be done on the other nodes and these are called out by OUI at various points during the process. If you run the installation from the system console, you will need to start an XWindows session. If you are installing from a network attached workstation, you can:
June 2007
Start vncserver on the node and run a vnc client from the workstation. Run XWindows on the workstation
Invoke OUI as follows:

cd <stage directory or mount point>/clusterware ./runInstaller
1. The first screen will be the Oracle Universal Installer Welcome screen. Click Next to continue. 2. The next screen will ask for the Inventory directory and the Operating System group name. Change the values as appropriate and click Next. 3. The next screen asks for Home Details. Modify the path for crs home directory (different from ORACLE_HOME) and click Next to continue. 4. The next screen, shown in Figure 8, shows the result of prerequisite checks performed by OUI
Figure 8: OUI prerequisite checks
Review the details for each each prerequisite check and correct the causes for those that did not succeed. Once corrected, click Retry to make OUI redo the prerequisite check. Once all the checks have succeeded, click Next to continue. 5. The next screen is to specify the cluster configuration. Initially, the screen will show only the current node. Click Add to add the other cluster nodes. For example, Figure 9 below shows an example of a 2-node cluster. We are running OUI on node x1 and added a second node, x2. Be sure that the host names are in all the /etc/host files.
38/82
June 2007
Figure 9: Specify cluster configuration
When finished, click Next to continue. 6. The next screen asks you to specify network interface usage. Examine the information in the Interface Type column. Make sure that the public and private networks are correctly designated. If not, highlight the one that is incorrect and click Edit. A window will pop up, allowing you to change the interface. When done, click OK. The resulting screen should designate one public and one private network, as shown in Figure 10.
Figure 10: Network interface usage
Notice in Figure 10 that we are using ip bonding to team NICs for redundancy. When you are finished with the network interface usage screen, click Next to continue. 7. The next screen asks you to specify the devices for the Oracle Cluster Registry (OCR).
June 2007
Here is where you will enter the names of the raw devices that were configured earlier. In our example, we leave Normal Redundancy selected, and enter /dev/raw/raw1 and /dev/raw/raw2 in the input fields for OCR locations as shown inFigure 11
Figure 11: OCR location
Note that in Figure 11, we have chosen Normal Redundancy and provided two OCR devices. Click Next to continue. 8. The next screen asks for the voting disk locations. Shown in Figure 12 is our selection of Normal Redundancy and the three voting disk raw devices, /dev/raw/raw3, raw4 and raw5. Click Next to continue.
Figure 12: Voting disk location
As we have done in the previous step, here we have also chosen Normal Redundancy
June 2007
and provided three voting disk devices. This means that Oracle will manage mirroring of the voting disks. Click Next to continue. 9. The next screen shows a summary of Oracle Clusterware that will be installed. Click Install to begin the installation. 10. The next screen shows the progress of the Installation. Note the installation log file, which provides more information on the steps being performed. 11. After the Installation progress screen, you will see the window shown in Figure 13, asking you to execute scripts as root user. IMPORTANT: Do this first on the node where you are running OUI. Execute both scripts on the first node before going to the 2nd and other nodes. DO NOT RUN THE SCRIPTS SIMULTANEOUSLY ON THE LISTED NODES.
Figure 13: Configuration scripts
12. The example below is the output from running root.sh on the first RAC node, x1, in this case.
41/82
June 2007
x1:~ # /oracle_crs/product/10.2.0/CRS/root.sh Checking to see if Oracle CRS stack is already configured /etc/oracle does not exist. Creating it now. Setting the permissions on OCR backup directory Setting up NS directories Oracle Cluster Registry configuration upgraded successfully Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: x1 x1-priv x1 node 2: x2 x2-priv x2 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Now formatting voting device: /dev/raw/raw3 Now formatting voting device: /dev/raw/raw4 Now formatting voting device: /dev/raw/raw5 Format of 3 voting devices complete. Startup will be queued to init within 90 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. CSS is active on these nodes. x1 CSS is inactive on these nodes. x2 Local node checking complete. Run root.sh on remaining nodes to start CRS daemons. x1:~ #
The example below shows the output from running root.sh on the second node, x2.
42/82
June 2007
x2:~ # /oracle_crs/product/10.2.0/CRS/root.sh Checking to see if Oracle CRS stack is already configured /etc/oracle does not exist. Creating it now. Setting the permissions on OCR backup directory Setting up NS directories Oracle Cluster Registry configuration upgraded successfully clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: x1 x1-priv x1 node 2: x2 x2-priv x2 clscfg: Arguments check out successfully. NO KEYS WERE WRITTEN. Supply -force parameter to override. -force is destructive and will destroy any previous cluster configuration. Oracle Cluster Registry for cluster has already been initialized Startup will be queued to init within 90 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. CSS is active on these nodes. x1 x2 CSS is active on all nodes. Waiting for the Oracle CRSD and EVMD to start Oracle CRS stack installed and running under init(1M) Running vipca(silent) for configuring nodeapps Creating Creating Creating Starting Starting Starting VIP GSD ONS VIP GSD ONS application application application application application application resource resource resource resource resource resource on on on on on on (2) (2) (2) (2) (2) (2) nodes... nodes... nodes... nodes... nodes... nodes...
Done. x2:~ #
If root is not the owner of the ORA_CRS_HOME mount point as described in the section Pre-Installation Tasks, you will see warning messages which should not adversely affect the installation at this point. If the scripts hang after the message Expecting the CRS daemons to be up within 600 seconds for more than 10 minutes, check the following: network interconnect configurations ssh equivalence /etc/hosts for the correct host names and addresses host names raw device configuration, definition and permissions You will need to restart the installation. Be sure to clean up the failed installation following Metalink note 239998.1, 10g RAC: How to Clean Up After a Failed CRS Install, Last Revision: 25-OCT-2005. 13. The next screen shows Configuration Assistants that configure and start Clusterware components configured. Check under the Status column for successful execution. Click Next to continue.
June 2007
14. The next screen indicates the end of Clusterware installation. Click Exit to end. 15. Change permissions of ORA_CRS_HOME, leaving write access only to root.
chmod -R g-w /oracle_crs
16. Hangcheck-timer In Red Hat, vi /etc/rc.d/rc.local, and add the following line at the end of the file. This will load the hangcheck-timer module at server startup. /etc/rc.d/rc.local
/sbin/insmod /lib/modules/<kernel version>/kernel/drivers/char/hangcheck-timer.ko hangcheck_tick=30 hangcheck_margin=180
In SLES, there is no equivalent to rc.local, therefore, insert this line in /etc/rc.d/boot.local.
Oracle 10g R2 database installation

Invoke OUI as follows:
cd <stage directory or mount point>/database ./runInstaller
1. The first screen will be the Oracle Universal Installer Welcome screen. Click Next to continue. 2. The next screen asks for Installation type. Select Enterprise Edition as shown in Figure 14, click Next to continue.
Figure 14: Installation type
3. The next screen asks for Home details. Enter the path of your ORACLE_HOME directory, if not already shown. Click Next to continue.
44/82
June 2007
4. The next screen, shown in Figure 15 asks for the Cluster Installation Mode. Select all the nodes to be in the RAC cluster, for example, x2 in the figure. When finished, click Next to continue.
Figure 15: Installation mode
5. The next screen, Figure 16 shows prerequisite checks performed by the installer. Review the details for the prerequisite checks that did not succeed. Select the line to display the detailed messages in the lower box. For those that have status of Not executed, verify the requirement and click the check box in front of the Status column. For those with Warning status, review and correct the condition as appropriate and either click Retry or check the box in front of Status. When done, click Next to continue.
Figure 16: OUI prerequisite checks - DB installation
6. The next screen asks whether you want to Create a database or Configure ASM. You
June 2007
could also select to install database software only. Make your selection and click Next to continue. In this procedure, we've chosen to install database software only. 7. The next screen shows a summary of the Oracle Database components that will be installed. Click Install to begin the installation. 8. Next you will see a window showing the progress of the installation. Again, note the location of the installation log file, which will contain more details about the steps being performed. 9. When the installation has completed, the window shown in Figure 17 will pop up asking for scripts to be executed as root user. Execute the scripts as root user, first on the node where OUI is being run, then the other nodes. During script execution, you will be prompted for information take the defaults unless you need to do otherwise. When done, return to the popup window and click OK to continue.
Figure 17: Configuration scripts
10. The End of Installation screen displays. Click Exit to end OUI.
46/82
June 2007
Database creation
Database Files
Space requirements for the database vary, depending on the size of the database. The database data files can be stored using Oracle Automatic Storage Management (ASM), a cluster file system or raw devices. Raw devices are more difficult to administer and we will not cover raw devices in this section. In the following sections, we describe database creation using ASM.
ASM configuration
As previously mentioned, if you are using ASM, you can configure the storage devices using either ASMLib or as raw devices.
Using ASMLib
You will need to download and install the packages described earlier. If you are running RHEL4, make sure that SELinux's security level configuration is set to Permissive. Verify this by invoking system-config-securitylevel as root user. Click the tab labeled SELinux and review the Enforcing parameter. Uncheck it, or make sure it is not checked, as shown in Figure 18. If Enforcing is checked, you will not be able to configure ASMLib. With SLES, selinux is not enabled, so this is not an issue.
Figure 18: system-config-securitylevel/SELinux
If you encounter further difficulty, uncheck the Enabled parameter. At present, Oracle's
June 2007
workaround is to set selinux=0 (disabled). Configure ASMLib as follows: 1. Use fdisk to create partitions on your LUNs. Oracle recommends whole-disk partitions. When finished, reboot all the nodes to make them discover the newly created partitions. 1. As root user, enable and start ASMLib
x1:~ # /etc/init.d/oracleasm configure Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface [oracle]: Default group to own the driver interface [oinstall]: Start Oracle ASM library driver on boot (y/n) [y]: y Fix permissions of Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: Scanning system for ASM disks: x1:~ # /etc/init.d/oracleasm start Loading module "oracleasm": Mounting ASMlib driver filesystem: Scanning system for ASM disks: x1:~ #
done done done done done
3. Prepare the disks as shown below. Repeat for all the disk partitions being prepared.
x1:~ # /etc/init.d/oracleasm createdisk DISK1 /dev/vpathb1 Marking disk "/dev/vpathb1" as an ASM disk: x1:~ # done
4. When done, review the ASMLib disks that have been created. For example,
x1:~ # /etc/init.d/oracleasm listdisks DISK1 DISK2 DISK3 DISK4 DISK5 x1:~ # /etc/init.d/oracleasm querydisk DISK2 Disk "DISK2" is a valid ASM disk on device [8, 65] x1:~ #
4. On all the other nodes, run scandisks to make the disks available. In our example,
48/82
June 2007
x2:~ # /etc/init.d/oracleasm scandisks Scanning system for ASM disks: x2:~ # /etc/init.d/oracleasm listdisks DISK1 DISK2 DISK3 DISK4 DISK5 x2:~ #
done
If you do not see the disks, check the storage configuration mapping and the device permissions.
Using Raw Devices with ASM

2. Use fdisk to create partitions on your LUNS. Reboot the nodes to make them discover the new partitions. 3. Bind raw devices to the disk partitions. Insert an appropriate statement to bind the character raw device to the block device. On SLES, edit /etc/raw and insert a statement similar to the one below for each raw device being configured
raw<n>:sd<xy>
where n is the raw device number and sdxy is the scsi disk name (such as sda1) for the LUN. If you are implementing DS6800 with the SDD driver, you will be using vpath<xy> devices. On RHEL9 edit /etc/sysconfig/rawdevices and insert a statement similar to the one below for each raw device being configured:
/dev/raw/raw<n> /dev/sd<xy>
4. Give ownership and permission of the raw devices to oracle user:

chown oracle:oinstall /dev/raw/raw<n> /dev/raw/raw<m> ... chmod 660 /dev/raw/raw<n> /dev/raw/raw<m> ...
5. Bind the raw devices On SLES,

rcraw start
Note: If the response message indicates missing, you may have raw disks already in use. The bindings for these are not affected. On RHEL3/RHEL4,
service rawdevices restart
Create ASM instance

Use Oracle's Database Configuration Assistant (dbca) to create the ASM instance. Invoke dbca as oracle user:
oracle@x1:~> dbca
1. On the dbca Welcome screen, select Oracle Real Application Clusters database. Click Next to continue.
49/82
June 2007
2. On the next screen, select Configure Automatic Storage Management. Click Next to continue. 3. Then select the nodes where ASM is to be configured. Click Next to continue. 4. On the next screen (Figure 19), select Create initialization parameter file (IFILE).
9.
Figure 19: Initialization parameters
Click OK on the popup window to create and start the ASM instance. 5. The next screen will ask you to create the listeners (Figure 20). Click Yes.
Figure 20: dbca warning about Listener
50/82
June 2007
6. The next screen, shown in Figure 21, allows you to create disk groups. In this example, we have not yet created any disk groups; click Create to do so now.
Figure 21: Disk group creation
7. The next screen, shown in Figure 22, shows the candidate disks. Note that the candidate disks prepared by ASMLib will show up with ORCL:<disk name> under Disk Path. Provide a Disk Group Name and select the candidate disks to be included in the disk group. You can let Oracle manage redundancy or chose External if redundancy is to be managed by the storage subsystem.
Figure 22: Disk group creation - detail
If you are using raw devices, the raw devices would be listed under Disk Path.
51/82
June 2007
8. Once the disk group has been created, dbca returns to the screen in Figure 23, where you can select a disk group:
Figure 23: Disk group selection
Click Finish, at which point a popup window will ask if you want to perform another operation. Clicking Yes will return to an earlier screen allowing you to select creation of the database instance.
Create Database Instance

The Oracle initialization file can be a binary (spfile) or text (pfile) file. The spfile is shared between all nodes in the cluster and the pfile is local to each node. If you chose to use an spfile, note that the spfile cannot be placed in ASM, so you will need to use either a raw device or a cluster file system. 1. Select Create a Database. click Next to continue. On the next screen, select the nodes on which to create the cluster database. Click Next. 2. The next screens guide you through selection of a database template, input of the Global Database Name and SID prefix, whether to configure with Oracle Enterprise Manager, and input of passwords for SYS and SYSTEM accounts. When you reach the screen shown in Figure 24, select Automatic Storage Management (ASM).
52/82
June 2007
Figure 24: Database storage options
9. On the ASM Disk Groups screen, shown in Figure 25, select the disk group to be used or create a new disk group. Click Next to continue.
Figure 25: ASM disk group selection for database
10. On the Database File Locations screen, select Use Oracle-Managed files as shown in Figure 26. Click Next to continue.
53/82
June 2007
Figure 26: Database file location
11. The next screens guide you through the selection of recovery options, sample schemas, and creation of database services. Make your selections as appropriate and continue. 12. Next, you will see the screen shown in Figure 27. Here you can change Oracle initialization parameters. If you are running dbca on a 32-bit operating system and a server with a large amount of memory, the defaults may exceed the SGA limits associated with 32-bit addressing. If so, reduce the SGA and PGA sizes, then click Next.
Figure 27: Database initialization parameters
13. The next screen, shown in Figure 28, allows you to specify storage parameters. Make the appropriate modifications and click Next to continue.
54/82
June 2007
Figure 28: Storage specification
14. The next screen, shown in Figure 29, provides an opportunity to save the database creation scripts. It is a good idea to do this, either to repeat the procedure without using dbca, or to use as a template for your own database creation scripts.
Figure 29: Create options
15. The next screens show a summary of the options and parameters used to create the database, then the status of creating and saving database creation scripts. Finally, a screen is displayed, showing the status of database creation (Figure 30).
55/82
June 2007
3.
Figure 30: Database creation status
16. Once database creation is completed, the screen in Figure 31 is shown. If desired, click Password Management to manage database accounts. Otherwise, click Exit to complete the database creation.
Figure 31: Database creation competion
17. Before ending, dbca starts up the cluster database, shown in Figure 32. Once started, dbca ends.
56/82
June 2007
Figure 32: Status of database startup
57/82
June 2007
Administration and tuning

Asynchronous IO
Asynchronous IO is enabled by default in Oracle 10g R2. The Asynchronous IO libraries have to be installed on the system before Oracle is installed (see the pre-requisites). Make sure the following Oracle database parameters in the initialization file (init<SID>.ora) are set:
disk_asynch_io=true filesystemio_options=asynch
The disk_asynch_io parameter is for datafiles on raw devices, and filesystemio_options is for datafiles on a filesystem. To get better I/O throughput on DSS workloads, the /proc/sys/fs/aio-max-size has to be increased from the default 131072 bytes to >=1M. For OLTP workloads, the default size of 131072 would suffice, as these workloads perform very small writes. You can set this value permanently by adding the following line in the sysctl.conf file:
fs.aio-max-size = 1048576
For SLES, the aio-max-size cannot be larger than 524288 because of an OS limit, and if this limit exceeds this limit, the LGWR process (Oracle Redolog writer) will crash.
Direct IO
Oracle can skip the filesystem buffer cache when it is doing IO. This capability is direct IO. This is a requirement for Oracle RAC and a recommended option for an Oracle single instance database. Oracle has a buffer cache so there is no need to use the filesystem buffer cache. For an Oracle database, the filesystem buffer cache is consuming memory but does not increase the overall performance. ASM is doing Direct IO by default, thanks to ASMlib. If ASMlib is not used, ASM has to be configured with raw devices (i.e. /dev/raw/raw3) instead of block devices (i.e /dev/sdc). The filesystems OCFS, ReiserFS, EXT2 and EXT3 support direct IO on SCSI drives (locally attached or from a SAN). IDE drives don't have this capability. In order to make Oracle do Direct IO on a filesystem, the datafiles have to be opened with a certain option. Set the following Oracle database parameters as follow:
filesystemio_options=setall
With this value, you benefit from asynchronous IO and Direct IO.
Swappiness
The swap behavior of a 2.6 kernel can be changed with the kernel parameter swappiness. As an Oracle database server is normally configured to use only the physical memory, the swappiness kernel parameter can be set to 0. This will reduce the chance of swapping some memory pages to disk. As we haven't tested this feature for an Oracle server in
June 2007
production, test it and check the behavior before implementing it in production. For more information, read the article at http://lwn.net/Articles/83588/. Edit sysctl.conf
vm.swappiness = 0
Allocating the SGA in large pages

To increase the performance of the Oracle instance, you may choose to place the SGA in large pages. The large pages area has to be allocated as follows for the different Linux distributions. When the Oracle instance is restarted, the SGA will be allocated in the large pages area.
RHEL3
Edit /etc/sysctl.conf, add:
vm.hugetlb_pool = XXX
Edit /etc/security/limits.conf, add:

oracle soft memlock = <SGA size or -1>
RHEL4
vm.nr_hugepages = XXX vm.hugetlb_shm_group = <oracle group id>

SLES10
vm.nr_hugepages = XXX vm.disable_cap_mlock = 1

Apply the changes from sysctl.conf by running sysctl -p.
CRS administration
The command crsctl controls the Oracle clusterware processes (CRS) locally on the node. For more information, check the Metalink note 309542.1. To stop CRS:
$ crsctl stop crs
or
59/82
June 2007
$ /etc/init.d/init.crs stop
To start CRS:
$ crsctl start crs
or
$ /etc/init.d/init.crs start
To disable the Clusterware from being started in a subsequent reboot: $ /etc/init.d/init.crs disable To enable the Clusterware to be started in a subsequent reboot: $ /etc/init.d/init.crs enable
OCR administration
OCR resources
When a RAC cluster is up and running, you can view the CRS resources with the crs_stat utility, as follow:
$ORA_CRS_HOME/bin/crs_stat NAME=ora.rac1.gsd TYPE=application TARGET=ONLINE STATE=ONLINE NAME=ora.rac1.oem TYPE=application TARGET=ONLINE STATE=ONLINE NAME=ora.rac1.ons TYPE=application TARGET=ONLINE STATE=ONLINE NAME=ora.rac1.vip TYPE=application TARGET=ONLINE STATE=ONLINE
In normal operation, all resources are supposed to be online. When database instances and listeners are created with DBCA, they are listed as resources. The OCR is backed up automatically periodically. In case of broken OCR, you can restore it from a backup: disable CRS, reboot the node in single user mode, check that CRS is not running, and run the following command:
$ORA_CRS_HOME/bin/ocrconfig -restore /oracle/product/10.2.0/CRS/cdata/crs/week.ocr
SRVCTL
The command srvctl controls the database instances, listeners, services in a RAC cluster from a centralized point. It is run as the oracle user.
June 2007
To start/stop/check all instances:

$ srvctl start|stop|status database -d <db_name>
To start/stop a given instance:

$ srvctl start|stop|status instance -d <db_name> -i <instance_name>
To list the instances:

$ srvctl config database -d <db_name>
To start/stop/check the VIP, GSD, listener, ONS on a node:

$ srvctl start|stop|status nodeapps -n <node_name>
To start/stop the ASM instance: $ srvctl start|stop asm -n <node_name> [-i <asm_inst_name>] [-o <oracle_home>] To get environment information:
$ srvctl getenv database -d <db_name> [-i <instance_name>]
To set an env. variable globally:

$ srvctl setenv database -d <db_name> -t LANG=en
To delete a database definition from the OCR:

$ srvctl remove database -d <db_name>
To add a database definition in the OCR:

$ srvctl add database -d <db_name> -o <oracle_home> [-m <domain_name>] [-p
<spfile>] [-A <name|ip>/netmask] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY}] [-s <start_options>] To add instances definition for a database in the OCR:
$ srvctl add instance -d <db_name> -i <instance_name> -n <node1> $ srvctl add instance -d <db_name> -i <instance_name> -n <node2>
To add an ASM instance in the OCR: srvctl add asm -n <node_name> -i <asm_inst_name> -o <oracle_home> To add a service: srvctl add service -d <db_name> -s <service_name> -r <preferred_list> [-a <available_list>] [-P <TAF_policy>] [-u] To modify a service to execute on another node: srvctl modify service -d <db_name> -s <service_name> -i <orig_instance_name> -t <target_instance_name> To relocate a service from one instance to another srvctl relocate service -d <db_name> -s <service_name> -i <orig_instance_name> -t <target_instance_name> SRVCTL START/STOP/STATUS options: -d -i database name instance name
61/82
June 2007
-n -A is:
node name (for nodeapps) vip range, node, and database, address specification. The format of address string
[<logical host name>]/<VIP address>/<net mask> [/<host interface1 [ | host interface2 |..]>] [,] [<logical host name>]/<VIP address>/<net mask> [/<host interface1[ | host interface2 |..]>]
SRVCTL ADD options:
-a for services, list of available instances, this list cannot include preferred instances -m domain name with the format us.domain.com -n node name that will support one or more instances -o $ORACLE_HOME to locate Oracle binaries -P for services, TAF preconnect policy - NONE, PRECONNECT -r for services, list of preferred instances, this list cannot include available instances. -s spfile name -u updates the preferred or available list for the service to support the specified instance. Only one instance may be specified with the -u switch. Instances that already support the service should not be included. SRVCTL MODIFY options: -i <instance-name> -t <instance-name> : the instance name (-i) is replaced by the instance name (-t) -i <instance-name> -r : the named instance is modified to be a preferred instance -A address-list for VIP application, at node level -s <asm_inst_name> : add or remove ASM dependency SRVCTL RELOCATE options: -i <instance-name> -t <instance-name> : the instance name (-i) is replaced by the instance name (-t) For more information on SRVCTL, check the Oracle documentation and the Metalink note 259301.1.
ASM administration
The default name for the ASM instance is +ASMn, n is the node number. Check the name of your ASM instance during the creation or when it is running through the ASM background process names:
oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle 1421 1423 1419 1417 1427 1582 1425 1429 1584 1615 1586 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 Sep Sep Sep Sep Sep Sep Sep Sep Sep Sep Sep 22 22 22 22 22 23 22 22 23 23 23 ? ? ? ? ? ? ? ? ? ? ? 0:00 0:00 0:00 0:00 0:00 0:00 0:00 0:01 0:00 0:00 0:00 asm_dbw0_+ASM1 asm_lgwr_+ASM1 asm_mman_+ASM1 asm_pmon_+ASM1 asm_smon_+ASM1 asm_asmb_+ASM1 asm_ckpt_+ASM1 asm_rbal_+ASM1 oracle+ASM1 (DESCRIPTION=(LOCAL=Y) oracle+ASM1 (DESCRIPTION=(LOCAL=Y) asm_o000_+ASM1
In dbca, you have created a diskgroup in the ASM instance. A diskgroup is the storage space where database files will be stored. Access the ASM instance with sqlplus as follows (assuming the ASM instance name is +ASM1):
62/82
June 2007
$ export ORACLE_SID=+ASM1 $ sqlplus / as sysdba
To manually create an additional diskgroup: Using ASMLib

SQL> create diskgroup DG01 external redundancy disk 'ORCL:DISK1', 'ORCL:DISK2';
To add a storage device to an existing ASM diskgroup:

SQL> alter diskgroup DG01 add disk 'ORCL:DISK3'
To get information about the existing diskgroups:

SQL> select name, state, type, total_mb, free_mb from v$asm_diskgroup; NAME STATE TYPE TOTAL_MB FREE_MB ------------------------------ ----------- ------ ---------- ---------DG01 MOUNTED EXTERN 49150 49055
To get information about the defined disks on the system:

SQL> select substr(a.group_number,1,5) Group#, substr(a.name,1,10) Diskgroup, substr(b.name,1,15) Diskname, substr(path,1,15) Path, mode_status, substr(disk_number,1,5) Disk#, b.state from v$asm_diskgroup a, v$asm_disk b where a.group_number=b.group_number GROUP ----1 1 1 DISKGROUP ---------DG01 DG01 DG01 DISKNAME --------------DISK1 DISK2 DISK3 PATH --------------ORCL:DISK1 ORCL:DISK2 ORCL:DISK3 MODE_ST ------ONLINE ONLINE ONLINE DISK# ----0 1 2 STATE -------NORMAL NORMAL NORMAL
Since an ASM diskgroup can be shared by more than one database, it is good to create directories for each database, and for each database a directory for the type of files. For example, proceed as follows to create the 'TEST' directory in the disk group DG01: SQL> alter diskgroup DG01 add directory '+DG01/TEST';
To create the directories for the types of files in the TEST directory:
SQL> SQL> SQL> SQL> alter alter alter alter diskgroup diskgroup diskgroup diskgroup DG01 DG01 DG01 DG01 add add add add directory directory directory directory '+DG01/TEST/datafile'; '+DG01/TEST/index'; '+DG01/TEST/onlinelog'; '+DG01/TEST/controlfile';
A database can be created with data files in the paths defined in the ASM instance. The path names have the following format:
+diskgroup_name/database_name/database_file type/ASM configuration
If using raw devices, the names of the raw devices would be used, as follows:
63/82
June 2007
SQL> create diskgroup DG01 external redundancy disk '/dev/raw/raw6', '/dev/raw/raw7'; SQL> select substr(a.name,1,15) Diskgroup, substr(b.name,1,15) DiskName, substr(path,1,15) Path, mode_status, b.state, disk_number from v$asm_diskgroup a, v$asm_disk b where b.name is not NULL order by a.name DISKGROUP --------------DG01 DG01 DISKNAME --------------DG01_0000 DG01_0001 PATH --------------/dev/raw/raw6 /dev/raw/raw7 MODE_ST ------ONLINE ONLINE STATE DISK_NUMBER -------- ----------NORMAL 0 NORMAL 1
Adding/removing nodes in a RAC cluster

Adding a node
For details, follow the Metalink Note 270512.1. Adding a Node to a 10g RAC Cluster, Last Revision Date: 25-APR-2005. As a summary, run the following steps on one existing node:

run the pre-installation steps on the new node configure the node to access the shared storage (OCFS, RAW, ASM...) run addNode.sh from $ORA_CRS_HOME run racgons from $ORA_CRS_HOME with the new node information run addNode.sh from $ORACLE_HOME configure the new Virtual IP configure the listener.ora and tnsnames.ora add instance to the database
Remove a node
For details, follow the Metalink Note 269320.1, Removing a Node from a 10g RAC Cluster, Last Revision Date: 11-JUL-2005. As a summary, run the following steps:

remove the instance using DBCA remove the node from the cluster clean up the node
Performance report
In Oracle9i, the statspack package has to be installed. Do this as follows:
SQL> connect / as sysdba SQL> @?/rdbms/admin/spcreate
Snapshot creation:
SQL> connect perfstat/perfstat SQL> execute statspack.snap;
Report creation:
SQL> connect perfstat/perfstat SQL> @?/rdbms/admin/spreport
In Oracle 10g, the installation of the package is done by default.
64/82
June 2007
Snapshot creation: SQL> EXECUTE dbms_workload_repository.create_snapshot(); Report creation:

SQL> @?/rdbms/admin/awrrpt.sql
The snapshot have to be created before and after the time window during which the instance has to be monitored. It has to be a long enough period, minimum 15 min.
65/82
June 2007
Appendix A: ip Bonding
During the Linux installation, the hardware is probed and appropriate drivers are loaded. The configuration is stored in the /etc/modules.conf (for 2.4 kernels) or in /etc/modprobe.conf (for 2.6 kernels). On our x366 servers, the two onboard Gigabit network adapters (Broadcom) are managed by the tg3 driver, and 2 additional PCI 10/100 Ethernet adapters (Intel) managed by the e100 driver. We are paired up the 2 Gigabit NICs and the 2 10/100 adapters:

Bond0 is the master device for eth0 and eth1 (slaves). Bond1 is the master device for eth2 and eth3 (slaves).
The configuration of bonding is slightly different between RHEL and SLES. The setting is outlined in the following sections.
ip bonding configuration - RHEL4

The bonding is configured in the modules.conf or modprobe.conf, and in the network devices configuration files in /etc/sysconfig/network-scripts. In the /etc/modules.conf or /etc/modprobe.conf, the line add above bonding e1000 tg3 determines what network driver will have to be loaded before the bonding driver. /etc/modules.conf or /etc/modprobe.conf
alias alias alias alias alias alias alias alias eth0 tg3 eth1 tg3 eth2 e100 eth3 e100 scsi_hostadapter ips scsi_hostadapter1 qla2300 usb-controller usb-uhci usb-controller1 ehci-hcd
add above bonding e100 tg3 alias bond0 bonding options bond0 miimon=50 mode=1 alias bond1 bonding options bond1 miimon=50 mode=1
The following three files define the devices for the bonded public network. /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0 ONBOOT=yes IPADDR=100.100.100.101 NETMASK=255.255.255.0 NETWORK=100.100.100.0 BROADCAST=100.100.100.255 BOOTPROTO=none USERCTL=no
66/82
June 2007
/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0 USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none
The following 3 files define the devices for the bonded private network. /etc/sysconfig/network-scripts/ifcfg-bond1
DEVICE=bond1 ONBOOT=yes IPADDR=100.100.100.101 NETMASK=255.255.255.0 NETWORK=100.100.100.0 BROADCAST=100.100.100.255 BOOTPROTO=none USERCTL=no
Once the files have been created or modified, restart the network with the following command:
service network restart
Check the settings with the ifconfig command.
ip bonding configuration - SLES

Using yast2, configure the first network interface with the IP address that you want the bonded interface to have. Configure the remaining network card(s) with dummy IP addresses for now. These configurations won't be used, but if you are defining multiple
June 2007
bonded interfaces, it is helpful to assign dummy addresses from the same subnet to the NICs being teamed. The /etc/sysconfig/network directory contains a number of files named idcfg-eth-id-<MAC address> for each network interface configured. Each of these files will have a line starting with the keyword _nm_name, which specifies the PCI bus ID of the interface. Note down the names of the files and the values of the IPADDR and _nm_name parameters. You will need to pair up the PCI bus IDs for the NICs being paired up. Copy the ifcfg-eth-id-<MAC address> file with the desired bonded IP address to ipcfgbond<n>, where n=0, 1, etc. The first bonded interface should be ifcfg-bond0, the second ifcfg-bond1, and so on. Next, obtain the PCI bus ID for the network cards being teamed. Since we copied the first ifcfg-eth-id-<MAC address> file to ifcfg-bond0, we have the first one and only need the PCI bus ID of the other NICs. At the prompt, type
grep bus-pci ifcfg-eth-id*
For example, with two NICs, you will see something like this ifcfg-eth-id-<MAC address>:_nm_name='bus-pci-0000:00:11.0'
ifcfg-eth-id-<MAC address>:_nm_name='bus-pci-0000:00:12.0' In our example, the first line above is from our first NIC, and is already in the ifcfg-bond0 file (since we copied the ifcfg-eth-id-<MAC address> file earlier). Edit ifcfg-bond0:

change _nm_name to BONDING_SLAVE_0 delete the line starting with UNIQUE= insert the lines shown highlighted below BONDING_MASTER=yes BONDING_SLAVE_0='bus-pci-0000:00:11.0'
BONDING_SLAVE_1='bus-pci-0000:00:12.0' BONDING_MODULE_OPTS='miimon=100 mode=0 use_carrier=0'
Save the file. The 'miimon=100' value tells the driver to use MII link monitoring, watching every 100 milliseconds for a link failure. The 'mode' parameter specifies one of four bonding policies. The default is round-robin. For more information, refer to /usr/src/linux/Documentation/networking/bonding.txt. Repeat the procedure for additional bonded interfaces. Now that we've built the configuration, the old ifcfg files are no longer needed. Take a backup and remove the ifcfg-eth-id-<MAC address> files in /etc/sysconfig/network directory. Alternatively, edit the idcfg-eth-id-<MAC address> files and set STARTMODE='off'. Restart the network:
68/82
June 2007
x1:~ # rcnetwork restart Shutting down network interfaces: bond0 Nothing to flush. Nothing to flush. done bond1 Nothing to flush. Nothing to flush. done eth0 device: Intel Corporation 82544GC Gigabit Ethernet Controller (LOM ) (rev 02) eth0 configuration: eth-id-00:02:b3:55:52:15 done eth1 device: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 0d) eth1 configuration: eth-id-00:02:b3:55:52:14 done eth2 device: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 0d) eth2 configuration: eth-id-00:02:b3:cb:2e:ad done eth3 device: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 0d) eth3 configuration: eth-id-00:02:b3:a4:2e:4f done Shutting down service network . . . . . . . . . . . . . done. Setting up network interfaces: lo lo IP address: 127.0.0.1/8 done eth0 device: Intel Corporation 82544GC Gigabit Ethernet Controller (LOM ) (rev 02) eth0 configuration: eth-id-00:02:b3:55:52:15 eth0 Startmode is 'off' skipped eth1 device: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 0d) eth1 configuration: eth-id-00:02:b3:55:52:14 eth1 Startmode is 'off' skipped eth2 device: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 0d) eth2 configuration: eth-id-00:02:b3:cb:2e:ad eth2 Startmode is 'off' skipped eth3 device: Intel Corporation 82557/8/9 [Ethernet Pro 100] (rev 0d) eth3 configuration: eth-id-00:02:b3:a4:2e:4f eth3 Startmode is 'off' skipped bond0 bond0 enslaving interfaces: eth1 eth0 bond0 IP address: 9.38.158.136/25 as bonding master done bond1 bond1 enslaving interfaces: eth2 eth3 bond1 IP address: 1.1.1.1/24 as bonding master done Setting up service network . . . . . . . . . . . . . . done.
As in the previous example, we see the bond0 interface start up, enslaving the two NICs (eth0 and eth1), and using the desired IP address for the bonded pair 'as bonding master'. In this example, we also created bond1 for the private interconnect with IP address 1.1.1.1. Since we modified the ifcfg-eth-id-<MAC address> files, changing STARTMODE to off, these configurations are ignored by the restart command. Verify the configuration using ifconfig and you'll notice the MAC addresses for all the cards are identical, just as the IP addresses for eth0, eth1 and bond0 are identical. We observed that after a system reboot, not all the interfaces come up. In the example above, the Gigabit Ethernet interface would not come up, even though it would after you restart the network. We found that we needed to start the driver earlier during the boot process, and added the e1000 driver to /etc/sysconfig/kernel, in the MODULES_LOADED_ON_BOOT parameter. For additional information, refer to http://portal.suse.com/sdb/en/2004/09/tami_sles9_bonding_setup.html
June 2007
Appendix B: Kernel crash utilities

Linux Kernel Crash Dump(LKCD) for SLES and netdump/diskdump for RHEL are utilities that provide the ability to gather, save, and analyze information about the kernel when the system dies due to software failure.
LKCD for SLES

The LKCD setup is well documented in the following URL. Follow these installation steps. http://www.novell.com/coolsolutions/feature/15284.html
netdump for RHEL3 and RHEL4

Follow the installation steps at this URL: http://www.redhat.com/support/wpapers/redhat/netdump/index.html We only give an outline of the installation steps below. The nodes in the RAC cluster should be configured as netdump clients. The netdump package must be installed on all nodes. Another server has to be configured as a netdump server. A single CPU server with 512MB of memory is sufficient. It has to be connected to the same network as the cluster. It has to have a sufficient amount of disk space to store the dump files, we would recommend 20GB available on the /var/crash mount point. The "netdump-server" package must be installed on this server. The dump files size is a function of the servers' memory, it can be around 4 or 5GB for a server with a lot of memory. The network client must have a network interface of a certain type for netdump. Netdump cannot be done over bonded interfaces. Below is the list of supported network interfaces relevant for System x servers and their associated drivers:
Broadcom NICs using tg3; the bcm5700 has support at the time this paper was written Intel NICs using e100 and eepro100
Netdump server setup As root, set a password for the netdump user:
passwd netdump
Start the netdump-server service and make it start after reboot:

chkconfig --level 35 netdump-server on service netdump-server start
Netdump client setup As root, edit /etc/sysconfig/netdump, uncomment and set the NETDUMPADDR variable to the IP address of the netdump server, as follow:
70/82
June 2007
# LOCALPORT=6666 # DEV= NETDUMPADDR= 100.100.100.025 # NETDUMPPORT=6666 # NETDUMPMACADDR=
Then register the client to the netdump server, run the following command and provide the netdump user password that you set on the netdump server:
service netdump propagate
Start the netdump (client) service and make it started after reboot:
chkconfig --level 35 netdump on service netdump start
System Request keys

Even if the Linux kernel is developed in order to avoid system hangs, it has happened in the past. The system request keys (sysRq keys) allow you to perform emergency operations on a server in case of a kernel hang. The best facility is to generate a kernel dump with a simple key combination. To enable SysRq keys, set kernel.sysrq = 1 in /etc/sysctl.conf. To disable them, set kernel.sysrq = 0. Run the following command to apply the changes:
sysctl -p
If you have netdump, diskdump or LKCD set up and configured, Alt-Sysrq-c (on RHEL3/4) or Alt-Sysrq-d (on SLES) from the console will panic the machine and trigger a kernel dump. For full information about the SysRq mechanism, refer to the SysRq documentation in Documentation/sysrq.txt in your kernel tree.
71/82
June 2007
Appendix C: Sample Configurations

BladeCenter and DS4500
Figure 33 below shows the cabling for a BladeCenter and DS4500 configuration. Note that we have cabled directly from the BladeCenter fibre channel switch modules directly to the DS4500 host minihubs. If you have existing SAN switches, you can run the fibre channel switches in interoperabilty mode, or in the BladeCenter, you can use Optical Passthru Modules (OPM).
ProductionNetw ork Fibrechannel eth0 eth0
Fibrechannel
BladeCenter HS20s eth1 ClusterInterconnect eth1
FAStT900Storage Server & FAStT EXP700Storage ExpansionUnits
Figure 33: Sample BladeCenter RAC configuration
72/82
June 2007
Figure 34: BladeCenter to DS4500 cabling
73/82
June 2007
Part 86771XX
48P7062 48P7054
8832N1X
33L5039 48P7061 48P7063 174290U 17401RU 32P0768 19K1271
Description IBM BladeCenter Chassis IBM BladeCenter 2-port Fibre Channel Switch Module IBM BladeCenter 4-port GB Ethernet Switch Module Blade server (per node) IBM BladeCenter HS20, 2x Xeon 3.06GHz (1MB L2 Cache) with EM64T, 2x 1GB PC2100 CL2.5 ECC DDR SDRAM DIMMS, ATA-100 IDE controller, Ultra320 controller, Dual Gigabit Ethernet, ATI RageXL (8MB) SVGA video 1GB PC2100 CL2.5 ECC DDR SDRAM RDIMM IBM HS20 Fibre Channel Expansion Card IBM HS20 40GB 5400-rpm ATA-100 HDD DS4500 IBM System Storage DS4500 Storage Server FAStT700 Storage Expansion Unit 2Gbps FC 73.4 GB 15K Hot-Swap HDD
Short Wave SFP Module LC-LC Fibre Channel Cables
Table 4: BladeCenter and DS4500 configuration components
Qty 1
2 2
2 1 1 1 2 28 12 6
74/82
June 2007
x366 and DS6800

The x366 and DS6800 configuration is shown in Figure 35. Each x366 contained four network interfaces, two onboard Gigabit and two PCI 10/100Mbps adapters. We bonded the network interfaces, using PCI adapters for the production network and the Gigabit adapters for the network interconnect. Each x366 had two fibre channel HBAs, connected to the DS6800 through two SAN switches for redundancy.
Figure 35: Sample x366 and DS6800 configuration
75/82
June 2007
Part
88631RU 13N0694 73P2866 24P0960 26K5654 31P9601 2109F16 19K1271
1750-511 0801 0945 1310 1351 2002 5001 9009
Description x366 server each node x366, Xeon 3.16GHz/667MHz with EM64T, 1MB L2, 2x1GB, DVD-ROM, 1300W p/s System x 3.16GHz/667MHz, 1MB L2 Upgrade with Xeon Processor MP 2 x 1GB PC2-3200 ECC DDR2 SDRAM RDIMM Kit IBM Total Storage DS4000 FC2-133 Host Bus Adapter 36.4 GB 10K 2.5-inch SAS HDD Intel PRO/1000 MT Server Adapter SAN Switch SAN Fibre Channel Switch, 16-port Short Wave SFP Module LC-LC Fibre Channel Cables DS6800 Single Drawer: IBM System Storage DS6800 Integrate in rack enclosure Intel Linux Systems (22R2177) 2 Gbps Fibre Channel Short Wave SFP Pair (22R2180) 7M 50 micron LC-LC cable (22R2174) 2Gb-146 GB 10K Drive Set (22R2426) OEL - 5TB English Translation Materials
Table 5: x366 and DS6800 configuration components
Qty
1 3 1 2 3 2 2 8 6 1 1 1 2 4 4 1 1
76/82
June 2007
Appendix D: Miscellaneous Notes

Enabling Async IO in Oracle 9i and 10gR1
To enable asynchronous I/O for Oracle 9i and 10g R1, you must re-link Oracle using the Async I/O library, libaio, as follows:
cd to $ORACLE_HOME/rdbms/lib make -f ins_rdbms.mk async_on make -f ins_rdbms.mk ioracle
LD_ASSUME_KERNEL Environmental Variable

The variable LD_ASSUME_KERNEL has to be set to 2.4.19 in order to run some Java based program that are not compatible with the NTPL threading architecture. We do not recommend you set this variable all the time. The error generated when a program is not compatible with NTPL is:
symbol error, version GLIBC_2.0 not defined
OCFS
This section is about the setup of OCFS 1, it is supported with Linux 2.4 kernels, not with 2.6. OCFS 2 will be supported with Linux 2.6 kernels in the future.
OCFS1
Use ocfstool to configure OCFS and create OCFS filesystems. Generate the OCFS configuration file on the system, and a unique identifier. Check /etc/ocfs.conf. Create the filesystem with ocfstool, or from the command line as follows. This definition has to be done on one node only in the RAC cluster. mkfs.ocfsFb128Locfsm/oracle/oradata/ocfs01u500g500p0775 /dev/sdd Mount the filesystem from ocfstool, or from the command line as follows: mounttocfs/dev/sdd/oracle/oradata/ocfs01 Make the filesystem mount permanent. Edit /etc/fstab and add the information for the defined OCFS filesystem. The option _netdev will make the system wait for the network devices to be ready before mounting the filesystem. /dev/sdd/oracle/oradata/ocfs01ocfs_netdev 13
Prevent updatedb from scanning the OCFS filesystems. Edit /etc/updatedb.conf as shown below:

add ocfs to PRUNEFS, which is the list of filesystem types to be avoided by updatedb add the created OCFS filesystems to PRUNEPATHS, which is the list of filesystems to be avoided.
77/82
June 2007
PRUNEFS="devptsnfsafsprocsmbfsautofsautoiso9660ocfs" PRUNEPATHS="/tmp/usr/tmp/var/tmp/afs/net/oracle/oradata/ocfs01" exportPRUNEFS exportPRUNEPATHS It is recommended that you have more than one OCFS filesystem for a high performance database. One filesystem per type of access is recommended:

one filesystem for datafiles and index files. For example: ocfs01 one filesystem for each redo log group. In a RAC cluster, each node will write its redo log information to a specific OCFS filesystem in order to keep the access sequential. For example: ocfs02 for redo log group 1 for node 1, ocfs03 for redo log group 2 for node 1, ocfs04 for redo log group 3 for node 2, ocfs05 for redo log group 4 for node 2. one filesystem for each archive log and for each node. For example: ocfs06 for archive log from node 1, ocfs07 for archive log from node 2.
A database can now be created with datafiles in the paths of the OCFS filesystems, such as /oracle/oradata/ocfs01/, /oracle/oradata/ocfs02/, etc.
78/82
June 2007
References
Oracle Documentation: http://www.oracle.com/technology/documentation/database10gR2.html 1. Oracle Database, Release Notes, 10g Release 2 (10.2) for Linux x86, B15659-03, September 2005. 2. Oracle Database, Release Notes, 10g Release 2 (10.2) for Linux x86-64, B15666-01, October 2005. 3. Oracle Database, Installation Guide, 10g Release 2 (10.2) for Linux x86, B15660-01, June 2005. 4. Oracle Database, Installation Guide, 10g Release 2 (10.2) for Linux x86-64, B1566701, September 2005. 5. Oracle Database, Oracle Clusterware and Oracle Real Application Clusters, Installation Guide, 10g Release 2 (10.2) for Linux, B14203-02, September 2005. 6. Oracle Metalink Note: 341782.1, Linux Quick Reference, Last Revision Date: 24-NOV2005. 7. Oracle Metalink Note: 220970.1, RAC: Frequently Asked Questions, Last Revision Date: 19-JUL-2005. 8. Oracle Metalink Note:169706.1, Oracle RDBMS on AIX,HP-UX, Solaris, Tru64, Linux, MacOSX: Versions, Sizes, Requirements Quick Reference, Revision Date: 11-DEC2005. 9. Oracle Cluster File System (OCFS2) Users Guide, available at http://oss.oracle.com/projects/ocfs2/. 10. Metalink Note 270512.1, Adding a Node to a 10g RAC Cluster, Revision Date: APR-2005. 25-
11. Metalink Note 269320.1, Removing a Node from a 10g RAC Cluster, Last Revision Date: 11-JUL-2005. 12. Oracle Real Application Clusters 10g, An Oracle Technical White Paper, by Barb Lundhild and Peter Sechser, May 2005. IBM Documentation 1. IBM System Storage, Multipath Subsystem Device Driver User's Guide, SC30-409601, Second Edition (October 2005). 2. IBM Redbook, IBM eServer xSeries and BladeCenter Server Management, David Watts, Lesley Bain, and Christian Leitenberger, International Technical Support Organization, SG24-6495-00, March 2005. Novell Documentation
June 2007
1. Setting up bonding on SLES 9 http://portal.suse.com/sdb/en/2004/09/tami_sles9_bonding_setup.html
80/82
June 2007
Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: IBM the IBM logo the e-business logo IBM eServerTM System xTM BladeCenterTM Intel, Xeon and Itanium are trademarks or registered trademarks of Intel Corporation. Microsoft, Windows, Windows NT and the Windows logo are trademarks or registered trademarks of Microsoft Corporation. Oracle is a registered trademark, and Oracle 10g, Oracle 10g Database, Real Application Clusters (RAC), Oracle9i, Oracle MetaLink, and SQL*Plus are trademarks or registered trademarks of Oracle Corporation. Linux is a registered trademark of Linus Torvalds. Red Hat, the Red Hat Shadow Man logo, RPM and all Red Hat-based trademarks and logos are trademarks or registered trademarks of Red Hat, Inc. in the United States and other countries. SuSE is a registered trademark of Novell, Inc. PolyServe and the PolyServe logo are trademarks of PolyServe, Inc. VERITAS is a registered trademark of VERITAS Operating Corporation. Symantec is a registered trademark of Symantec Corporation. EMC is a registered trademark of EMC Corporation. Hitachi is a registered trademark of Hitachi Corporation. QLogic is a registered trademark of QLogic Corporation. Other company, product, and service names may be trademarks or service marks of others.
81/82
June 2007
IBM Corporation 2002 IBM Server Group Advanced Technical Support Produced in the USA All rights reserved IBM reserves the right to change specifications or other product information without notice. References in this publication to IBM products or services do not imply that IBM intends to make them available in all countries in which IBM operates. IBM PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties in certain transactions; therefore, this statement may not apply to you. This publication may contain links to third party sites that are not under the control of or maintained by IBM. Access to any such third party site is at the user's own risk and IBM is not responsible for the accuracy or reliability of any information, data, opinions, advice or statements made on these sites. IBM provides these links merely as a convenience and the inclusion of such links does not imply an endorsement.
82/82

Oracle10gRAC SystemX-06302007

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Oracle10gRAC SystemX-06302007

Загружено:

Авторское право:

Доступные форматы

Oracle 10g R2 RAC implementation with IBM System x, BladeCenter, System Storage and Linux

June 30th, 2007

COPYRIGHT IBM CORPORATION, 2007

Gregory Verstraeten IBM ATS Americas gregvers@us.ibm.com

Oracle 10g R2 RAC Implementation with System x

Hardware Requirements................................................................................................................... ....................7

Software requirements................................................................................................................................. .......20

Set Up Server Environment .......................................................................................................... .....................23

Shared Storage setup............................................................................................................................. .............27

Installing Oracle 10g R2 RAC..................................................................................................... ........................32

Appendix D: Miscellaneous Notes.................................................................................................................... ...73

COPYRIGHT IBM CORPORATION, 2007. All rights reserved.

Oracle 10g R2 RAC Implementation with System x

COPYRIGHT IBM CORPORATION, 2007. All rights reserved.

Oracle 10g R2 RAC Implementation with System x

About Oracle Real Application Clusters

COPYRIGHT IBM CORPORATION, 2007. All rights reserved.

Oracle 10g R2 RAC Implementation with System x

About IBM System x

About IBM BladeCenter

About IBM System Storage

COPYRIGHT IBM CORPORATION, 2007. All rights reserved.

Oracle 10g R2 RAC Implementation with System x

For more information about the DS6000, visit: http://www-03.ibm.com/servers/storage/index.html

COPYRIGHT IBM CORPORATION, 2007. All rights reserved.

Oracle 10g R2 RAC Implementation with System x

Figure 1: RAC Overview

Oracle 10g R2 RAC Implementation with System x

Oracle 10g R2 RAC Implementation with System x

COPYRIGHT IBM CORPORATION, 2007. All rights reserved.

Oracle 10g R2 RAC Implementation with System x

IBM GPFS is not supported for Oracle RAC.

System x and BladeCenter Servers

Table 1: Selected System x and BladeCenter servers

Oracle 10g R2 RAC Implementation with System x

Oracle 10g R2 RAC Implementation with System x

Oracle 10g R2 RAC Implementation with System x

High Availability Considerations

Oracle 10g R2 RAC Implementation with System x

COPYRIGHT IBM CORPORATION, 2007. All rights reserved.

Oracle 10g R2 RAC Implementation with System x

Figure 2: Front view of the BladeCenter H

Oracle 10g R2 RAC Implementation with System x

Figure 3: Rear view of the BladeCenter H

Oracle 10g R2 RAC Implementation with System x

Figure 4: DS4800 with enclosures EXP810

Cluster of x3650/x3655/x3755/x3850/x3950 attached to a DS4800

Oracle 10g R2 RAC Implementation with System x

Oracle 10g R2 RAC Implementation with System x

Figure 6: Four node cluster directly attached to the DS4800

Oracle 10g R2 RAC Implementation with System x

Figure 7: Three RAC nodes attached to a DS3200

COPYRIGHT IBM CORPORATION, 2007. All rights reserved.

Oracle 10g R2 RAC Implementation with System x

Red Hat https://hardware.redhat.com/hwcert/index.cgi#list Novell SuSEs web site: http://developer.novell.com/yessearch/Search.jsp

Storage related software

Oracle 10g R2 RAC Implementation with System x

Optional Software Products

Oracle 10g R2 RAC Implementation with System x

Automatic Storage Management Library (ASMlib)

Drivers for specific kernels oracleasm-<kernel version>.rpm

Oracle 10g R2 RAC Implementation with System x

COPYRIGHT IBM CORPORATION, 2007. All rights reserved.