HA With Sap en Queue Replication and Sun Cluster V02

Sun Microsystems GmbH

High Availability with SAP enqueue replication and
Sun Cluster
Author: Jörg Meiners, Sun Microsystems GmbH

Created: 16.01.2009
Last Changed: 17.03.2009
Version: 02
Page 1 of 21
Table of Contents
1 - Conceptual overview...........................................................................................................................................................3
2 - About this paper..................................................................................................................................................................4
3 - Description of the target setup.............................................................................................................................................5
3.1 - Cluster configuration...................................................................................................................................................5
3.2 - Oracle database..........................................................................................................................................................5
3.3 - SAP System Central Services (SCS)................................................................................................................................6
3.4 - SAP Replicated Enqueue Server....................................................................................................................................6
3.5 - SAP Central Instance....................................................................................................................................................6
4 - Performing the installation..................................................................................................................................................8
4.1 - Solaris and Sun Cluster installation..............................................................................................................................8
4.2 - Basic Cluster configuration...........................................................................................................................................8
4.3 - Cluster data services configuration (part I)....................................................................................................................8
4.4 - SAP/Oracle Installation..............................................................................................................................................10
4.5 - Prepare SAP/Oracle switchover...................................................................................................................................11
4.6 - Test SAP/Oracle switchover........................................................................................................................................12
4.7 - Enable enqueue replication........................................................................................................................................12
4.8 - Test the enqueue replication......................................................................................................................................13
4.9 - Cluster data services configuration (part II).................................................................................................................14
4.10 - Enable and test the cluster.......................................................................................................................................16
5 - Appendix...........................................................................................................................................................................17
5.1 - Default profile...........................................................................................................................................................17
5.2 - ERS11 start profile......................................................................................................................................................17
5.3 - ERS11 instance profile................................................................................................................................................17
5.4 - SCS01 start profile......................................................................................................................................................18
5.5 - SCS01 instance profile................................................................................................................................................19
5.6 - JC00 start profile........................................................................................................................................................19
5.7 - JC00 instance profile...................................................................................................................................................20
5.8 - Script startsap_cluster.sh...........................................................................................................................................20
5.9 - Script stopsap_cluster.sh...........................................................................................................................................21
Page 2 of 21
1 - Conceptual overview
The Enqueue Server is a critical single point of failure in SAP architectures. And furthermore - as the Enqueue
Server holds its data in memory - a failure would mean to loose the central SAP enqueue table. As a consequence,
all ongoing SAP transactions having an active lock would abort. And special only to the SAP JAVA-stack, all SAP
application instances will automatically restart as soon as the Enqueue Server is online again (and having a new
empty enqueue table with a new ID).
To overcome this drawback, SAP strongly recommends to use the standalone Enqueue Server (as part of the SAP
System Central Services) together with the Replication Enqueue Server especially in productive JAVA-stack
systems.
The Replication Enqueue Server runs on another host and contains a replica of the enqueue table (the replication
table). Enqueue Server and Replicated Enqueue Server are running in a synchronous replication mode. Thus if the
Enqueue Server fails and becomes online again, the enqueue table is not lost. Instead, the Enqueue Server can
recover the enqueue table from the replication table.
The SAP enqueue replication concept is based on an underlying HA software and can't work without that. Sun
Cluster supports this architecture out-of-the-box by providing best suitable product features:
• Provides a robust and reliable cluster framework.
• Provides data agents for the basic infrastructure (Storage, Network).
• Provides data agents for the SAP and database applications. In this context especially for the SAP SCS
(System Central Services: Message Server and Enqueue Server) and the Replicated Enqueue Server.
• Handles the particular needs of Enqueue Server and Replicated Enqueue Server in a failover environment
➢ If Enqueue Server fails, Sun Cluster will restart the Enqueue Server on the cluster node (=
physical server) that is currently hosting the Replicated Enqueue Server. This is mandatory,
because the restarted Enqueue Server will recover the enqueue table from the replication table
using a server-local access to the shared memory segment that holds the replicated table.
➢ After table recovery has finished, the Replicated Enqueue Server will be stopped. Sun Cluster will
restart the Replicated Enqueue Server on another cluster node (different than the node hosting
the Enqueue Server) – if another cluster node is available. Thus keeping up the enqueue
replication concept and maximal availability.
To learn more about the SAP enqueue replication and Sun Cluster, please refer to:
• Sun Cluster 3.2 documentation (http://docs.sun.com/app/docs/prod/sun.cluster32)
• Sun Cluster 3.2 Data Service for SAP Web Application Server Guide
(http://docs.sun.com/app/docs/doc/820-2568)
• SAP documentation (http://help.sap.com → SAP NetWeaver 7.0 Library → Look for the chapter
"Standalone Enqueue Server")
Page 3 of 21
2 - About this paper

This paper describes the implementation of a SAP High Availability Architecture based on the “SAP enqueue
replication” concept and a Sun infrastructure.
An installation-”HowTo” is being provided, using:
• Sun Solaris 10 (Update 5/08)
• Sun Cluster 3.2.0 (Update 2/08)
• Oracle Database 10g Enterprise Edition Release 10.2.0.2.0
• SAP NetWeaver Portal 7.0 (SP16, Kernel 7.00 patch# 159)
The HowTo is focused on all necessary implementation and configuration tasks that are specific for the “SAP
Enqueue replication” scenario. A “standard” installation of Solaris, Sun Cluster and SAP Portal is assumed and a
description of this process is out of scope.
Also it is not intended to describe all aspects of an “High End” SAP/Oracle/Cluster implementation. Many things in
this example installation are implemented as simple as possible (like filesystem layout, parameter tuning, ...) to
hold this paper small and to focus on “SAP enqueue replication” setup.
Page 4 of 21
3 - Description of the target setup

3.1 - Cluster configuration
Cluster Name: sap1
Node Names: sscsapx1,sscsapx2
Transport Adapters: bge1,e1000g1 (on both nodes)
Quorum Device Name: d11 (c4t600015D00022B30000000000000072CDd0)
Public Network sscsapx1
• IPMP-group: sc_ipmp0
➢ bge0 (sscsapx1 192.168.14.35)
➢ e1000g0 (sscsapx1b 192.168.14.1)
Public Network sscsapx4
• IPMP-group: sc_ipmp0
➢ bge0 (sscsapx2 192.168.14.36)
➢ e1000g0 (sscsapx2b 192.168.14.6)
SAP CI
Logical Hostname
SAP NW WAS
Oracle DB Group
Logical Hostname
Oracle Listener
Oracle Database
Failover Filesystem ZFS (/oracle/PE2)
SAP Replicated Enqueue Group
Logical Hostname
SAP Replicated Enqueue Server
SAP SCS Group
Logical Hostname
NFS Share (/sapmnt/PE2, /usr/sap/trans)
SAP Message Server
SAP Enqueue Server
Global Filesystem UFS+SVM (/usr/sap/PE2, /sapmnt/PE2, /usr/sap/trans)
Local (root FS): Local (root FS):

/oracle/client /oracle/client
/oracle/stage/102_64 /oracle/stage/102_64
Figure 1: Overview Cluster Resourcegroups
3.2 - Oracle database

DB name PE2, installed on logical hostname pe2oralh.
Cluster-Resourcegroup: pe2ora; preferred node: sscsapx1

Cluster-resource Type Properties
pe2ora_stor SUNW.HAStoragePlus zpool, zfs
Devices: d10 (c4t600015D00022B30000000000000072D1d0)
Failover filesystem
Mountpoints: /oracle/PE2 (150 GB)
Page 5 of 21
pe2ora_lh SUNW.LogicalHostname ip / hostname = 192.168.14.24 / pe2oralh

pe2ora_lsnr SUNW.oracle_listener
pe2ora_db SUNW.oracle_server
Notes regarding SAP/Oracle filesystem configuration:

• /oracle/client: Local directory in root-fs. To be copied on both cluster nodes.
• /oracle/stage/102_64: Local directory in root-fs of the install server. Not needed for running the database.
• /sapmnt/PE2 and /usr/sap/PE2: Will be accessible to the database by the cluster global filesystem. No
need to copy anything.
3.3 - SAP System Central Services (SCS)

SAP SID PE2, Instance No. 01 (instance name SCS01), installed on logical hostname pe2scslh.
Cluster-Resourcegroup: pe2scs; preferred node: sscsapx1

pe2scs_stor SUNW.HAStoragePlus SVM diskset pe2scs_stor
Devices: d11 (c4t600015D00022B30000000000000072CDd0)
Global filesystems
Mountpoints: /usr/sap/PE2 (20 GB), /sapmnt/PE2 (10 GB), /usr/sap/trans (10 GB)
pe2scs_lh SUNW.LogicalHostname ip / hostname = 192.168.14.23 / pe2scslh
pe2scs_ms SUNW.sapscs (Just FYI: SAP Message Server)
pe2scs_enq SUNW.sapenq (Just FYI: SAP Enqueue Server)
pe2scs_nfs SUNW.nfs Exports “/sapmnt/PE2” and “/usr/sap/trans”
Note: Any necessary application shared data access (for example to “/sapmnt/PE2”) is done by using the global
filesystem(s) within the cluster. This is also to avoid any problems with “NFS cross-mounts” between cluster nodes
(that are generally not supported in a Sun Cluster configuration). The NFS server configured in this resourcegroup
is to provide all cluster-external NFS-clients – like additional SAP application instances – with access to the shared
data.
3.4 - SAP Replicated Enqueue Server

SAP SID PE2, Instance No. 11 (instance name ERS11), installed on logical hostname pe2erslh.
Cluster-Resourcegroup: pe2ers; preferred node: sscsapx2

pe2ers_lh SUNW.LogicalHostname ip / hostname = 192.168.14.25 / pe2erslh
pe2ers_enq SUNW.saprepl (Just FYI: SAP Replicated Enqueue Server)

• No dedicated filesystems needed. Global filesystems (/usr/sap/PE2, /sapmnt/PE2) will be used for
instance specific data.
3.5 - SAP Central Instance

SAP SID PE2, Instance No. 00 (instance name JC00), installed on logical hostname pe2cilh.
Cluster-Resourcegroup: pe2ci; preferred node: sscsapx1

Page 6 of 21
pe2ci_lh SUNW.LogicalHostname ip / hostname = 192.168.14.22 / pe2cilh

pe2ci_jc00 SUNW.sapwebas (Just FYI: SAP Web Application Server)

• No dedicated filesystem needed. Global filesystems (/usr/sap/PE2, /sapmnt/PE2) will be used for instance
specific data.
Page 7 of 21
4 - Performing the installation

4.1 - Solaris and Sun Cluster installation
Install Sun Solaris 10 (Update 5/08) and Sun Cluster 3.2.0 (Update 01) on the servers sscsapx1 and sscsapx2. A
description is out of scope of this paper.
Note: The documentation “Sun Cluster Quick Start Guide for Solaris OS” (http://docs.sun.com/app/docs/doc/820-
2556) provides a good example (cookbook) for setting up the cluster hardware, installing the Solaris and Sun
Cluster Software.
Note: Using Solaris Volume Manager (SVM), take care not to use the same metadevice name for the filesystem “/
globaldevices” on both nodes. The cluster will not work properly. Instead take different metadevice names (for
example d200 on sscsapx1 and d400 on sscsapx2).
4.2 - Basic Cluster configuration

This step describes some basic cluster configurations. To be executed, if not done before.
Add Quorum Device

# cldevice list v (… FYI: to view the available did devices)
# clquorum add d11 (… adds d11 as quorum device)
# clquorum list (… FYI: to view the configured quorum devices)
# clquorum status (… FYI: to view the quorum status)
Network configuration
# eeprom localmacaddress?=true (… to be executed on both cluster nodes)
Data Services registration

# clresourcetype register SUNW.HAStoragePlus
# clresourcetype register SUNW.LogicalHostname
# clresourcetype register SUNW.oracle_listener
# clresourcetype register SUNW.oracle_server
# clresourcetype register SUNW.sapscs
# clresourcetype register SUNW.sapenq
# clresourcetype register SUNW.saprepl
# clresourcetype register SUNW.nfs
# clresourcetype register SUNW.sapwebas
Hostnames
Add all logical hostnames needed to the local /etc/hosts on sscsapx1 and sscsapx2
192.168.14.24 pe2oralh pe2oralh.abc.de # logical host for Oracle DB
192.168.14.23 pe2scslh pe2scslh.abc.de # logical host for SAP SCS01
192.168.14.25 pe2erslh pe2erslh.abc.de # logical host for SAP ERS11
192.168.14.22 pe2cilh pe2cilh.abc.de # logical host for SAP JC00
4.3 - Cluster data services configuration (part I)

This step will configure the storage, filesystems and IP addresses. Hereby providing the necessary infrastructure
platform to install SAP/Oracle afterwards.
Oracle database
a) Create failover filesystem (zpool and zfs filesystem)
# zpool create pe2ora_stor /dev/dsk/c4t600015D00022B30000000000000072D1d0
# zfs create pe2ora_stor/oraclePE2
# zfs set mountpoint=/oracle/PE2 pe2ora_stor/oraclePE2
Test:
# zpool export pe2ora_stor (… on the currently hosting node)
# zpool import pe2ora_stor (… on the target node)
b) Create resourcegroup with storage and logical hostname
Page 8 of 21
# clresourcegroup create n sscsapx1,sscsapx2 pe2ora
# clresource create t HAStoragePlus g pe2ora p zpools=pe2ora_stor p AffinityOn=true pe2ora_stor
# clreslogicalhostname create g pe2ora h pe2oralh N sc_ipmp0@sscsapx1,sc_ipmp0@sscsapx2 pe2ora_lh
c) Enable and test Resourcegroup

# clresourcegroup online pe2ora
# clresourcegroup switch n sscsapx2 pe2ora
# clresourcegroup switch n sscsapx1 pe2ora
SAP System Central Services (SCS)

a) Create SVM Diskset and cluster devicegroup
Check if /kernel/drv/md.conf identical on both cluster nodes
# cldevice list v (… FYI: Just to review; look for d11)
# metaset s pe2scs_stor a h sscsapx1 sscsapx2 (… FYI: creates devicegroup “pe2scs_stor”)
# cldevicegroup show (… FYI: To view existing device groups)
# cldevicegroup status (… FYI: To view status of device groups)
# cldevicegroup set p <name>=<value> pe2scs_stor (… Optional. Nothing mandatory here !!)
Test cluster devicegroup:
# cldevicegroup switch n sscsapx2 pe2scs_stor
# cldevicegroup switch n sscsapx1 pe2scs_stor
b) Create global filesystems (UFS)

# metaset s pe2scs_stor a /dev/did/rdsk/d11
# metainit s pe2scs_stor d100 p e /dev/did/rdsk/d11 10G
# metainit s pe2scs_stor d101 p /dev/did/rdsk/d11s0 20G
# metainit s pe2scs_stor d102 p /dev/did/rdsk/d11s0 all
# newfs /dev/md/pe2scs_stor/rdsk/d100
# mkdir p /sapmnt/PE2 (… to be executed on both cluster nodes)
# mkdir p /usr/sap/PE2 (… to be executed on both cluster nodes)
# mkdir p /usr/sap/trans (… to be executed on both cluster nodes)
c) Add entries to /etc/vfstab (both cluster nodes):

/dev/md/pe2scs_stor/dsk/d100 /dev/md/pe2scs_stor/rdsk/d100 /sapmnt/PE2 ufs 2 yes global,logging
/dev/md/pe2scs_stor/dsk/d101 /dev/md/pe2scs_stor/rdsk/d101 /usr/sap/PE2 ufs 2 yes global,logging
/dev/md/pe2scs_stor/dsk/d102 /dev/md/pe2scs_stor/rdsk/d102 /usr/sap/trans ufs 2 yes global,logging
# sccheck
# mount /sapmnt/PE2
# mount /usr/sap/PE2
# mount /usr/sap/trans
d) Create resourcegroup with storage, logical hostname and NFS server

# clresourcegroup create n sscsapx1,sscsapx2 pe2scs
# clresource create t HAStoragePlus g pe2scs p GlobalDevicePaths=pe2scs_stor p AffinityOn=true
pe2scs_stor
# clreslogicalhostname create g pe2scs h pe2scslh N sc_ipmp0@sscsapx1,sc_ipmp0@sscsapx2 pe2scs_lh
# clresourcegroup set p Pathprefix=/usr/sap/PE2 pe2scs
# mkdir p /usr/sap/PE2/SUNW.nfs
Add these two entries (shares) to /usr/sap/PE2/SUNW.nfs/dfstab.pe2scs_nfs
share F nfs o rw /sapmnt/PE2
share F nfs o rw /usr/sap/trans
# clresource create g pe2scs t SUNW.nfs p Resource_dependencies=pe2scs_stor pe2scs_nfs
e) Enable and test Resourcegroup

# clresourcegroup online pe2scs
# clresourcegroup switch n sscsapx2 pe2scs
# clresourcegroup switch n sscsapx1 pe2scs
SAP Central Instance

a) Create Resourcegroup with logical hostname
# clresourcegroup create n sscsapx1,sscsapx2 pe2ci
# clreslogicalhostname create g pe2ci h pe2cilh N sc_ipmp0@sscsapx1,sc_ipmp0@sscsapx2 pe2ci_lh
b) Enable and test Resourcegroup

# clresourcegroup online pe2ora
# clresourcegroup switch n sscsapx2 pe2ci

a) Create Resourcegroup with logical hostname
# clresourcegroup create n sscsapx2,sscsapx1 pe2ers
Page 9 of 21
# clreslogicalhostname create g pe2ers h pe2erslh N sc_ipmp0@sscsapx2,sc_ipmp0@sscsapx1 pe2ers_lh
b) Enable and test Resourcegroup

4.4 - SAP/Oracle Installation

SAP/Oracle standard installation
Perform a “standard” installation of SAP NetWeaver Portal 7.0 and Oracle Database 10g on the prepared storage
and logical hosts. Any special configuration needs regarding failover, Sun Cluster or enqueue replication will be
described/done not here, but in the following steps.
• Oracle DB on node sscsapx1: Name PE2, configured on logical hostname pe2oralh
• SAP SCS on node sscsapx1: SAP SID PE2, Instance No. 01 (instance name SCS01), configured on
logical hostname pe2scslh
• SAP CI on node sscsapx1: SAP SID PE2, Instance No. 00 (instance name JC00), installed on logical
hostname pe2cilh
SAP Replicated Enqueue Server installation

Perform a “standard” installation of the SAP Replicated Enqueue Server on the prepared storage and logical hosts.
Any special configuration needs regarding failover, Sun Cluster or Enqueue replication will be described/done not
here, but in the following steps.
•SAP Replicated Enqueue Server on node sscsapx2: SAP SID PE2, Instance No. 11 (instance name ERS11),
installed on logical hostname pe2erslh
As of today (January 2009, SAP NetWeaver 7.0 EHP 1, SPS02) there is no SAP tool like “sapinst” to install the
SAP Replicated Enqueue Server. Instead, it has to be done manually. You find the guide in the SAP
documentation (http://help.sap.com → SAP NetWeaver 7.0 Library → Look for the chapter "Setting Up the
Replication Server").
Here is a summary. It is simplified and adopted to the special needs of the cluster configuration:
• Directory structure
Create the instance-specific subdirectories in the global filesystem “/usr/sap/PE2”:
/usr/sap/PE2/ERS11/exe
/usr/sap/PE2/ERS11/log
/usr/sap/PE2/ERS11/data
/usr/sap/PE2/ERS11/work
• Instance profile
Create the instance profile “/sapmnt/PE2/profile/PE2_ERS11_pe2erslh”. You find the file for reuse in the
appendix (Chapter “ERS11 instance profile”). Please pay attention to comments given in the file. Some
changes have to be done (compared to the SAP documentation) because of Sun Cluster specifics.
• Start profile
You may want to create a start profile. This is optional, because Sun Cluster will start the server-process
directly (by executing the command: “/usr/sap/PE2/SYS/exe/run/enrepserver
pf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh”). In this way, no start profile is being used. If you want to
start the server-process manually, you can do it in the same way. Just if you want to use “startsap”
(“startsap r3 ERS11 pe2erslh”), you have to provide the start profile
“/sapmnt/PE2/profile/START_ERS11_pe2erslh”. You find the file for reuse in the appendix (Chapter
“ERS11 start profile”).
Disable “sapcpe”
Copying the SAP Kernel from “/sapmnt/PE2/exe” to the instance-specific directories
“/usr/sap/PE2/<instancename>/exe” does not make sense, because it's all on the global filesystem(s). Instead start
directly from /usr/sap/PE2/SYS/exe/run (that is a softlink to /sapmnt/PE2/exe).
• Set these parameters in all start profiles and instance profiles:
DIR_CT_RUN = /usr/sap/PE2/SYS/exe/run
DIR_EXECUTABLE = /usr/sap/PE2/SYS/exe/run
• Check and adjust environment of OS users pe2adm and orape2 to point to the correct directory
PATH=...:/usr/sap/PE2/SYS/exe/run:...
DIR_LIBRARY=.../usr/sap/PE2/SYS/exe/run:...
LD_LIBRARY_PATH=.../usr/sap/PE2/SYS/exe/run:...
• In the instance start profiles, disable all of the “sapcpe” calls like this one:
Execute_00 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG0)
Page 10 of 21
• Delete the existing (already copied) files in the instance-specific directories /usr/sap/PE2/<instancename>/
exe.
4.5 - Prepare SAP/Oracle switchover

Prepare both nodes (sscsapx1 and sscsapx2) to be capable to host the SAP/Oracle applications. If not done
before (during the installation), please check if these configurations are available and identical on both cluster
nodes.
Home directories
Relocate the home directories of the OS users pe2adm and orape2 onto global filesystems.
• Homedir of pe2adm: /usr/sap/PE2/pe2adm
• Homedir of orape2: /oracle/PE2/orape2
Create OS users and groups

Create the accounts on both hosts, if missing.
# groupadd g 100 sapsys
# groupadd g 101 dba
# groupadd g 102 oper
# groupadd g 103 sapinst
# useradd g 100 G 101,102,103 u 110 d /usr/sap/PE2/pe2adm s /bin/csh pe2adm
# useradd g 101 G 102,103 u 111 d /oracle/PE2/orape2 s /bin/csh orape2
Solaris project
Check “/etc/project” for this entry:
PE2:200:SAP System PE2:orape2,pe2adm,root::process.maxsemnsems=(priv,2048,deny);project.maxsem
ids=(priv,1024,deny);project.maxshmids=(priv,256,deny);project.maxshm
memory=(priv,18446744073709551615,deny)
Definition of default project

Check file “/etc/user_attr” for these entries:
pe2adm::::project=PE2
orape2::::project=PE2
Services
Check file “/etc/services” for these entries:
sapdp00 3200/tcp # SAP System Dispatcher Port
sapgw00 3300/tcp # SAP System Gateway Port
sapdp00s 4700/tcp # SAP System Dispatcher Security Port
sapgw00s 4800/tcp # SAP System Gateway Security Port
Hosts
Check file “/etc/hosts” to contain all logical hosts and physical nodes.
Mountpoints for failover filesystems

Make sure, that the mountpoint “/oracle/PE2” is available on all nodes.
Java J2SE
Install the most recent version of J2SE 1.4.2.x SDK on a global filesystem.
• Download self-extracting version von http://java.sun.com
• Install into directory /usr/sap/PE2/j2se
• Modify environment of user pe2adm: JAVA_HOME=/usr/sap/PE2/j2se
• Modify JC00 instance profile: jstartup/vm/home = /usr/sap/PE2/j2se
Note: For Solaris x64 you need to download a version from http://java.sun.com/j2se/1.4.2/SAPsite/download.html.
JCE policy files

Download from http://java.sun.com/javase/downloads and extract to:
Page 11 of 21
/usr/sap/PE2/j2se/jre/lib/security/local_policy.jar
/usr/sap/PE2/j2se/jre/lib/security/US_export_policy.jar
saposcol directory
Create a local working directory “/usr/sap/tmp” for saposcol on both nodes (owner: root, group: sapsys,
permissions: drwxrwxr-x).
Oracle client
Copy directory “/oracle/client” (from the Oracle-installation node) to the other nodes.
Oracle inventory
Relocate the Oracle inventory to shared storage. On the Oracle-installation node;
• Modify the entry in “/var/opt/oracle/oraInst.loc”: inventory_loc=/oracle/PE2/oraInventory
• Copy directory “/var/opt/oracle” to the other nodes
• Move directory “/oracle/oraInventory” to “/oracle/PE2/oraInventory”
4.6 - Test SAP/Oracle switchover

Check if SAP/Oracle is able to run on both cluster nodes. It makes sense to test application switchover at this point
in time, without a fully completed Sun Cluster integration. Because in the case of any errors, you could be sure to
have no impact of a potentially inaccurate cluster configuration. And you could focus your error analysis on the
native SAP stuff.
Start/Stop SAP and Oracle manually, relocate filesystems and IP adresses using Sun Cluster commands.
To stop SAP, use following commands and OS user pe1adm:

# stopsap r3 JC00 pe2cilh
# stopsap r3 SCS01 pe2scslh
# stopsap r3 ERS11 pe2erslh
(or kill the “enrepserver” process, if you have not provided a start profile for ERS11)
To stop Oracle, use following commands and OS user orape2:
# sqlplus
Enter username: / as sysdba
SQL> shutdown
# lsnrclt stop
To start Oracle, use following commands and OS user orape2:

# lsnrclt start
# sqlplus
Enter username: / as sysdba
SQL> startup
To start SAP, use following commands and OS user pe1adm:
# startsap r3 SCS01 pe2scslh
# startsap r3 JC00 pe2cilh
# startsap r3 ERS11 pe2erslh
(or “# /usr/sap/PE2/SYS/exe/run/enrepserver pf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh”, if you have not
provided a start profile for ERS11)
To switch resourcegroups (containing storage and IP) between the cluster nodes:
# clrg switch n <targetnode> <resourcegroup>
with <target-node> = sscsapx1 or sscsapx2
with <resourcegroup> = pe2scs, pe2ers, pe2ora or pe2ci
4.7 - Enable enqueue replication

You need to set some SAP parameters to enable the enqueue replication. For complete list of enqueue-
parameters please refer to the SAP documentation (http://help.sap.com → SAP NetWeaver 7.0 Library → Look
for the chapter "Profile Parameter Settings for the Standalone Enqueue Server").
This overview shows the enqueue (and enqueue replication) related parameterisation of this example installation.
All profiles (including all parameters) are listed in the appendix of this paper for reuse.
Parameters for SCS01

Configured in the instance profile:
Page 12 of 21
ipc/shm_psize_34 = 0
enque/server/replication = true
enque/serverinst = 01
enque/table_size = 4096
rdisp/enqname = $(rdisp/myname)
enque/snapshot_pck_ids = 100
Parameters for JC00 (and all other application instances)

Configured in the default profile:
enque/deque_wait_answer = TRUE
Parameters for ERS11

Nothing more to do, if you have reused the instance profile from the appendix of this paper.
4.8 - Test the enqueue replication

It makes sense to test the enqueue replication at this point in time, without a fully completed Sun Cluster
integration. Because in the case of any errors, you could be sure to have no impact of a potentially inaccurate
cluster configuration. And you could focus your error analysis on the native SAP stuff.
For more infos, please refer to the SAP documentation (http://help.sap.com → SAP NetWeaver 7.0 Library →
Look for the chapter "Replication Server: Check Installation").
Execute the following SAP-related commands (ensmon, enqt, …) with OS user pe2adm.
Attention: Take care to execute the commands on the correct nodes as shown below. Some of the commands
are working on both sides (Enqueue and Replicated Enqueue), others not. It may be astonishingly to create the
enqueue table test-entries on the Replicated Enqueue Server side (and not on the Enqueue Server side). But this
is NOT a typo.
1) Switch applications to their preferred node

Start the applications on their preferred node. Oracle DB, SCS01, JC00 on sscsapx1. ERS11 on sscsapx2. See
chapter “Test SAP/Oracle switchover” on how to start, stop and switch the applications (in this installation phase,
without having a complete cluster integration).
2) Check if replication is active

On sscsapx2 (the current node hosting ERS11, containing the Replicated Enqueue Server):
# ensmon pf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh 2
The output should look like this:

Try to connect to host pe2scslh service sapdp01
get replinfo request executed successfully
Replication is enabled in server, repl. server is connected
Replication is active
…
3) Check the enqueue table ID

On sscsapx1 (the current node hosting SCS01, containing the Enqueue Server):
# enqt pf=/sapmnt/PE2/profile/PE2_SCS01_pe2scslh 97
# enqt pf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh 97
The output should be identical and look like this:

REQ
EnqId: EnqTabCreaTime/RandomNumber = 14.01.2009 18:44:31 1231955071 / 2868
ReqOrd at Srv: TimeInSecs/ReqNumberThisSec = 18.01.2009 11:26:32 1232274392 / 1

4) Create enqueue table entries

# enqt pf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh 11 20
This command will create 20 dummy entries in the table.
5) View enqueue table entries
Page 13 of 21
On sscsapx1 (the current node hosting SCS01, containing the Enqueue Server):
# enqt pf=/sapmnt/PE2/profile/PE2_SCS01_pe2scslh 20
# enqt pf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh 20
The output should be identical and contain 20 entries like this (and maybe some others):
000 ENQT ENQ_PERF X enqueue performance test 11111111000
6) Perform switchover
Stop the instance SC01 on node sscsapx1 and start it on node sscsapx2. See chapter “Test SAP/Oracle
switchover” on how to start, stop and switch the applications (in this installation phase, without having a complete
cluster integration).
7) Check the success

• The Enqueue Server should have stopped the Replicated Enqueue Server on node sscsapx2. Check if the
OS process is gone. It is the process command name “er.sapPE2_ERS11”, if you have started ERS11
with “startssap” (having a start profile). Otherwise, if you have started the Replicated Enqueue Server
directly, the process command name is “enrepserver”.
• Check if the enqueue table ID is still the same from before the switchover. Repeat step 3, solely regarding
SCS01 (because ERS11 should be stopped). Remember that the node has changed from sscsapx1 to
sscsapx2 !
• Check if the 20 enqueue table entries still exist. Repeat step 4, solely regarding SCS01 (because ERS11
should be stopped). Remember that the node has changed from sscsapx1 to sscsapx2 !
Note. In the case, that the replication does not work correctly, you can use these files foe debugging:
• Look for the developer traces “dev_en*” in the directories “/usr/sap/PE2/SCS01/work” and
“/usr/sap/ERS11/SCS01/work”.
• Attention: If you have started the Replicated Enqueue Server directly (not using “startsap” and a start
profile), then you will find the trace files of the Replicated Enqueue Server in the home directory of user
pe2adm. Not yet, but in the final configuration, Sun Cluster will also start the Enqueue Server directly (not
using “startsap” and a start profile). Thus, you will find also these trace files in the home directory of user
pe2adm.
• If needed, increase the trace level. Set parameter “rdisp/TRACE = 3” in the instance profiles of SCS01 and
ERS11.
4.9 - Cluster data services configuration (part II)

This step will finalize the integration of SAP/Oracle into the Sun Cluster.
Provide start/stop script for SAP instances

Provide these two scripts (owner: pe2adm, group: sapsys, permissions: -rwxr—r--):
/usr/sap/PE2/pe2adm/startsap_cluster.sh
/usr/sap/PE2/pe2adm/stopsap_cluster.sh
You find them in the appendix. Sun Cluster will call them to start and stop the SAP instances SCS01 and JC00.
Oracle database
a) Create Oracle monitor-user and table
Start the oracle database.
Use OS user orape2 to connect to the db:
# sqlplus "/ as sysdba"
sql> create user haoracle identified by haoracle;
sql> alter user haoracle default tablespace system quota 1m on system;
sql> grant select on v_$sysstat to haoracle;
sql> grant select on v_$archive_dest to haoracle;
sql> grant create session to haoracle;
sql> grant create table to haoracle;
sql> exit;
b) Resource for Oracle database

Stop database, if running.
Then create the cluster resource:
# clresource create g pe2ora \
t SUNW.oracle_server \
p Resource_project_name=PE2 \
Page 14 of 21
p Connect_string=haoracle/haoracle \
p ORACLE_SID=PE2 \
p ORACLE_HOME=/oracle/PE2/102_64 \
p Alert_log_file=/oracle/PE2/saptrace/background/alert_PE2.log \
p Restart_type=RESOURCE_RESTART \
p resource_dependencies=pe2ora_stor \
pe2ora_db
c) Resource for Oracle listener

Stop listener, if running.
# clresource create g pe2ora \
t SUNW.oracle_listener \
p LISTENER_NAME=LISTENER \
p ORACLE_HOME=/oracle/PE2/102_64 \
p resource_dependencies=pe2ora_stor \
pe2ora_lsnr
SAP System Central Services (SCS)

a) Remove the Enqueue Server from instance start
Disable the startup in the start profile "/sapmnt/PE2/profile/START_SCS01_pe2scslh". Remove these lines in the
section ”Start SAP locking service”:
_EN = en.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)
Execute_03 = local rm f $(_EN)
Execute_04 = local ln s f $(DIR_EXECUTABLE)/enserver$(FT_EXE) $(_EN)
Start_Program_01 = local $(_EN) pf=$(DIR_PROFILE)/PE2_SCS01_pe2scslh
Note. You can start the Enqueue Server directly (and Cluster will do it in this way):
# /usr/sap/PE2/SCS01/exe/enserver pf=/sapmnt/PE2/profile/PE2_SCS01_pe2scslh
b) Resource for Enqueue Server

Stop instance SCS01, if running.
# clresource create d g pe2scs t SUNW.sapenq \
p enqueue_profile=/sapmnt/PE2/profile/PE2_SCS01_pe2scslh \
p enqueue_server=/usr/sap/PE2/SYS/exe/run/enserver \
p sap_user=pe2adm \
p enqueue_instance_number=01 \
p resource_dependencies=pe2scs_stor,pe2ora_db{ANY_NODE},pe2ora_lsnr{ANY_NODE} \
pe2scs_enq
c) Resource for SCS01 (remaining Message Server)

Stop instance SCS01, if running.
# clresource create d g pe2scs t SUNW.sapscs \
p sap_sid=PE2 \
p sap_instance_number=01 \
p sap_instance_name=SCS01 \
p msg_server_port=3901 \
p scs_startup_script=/usr/sap/PE2/pe2adm/startsap_cluster.sh \
p scs_shutdown_script=/usr/sap/PE2/pe2adm/stopsap_cluster.sh \
p resource_dependencies=pe2scs_stor,pe2ora_db{ANY_NODE},pe2ora_lsnr{ANY_NODE} \
pe2scs_ms
SAP Central Instance

a) Resource for JC00
Stop instance JC00, if running.
# clresource create d g pe2ci t SUNW.sapwebas \
p sap_sid=PE2 \
p sap_instance_number=00 \
p sap_instance_name=JC00 \
p sap_instance_type=J2EE \
p webas_startup_script=/usr/sap/PE2/pe2adm/startsap_cluster.sh \
p webas_shutdown_script=/usr/sap/PE2/pe2adm/stopsap_cluster.sh \
p resource_dependencies=pe2scs_stor{ANY_NODE},pe2ora_db{ANY_NODE}, \
pe2ora_lsnr{ANY_NODE},pe2scs_enq_1{ANY_NODE},pe2scs_ms{ANY_NODE} \
pe2ci_jc00
Page 15 of 21

a) Resource for Replicated Enqueue Server (instance ERS11)
Stop Replicated Enqueue Server (Instance ERS11), if running.
# clresource create d g pe2ers t SUNW.saprepl \
p replica_profile=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh \
p replica_server=/usr/sap/PE2/SYS/exe/run/enrepserver \
p sap_user=pe2adm \
p resource_dependencies=pe2scs_stor{ANY_NODE},pe2scs_enq{ANY_NODE} \
pe2ers_enq
Group affinities
Configure these resourcegroup affinities to reflect the special failover requirements of the SAP enqueue replication
concept.
# clresourcegroup set p RG_affinities=+pe2ers pe2scs
The weak positive affinity setting ensures that, in case of failover, the group pe2scs (SCS01, including Enqueue
Server) fails over to the node where the group pe2ers (ERS11, Replicated Enqueue Server) is running.
# clresourcegroup set p RG_affinities=pe2scs pe2ers
The strong negative affinity setting: The group pe2ers (ERS11, Replicated Enqueue Server) is never brought
online nor allowed to remain online on a node on which the group pe2scs (SCS01, including Enqueue Server) is
online.
4.10 - Enable and test the cluster

This chapter does not intend to describe a complete cluster acceptance test. Just the minimum functional test is
shown here.
Enable the resources

Enable the cluster resources, configured in the step before:
# clresource enable pe2ora_lsnr
# clresource enable pe2ora_db
# clresource enable pe2scs_enq
# clresource enable pe2scs_ms
# clresource enable pe2ci_jc00
# clresource enable pe2ers_enq
Switchover
Test if every resourcegroup can be switched over to every node.
# clrg switch n <targetnode> <resourcegroup>
with <target-node> = sscsapx1 or sscsapx2
with <resourcegroup> = pe2scs, pe2ers, pe2ora or pe2ci
Check if the evacuation of the Replicated Enqueue Server is working, as specified in the section “Group affinities”
above.
SAP enqueue replication

Test the enqueue replication again, as already described in the chapter “Test the enqueue replication”.
Verify that JC00 (as enqueue client) is not doing a restart, after group pe2scs (SCS01, including Enqueue Server)
has done an switchover or failover.
Check the enqueue clients

Check, if the JAVA enqueue clients (all SAP application instances) will continue to work without a restart, after a
SCS01 failover or switchover.
Page 16 of 21
5 - Appendix
5.1 - Default profile
File “/sapmnt/PE2/profile/DEFAULT.PFL”
SAPSYSTEMNAME = PE2
SAPGLOBALHOST = pe2scslh
#
# SAP Central Service Instance for J2EE
#
j2ee/scs/host = pe2scslh
j2ee/scs/system = 01
j2ee/ms/port = 3901
SAPDBHOST = pe2oralh
j2ee/dbtype = ora
j2ee/dbname = PE2
j2ee/dbhost = pe2oralh
DIR_PUT = /usr/sap/$(SAPSYSTEMNAME)/put
# Enable enqueue replication for all clients (= app instances)
enque/deque_wait_answer = TRUE
5.2 - ERS11 start profile

File “/sapmnt/PE2/profile/START_ERS11_pe2erslh”
SAPSYSTEMNAME = PE2
# See comments in the ERS11 instance profile.
SAPSYSTEM = 01
INSTANCE_NAME = ERS11
SETENV_00 = PATH=$(DIR_INSTANCE)/exe:%(PATH)
SETENV_01 = LD_LIBRARY_PATH=$(DIR_EXECUTABLE)
_PF = $(DIR_PROFILE)/PE2_ERS11_pe2erslh
#
# Disabled! See comments in the ERS11 instance profile.
#
# _CPARG0 = list:$(DIR_EXECUTABLE)/ers.lst
# Execute_00 = immediate $(DIR_EXECUTABLE)/sapcpe$(FT_EXE) $(_CPARG0) pf=$(_PF)
# OS_UNICODE = uc
#
# Start Replicated Enqueue Server
#
_ER = er.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)
Execute_01 = immediate rm f $(_ER)
Execute_02 = local ln s f $(DIR_EXECUTABLE)/enrepserver $(_ER)
# Not starting it in the SAPdescribed way.
# Restart_Program_00 = local $(_ER) pf=$(_PF) NR=01
Restart_Program_00 = local $(_ER) pf=$(_PF)
5.3 - ERS11 instance profile

File “/sapmnt/PE2/profile/PE2_ERS11_pe2erslh”
SAPSYSTEMNAME = PE2
Page 17 of 21
# !!ATTENTION!!
# Even this instance was installed on #11 and instance name is ERS11,
# you have to set the internal number to 01. That is the instance number of SCS01.
# This assures that both instances (SAP Enqueue Server AND SAP Replicated Enqueue Server)
# are using the same internal memory representation for the LOCK table. Thus being able to
# synchronize in the case of a failover (otherwise the synchronisation will fail!)
# The SAP documentation is describing a different way, which comes obviously to the same
# result. It starts the process within the start profile with this parameterisation:
# “Restart_Program_00 = local $(_ER) pf=$(_PF) NR=01” (note the ending NR=01).
# But as the cluster starts the Replicated Enqueue process directly (without using the
# start profile), we have to set this parameter in the instance profile.
SAPSYSTEM = 01
INSTANCE_NAME = ERS11
# Start directly from /sapmnt/PE2/exe (the dirs specified here are softlinks to /sapmnt/PE2/exe).
# No “sapcpe” to instancespecific dirs make sense, because it's all on the global filesystem(s).
SAPLOCALHOST = pe2erslh
OS_UNICODE = uc
# Infos about SAP Enqueue Server (runnning in SCS01)
enque/process_location = REMOTESA
enque/serverhost = pe2scslh
# General instance parameter
ipc/shm_psize_10 = 130000000
ipc/shm_psize_40 = 115000000
5.4 - SCS01 start profile

File “/sapmnt/PE2/profile/START_SCS01_pe2scslh”
SAPSYSTEMNAME = PE2
SAPSYSTEM = 01
INSTANCE_NAME = SCS01
# DIR_CT_RUN = $(DIR_EXE_ROOT)/run
# DIR_EXECUTABLE = $(DIR_INSTANCE)/exe
SAPLOCALHOST = pe2scslh
DIR_PROFILE = $(DIR_INSTALL)/profile
_PF = $(DIR_PROFILE)/PE2_SCS01_pe2scslh
SETENV_00 = LD_LIBRARY_PATH=$(DIR_LIBRARY):%(LD_LIBRARY_PATH)
SETENV_01 = SHLIB_PATH=$(DIR_LIBRARY):%(SHLIB_PATH)
SETENV_02 = LIBPATH=$(DIR_LIBRARY):%(LIBPATH)
#
# Copy SAP Executables !!! Disabled !!!
#
# _CPARG0 = list:$(DIR_CT_RUN)/scs.lst
# Execute_00 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG0)
# OS_UNICODE = uc
#
# Start SAP messaging service
#
_MS = ms.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)
Execute_01 = local rm f $(_MS)
Execute_02 = local ln s f $(DIR_EXECUTABLE)/msg_server$(FT_EXE) $(_MS)
Start_Program_00 = local $(_MS) pf=$(DIR_PROFILE)/PE2_SCS01_pe2scslh
#
# Start SAP locking service
# !!! Disabled !!! Will be started directly by Sun Cluster
#
# _EN = en.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)
# Execute_03 = local rm f $(_EN)
# Execute_04 = local ln s f $(DIR_EXECUTABLE)/enserver$(FT_EXE) $(_EN)
# Start_Program_01 = local $(_EN) pf=$(DIR_PROFILE)/PE2_SCS01_pe2scslh
Page 18 of 21
5.5 - SCS01 instance profile

File “/sapmnt/PE2/profile/PE2_SCS01_pe2scslh”
SAPSYSTEMNAME = PE2
SAPSYSTEM = 01
INSTANCE_NAME = SCS01
SAPLOCALHOST = pe2scslh
OS_UNICODE = uc
#
# SAP Messaging Service for Java
#
rdisp/msserv = 0
rdisp/msserv_internal = 3901
ms/standalone = 1
ms/server_port_0 = PROT=HTTP,PORT=81$$
#
# SAP Locking Service
#
enque/table_size = 4096
enque/snapshot_pck_ids = 100
enque/server/replication = true
ipc/shm_psize_34 = 0
ipc/shm_psize_10 = 130000000
ipc/shm_psize_40 = 115000000
5.6 - JC00 start profile

File “/sapmnt/PE2/profile/START_JC00_pe2cilh”
SAPSYSTEMNAME = PE2
SAPSYSTEM = 00
INSTANCE_NAME = JC00
SAPLOCALHOST = pe2cilh
DIR_PROFILE = $(DIR_INSTALL)/profile
_PF = $(DIR_PROFILE)/PE2_JC00_pe2cilh
SETENV_00 = LD_LIBRARY_PATH=$(DIR_LIBRARY):%(LD_LIBRARY_PATH)
SETENV_01 = SHLIB_PATH=$(DIR_LIBRARY):%(SHLIB_PATH)
SETENV_02 = LIBPATH=$(DIR_LIBRARY):%(LIBPATH)
#
# Copy SAP Executables !!! Disabled !!!
#
# _CPARG0 = list:$(DIR_CT_RUN)/j2eeinst.lst
#
# Copy SAP Executables
#
# _CPARG1 = list:$(DIR_CT_RUN)/igsexe.lst
#
# Start Java application server
#
_JC = jc.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)
Execute_02 = local rm f $(_JC)
Execute_03 = local ln s f $(DIR_EXECUTABLE)/jcontrol$(FT_EXE) $(_JC)
Start_Program_00 = local $(_JC) pf=$(_PF)
#
# Start internet graphics server
#
_IG = ig.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)
Execute_04 = local rm f $(_IG)
Page 19 of 21
Execute_05 = local ln s f $(DIR_EXECUTABLE)/igswd_mt $(_IG)
Start_Program_01 = local $(_IG) mode=profile pf=$(_PF)
5.7 - JC00 instance profile

File “/sapmnt/PE2/profile/PE2_JC00_pe2cilh”
SAPSYSTEMNAME = PE2
SAPSYSTEM = 00
INSTANCE_NAME = JC00
SAPLOCALHOST = pe2cilh
jstartup/trimming_properties = off
jstartup/protocol = on
jstartup/vm/home = /usr/sap/PE2/j2se
jstartup/max_caches = 500
jstartup/release = 700
jstartup/instance_properties = $(jstartup/j2ee_properties):$(jstartup/sdm_properties)
j2ee/dbdriver = /oracle/client/10x_64/instantclient/ojdbc14.jar
igs/listener/rfc/disable = 1
PHYS_MEMSIZE = 512
exe/saposcol = $(DIR_CT_RUN)/saposcol
#
# Jcontrol: Migrated Profile Parameter
# create at Thu Jun 12 15:06:30 2008
#
j2ee/instance_id = ID0016265
#
5.8 - Script startsap_cluster.sh

File “/usr/sap/PE2/pe2adm/startsap_cluster.sh”
#!/bin/sh
###################### please customize .... #######################
HOME=`getent passwd pe2adm | cut d: f 6 s`
. $HOME/.profile
INSTANCE=$2
LOGICAL_HOST=ùname n`
if [ "$INSTANCE" = "SCS01" ]; then
LOGICAL_HOST=pe2scslh;
fi
if [ "$INSTANCE" = "JC00" ]; then
LOGICAL_HOST=pe2cilh;
fi
SAPSID=PE2
STARTSAP="/usr/sap/${SAPSID}/SYS/exe/run/startsap"
####################### end of customizing .... #######################
SCRIPTNAME=`basename $0 `
LOG_FILE=/tmp/${SCRIPTNAME}_${INSTANCE}.log
exec >> $LOG_FILE
echo "================================================================================"
echo "`date`: Script $SCRIPTNAME started with options $* ... "
GREP=/usr/bin/grep
NEWTASK=/usr/bin/newtask
PS=/usr/bin/ps
SLEEP=/usr/bin/sleep
INSTID=ècho $INSTANCE | sed 's/^[AZaz]*//g'`
# start $INSTANCE ....
Page 20 of 21
echo " execute $STARTSAP $INSTANCE $LOGICAL_HOST "
$NEWTASK $STARTSAP $INSTANCE $LOGICAL_HOST
5.9 - Script stopsap_cluster.sh

File “/usr/sap/PE2/pe2adm/stopsap_cluster.sh”
#!/bin/sh
###################### please customize .... #######################
HOME=`getent passwd pe2adm | cut d: f 6 s`
. $HOME/.profile
INSTANCE=$2;
LOGICAL_HOST=ùname n`
if [ "$INSTANCE" = "SCS01" ]; then
LOGICAL_HOST=pe2scslh;
fi
if [ "$INSTANCE" = "JC00" ]; then
LOGICAL_HOST=pe2cilh;
fi
SAPSID=PE2
STOPSAP="/usr/sap/${SAPSID}/SYS/exe/run/stopsap"
SAPCCMSR="/usr/sap/${SAPSID}/SYS/exe/run/sapccmsr"
####################### end of customizing .... #######################
SCRIPTNAME=`basename $0 `
LOG_FILE=/tmp/${SCRIPTNAME}_${INSTANCE}.log
exec >> $LOG_FILE
echo "================================================================================"
echo "`date`: Script $SCRIPTNAME mit Optionen $* gestartet ... "
###################### stop ccmsr ######################
echo " execute ${SAPCCMSR} stop j2ee pf=/usr/sap/${SAPSID}/SYS/profile/${SAPSID}_${INSTANCE}_$
{LOGICAL_HOST} "
${SAPCCMSR} stop j2ee pf=/usr/sap/${SAPSID}/SYS/profile/${SAPSID}_${INSTANCE}_${LOGICAL_HOST}
###################### stop $INSTANCE .... ######################
echo " execute $STOPSAP startupsrv $INSTANCE2 $LOGICAL_HOST "
$STOPSAP startupsrv $INSTANCE $LOGICAL_HOST
echo " execute $STOPSAP $INSTANCE $LOGICAL_HOST "
$STOPSAP $INSTANCE $LOGICAL_HOST
# IGS needs some time for stoppin all precesses, so wait ...
if [ "`pgrep f /usr/sap/${SAPSID}/${INSTANCE}/exe/igs`" != "" ]
then
/usr/bin/sleep 30
fi
###################### stop jlaunch processes ######################
INSTANCEtemp=ècho ${INSTANCE} | awk '/^D[09][09]$/{print $1}'`
if [ "$INSTANCEtemp" ]; then
echo " killing jlaunch processes "
/usr/bin/pkill f /usr/sap/${SAPSID}/${INSTANCE}/exe/jlaunch
fi
###################### cleanup IPC ######################
instance_nr=ècho $INSTANCE | sed 's/[AZ]*//'`
echo " /usr/sap/${SAPSID}/SYS/exe/run/cleanipc $instance_nr remove"
/usr/sap/${SAPSID}/SYS/exe/run/cleanipc "$instance_nr" remove
###################### stop saposcol ######################
# Note: Not really correct. saposcol is not instance specific
# Best would be to start it outside of SAP or Cluster control as system service (SMF)
echo " execute /usr/sap/${SAPSID}/SYS/exe/run/saposcol k"
/usr/sap/${SAPSID}/SYS/exe/run/saposcol k
echo "`date`: Script $SCRIPTNAME finished
Page 21 of 21

HA With Sap en Queue Replication and Sun Cluster V02

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

HA With Sap en Queue Replication and Sun Cluster V02

Загружено:

Авторское право:

Доступные форматы

Sun Microsystems GmbH

Sun Microsystems GmbH

Author: Jörg Meiners, Sun Microsystems GmbH

2 - About this paper

3 - Description of the target setup

Local (root FS): Local (root FS):

Figure 1: Overview Cluster Resourcegroups

3.2 - Oracle database

Cluster-Resourcegroup: pe2ora; preferred node: sscsapx1

pe2ora_lh SUNW.LogicalHostname ip / hostname = 192.168.14.24 / pe2oralh

Notes regarding SAP/Oracle filesystem configuration:

3.3 - SAP System Central Services (SCS)

Cluster-Resourcegroup: pe2scs; preferred node: sscsapx1

3.4 - SAP Replicated Enqueue Server

Cluster-Resourcegroup: pe2ers; preferred node: sscsapx2

Notes regarding SAP/Oracle filesystem configuration:

3.5 - SAP Central Instance

Cluster-Resourcegroup: pe2ci; preferred node: sscsapx1

pe2ci_lh SUNW.LogicalHostname ip / hostname = 192.168.14.22 / pe2cilh

Notes regarding SAP/Oracle filesystem configuration:

4 - Performing the installation

4.2 - Basic Cluster configuration

Add Quorum Device

Data Services registration

4.3 - Cluster data services configuration (part I)

b) Create resourcegroup with storage and logical hostname

c) Enable and test Resourcegroup

SAP System Central Services (SCS)

b) Create global filesystems (UFS)

c) Add entries to /etc/vfstab (both cluster nodes):

d) Create resourcegroup with storage, logical hostname and NFS server

e) Enable and test Resourcegroup

SAP Central Instance

b) Enable and test Resourcegroup

SAP Replicated Enqueue Server

b) Enable and test Resourcegroup

4.4 - SAP/Oracle Installation

SAP Replicated Enqueue Server installation

4.5 - Prepare SAP/Oracle switchover

Create OS users and groups

Definition of default project

Mountpoints for failover filesystems

JCE policy files

4.6 - Test SAP/Oracle switchover

To stop SAP, use following commands and OS user pe1adm:

To start Oracle, use following commands and OS user orape2:

4.7 - Enable enqueue replication

Parameters for SCS01

Parameters for JC00 (and all other application instances)

Parameters for ERS11

4.8 - Test the enqueue replication

1) Switch applications to their preferred node

2) Check if replication is active

The output should look like this:

3) Check the enqueue table ID

The output should be identical and look like this:

4) Create enqueue table entries

5) View enqueue table entries

7) Check the success

4.9 - Cluster data services configuration (part II)

Provide start/stop script for SAP instances

b) Resource for Oracle database

c) Resource for Oracle listener