Oracle11g RAC

Oracle11gR2 RAC Data Guard Build

Oracle 11.2.0.3 on Linux Version 1.4 Leveraged Build Oracle10g Grid

Page 1 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

Table of Contents
1 Document Control ................................................................................................................. 4 1.1 1.2 1.3 2 3 4 5 6 7 8 9 Distribution List ...................................................................................................................... 4 Amendment Record ............................................................................................................... 4 Terminology ........................................................................................................................... 4

Summary............................................................................................................................... 5 Exclusions ............................................................................................................................. 5 Assumptions ......................................................................................................................... 5 Oracle References ................................................................................................................. 5 Requirements........................................................................................................................ 6 CRS and ASM ......................................................................................................................... 6 Ports and Firewalls ................................................................................................................ 6 Summary of Build Process...................................................................................................... 6

10 Configure Oracle Net Services ................................................................................................ 7 11 Initial Configuration ............................................................................................................... 8 11.1 DB Unique Name .................................................................................................................... 8 11.2 Primary Database: Forced Logging ......................................................................................... 8 11.3 Create Standby Logs on Primary ............................................................................................ 8 11.4 Create Directories (Standby) .................................................................................................. 8
Page 1 of 30

Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised October 18, 2010

11.5 Parameter File ........................................................................................................................ 8 12 Create Standby with Active Database Duplication .................................................................. 9 12.1 Parameter File ........................................................................................................................ 9 12.2 Create Directory ................................................................................................................... 11 12.3 Duplication ........................................................................................................................... 11 12.4 Create Standby Redo Logs on Standby ................................................................................ 11 13 Modify Primary and Start Replication................................................................................... 11 13.1 Modify tnsnames.ora ........................................................................................................... 11 13.2 Start Managed Recovery ...................................................................................................... 11 13.3 Verify Log Application .......................................................................................................... 12 13.4 RMAN configuration change on Primary (optional) ............................................................. 12 14 Registration of RAC with OCR............................................................................................... 12 15 Data Guard Broker Setup ..................................................................................................... 13 16 Automatic Client Failover..................................................................................................... 14 16.1 Services for READ/WRITE Access on Primary and Standby Servers ..................................... 14 16.2 Add Service for Read Only on Primary and Standby Servers ............................................... 14 16.3 tnsnames.ora and sqlnet.ora ............................................................................................... 15 16.4 Restart for Active Data Guard .............................................................................................. 15 17 Verification (Switchover) ..................................................................................................... 16 18 Known Issues ...................................................................................................................... 16 19 Appendices ......................................................................................................................... 17 19.1 Example ................................................................................................................................ 17 19.2 Data Guard Log Apply Verification Script ............................................................................. 22
Page 2 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

19.3 Maximum Availability Verification Script (lgwr.sql) ............................................................. 22 19.4 Script to Check Status at Primary (check_primary.sql) ........................................................ 23 19.5 asmfiles script....................................................................................................................... 23 19.6 RI for Active Data Guard Standby Creation .......................................................................... 24

Page 3 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

1
1.1

Document Control
Distribution List
Name
Barbara Pascavage Sherri Trojan

Role
Oracle Technical Lead Oracle CTO HP HP

Representing

1.2

Amendment Record
Version
1.0 1.1

Date
04/02/12 04/06/12

Status
Draft Draft 2

Comment
Based on 11.2.0.1 version Moved some things and added RI in Appendices Add to Backout Section for RI (in Appendices) Added more on Services Added note on TDE

1.2

04/13/12

1.3 1.4

04/18/12 4/21/12

1.3

Terminology
Term
grid

Definition
Term used to refer to the configuration of several servers (nodes) using Oracle Clusterware and Real Application Clusters; one database and multiple instances.

OMF Real Time Apply

Oracle Managed Files; filenames are generated by Oracle replication of changes from primary to standby as the logs are written instead of after the log switches complete.

Page 4 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

2 Summary
This document provides information on how to build Oracle 11.2.0.3 Active Data Guard physical standby databases. The grid is using Oracle 11.2.0.3 Clusterware (CRS) and Automatic Storage Management (ASM). This build can be completed with no downtime for the primary database. A bounce of the listener on the standby is required. That should take less than one minute. During that time current connections to the database are not impacted. However, no new connections can be made during that short outage. An example for the Oracle net files (listener.ora and tnsnames.ora) as well as the full rman duplication is provided in the Appendices. A sample RI is also provided in the Appendices.

3 Exclusions
This document does not include: instructions for installation and configuration of CRS and ASM instructions for installation of database binaries build of primary database.

4 Assumptions
This document is written for the experienced Oracle10 /11 DBA. This process should not be performed by anyone who has not supported and/or installed an Oracle grid configuration on a Unix (or Linux) platform. It is assumed that the Oracle DBA has access to both Oracle Support CSI and Oracle Technology Network. While the information included in this document has been tested, that is no guarantee that additional problems will not be encountered.

5 Oracle References
This document is based on Oracle Best Practices and Data Guard documentation. Metalink: How to create physical standby database with 11gRMAN DUPLICATE FROM ACTIVE DATABASE [ID 747250.1] Metalink: 11g Active Database Duplication [ID 568034.1] Metalink: Manual Completion of a Filed RMAN Duplicate [ID 360962.1] Oracle Data Guard Broker 11.2 Oracle Data Guard: Concepts and Administration 11g R2 Metalink Doc. 360962.1 Manual Completion of a Failed RMAN Duplicate.

Page 5 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

6 Requirements
The following are required before starting this build: The executables for the binaries have already been installed at the same patch level as the production database. The CRS, ASM instance and diskgroups have been configured on both primary and standby. The ASM diskgroups on standby are the same as those in primary; no file name conversions are required. Listener is configured on both primary and standby server. Primary is in archivelog mode Primary and standby are using db_recovery_file_dest for archivelogs

7 CRS and ASM

Because the standby is a physical copy of the primary, the storage allocated to each must be equivalent in size. If the primary is expected to expand, the standby must be able to expand to the same size. If the primary is expanded beyond the space available on the standby, then the standby will no longer be valid until the space is available. For this configuration, ASM diskgroups at standby site will be same as primary. Both are using two diskgroups: DATA for all data and FLASH for archivelogs. The online redo logs, controlfile and Data Guard Broker files are multiplexed across both diskgroups.

8 Ports and Firewalls

There must be 2 way communications between the standby servers (physical data guard databases) and the database servers at the primary (production) site. General information on firewalls: 1521 should be opened between all servers 22 (ssh) should be opened between all servers

9 Summary of Build Process

The following tasks will be performed to create the Physical Data Guard Standby instance(s): 1. Configure Oracle Net Services 2. Initial Configuration 3. Create Standby with Active Database Duplication 4. Modify Primary and Start Replication 5. Data Guard Broker Setup 6. Configuration of Automatic Client Failover

For step-by-step instructions, see the Appendices: RI for Active Data Guard Standby Creation. Page 6 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

10 Configure Oracle Net Services

The entries for the primary database must be on the standby servers and the standby database must be on the primary servers. This can be tested with a tnsping command before starting the build. Replication, including fal_server and fal_client, will be configured to use the default service that is automatically created using for the db_unique_name. In order to use Active Database Duplication, the connections must use a pre-configured static Oracle Net connection for the standby server. This is not a connection configured as a CRS Service. There must be an entry in the listener.ora file under the SID_LIST_LISTENER: (SID_LIST_LISTENER = (SID_DESC=(GLOBAL_DBNAME=<db_unique_name>) (ORACLE_HOME=<full path for ORACLE_HOME>) (SID_NAME=<instance name>)) ) Refer to the sample listener.ora file entry in the Appendices. Warning: While Oracle Support articles indicate that a static entry in the listener.ora file is not necessary when using the UR=A in the tnsnames.ora file, this will not work for an active database duplication. The connections can be made with sqlplus. The rman duplicate fails just after the addition of the controlfile entries to the standby spfile. There must be a corresponding tnsnames.ora entry for the standby on all servers in the primary grid and on the server that is used for initial creation of the standby: <alias>= (DESCRIPTION= (ADDRESS=(PROTOCOL=TCP) (HOST=<fully qualified vip name or ip>)(PORT=1521)) (CONNECT_DATA=(SERVER=DEDICATED) (SERVICE_NAME=<db unique name>) (INSTANCE_NAME=<instance_name>) ) There should also be an entry in the standby tnsnames.ora file referencing the primary database. Refer to the sample tnsnames.ora file entry in the Appendices. (UR=A) )

Page 7 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

11 Initial Configuration
The commands and actions in this section should be completed before the standby can be created.

11.1 DB Unique Name

The database name (db_name) must be the same for both the primary and standby. The database unique name (db_unique_name) must be different. If the db_unique_name is not specified in the init.ora file for the standby, it will default to the db_name and all files will be in the wrong ASM directory structure. It is the db_unique_name that is used in all srvctl commands.

11.2 Primary Database: Forced Logging

Set the primary database to FORCED LOGGING in order to prevent loss of data. alter database force logging;

11.3 Create Standby Logs on Primary

Create standby logs the same size as the redo logs. There should be at least one more standby log per instance than the number of redologs. The group numbers must not be duplicates of the current redo log groups. Use group numbers that allow for additional groups on both the primary and standby. The group numbers cannot be the same for the standby and primary logs.

11.4 Create Directories (Standby)

The only file system directory that must be pre-created is the directory for the audit_file_dest. The remaining directories will be automatically created under the diagnostic_dest (/u02/app/oracle/diag) directory. In ASM, under FLASH and DATA, create a directory with the db_unique_name for the standby.

11.5 Parameter File

Modify parameters on the primary and create a pfile for use on the standby.

Parameter _log_deletion_policy ALL OPTIONAL

Value

Note: This requires reboot of database. This should be set at initial database creation. If not, it can be delayed until a scheduled reboot. This does not impact the build; however, the rman archive log deletion policy cannot be set to applied on standby until this parameter is set. fal_server fal_client Net service name for the standby database Net service name for the primary database Page 8 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

Parameter log_archive_config

Value dg_config=(<primary net service name>,<standby net service name>) service=<net service name for other database> lgwr sync affirm valid_for=(online_logfiles,primary_role) db_unique_name=<name for other database>) DEFER Note: To prevent generation of error messages on the primary, this can be set to DEFER until the initial duplication is complete and standby archivelogs are created for both the primary and standby.

log_archive_dest_2

log_archive_dest_state_2

standby_file_management dg_broker_config_file1

AUTO set this to use an ASM file: +DG_DATA/<db unique name>/dr1<db unique name>.dat set this to use an ASM file: +DG_LOGS/<db unique name>/dr2<db unique name>.dat

dg_broker_config_file2

12 Create Standby with Active Database Duplication

This creates the standby database directly from the primary database. It must use sqlnet connections and be executed from the primary side.

12.1 Parameter File

Create a pfile from the primary and copy it to the standby. Review and modify the standby parameters. Change all occurrences of the db_unique_name from the primary name to the standby db_unique_name, with the following exceptions: excluding the db_name, which must be the same on both the primary and standby log_archive_config

Be sure to make the following changes to the standby init.ora file:

Parameter audit_trail cluster_database_instances cluster_database control_files NONE

Value

# of instances, if different FALSE (temporarily for rman duplicate) Remove this from the initial spfile used on the standby; it will be replaced with the appropriate values during the rman duplicate Page 9 of 30
Oracle 11gR2 RAC Data Guard Build

HP INTERNAL DOCUMENT/Revised April 20, 2012

Parameter db_unique_name <SID>.instance_number <SID>.instance_name _log_deletion_policy

Value unique name for the standby Number assigned to each SID Name assigned to each SID ALL optional Note: This requires reboot of database. This should be set at initial database creation. If not, it can be delayed until a scheduled reboot. This does not impact the build; however, the rman archive log deletion policy cannot be set to applied on standby until this parameter is set.

<SID>.local_listener

(ADDRESS=(PROTOCOL=TCP) (HOST=fully qualified hostname) (PORT=1521)) Fully qualified SCAN name:1521 Thread number assigned to each SID Audit file destination; Make sure this points to a valid directory on the standby server. (PROTOCOL=TCP)(SERVICE=<name for XDB service) Net service name for the primary database. Net service name for the standby database dg_config=(<primary net service name>,<standby net service name>) service=<net service name for other database> lgwr sync affirm valid_for=(online_logfiles,primary_role) db_unique_name=<name for other database>) DEFER Note: To prevent generation of error messages on the primary, this can be set to DEFER until the initial duplication is complete and standby archivelogs are created for both the primary and standby.

<SID>.remote_listener <SID>. Thread audit_file_dest

dispatchers fal_server fal_client log_archive_config

log_archive_dest_2

log_archive_dest_state_2

standby_file_management dg_broker_config_file1

AUTO set this to use an ASM file: +DG_DATA/<db unique name>/dr1<db unique name>.dat set this to use an ASM file: +DG_LOGS/<db unique name>/dr2<db unique name>.dat

dg_broker_config_file2

Page 10 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

Create a directory in ASM for the standby under FLASH and DATA. Create the spfile in ASM from the modified pfile. On each RAC standby server, point to the spfile in ASM from the $ORACLE_HOME/dbs/init<SID>.ora. Note: Ignore the parameter for standby_archive_dest. It is deprecated with Oracle11R2.

12.2 Create Directory

Create an adump directory to match the value in the parameter file.

12.3 Duplication
Start the standby in nomount using the modified parameter file. Connections in rman must use Oracle Net. Failure to use Oracle Net will result in the following error:
RMAN-03002: failure of Duplicate Db command at 09/03/2010 18:57:27 RMAN-06217: not connected to auxiliary database with a net service name

Startup rman, connect to the target & auxiliary, then issue the duplicate command, as follows: connect target sys/<pwd>@<PRIMARY instance name>_DGMGRL connect auxiliary sys/<pwd>@<STANDBY instance name>_DGMGRL DUPLICATE TARGET DATABASE FOR STANDBY FROM ACTIVE DATABASE; Refer to the sample RMAN Dup in the Appendices for full results from this command. When the duplication completes, the standby database should be mounted; but not open. If the standby is in a RAC, immediately update the cluster_database parameter to TRUE.

12.4 Create Standby Redo Logs on Standby

Create Standby Redo Logs on the standby database. The standby log files can be created while the database is mounted; but, not while it is in managed recovery mode. Refer to Create Standby Redo Logs on Primary for additional information.

13 Modify Primary and Start Replication

The duplicate command will leave the standby database mounted (as a standby). It will create all the files in ASM, including a temporary tablespace. The next step is to modify the primary database and complete the setup for replication to the standby.

13.1 Modify tnsnames.ora

Modify the tnsnames.ora entry for the standby. Change it to point to the scan address and remove the instance_name parameter. This should be changed on all tnsnames.ora entries for the primary and standby.

13.2 Start Managed Recovery

If log_archive_dest_state_2 was set to DEFER on the primary, it should be changed to ENABLED at this point.

Page 11 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

On the primary, change the database to maximum availability: alter database set standby database to maximize availability; Optionally, on the standby server, clean up the trace files that were generated during the configuration. Alter the standby to begin managed recovery: alter database recover managed standby database using current logfile disconnect; Note: Until the Data Guard Broker is configured, no automation of the managed recovery is in place. Always restart managed recovery manually after a database restart or server reboot until the Data Guard Broker is configured.

13.3 Verify Log Application

Perform a log switch on each of the primary RAC instances. Then verify that the logs are being sent to the standby. Check to be sure the logs are going to the db_recovery_file_dest. Verify that the logs are applied to the standby and correct any problems. Refer to the scripts in the Appendices: Data Guard Log Apply Verification Script, Maximum Availability Verification Script (lgwr.sql) and Script to Check Logs at Primary (check_primary.sql). RFS errors may appear in the standby alert log; these errors can be ignored.

13.4 RMAN configuration change on Primary (optional)

Optionally, set rman configuration for archivelog deletion policy to applied on standby. Note: This can only be set if the _log_deletion_policy=ALL has been set. Since this is an underscore parameter, it requires a database shutdown/restart. This should only be set if there is sufficient space to hold at least 3 days of archivelogs on the primary.

14 Registration of RAC with OCR

If the physical Data Guard standby is a RAC configuration, then the database and instances must be manually registered with srvctl commands. srvctl add database d <db unique name> -o <$ORACLE_HOME> -s open n <db name> -r PHYSICAL_STANDBY -p <location of spfile> -a DG_DATA,DG_LOGS

srvctl add instance d <db unique name> -i <instance name> -n <node name>

Use the srvctl config database d <db unique name> of the standby.

command to verify the configuration

Page 12 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

15 Data Guard Broker Setup

After the Data Guard physical standby is created, the data guard broker configuration is created. No outages are required to configure and enable the Data Guard Broker. The Data Guard Broker in a RAC environment requires that the broker configuration files be created in ASM or on shared disk. This configuration uses ASM. If not already configured to use ASM, set the following database parameters for the broker configuration files for both the primary and standby (scope=both). The following assumes the standard DATA and FLASH diskgroups: dg_broker_config_file1=+DATA/<db unique name>/dr1<db name>.dat dg_broker_config_file2=+FLASH/<db unique name>/dr2<db name>.dat

Turn on the Data Guard broker by setting the following parameter in both the primary and standby: dg_broker_start = true

Start dgmgrl and connect as sys to the primary database (sysdba connection is assumed). Create the data guard configuration: create configuration configuration name AS primary database is db unique name for primary connect identifier is <tnsnames.ora net service name> ; Add the database to the configuration: add database db unique name for standby AS connect identifier is <tnsnames.ora net service name> maintained as physical ; Note: Use the following command to verify that the properties are configured correctly for each database: show database verbose <name> For MaxAvailability mode , LogXptMode should be SYNC for both primary and standby databases. The value of LogXptMode can be found in show database verbose. If necessary correct the LogXptMode or other properties using the command: edit database '<Database Name>' set property LogXptMode='SYNC'; Turn on the configuration: ENABLE CONFIGURATION;

When bringing the broker configuration online (enable configuration). After the configuration is enabled, Data Guard may still be performing some background work. Wait approximately ten minutes for this work to complete before verification of the configuration. Check the Broker logs (drc<SID>.log), located in the trace directory where the text version of the alert log is located, to view the actions being taken by the broker. Use the show configuration command to verify that the broker configuration is successful. For additional information and commands, refer to the Data Broker manual at Oracle Support 11gR2 Manuals. Page 13 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

Note: The Broker will create a static service (ending in _DGMGRL) that it uses for switchover and failover. This is only used by the Broker and is not part of the normal Oracle Services. When you execute a show instance verbose <INSTANCE_NAME>, it will display a StaticConnectIdentifier in the following format: '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.209.116.60) (PORT=1521))(CONNECT_DATA=(SERVICE_NAME=TESTDB_DGMGRL) (INSTANCE_NAME=TESTDB1)(SERVER=DEDICATED)))'

16 Automatic Client Failover

By configuring automatic client failover, there is no need for the client to make any changes to accommodate switchover/failover of the primary. For Automatic Client Failover, addresses for both the primary and standby must be specified in the tnsnames.ora entry for the service. Note: If a service has already been created, use srvctl modify service to make changes to the service.

16.1 Services for READ/WRITE Access on Primary and Standby Servers

Add or modify the service(s) used for read/write connections. The srvctl option l indicates the database role for the CRS to start the service. If the service already exists, it can be modified to add the l PRIMARY option. ON PRIMARY: srvctl add service d <PRIMARY DB UNIQUE NAME> -s <SERVICE for read/write access> -r <comma separated list of instances> -l PRIMARY q TRUE e SELECT m BASIC w 10 z 150 ON STANDBY: srvctl add service d <STANDBY DB UNIQUE NAME> -s <SERVICE for read/write access> -r <comma separated list of instances> -l PRIMARY q TRUE e SELECT m BASIC w 10 z 150 Start the service on the primary database server. Verify the service is correct with the srvctl config service d <db unique name> -s <service name> command. Use the srvctl status database d <db unique name> -f v command to check that the instances and service is running.

16.2 Add Service for Read Only on Primary and Standby Servers
The Read Only service for access to the Active Data Guard standby must be created with both srvctl commands and with dbms_service package. First, in the primary database: exec DBMS_SERVICE.CREATE_SERVICE(SERVICE for read only access>, SERVICE for read only access>,NULL,NULL,TRUE,BASIC,SELECT,150,10,NULL); Page 14 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

Perform log switches to ensure the change is propagated to the standby side. Then create the service with srvctl commands on both the primary and standby grids. srvctl add service d <PRIMARY DB UNIQUE NAME> -s <SERVICE for read only access> -r <comma separated list of instances> -l PHYSICAL_STANDBY q TRUE e SELECT m BASIC w 10 z 150 srvctl add service d <STANDBY DB UNIQUE NAME> -s <SERVICE for read only access> -r <comma separated list of instances> -l PHYSICAL_STANDBY q TRUE e SELECT m BASIC w 10 z 150 The service can now be started on the standby side. If it fails to start, allow time for the replication of the dbms_service command. Verify the service is correct with the srvctl config service d <db unique name> -s <service name> command. Use the srvctl status database d <db unique name> -f v command to check that the instances and service is running.

16.3 tnsnames.ora and sqlnet.ora

The entry for the client tnsnames.ora must include both SCAN addresses for each entry. For example: # Address for read only access to the Active Standby database <ALIAS READ ONLY> = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = <scan for primary>)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = <scan for standby>)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVICE_NAME = <service name for read only access>) ) ) # Address for read/write access to the Primary database <ALIAS READ WRITE> = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = <scan for primary>)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = <scan for standby>)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVICE_NAME = <service name for read write>) ) ) In sqlnet.ora file on all the servers, set to 3 seconds to avoid longer TCP timeout. SQLNET.OUTBOUND_CONNECT_TIMEOUT=3

16.4 Restart for Active Data Guard

Verify that the orapw and init.ora files and the adump directory are on the additional standby servers. Verify that the standby cluster_database parameter is set to TRUE. Stop managed apply: alter database recover managed standby database cancel; Page 15 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

Shutdown the single instance of standby the is running. Start the standby database using srvctl start database command. Verify that the instances and services are running with the srvctl status command. Verify that the logs are being applied.

17 Verification (Switchover)
Schedule a downtime to verify the standby database switchover is working properly. Perform a switchover to the standby and then back to the original configuration. Switchover with 11gR2 is a two-step process: 1. SWITCHOVER: Connect to Data Guard Broker using <SID>_DGMGRL connection and check the status. If the result is SUCCESS, issue the switchover command. If there is a problem, resolve the problem before doing the switchover. 2. VERIFICATION: After the switchover is complete, verify that all instances & services are running and the logs are being applied. Check dgmgrl to verify the configuration shows the change of standby and primary. Check the srvctl configuration to verify that CRS has recorded the database role change.

18 Known Issues
Standby databases that are open for Active Data Guard may generate excessive audit logs in the adump directory. This is related to Oracle Bug 7006588. Until this is resolved, create a cron job to remove the aud logs at least hourly. If using TDE, the recommendation is to set the wallet for manual open. The encrypted data will replicate. However, there may be some problems when the Data Guard Broker attempts to open the wallet on the standby before the database open command completes. If there is an auto open wallet, additional testing should be completed to ensure that this does not interfere with opening the standby for Active Data Guard on the standby.

Page 16 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

19 Appendices
This section provides samples and a checklist that can be used when creating a physical Data Guard standby for Disaster Recovery.

19.1 Example
These examples are based on the following: Primary database is TESTDB Standby database is TESADG

Listener Entry on Standby:

SID_LIST_LISTENER = (SID_LIST= (SID_DESCRIPTION = (GLOBAL_DBNAME = TESTADG) (ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1) (SID_NAME = TESTADG1) ) )

tnsnames.ora entry on primary and standby servers:

TESTDB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = orthlt1-scan)(PORT = 1521)) (CONNECT_DATA = ) TESTADG = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.210.108.30) (PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TESTADG) (INSTANCE_NAME = TESTADG1) ) (UR=A ) ) (SERVER = DEDICATED) (SERVICE_NAME = TESTDB) )

Duplication results:
RMAN> connected to target database: TESTDB (DBID=2561044898) RMAN> connected to auxiliary database: TESTDB (not mounted)

RMAN>

Page 17 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

Starting Duplicate Db at 01-APR-12 using target database control file instead of recovery catalog allocated channel: ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: SID=128 device type=DISK contents of Memory Script: { backup as copy reuse targetfile '/u01/app/oracle/11.2.0.3/db_1/dbs/orapwTESTDB1' auxiliary format '/u01/app/oracle/product/11.2.0/db_1/dbs/orapwTESTADG1' } executing Memory Script Starting backup at 01-APR-12 allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=75 instance=TESTDB1 device type=DISK Finished backup at 01-APR-12 contents of Memory Script: { sql clone "alter system set control_files = ''+FLASH/testadg/controlfile/current.280.779460647'', ''+DATA/testadg/controlfile/current.270.779460647'' comment= ''Set by RMAN'' scope=spfile"; backup as copy current controlfile for standby auxiliary format '+FLASH/testadg/controlfile/current.279.779460647'; restore clone controlfile to '+DATA/testadg/controlfile/current.289.779460647' from '+FLASH/testadg/controlfile/current.279.779460647'; sql clone "alter system set control_files = ''+FLASH/testadg/controlfile/current.279.779460647'', ''+DATA/testadg/controlfile/current.289.779460647'' comment= ''Set by RMAN'' scope=spfile"; shutdown clone immediate; startup clone nomount; } executing Memory Script ;

sql statement: alter system set control_files = ''+FLASH/testadg/controlfile/current.280.779460647'', ''+DATA/testadg/controlfile/current.270.779460647'' comment= ''Set by RMAN'' scope=spfile

Page 18 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

Starting backup at 01-APR-12 using channel ORA_DISK_1 channel ORA_DISK_1: starting datafile copy copying standby control file output file name=/u800/backup/TESTDB/controlfiles/snapcf_TESTDB1.f tag=TAG20120401T125046 RECID=17 STAMP=779460647 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01 Finished backup at 01-APR-12 Starting restore at 01-APR-12 using channel ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: copied control file copy Finished restore at 01-APR-12 sql statement: alter system set control_files = ''+FLASH/testadg/controlfile/current.279.779460647'', ''+DATA/testadg/controlfile/current.289.779460647'' comment= ''Set by RMAN'' scope=spfile Oracle instance shut down connected to auxiliary database (not started) Oracle instance started Total System Global Area Fixed Size Variable Size Database Buffers Redo Buffers 2471931904 bytes

2230872 bytes 553649576 bytes 1895825408 bytes 20226048 bytes

contents of Memory Script: { sql clone 'alter database mount standby database'; } executing Memory Script sql statement: alter database mount standby database contents of Memory Script: { set newname for clone tempfile 2 to new; switch clone tempfile all; set newname for clone datafile 1 to new; set newname for clone datafile 2 to new; set newname for clone datafile 3 to new;

Page 19 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

set newname for clone datafile 4 to new; set newname for clone datafile 5 to new; set newname for clone datafile 6 to new; backup as copy reuse datafile 1 auxiliary format new datafile 2 auxiliary format new datafile 3 auxiliary format new datafile 4 auxiliary format new datafile 5 auxiliary format new datafile 6 auxiliary format new ; sql 'alter system archive log current'; } executing Memory Script executing command: SET NEWNAME renamed tempfile 2 to +DATA in control file executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME Starting backup at 01-APR-12 using channel ORA_DISK_1 channel ORA_DISK_1: starting datafile copy input datafile file number=00002 name=+DATA/testdb/datafile/sysaux.277.777741627 output file name=+DATA/testadg/datafile/sysaux.290.779460671 tag=TAG20120401T125111 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15 channel ORA_DISK_1: starting datafile copy input datafile file number=00001 name=+DATA/testdb/datafile/system.276.777741623 output file name=+DATA/testadg/datafile/system.291.779460687 tag=TAG20120401T125111 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07 channel ORA_DISK_1: starting datafile copy input datafile file number=00005 name=+DATA/testdb/datafile/example.278.777741629 output file name=+DATA/testadg/datafile/example.292.779460695 tag=TAG20120401T125111

Page 20 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07 channel ORA_DISK_1: starting datafile copy input datafile file number=00003 name=+DATA/testdb/datafile/undotbs1.279.777741633 output file name=+DATA/testadg/datafile/undotbs1.293.779460701 tag=TAG20120401T125111 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03 channel ORA_DISK_1: starting datafile copy input datafile file number=00006 name=+DATA/testdb/datafile/undotbs2.280.777741633 output file name=+DATA/testadg/datafile/undotbs2.298.779460705 tag=TAG20120401T125111 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01 channel ORA_DISK_1: starting datafile copy input datafile file number=00004 name=+DATA/testdb/datafile/users.281.777741635 output file name=+DATA/testadg/datafile/users.297.779460705 tag=TAG20120401T125111 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01 Finished backup at 01-APR-12 sql statement: alter system archive log current contents of Memory Script: { switch clone datafile all; } executing Memory Script datafile 1 switched to datafile copy input datafile copy RECID=17 STAMP=779460712 file name=+DATA/testadg/datafile/system.291.779460687 datafile 2 switched to datafile copy input datafile copy RECID=18 STAMP=779460712 file name=+DATA/testadg/datafile/sysaux.290.779460671 datafile 3 switched to datafile copy input datafile copy RECID=19 STAMP=779460712 file name=+DATA/testadg/datafile/undotbs1.293.779460701 datafile 4 switched to datafile copy input datafile copy RECID=20 STAMP=779460713 file name=+DATA/testadg/datafile/users.297.779460705 datafile 5 switched to datafile copy input datafile copy RECID=21 STAMP=779460713 file name=+DATA/testadg/datafile/example.292.779460695

Page 21 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

datafile 6 switched to datafile copy input datafile copy RECID=22 STAMP=779460713 file name=+DATA/testadg/datafile/undotbs2.298.779460705 Finished Duplicate Db at 01-APR-12 RMAN> Recovery Manager complete.

19.2 Data Guard Log Apply Verification Script

This script can be used to verify that the logs are being received and applied to the physical Data Guard standby. PROMPT Last log received and last applied select al.thread#, max(al.sequence#) "Last Recd" , max(lh.sequence#) "Last Applied" from (select thread#, sequence# from v$archived_log) al , (select thread#, sequence# from v$archived_log where applied=YES and registrar=RFS) lh where al.thread#=lh.thread# group by al.thread# order by al.thread# /

19.3 Maximum Availability Verification Script (lgwr.sql)

This script can be used to check the realtime apply process when using maximum availability mode: -- This script provides a report on the status of the standby database. -- It should be executed when connected to the standby database. PROMPT Instance select instance_name from v$instance; PROMPT Last log received and last applied PROMPT If using Real Time Apply, received and applied should be equal select al.thread#, max(al.sequence#) "Last Recd" , max(lh.sequence#) "Last Applied" from (select thread#, sequence# from v$archived_log) al , (select thread#, sequence# from v$archived_log where applied=YES and registrar=RFS) lh where al.thread#=lh.thread# group by al.thread# order by al.thread# / PROMPT LGWR processes select process, status, client_process, thread#, sequence#, block# from v$managed_standby where client_process in ('ARCH','LGWR','APPLYING_LOG') and thread#> 0 order by thread# Page 22 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

/ PROMPT Report archive gaps select thread#, low_sequence#, high_sequence# from v$archive_gap / PROMPT Currently waiting for the following log PROMPT If using Real Time Apply, no rows will be returned select thread#, sequence# from v$managed_standby where status = 'WAIT_FOR_LOG' / PROMPT Archivelog Space Report col limit format 999,999,999 heading "Limit (mb)"; col used format 999,999,999 heading "Space Used (mb)"; col free format 999,999,999 heading "Space Free (mb)"; select space_limit/(1024*1024) limit , (space_used + space_reclaimable)/(1024*1024) used , (space_limit-(space_used+space_reclaimable))/(1024*1024) free ,number_of_files from v$recovery_file_dest /

19.4 Script to Check Status at Primary (check_primary.sql)

The following script can be used to pull the last archive log from the primary database: select distinct db_unique_name, recovery_mode, synchronization_status from gv$archive_dest_status where db_unique_name !='NONE' /

19.5 asmfiles script

This script can be used to generate a list of all the files in ASM: select gname||sys_connect_by_path(aname,'/') full_path from (select g.name gname, a.parent_index pindex ,a.name aname, a.reference_index rindex ,a.alias_directory dir, a.system_created sys from v$asm_alias a, v$asm_diskgroup g where a.group_number = g.group_number) start with (mod(pindex, power(2,24))) = 0 connect by prior rindex=pindex order by full_path asc;

Page 23 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

19.6 RI for Active Data Guard Standby Creation

For the RI for the database build, copy the following and modify the primary and standby information as applicable. It can either be an attachment referenced in the Implementation Plan or cut/pasted into the RI tabs.

BACKOUT: 1. On primary, change the following parameters: alter database set log_archive_dest_state_2 = DEFER ; alter database set dg_broker_start=FALSE; 2. On primary and standby: remove Data Broker files from ASM 3. On standby: shutdown standby database comment out cron job for deletion of archive logs IMPLEMENTATION PLAN ** Secondary Contacts ** Oracle DBA performing database duplicate Name: Phone: Email: ** Bridge Line - N/A ** Teams HP Oracle DBA ** Implementation StepsContact MRO before starting. 855-724-0190 or www.rooms.hp.com/attend/default.aspx?key=RPL2ZQJLR2
Primary Database Unique Name: Primary Servers: SCAN and Clustername for Primary:

Standby Database Unique Name: Standby Server for Build: Additional Standby Servers: SCAN and Clustername for Standby:

Page 24 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

1. Standby Build Server: Add the following entry to the listener.ora file and restart the listener. (SID_LIST_LISTENER = (SID_DESC=(GLOBAL_DBNAME=<db_unique_name>) (ORACLE_HOME=<full path for ORACLE_HOME>) (SID_NAME=<instance name>)) )

2. Add entry to all tnsnames.ora files on the primary and standby servers:
<STANDBY DB UNIQUE NAME> = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = <ip or name for standby vip>) (PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = <db unique name for standby>) (INSTANCE_NAME = <db unique name for standby instance) (UR=A ) ) )

3. Add entry for primary database to all the standby servers:

<PRIMARY DB UNIQUE NAME> = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = <scan for primary>) (PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED)

(SERVICE_NAME = <db unique name for standby>)) )

4. Make the following changes on the Primary Database: alter database force logging (if force logging is already in place, this will error out without any impact) create standby logs check number and size of current logs: select group#, bytes/1024/1024 from v$log; add standby logs (at least one more than the number of log groups), using the following command syntax where # is a number: alter database add standby logfile group 20#; verify the standby logs are there: select group#, type from v$logfile; 5. Add parameter values for primary (alter system command with scope=both): fal_server=<standby db unique name> Page 25 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

fal_client=<primary db unique name> log_archive_config=dg_config=(<primary net service name>,<standby net service name>) log_archive_dest_2=service=<net service name for standby> lgwr sync affirm valid_for=(onine_logfiles,primary_role) db_unique_name=<db unique name for standby> standby_file_management=AUTO dg_broker_config_file1=+DATA/<db unique name for primary>/dr1<db unique name for primary>.dat db_broker_config_file2=+FLASH//<db unique name for primary>/dr2<db unique name for primary>.dat

6. On primary, create a pfile from the spfile: create pfile=<path and file name> from spfile; 7. Create the following directories on Standby: file for adump (audit logs) on each Standby server ASM DATA and FLASH directories for standby unique name

8. Copy from $ORACLE_HOME/dbs on primary to all standby servers and rename with standby SID: orapwd for primary spfile for primary (modify primary unique name in the file to the standby unique name)

9. Copy the pfile created from the primary spfile to the standby server that will be used for the build. Modify the copy of the pfile on the standby: change references from db unique name of primary to db unique name for standby with the following exceptions: Do NOT change db_name and log_archive_config parameters audit_trial=NONE cluster_database=FALSE remove references to controlfiles modify local_listener to reference listener on standby modify remote_listener to SCAN for standby dispatcher changed to use standby unique name fal_server=<primary net service name> fal_client=<standby net service name> log_archive_dest_2=service=<net service name for primary> lgwr sync affirm valid_for=(onine_logfiles,primary_role) db_unique_name=<db unique name for primary> dg_broker_config_file1=+DATA/<db unique name for standby>/dr1<db unique name for standby>.dat Page 26 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

db_broker_config_file2=+FLASH//<db unique name for standby>/dr2<db unique name for standby>.dat

10. Perform duplicate command (after duplicate finishes the standby database should be in mount state): sqlplus: startup nomount rman connect target sys/****@<net service name for primary> connect auxiliary sys/****@<net service name for standby> duplicate target database for standby from active database; exit; cluster_database=TRUE scope=spfile

11. Modify spfile to change parameter:

12. Modify tnsnames.ora on all the primary and standby servers to point to the scan and remove the instance specification:
<STANDBY DB UNIQUE NAME> = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = <scan for standby grid>) (PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = <db unique name for standby>) (UR=A ) ) )

13. Create standby redo logs on standby database (mounted) and start recovery: add same number of standby logs as added on primary database: alter database add standby logfile group 30#; verify the standby logs are there (select group#, type from v$logfile) start log apply in sqlplus: alter database recover managed standby database disconnect from session; 14. On primary, if log_archive_dest_state_2=DEFER change it to ENABLE and switch logfile several times. 15. Change to maximum availability: on primary: alter database set standby database to maximize availability; on standby: alter database recover managed standby database cancel; on standby: alter database recover managed standby database using current logfile disconnect;

16. Verify log apply on standby. select al.thread#, max(al.sequence#) Last Recd ,max(lh.sequence#) Last Applied from Page 27 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

(select thread#, sequence# from v$archived_log) al ,(select thread#, sequence# from v$archived_log where applied=YES and registrar=RFS) lh where al.thread#=lh.thread# group by al.thread# order by al.thread#; 17. Register standby RAC with OCR: srvctl add database d <db unique name> \ -o <$ORACLE_HOME full path> \ -s open n <db name> -r PHYSICAL_STANDBY \ -p <location of spfile> -a DATA,FLASH for each instance: srvctl add instance d <db unique name> -i <instance name> \ -n <node name> VERIFY: srvctl config database d <db unique name>

18. Configure broker from primary server: Set parameter in both standby and primary: dg_broker_start=TRUE Log into Data Guard Broker and create configuration: dgmgrl connect /

create configuration <name> AS primary database IS <db unique name for primary> connect identifier is <net service name for primary>; add database <db unique name for standby> AS connect identifier IS <net service name for standby> maintained as physical; show database verbose <name>; if necessary: edit database <database name> set property LogXpMode=SYNC; enable configuration; show configuration; 19. Modify service(s) for connections to primary (if a service is added an entry should be added to all the tnsnames.ora files on the primary and standby servers): srvctl modify service d <primary db unique name> \ -s <service name> -l PRIMARY q TRUE e SELECT -w 10 z 150 Verify with command: srvctl config service d <primary db unique name> -s <service name> Page 28 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

20. Add service(s) for connections to standby when it is activated (if a service is added an entry should be added to all the tnsnames.ora files on the primary and standby servers): srvctl add service d <standby db unique name> \ -s <service for read write access> -r <comma separated list of instances> \ -l PRIMARY q TRUE e SELECT m BASIC w 10 z 150 Verify with command: srvctl config service d <standby db unique name> -s <service name> 21. Add service for read only connections to standby: sqlplus on PRIMARY: execute dbms_service.create_service(<name for readonly access>,<name for read only service>,null, null,TRUE,BASIC,SELECT,150,10,null);

On PRIMARY: srvctl add service d <primary db unique name> \ -s <service for read only access> -r <comma separated list of instances> \ -l PHYSICAL_STANDBY q TRUE e SELECT m BASIC w 10 z 150 Verify with command: srvctl config service d <primary db unique name> -s <service name>

On STANDBY: srvctl add service d <standby db unique name> \ -s <service for read only access> -r <comma separated list of instances> \ -l PHYSICAL_STANDBY q TRUE e SELECT m BASIC w 10 z 150

On STANDBY, verify with command: srvctl config service d <standby db unique name> -s <service name> 22. Add tnsnames.ora on primary and standby for read only service created: <ALIAS for READ ONLY> = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = <scan for primary>)(PORT=1521)) (ADDRESS = (PROTOCOL = TCP) (HOST = <scan for standby>)(PORT=1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVICE_NAME=<service name for read only)) ) Page 29 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012

23. Add to sqlnet.ora: SQLNET.OUTBOUND_CONNECT_TIMEOUT=3 24. Stop managed recovery and shutdown standby: alter database recover manage standby database cancel; shutdown immediate; 25. Restart standby with srvctl start database command. The standby should be started along with the read only service. Since the broker is configured, the log apply will be started automatically. 26. Execute srvctl status command to verify standby is started with services. 27. Verify log apply is working (from standby): select process, status, thread#, sequence#, block# from v$managed_standby; 28. Set up an hourly cron job to delete the aud logs from $ORACLE_BASE/admin/<db unique name>/adump.

Page 30 of 30
Oracle 11gR2 RAC Data Guard Build HP INTERNAL DOCUMENT/Revised April 20, 2012