Diagnostic information for Database Replay issues

CAPTURE DATABASE
Following is the recommended information to be collected to debug Capture issues. Additional information may be requested. Capture Database information: Platform: RDBMS Version: RDBMS RAT patches: RAC: YES/NO RDA Report Check RAT patches at least as per MOS note 560977.1

Type of workload: Upload Information: ASH report AWR report Capture Report AWR export dump

Application (EBS, Siebel, SAP, and Homegrown) Try to find out details about the application, Is SYSDATE used extensively, is the application using PL/SQL are those PL/SQL procedures complex or simple? Is the application using RAT non supported features? SQL Loader direct path load, import/export OCI based object navigation (ADTs) and REF binds Non-PL/SQL based AQ Distributed transactions, remote describe/commit operations Flashback queries MTS (11.2.0.1 and earlier)

WRC trace files

This must be done at the end of capture. Before enabling the tracing Backup existing trace files to an alternate location and clean the trace directory Be careful in doing it in production system. This

will generate lot of files & It will be an overhead. The overhead will depend on the type of the application workload, Like OLTP workload will be have more overhead as compared to DSS workload , do not do it without considering consequences of the overhead. Avoid doing it on production systems. Minimize the impact by doing very small targeted workload that is having an issue. And best thing is do it on a test system and on a much targeted workload. This kind of tracing will be rarely required on Capture side. This needs to be done only on Oracle Supports recommendation or as requested by Oracle development. Identify trace files location. Sqlplus / as sysdba Select value from v$diag_info where name =Diag Trace; ( for 11 g ) For earlier version it will be udump. Turn on tracing on all instances sqlplus / as sysdba Alter system set _wcr_control=8; Execute Workload Capture. disable the tracing on all instances sqlplus / as sysdba Alter system set _wcr_control=0;

Additional information to be uploaded:

1) The entire capture files may be required in certain cases. 2) Provide exact steps used to do the capture. 3) Sql traces with binds may be required in certain cases. 4) Alert.log, Incident trace file if any

PREPROCESS STEP
Ensure that the preprocess has completed successfully. Preprocessed files are platform independent but database version dependent. Preprocess can be done on any operating platform and database, only requirement is it must be done on the same version of database which it going to be replayed on.Preprocess has to be done only once, There is no need to do preprocess again & again.Preprocessed capture can be reused for Database Replay any number of times on the same version of database. Preprocess is not recomeded to be done on Production Box,it should be done in an Test Box. Preprocess is a resource intensive job. Following is the recommended information to be collected to debug Preprocess issues. Additional information may be requested. Backup existing trace files to an alternate location and clean the trace directory before starting the tracing. Be careful in doing it in production system. This will generate lot of files & it will be an overhead. Identify trace files location Sqlplus / as sysdba Select value from v$diag_info where name =Diag Trace; Turn on tracing Sqlplus / as sysdba Alter system set _wcr_control=8; Execute the Preprocess Disable the tracing sqlplus / as sysdba Alter system set _wcr_control=0; For slow running preprocess provide an AWR or Statspack report. If there is any incident during Preprocess it will be in ADR_BASE/diag/rdbms/<DATABASE NAME>/<INSTANCE_NAME>/incident If possible send the entire capture files to Oracle support. This kind of tracing will be rarely required. This needs to be done only on Oracle Supports recommendation or as requested by Oracle development.

WORKLOAD ANALYZER REPORT

Review Workload Analyzer Report before doing Database Replay. Execute Workload Analyzer on captured workload & follow recommendations. This will help Identify any potential problems with Database Replay upfront. Use a dummy 11.2.0.2.0 database to generate this Workload analyzer Report if your Database Replay version is not 11.2.0.2.0. There is no need to have schema data to generate the Workload Analyzer Report. The results of the workload analysis are saved to an HTML report named wcr_cap_analysis.html located in the capture directory that is being analyzed.The capture files can be of any version. Workload captured from 9.2.0.X database will provide limited information as we do not have AWR export. Workload Analyzer does import AWR schema from captured directory. Refer to MOS Note 1268920.1 Real Application Testing: Workload Analyzer TIP: Before using Database Replay always complete SPA trials, analysis & fix any identified SQL regressions in the test system where replay will be done. This is a good way to verify the schema as well as it will miminze the chances of Database Replay being slow due to sql regression. TIP: It is recommended that you start the Database Replay with a smaller duration of capture. As a thumb rule we recommend Database Replay of one hour capture of representative workload. Do not jump directly to a larger duration capture like 12 hour or 24 hour. Once a one hour Database Replay run has worked well to satisfaction one can move towards a larger duration of Database Replay.This will allow you find issues quickly and rapidly retest to validate resolution actions.

REVIEW TEST DATABASE SETUP

Review Database Restore point How was the database restored, till what exact SCN & how exactly? Any missing User, Views, Tables , Synonyms, Procedure, Packages or any other object Any external dependency Review init.ora Parameters Network configuration like listener.ora & tnsnames.ora Isolate Test Database from Production Databases Reason If any of this are incorrect or inappropriate This can lead to Error divergence Data divergence May possibly reflect as some kind of performance problem as well Failure of Database Replay clients or Database Replay itself

Tip: Setup Guaranteed Restore Point using Database Flashback on the Test database before starting Database Replay. Isolate test database from production databases.

DATABASE REPLAY

Replay Database information: Platform: RDBMS Version: RDBMS RAT patches: RDA Report RAC: YES/NO Or Exadata Verify Database Server side RAT patches as well Database Replay client WRC side patches

Database Restore Point & Setup SPA usage: Have SPA been used for tuning? Upload information: ASH report AWR report Replay Report Compare period report Please Note This tracing will have a performance Overhead. Since Database Replay must always be done on a Test database, it is ok to do this tracing as long as production is not impacted in anyway. Ensure sufficient disk space. First wipe out the existing traces after backing it up, as this will generate lot of traces & we only want relevant traces. Identify trace files location sqlplus / as sysdba Select value from v$diag_info where name =Diag Trace; Turn on tracing on all instances Alter system set _wcr_control=8; Run workload Replay Disable the tracing once replay is done or cancelled. sqlplus / as sysdba Alter system set _wcr_control=0;

Server side tracing 1) Alert.log 2) In few cases sql traces with binds may be required 3) Exact steps taken to do the Database Replay including connection remapping

Additional information to be uploaded:

4) Any incident trace files if any will be in ADR_BASE/diag/rdbms/<DATABASE NAME>/<INSTANCE_NAME>/incident oswatcher or vmstat Verify that resources are not exhausted on servers

DATABASE REPLAY CLIENTS. 1) Ideally separate Database Replay clients (WRC) from database server so that you do not run into resource issue on database server. 2) Patch the Oracle Home from where the WRC is executed with the latest patch as recommended.

Replay Client information: Platform: Client Version: Verify that the amount of clients is sufficient for the workload. Execute WRC in calibrate mode to find out required number of WRCs Verify that the amount of CPUs are sufficient for the workload a thumb rule is 4 replay clients/CPU

# Clients # cpu/cores on client machines # client machines Upload information: WRC calibration info

On the Client machine, set up ADR_BASE in sqlnet.ora to a directory with write permissions. Possible incidents if any will be in $ADR_BASE/oradiag_oracle/diag/clients/user_oracle/ host_<host-id>

WRC Client side tracing

Backup existing trace files to an alternate location and clean the whole trace directory Start the replay client with the debug=ON option wcr user/passwd@server replaydir=<capture-dir> debug=ON client traces will be in $ADR_BASE/oradiag_oracle/diag/clients/user_oracle/ host_<host-id> Or Alternatively, You can specify the client trace directory using the workdir option of WRC wrc user/passwd@server replaydir=<capture-dir> workdir=<client-traces-dir> debug=ON Traces will be in the workdir directory Additional information to be uploaded: oswatcher or vmstat

Verify that resources are not exhausted on servers

NOTES

1) After fixing any identified SQL regressions in the test system where Database Replay was done using SPA, if there is still a performance problem during Database Replay this will be mostly indicated with waits on replay clock by wait event WCR: replay clock. Database Replay can be executed with SYNCRONIZATION=FALSE and the results be analyzed. The Workload Analyzer Report will show pointer towards indicating Database Replay may be tried with SYNCRONIZATION=FALSE.

2) Incorrect database restore or incorrect test system setup like missing users, schema, indexes etc is one of the common most causes of error or other issues during Database Replay. 3) One can expect more error or divergence when percentage of in-flight transactions is relatively more. 4) Always review Divergence section first & than move on to Performance analysis. Review Compare Period Report. Please look at the full divergence report if this report shows significant divergence. The possible divergence levels are: (NONE) no divergence detected at all. (LOW) minimal divergence detected but the performance comparison is most likely still valid. (MEDIUM) some non-trivial divergence is detected and the performance comparison is suspect. (HIGH) severe divergence detected and the performance comparison is unlikely to be informative. Besides using Replay divergence information to analyze Database Replay characteristics of a given system change, you should also use an application-level validation procedure to assess the system change. Consider developing a script to assess the overall success of the replay. For example, if 10,000 orders are processed during workload capture, you should validate that a similar number of orders are also processed during Database Replay. 5) For Performance comparison focus on Database Time, Not on Database Replay elapsed time.Even if the Database performance improved the Database Replay elapsed time is not going to change with the default Database Replay options, the Database Replay elapsed time will be pretty much very close to Capture elapsed time. But the Database Time will reduce if the performance has improved during Database Replay. The Database Replay elapsed time is likely to increase if the Database is running slow. The Database Time will increase if the performance is slow as compared to Capture during Database Replay. 6) For Workload Replay Analysis compare Replay to Replay in the same environment & Database Server to the best effort possible.

7) Use the scripts in <Document 760402.1> SCRIPTS TO DEBUG SLOW REPLAY for debugging slow Database Replay. This scrpts should be used once you have followed the best practices like using doing the SPA trials befor the Database Replay, Reviewing the Workload Analyzer reports, ensuring that the test database setup is correct and as expected and in conjuction with other performance and hardware data. 8) To find Database Replay divergence use <Document 1388309.1> How To Find Database Replay Divergence Details.