1

<Insert Picture Here>

Oracle E-Business Suite Release 12.1 Upgrade Best Practices Technical Insight
Udayan Parvate, Director, EBS Release Engineering Lester Gutierrez, Senior Architect, EBS Applications Performance

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracles products remains at the sole discretion of Oracle.

Objectives Provide an overview of the R12.1 release, the technology stack and supported upgrade paths Outline best practices for the R12.1 technical upgrade, with a focus on downtime reduction Review runtime performance monitoring and tuning tips Provide sample customer upgrade profiles and share Oracles GSI upgrade experience

Agenda R12.1 Upgrade Overview Upgrade Best Practices Customer Upgrade Snapshots References Additional Resources Q&A

R12.1 Upgrade Overview

R12.1 Rapid Install (RI) R12.1 Maintenance Pack (MP) R12.1 was generally available (GA) in May 2009
Via Oracle Electronic Product Delivery (EPD) and Oracle Store For all the supported platforms and languages

Can be used by new and upgrading customers (11.5.9 and above) to go directly to R12.1
If you are on R11i, use R12.1 RI from EPD. Follow instructions from the Upgrade Guide: 11i to 12.1 and 12.1.1 Release notes If you are on R12.0.X, use R12.1 MP (7303030) from My Oracle Support (MOS). Follow instructions from R12.1 Maintenance Pack Install Instructions (752619.1)
7

R12.1 Technology Stack

TECHNOLOGY COMPONENT Apps Mid tierForms/Reports Apps Mid tierJava Oracle Home/ Apps Mid tierJDK VERSION INCLUDED 11i10CU2 6.0.8.25 VERSION INCLUDED 12.0.4 RI 10.1.2.2 VERSION INCLUDED 12.1 RI 10.1.2.3 VERSION CERTIFIED WITH MINIMUM REQUIRED VERSION

10.1.2.3

1.0.2.2/1.4.2

10.1.3.0/1.5

10.1.3.4/1.6.0

10.1.3.5

10.1.3.4/1.6

Database/ JDBC drivers

9.2.0.6/9.2.0.6

10.2.0.3

11.1.0.7/11.1.0.7

10.2.0.5/11.2.0.1

10.2.0.4/11.1.0.7

For details Oracle E-Business Suite Release 12.1 Maintenance Pack Installation Instructions (752619.1)

R12.1 Upgrade Paths

Minimum EBS suite level for direct upgrade to R12.1 11i9, 11i9CU1, 11i9CU2 or above 11i10, 11i10CU1, 11i10CU2 or above R12.0 and above Minimum EBS suite level required for database versions 10.2.0.4 11i9CU2, 11i10CU2 10.2.0.5/11.0.7/11.2.0.1 11i10CU2, R12.0.4 Based on R12.1 DB guide (761570.1), certified upgrade path options can be categorized into : A. Upgrade database and EBS level in a single downtime B. Upgrade database and EBS level in separate downtimes C. Apply upgrade interoperability DB patches and then upgrade EBS level
9

R12.1 Upgrade Paths Continued..

11.0/11i.X => 12.1
SOURCE

TARGET

12.1
B ,C

< = 11.5.8 11i10cu2 11.0

11.2.0.1 12.1

B ,C B ,C

10.2.0.5 12.1

11i9cu2 11.5.9/cu1 11.5.10/cu1

B ,C A

B ,C

11.1.0.7 12.1 10.2.0.4

A. Upgrade database and EBS level in a single downtime B. Upgrade database and EBS level in separate downtimes C. Apply upgrade interoperability DB patches and then upgrade EBS

10

R12.1 Upgrade Paths Continued..

12.0.X => 12.1
SOURCE

TARGET

12.1
B ,C B ,C A A

10.2.0.4

< = 12.0.3

12.1
B ,C

11.2.0.1 12.1 11.1.0.7

> = 12.0.4

B ,C

B ,C
A. Upgrade database and EBS level in a single downtime B. Upgrade database and EBS level in separate downtimes C. Apply upgrade interoperability DB patches and then upgrade EBS

12.1 10.2.0.5

11

R12.1 .1 Key Facts

11i10CU2 # of Product Schemas R12.0.4 RI R12.1 RI

209

195 (25 removed, 11 added when compared with 11.5.10 CU2)

201 ( 25 removed, 17 added when compared with 11.5.10 CU2 )

#file calls in DB portion of the U driver

PROD database size File system size #files shipped in RI #of Changed+ New files in DB portion of the U driver

104242

144940

156622

31 GB 26 GB 268359 NA

45 GB 28 GB 357778 ~95488 ( Vs 11.5.10.2 )

50 GB 28 GB 385755 ~104057 ( Vs 11.5.10.2 ) ~31843 ( Vs 12.0.4 )

12

Upgrade Best Practices

13

Upgrade Best Practices Identification of required patches

Prepare a complete list of pre and post patches and recommended code levels Keep the system current on AD/ATG/OAM code e.g. latest AD/ATG RUPs on 11i/R12.0 and once on R12.1 High priority patches from MOS. Consolidated Upgrade Patches (CUP) and required one-offs that need to be applied in pre-install mode. R12.1 CUP1 (7303029:12.1.0) is available now Review Known-issues sections from key documents to determine additional pre or post upgrade patches: Release notes, MP Install Instructions
14

Upgrade Best Practices Continued.. Patch merging and sequencing

Take advantage of patch merge, hot patching and apply patches in the correct sequence Merge patches Merge NLS patches per language. Apply NLS patches as hot patches Perform uptime maintenance when possible e.g. hot patching of iHelp or NLS patches, upload patch history HRGLOBAL step before NLS ( Document 145837.1 & 414434.1 ) Stage patches locally accessible by mid-tier machines instead of on nfs mounted or remote servers
15

Upgrade Best Practices Continued.. Adpatch options and other reporting tools
Use applicable adpatch options to reduce downtime and track key upgrade aspects using in-built tools Use non-interactive patching Use adpatch options such as nomaintainmrc, phtofile, nocompiledb, nolink, nogenform, nogenrep, nocompile jsp, noautoconfig, novalidate (1078973.1) Use optimal values for batch size and #workers Use OAM reports and Patch Wizard functionality to track patch timing, files applied, functionality patched, impacted customized files etc. (976188.1, 976688.1)
16

Upgrade Best Practices Continued.. Downtime reduction procedures

Use Shared APPL_TOP with Distributed AD for upgrades and regular maintenance for multi-node instances No need to apply the same patch on multiple tiers Distributed AD adds to the degree of parallelism by distributing AD workers across application tier nodes and improves D/G portion of the patch driver. Use Staged APPL_TOP for regular maintenance and upgrade Saves time to patch the file system (C/G portion) by using a patched up copy of production instance file system Use in 11i => R12.1 upgrade to avoid applying NLS C/G portion Can use for R12.0.X => R12.1 upgrade and once on R12.1
17

Upgrade Best Practices Continued.. Downtime reduction procedures continued..

For the duration of the upgrade, consider Disabling custom triggers and business events Disabling auditing if enabled If possible run in noarchivelog mode Disabling flashback DB Removing TDE from volume tables

18

Upgrade Best Practices Continued.. Pre-upgrade activities

Identify and execute tasks that could be completed in a separate downtime period, prior to the production upgrade Use applicable steps mentioned in the "Downtime reduction" and Upgrade By Request appendices E and G of the R12.1 upgrade guide Upgrade RDBMS version to latest certified for the current APPS level ( 11.2.0.2 / 11.1.0.7 / 10.2.0.5 ) Convert to Oracle Applications Tablespace Model (OATM) Compacts data, optimizes storage settings and I/O Other planned HW or OS upgrades
19

Upgrade Best Practices Continued.. Pre-upgrade activities continued...

Use TUMS (The Upgrade Manual Script) to avoid running tasks that are not relevant for your system Convert to multiple organization architecture (Document 210193.1) Drop MRC schema ( 11.5.10 and above ) Assign post upgrade jobs to specialized CM queue (by request_type, see Document 399362.1)

20

Upgrade Best Practices Continued.. Pre-upgrade activities continued...

Flush all the interfaces, such as Autoinvoice, Journal entry import, order import etc.. Review and disable all debug, logging at site, responsibility, user level Gather stats with GATHER_AUTO option for all schemas close to the start of downtime

21

Upgrade Best Practices Continued.. Pre-upgrade activities continued...

Pre-create new large indexes created by upgrade (constrained to indexes on existing columns) Minimize historical data to be upgraded as per business requirements Upgrade By Request Purge old and/or transient data before upgrading Over 260 standard purge programs in R12 Use new Purge Portal in OAM to administrate

22

Upgrade Best Practices Continued.. Purging

Use OAM to configure, initiate and monitor purge programs
Set the execution frequency and view program history Programs tagged with the Purge program type System Administrator >Oracle Applications Manager >Purging/Critical Activities

23

Upgrade Best Practices Continued.. Key recommended fixes

Affecting Upgrade Performance AD-Parallel improved scalability: 8917381 (Part of 9179588 AD CUP) Forms/reports generation regression with 11g: 8557019 General Review document 244040.1 for latest EBS recommended performance fixes

24

Upgrade Best Practices Continued.. Database tier configuration

Maximize SGA and PGA sizing An upgrade only involves 10's of concurrent sessions; starting rules of thumb ... log buffer = 30 to 100 Mb shared pool = 1 to 4 Gb pga target = 3 to 20 Gb buffer cache = multi Gb, be generous without causing excessive paging or swapping Adjust with help from AWR pool advisories

25

Upgrade Best Practices Continued.. Database tier configuration

Other upgrade specific init.ora changes If specified, remove db_file_multiblock_read_count Maximize multiblock I/O sizes Set job_queue_processes = # of CPUS adobjcmp.sql (Phases : plb+90 and last+63) Set parallel_max_servers = 2 X CPUs Helps with large index creation, stats gathering and some large upg+ phase jobs Need to test with production-like DB server and I/O subsystem Shutdown other RAC instances

26

Upgrade Best Practices Continued.. Batch size and #workers

Batch size 10K is suitable for most installs, you can test other values from 1K up to 100K if time allows # Workers Starting rule-of-thumb is between 1 and 1.5 x #CPUs It is critical to do multiple rounds of testing, adjusting above settings to maximize server utilization, but constrained by factors such as Memory utilization (no swapping/ excessive paging) CPU utilization (scale down if at 100%) I/O response times (scale down if averages > 20 ms)
27

Upgrade Best Practices Continued.. Performance testing, monitoring and additional optimizations...
Analyze long runners via timing report, ad_task_timing analysis Mine ad_task_timing to identify low worker utilization due to phasing waits and review responsible culprits Review targeted AWR Instance and SQL reports awrrpt.sql and awrsqrpt.sql Use My Oracle Support to check for known issues and workarounds for the longest running jobs

28

Upgrade Best Practices Continued.. Performance testing, monitoring and additional optimizations continued...
Outside-the-box optimizations Requires Support Approval* Long running index creation and Statistics gathering As these tasks run with little or no concurrent tasks, consider using alter system commands to increase parallel_max_servers and pga_aggregate_target (Need to test so as to not exhaust machine resources) Large indexes: Pre-create, but do not alter column definitions For stats (adsstats.sql ; phase: last+63) : On RAC, consider using all nodes when gathering stats or use exp/imp of stats

29

Upgrade Best Practices Continued.. Performance testing, monitoring and additional optimizations continued...
Outside-the-box optimizations Requires Support Approval*
Long running mv related xdfs (en phases) Cleanup and truncate large MV logs (requires MV complete refresh) Look more closely at long running jobs that you suspect may not be required for your system Thousands of jobs and customer product/data/setup configurations Some of the heavy lifting work may not be required for a specific site Check with Oracle Support to validate A fix maybe available or possible to optimize for your case New 12.1.3 AD timing report by module

30

Upgrade Best Practices Continued.. Runtime Performance Testing Tips

Use Automated, scripted tools EBS Test Started Kits (Winrunner/QTP) Bundled QA based automated scripts for EBS testing - Patch 8408886 Oracle Applications Testing Suite (Accelerators for EBS) Web and Forms based flows Complement with user participation tests and batch load tests with frequent and critical jobs References
http://blogs.oracle.com/stevenChan/2009/10/oats_ebs_certified.html http://blogs.oracle.com/stevenChan/2009/08/evolutionary_steps_automated_testing_ebs.html

31

Upgrade Best Practices Continued.. Runtime Performance Testing Tips

Review the white paper: Upgrade to Oracle 11g - Performance Best Practices
http://www.oracle.com/apps_benchmark/html/white-papers-e-business.html

Points to note include11g features for Testing of 11g or 10g workloads with Real Application Testing Managing SQL Execution Plans and minimizing performance degradation with11g SQL Plan Management

Session OOW2010: Tuning the Oracle E-Business Suite Environment

Review EBS Benchmark Publications
http://www.oracle.com/apps_benchmark/html/white-papers-e-business.html

32

Customer Upgrade Snapshots

33

Oracle E-Business Suite Release 12.1 Live Customers

34

Customer Upgrade Snapshots 11i to 12.1

CPS (Chicago Public Schools)

Release: 11.5.10.2 to 12.1 DB size: 900GB #Workers and batch size: 32, 10000 #CPUs on DB server: 2 node RAC, 8 CPUs per node Downtime reduction measures Distributed AD with shared APPL_TOP Upgrade RDBMS to 10.2.0.4 in a separate downtime # hrs for the D driver: 22 hrs Customer snapshot
http://www.oracle.com/customers/snapshots/chicago-public-schools-ebs-snapshot.pdf

35

Customer Upgrade Snapshots Continued... 11i to 12.1 Cisco

Release: 11i to R12 DB size: 600GB #Workers and batch size: 32, 20000 #CPUs on DB server: 16 Downtime reduction measures Distributed APPL_TOP #hrs for the D driver: 5.5 hrs

36

Customer Upgrade Snapshots Continued... 11i to 12.1 Dell

Release: 11i10 to R12.1.2 DB size: 15TB , 16 node RAC Cluster #Workers and batch size: 32, 10000 #CPUs on DB server: 8 Downtime reduction measures Distributed APPL_TOP Pre-create large indexes #hrs for the D driver: 30 hrs (latest test results)

37

Customer Upgrade Snapshots Continued... 12.0 to 12.1 Zebra Technologies Corporation

Release: 12.0.6 to 12.1 DB Size: 106GB #Workers and batch size: 32, 10000 #CPUs on DB server: 8 Downtime reduction measures Staged APPL_TOP #hrs for the U driver: 12 hrs Customer snapshot
http://www.oracle.com/customers/snapshots/zebra-technologies-corporation-ebs-snapshot.pdf

38

Customer Upgrade Snapshots Continued... 12.0 to 12.1 Oracle GSI

Release: 12.0.3+ to R12.1 DB size: 17TB #Workers and batch size: 60, 10000 #CPUs on DB server: 88 processors Downtime reduction measures Staged APPL_TOP for US and ten languages Ran data fixes for problems found in test upgrades prior to production upgrade to minimize stoppages Distributed APPL_TOP (4 servers - 15 workers each) Pre-create large indexes (6 hrs savings) #hrs for the D driver: 14 hrs

39

Customer Upgrade Snapshots Continued... 12.1 to 12.1.3 Oracle GSI

Release: 12.1+ to R12.1.3 -- (Latest Testing) DB size: 17TB #Workers and batch size: 200, 10000 #CPUs on DB server: 150 processors Downtime reduction measures Staged APPL_TOP for US and ten languages Ran data fixes for problems found in test upgrades prior to production upgrade to minimize stoppages Distributed APPL_TOP (4 servers - 50 workers each) Pre-create large indexes #hrs for the D driver: 4 hrs

40

Customer Upgrade Snapshots Continued... 12.0 to 12.1 AT&T

Release: 12.0+ to R12.1.2 DB size: 10 TB #Workers and batch size: 40, 10000 #CPUs on DB server: 32 Processors Downtime reduction measures Staged APPL_TOP for US and ten languages Distributed APPL_TOP #hrs for the D driver: 9 hrs

41

References

42

References
R12.1 documentation roadmap (790942.1) Oracle E-Business Suite Release 12.1 Info center (806593.1) Database preparation guidelines for R12.1 upgrade (761570.1) Patching FAQs (459156.1, 225165.1) Using staged or shared APPL_TOP and distributed AD (734025.1, 384248.1, 236469.1) OAM Patch Wizard overview and FAQ (976188.1, 976688.1) AD Command Line Options for Release R12 (1078973.1) Recommended Performance Fixes (244040.1) R12 Upgrade Sizing & Best Practices (399362.1)

43

Additional Resources

44

Additional Resources
White paper
Planning Your Oracle E-Business Suite Upgrade from Release 11i to Release 12.1 (987516.1) R12 Upgrade considerations by product: Financials (889733.1)

Have upgrade questions ?

Post on OTN R12 upgrade forum
http://forums.oracle.com/forums/forum.jspa?forumID=395&start=0

45

Finding Additional Information

Accelerate your evaluation and planning

http://launch.oracle.com/?OOW
Contains Presentations RCDs RVPs Videos Customer Stories White Papers etc
46

Oracle Products Available Online

Oracle Store
Buy Oracle license and support online today at oracle.com/store

47

We encourage you to use the newly minted corporate tagline Software. Hardware. Complete. at the end of all your presentations. This message should replace any reference to our previous corporate tagline Oracle Is the Information Company.

48

Oracle OpenWorld

Latin America 2010

December 79, 2010

49

Oracle OpenWorld

Beijing 2010
December 1316, 2010

50

EBS 12.1 UPGRADE DEMOPOD MOSCONE SOUTH, S-089 MON : 9:45 AM 5:30 PM TUE : 9:45 AM 5:30 PM

WED : 9:00 AM 4:00 PM

51

SUMMARY
DOs..
Read and follow official documentation and have a project plan

DONTs..
Ignore including relevant functional SMEs in the planning process from the beginning

Engage Support early on to validate your upgrade plan

Identify and execute tasks you can do today Test ! Test ! Test ! Keep track of patches applied in test upgrades Be smart about using the right tools and explore published downtime reduction techniques Optimize patch runs to suite your H/W

Skip failed or long running jobs as a regular practice

Drop indexes Run adpatch in serial mode Perform workarounds without approval from support/DEV Insufficient time and/or # of rounds of upgrade and critical functional flow testing

52