Internals of

Concurrent
Managers

2010 Tieto Corporation

UKOUG Conference Series

Technology & E-Business Suite 2010

Mris Elsi
Senior Oracle Applications DBA
Tieto Latvia,
Maris.Elsins@tieto.com

Who I am?
8 years in IT

3 years PL/SQL developer

5 years Oracle [Apps] DBA (started with 11.5.7 and 8.1.7)

Certificates

10g OCM

9i / 10g / 11g OCP

11i Applications Database Administrator OCP

11i System Administrator OCE

Conferences

UKOUG 2007/2008/2010

LVOUG 2009/2010

EMEA Harmony 2010

Current employer Tieto Latvia

All kinds of oracle DBA tasks - patching, upgrade, performance tuning,

troubleshooting, planning and implementation of backup and recovery procedures,
cross platform migration, etc

Planning of sydtem architecture, design and implementation of HA solutions (RAC,

Data Guard, Custom cold failover)

Implementation of system specific monitoring, automation of routine tasks

Technical project planning and coordination, management2010-11-29

of team of DBAs

2010 Tieto Corporation

Purpose of this session

Provide background knowledge for successful
troubleshooting of concurrent request problems
What this presentation is about?
Life cycle of a concurrent request
Implementation details of the most interesting phases of a life cycle
of a concurrent request
Internals of Concurrent Managers and Conflict Resolution Managers
DB objects that can be used for troubleshooting

Why it is important?
DBA has to be quick when solving real issues
Knowing the process before the problems happen decreases solving
time
Querying DB objects is faster then navigating through Forms
Knowing the process is important when tuning the setup of
Concurrent Managers
3

2010 Tieto Corporation

2010-11-29

Most interesting DB objects

Objects

FND_CONCURRENT_REQUESTS (FCR)
FND_CONCURRENT_PROGRAMS (FCP)
FND_CONCURRENT_QUEUES (FCQ)
FND_CONCURRENT_PROCESSES (FCPROC)
FND_CONCURRENT_PROGRAM_SERIAL (FCPS)
FND_CONC_RELEASE_CLASSES (FCRC)
FND_LOOKUPS (FL)
FND_CONCURRENT_WORKER_REQESTS (FCWR)
FND_CRM_HISTORY (FCH)
FND_CONC_WAITING_REQUESTS

Some examples how the tables are used will be uncovered

in this presentation

2010 Tieto Corporation

2010-11-29

Types of concurrent managers

Types of concurrent managers

Internal Concurrent Manager

Service Manager
Concurrent Manager
Conflict Resolution Manager
Internal Monitor
Transaction Manager
Scheduler/Prereleaser Manager

Today the accent is on

Concurrent Manager
Conflict Resolution Manager

2010 Tieto Corporation

2010-11-29

Life cycle of a concurrent request

INACTIVE (I)

Disabled (U)
On Hold (H)
No Manager (M)

PENDING (P)

Normal (I)
Standby (Q)
Scheduled (P)
Waiting (Z)

RUNNING (R)

COMPLETED (C)

Normal (R)
Paused (W)
Resuming (B)
Terminating (T)

FND_LOOKUPS contains the values of Phase/Status Codes

Oracle E-Business Suite System Administrator's Guide Maintenance
describes the meaning of each phase and status

2010 Tieto Corporation

2010-11-29

Normal (C)
Error (E)
Warning (G)
Cancelled (D)
Terminated (X)

Typical life cycle of most requests

Pending / Scheduled
Request is scheduled for the execution in future

Pending / Standby
Time to execute the request has arrived
Request waits for to be evaluated by the Conflict Resolution Manager

Pending / Normal
Request is allowed to be executed
Request is waiting to be picked up by Concurrent Manager

Running / Running
Request is beeing executed by a concurrent manager

Completed / Normal

2010 Tieto Corporation

2010-11-29

Phase/Status representation in tables

FND_CONCURRENT_REQUESTS
PHASE_CODE
STATUS_CODE

Is that so simple? NO!

Phase = Inactive practically not used
Inactive/Disabled: FCR.PHASE_CODE=P, FCP.ENABLED_FLAG=N
Inactive/On Hold: FCR.PHASE_CODE=P, FCR.HOLD_FLAG=Y
Inactive/No manager: FCR.PHASE_CODE=P and nonexistance of
specific rows in FND_CONCURRENT_WORKER_REQESTS view

Pending/Scheduled is actually
STATUS_CODE=Q+(FCR.REQUESTED_START_DATE>SYSDATE)

Why so complicated?
No need to update statuses too often

2010 Tieto Corporation

2010-11-29

Pending / Scheduled

2010 Tieto Corporation

2010-11-29

Schedule types
As Soon as Possible/Once
FCR.REQUESTED_START_DATE
Periodically/On Specific Days
FND_CONC_RELEASE_CLASSES (FCRC)
Join with FCR on column RELEASE_CLASS_ID
FCRC.DATE1 - Start at field in the form
FCRC.DATE2 - End at field in the form
FCRC.CLASS_TYPE P for Periodically, S for On Specific
days
FCRC.CLASS_INFO actual schedule data

FCR.REQUESTED_START_DATE always contains the time

of next execution, only this field used to determine when the
request should be run by the concurrent manager

10

2010 Tieto Corporation

2010-11-29

Periodic schedules
FCRC.CLASS_INFO contents
Values like X:Y:Z
X number of months/weeks/days/hours/minutes the request has to
be rescheduled from prior run.
Y time units: M months, D days, H hours, N minutes
Z rescheduling type: S from the start of the prior run, C
from the completion of the prior run.

Samples
30:N:S Repeat every 30 minutes from the start of the prior run
5:N:C Repeat every 5 minutes from the completion of the prior run
12:H:S Repeat every 12 hours from the start of the prior run

11

2010 Tieto Corporation

2010-11-29

On Specific Days schedules

FCRC.CLASS_INFO contents
Values like
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXYZZZZZZZ
X, Y, Z contains values 0 or 1, representing if the option is selected
or not
X digit in each position represent dates 1 31
Y Last Day of month
Z day of week, Su - Sa

Samples
000000000000000000000000000000000000001 Days of week:
Sa
111111111000000000000000000000000111110 Dates: 1 2 3 4 5 6
7 8 9. Days of week: Mo Tu We Th Fr
000000000000000000000000000000010000000 Last day of
month
12

2010 Tieto Corporation

2010-11-29

Reporting the schedules

Easy to schedule requests, but hard to keep track
Check if there are no obsolete schedules that can be removed or tuned
Might be few repeting schedules of the same concurrent program can
be consolidated into one
reporting_concurrent_request_schedules.sql

13

2010 Tieto Corporation

2010-11-29

Pending / Standby

14

2010 Tieto Corporation

2010-11-29

Conflict Resolution Manager (CRM)

CRM resolves conflicts in execution of requests enforced by
incompatibility rules
Simplified CRM workflow
Startup load all incompatibility rules
1. Loaded from FND_CONCURRENT_PROGRAM_SERIAL
2. Stored in memory by FNDCRM (CRM binary executable)

Each itereation
1. If FCQ.CONTROL_CODE=V, reload incompatibility rules (?)
2. Lock STATUS_CODE for all Pending/Normal and Pending/Scheduled
requests
3. Check each Pending request against the incompatibility rules and verify
there are running concurrent managers that can process them
(FND_CONCURRENT_WORKER_REQUESTS)
1. Release Pending/Standby requests, which dont break any rules
2. Return Pending/Normal requests, which break some rules (?)
4. Record statistics in FND_CRM_HISTORY
5. Sleep for Sleep Seconods
15

2010 Tieto Corporation

2010-11-29

Interesting DB objects
FND_CONCURRENT_WORKER_REQUESTS
View returning mappings of requests and concurrent managers able
to execute them
Definition of the view contains hardcoded IDs according to the
Specialization Rules of Concurrent Managers
Rebuilt by Build Concurrent Request Queue View request on
change of Specilaization rules

FND_CRM_HISTORY
Good information for tuning and troubleshooting
Records statistics for each CRM run, stotistics include:
REQUESTS_EXAMINED
REQUESTS_STANDBY
REQUESTS_RELEASED
REQUESTS_RETURNED

Can be joined with FCR to find out how many requests were
released in each RCM execution iteration
FCR.CRM_RELEASE_DATE between FCH.WORK_START AND FCH.WORK_END
16

Tieto Corporation
2010
Purged
by Purge Concurrent Request and/or2010-11-29
Manager Data,

Graphing FND_CRM_HISTORY

17

2010 Tieto Corporation

2010-11-29

Why is my request still pending?

Check the status!
Pending/Scheduled time for execution has not come yet
Pending/Normal waiting to be picked up by concurrent manager
The request has just been released
All concurrent manager processes are busy executing requests
Long request execution queue

Pending/Standby Waiting for CRM to be released

SELECT WREQID, PHASE, STATUS, WHY
FROM FND_CONC_WAITING_REQUESTS
WHERE REQID = &REQUEST_ID --5820658

18

2010 Tieto Corporation

2010-11-29

Pending / Normal

19

2010 Tieto Corporation

2010-11-29

Workflow of Concurrent Managers

20

2010 Tieto Corporation

2010-11-29

Building SQL for querying the requests queue

And (P.Execution_Method_Code != 'S' OR

Select R.Rowid
From Fnd_Concurrent_Requests R
Where R.Hold_Flag = 'N'
And R.Status_Code = 'I'
And R.Requested_Start_Date <= Sysdate
And (R.Node_Name1 is null or
(R.Node_Name1 is not null and
FND_DCP.target_node_mgr_chk(R.request_id) = 1))

(R.PROGRAM_APPLICATION_ID,
R.CONCURRENT_PROGRAM_ID) IN
((0, 98), (0, 100), (0, 31721), (0, 31722), (0, 31757)))
AND ((R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID)
NOT IN
((510, 40032),
(510, 40033),
(510, 42156),
(510, 42157),

AND EXISTS

(530, 43793),

(Select Null
From Fnd_Concurrent_Programs P
Where P.Enabled_Flag = 'Y'
And R.Program_Application_Id = P.Application_Id
And R.Concurrent_Program_Id = P.Concurrent_Program_Id
AND EXISTS
(Select Null
From Fnd_Oracle_Userid O
Where R.Oracle_Id = O.Oracle_Id
AND EXISTS (Select Null
From Fnd_Conflicts_Domain C

(530, 43794),
(535, 42626),
(535, 42627),
(535, 42628)) AND
((R.REQUEST_CLASS_APPLICATION_ID IS NULL AND
R.CONCURRENT_REQUEST_CLASS_ID IS NULL) OR
(R.REQUEST_CLASS_APPLICATION_ID,
R.CONCURRENT_REQUEST_CLASS_ID) NOT IN ((0, 2)))))
ORDER BY NVL(R.priority, 999999999), R.Priority_Request_ID,
R.Request_ID

Where P.Run_Alone_Flag = C.RunAlone_Flag

And R.CD_Id = C.CD_Id))

Distributed
Concurrent
Processing
implementation
Run
Alone
flag
implementation
with
Conflict
domains
Implementation
Exclusion
Exclusion
Execution
specialization
of
Immediate
Specialization
order
for
rules
concurrent
type
rule
forexecutables
concurrent
for Request
programs
(Subroutines)
type
Query
only
Pending/Normal
requests
21

2010 Tieto Corporation

2010-11-29

Building SQL for querying the requests queue

The query is built at the startup of the concurrent manager
It hardcodes all specialization rules for the manager
Any changes to specialization rules force restart of the
concurrent manager processes (and runs Build Concurrent
Request Queue View concurrent program too)
Be careful! It does restart automatically! What happens if there are
long running requests?
Use request types!

22

2010 Tieto Corporation

2010-03-26

(C.RunAlone_Flag = P.Run_Alone_Flag And

Locking the STATUS_CODE

(P.Run_Alone_Flag = 'N' OR Not Exists
(Select Null

From Fnd_Concurrent_Requests Sr

Where Sr.Status_Code In ('R', 'T')

SELECT ...
FROM fnd_concurrent_requests
fnd_concurrent_programs
fnd_application
fnd_user

R,

P,

A,

And R.Rowid = :reqname

fnd_oracle_userid

O,

fnd_conflicts_domain

And ((P.Execution_Method_Code != 'S' OR

C,

fnd_concurrent_queues
fnd_executables

And Sr.CD_id = C.CD_Id))))

And Q.Running_Processes <= Q.Max_Processes

U,

fnd_application

And Sr.Enforce_Seriality_Flag = 'Y'

Q,

A2,
E,

fnd_conc_request_arguments X
WHERE R.Status_code = 'I'
And ((R.OPS_INSTANCE is null) or (R.OPS_INSTANCE = -1) or
(R.OPS_INSTANCE =
decode(:dcp_on, 1, FND_CONC_GLOBAL.OPS_INST_NUM,
R.OPS_INSTANCE)))
And R.Request_ID = X.Request_ID(+)
And R.Program_Application_Id = P.Application_Id(+)
And R.Concurrent_Program_Id = P.Concurrent_Program_Id(+)
And R.Program_Application_Id = A.Application_Id(+)
And P.Executable_Application_Id = E.Application_Id(+)
And P.Executable_Id = E.Executable_Id(+)
And P.Executable_Application_Id = A2.Application_Id(+)
And R.Requested_By = U.User_Id(+)

(R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) IN
((0, 98), (0, 100), (0, 31721), (0, 31722), (0, 31757))) AND
((R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) NOT IN
((510, 40032),
(510, 40033),
(510, 42156),
(510, 42157),
(530, 43793),
(530, 43794),
(535, 42626),
(535, 42627),
(535, 42628)) AND
((R.REQUEST_CLASS_APPLICATION_ID IS NULL AND
R.CONCURRENT_REQUEST_CLASS_ID IS NULL) OR
(R.REQUEST_CLASS_APPLICATION_ID,
R.CONCURRENT_REQUEST_CLASS_ID) NOT IN
((0, 2)))))
FOR UPDATE OF R.status_code NoWait

And R.Cd_Id = C.Cd_Id(+)

And R.Oracle_Id = O.Oracle_Id(+)
And Q.Application_Id = :q_applid
And Q.Concurrent_Queue_Id = :queue_id
And (P.Enabled_Flag is NULL OR P.Enabled_Flag = 'Y')

23

2010 Tieto Corporation

And R.Hold_Flag = 'N

2010-11-29

Locking the STATUS_CODE

Query for locking the STATUS_CODE reimplements the same validation
criteria to make sure situation has not changed
All processes of a concurrent manager use the same query to fetch the
cache size number of requests
As more processes of the same manager are run, as higher the competition
for requests (ORA-00054: resource busy and acquire with NOWAIT
specified, or 0 rows updated by the query if the status has been changed
already)
As higher the competition, as faster runs out the list of cached request ids for
each manager
As sooner the list of cached queries runs out, as more often FCR is queried

We want to query FCR as saldom as possible

24

Not hard to get to point where FCR queries are TOP SQLs in DB
Even more important if you have RAC
The key is to minimize the number of concurrent manager processes
Cache size and sleep seconds have some effect

2010 Tieto Corporation

2010-11-29

Cache size and Sleep seconds

Sleep seconds
sleep time of Conflict Resolution Manager affects how soon the
request will be passed to execution
are spent only when no requests are in pending/normal status in
FCR
should be chosen based on max time the request is allowed to
spend in the queue
Max time 20s and 5 managers? == 100s sleep seconds?

Cache size
Large cache sizes make changes of request priorities less effective
(do you use different priorities)
Small cache size is OK for Long-running requests queue
Larger cache sizes are OK for Short-running requests queues that
have few concurrent manager instances.
Large cache size increases the number of failed attempts to lock the
status code.
25

2010 Tieto Corporation

2010-11-29

Running / Normal

26

2010 Tieto Corporation

2010-11-29

How to find processes and sessions?

select r.request_id req_id,
r.phase_code p,
r.status_code s,
(select node_name || ':'
from applsys.fnd_concurrent_processes cp
where concurrent_process_id = r.controlling_manager) ||
r.os_process_id cp_process,
gi.INSTANCE_NAME || ':' || ss.sid || ',' || ss.serial# inst_sid_serial#,
gi.HOST_NAME || ':' || pp.spid db_process,
ss.program,
ss.status,
ss.sql_id || ':' || ss.sql_child_number sql_id_chld,
ss.event,
ss.WAIT_TIME,
ss.STATE
from applsys.fnd_concurrent_requests r,
gv$session

ss,

gv$process

pp,

gv$instance

gi,

applsys.fnd_concurrent_processes cp
where request_id = &request_id

For completed requests CP_PROCESS field is still visible

and ss.audsid(+) = r.oracle_session_id

and pp.inst_id(+) = ss.inst_id
27and

pp.addr(+)
= ss.paddr
2010 Tieto
Corporation

2010-11-29

Completed / Normal
and also the presentation is
Completed/Normal

28

2010 Tieto Corporation

2010-11-29

Where to get more information?

OTN - Oracle E-Business Suite System Administrator's
Guide Documentation Set
http://etrm.oracle.com ER Diagrams and information about
the DB Objects
http://appsdbalife.wordpress.com Comment and ask the
questions, I will answer!

Thank you!

?
29

2010 Tieto Corporation

2010-11-29

 2010 Tieto Corporation

Mris Elsi
Senior Oracle Applications DBA
Tieto Latvia,
Maris.Elsins@tieto.com