Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

Note: 1457104.1
ARUPDATE Abends Overview

The recovery from an ARUPDATE abend depends on WHERE the abend occurred in the lifecycle of the ARUPDATE multiprocess job. Hence, it
is important to appreciate the 'structure' of the ARUPDATE job and the processes, executed steps and their sequence.

The Receivable Update multi-process job (ARUPDATE) consists of up to eight steps, but these 4 are always executed:

1. AR_UPDATE AE process partitions the data and, if needed, calls AE AR_POSTING to create Payment, Maintenance, Transfer, Direct Debit
and/or Draft Worksheets GROUP_CONTROL and PENDING_ITEM records.
2. AR_PGG job contains AE process AR_PGG1 that calls AE process AR_PGG_SERV to generate PENDING_DST accounting entries and, if
needed, VAT and IU entries for any groups that are set to post.
3. AR_POST job contains AE process AR_POST1 that calls AE process AR_POSTING to post the transactions in each selected group.
4. AR_UPDATE2 AE process performs cleanup tasks.

The AR_UPDATE, AR_PGG1, AR_POST1 and AR_UPDATE2 will generate Message Log and Application Engine (AE) trace files. Hence, when
ARUPDATE is run, you can examine the following traces (produced in HTM but you can also save as TXT files) but please note the sequence of
allocating Process Instances <PI#> for each AE is not sequential. :

AE_AR_UPDATE_<PI#>_AET, AE_AR_PGG1_<PI#>_AET, AE_AR_POST1_<PI#>_AET & AE_AR_UPDATE2_<PI#>_AET.

Troubleshooting Abends

ARUPDATE Main Processes

1. AR_UPDATE
1. Builds groups of pending items (GROUP_CONTROL, PENDING_ITEM) from Payment, Maintenance, Transfer, Direct Debit and Drafts

1 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

worksheets and
2. Executes the parallel preprocessor to add group records to RP_PARALLEL and partition all groups set to post among the specified parallel
processes.

2. AR_PGG
The AR_PGG parallel job calls the AR_PGG1 AE Process (As delivered in DEMO. At customer sites you may also have AR_PGG2, AR_PGG3,
etc.). The AR_PGG1 AE Process sets the partition to #1 and then calls AR_PGG_SERV to actually process the data; it is only 1 Step. The
AR_PGG_SERV main operations are:

1. Copies group records from RP_PARALLEL to a Temp table

2. Locks the PENDING_ITEM by setting the records PROCESS_INSTANCE number and updates DST_BAL_ITEM =’O’
3. Locks GROUP_CONTROL records for each group and updates DST_BAL_STATUS=’O’
4. Creates and inserts the new User then System accounting entry rows in a Temp table PS_PG_PENDST_TAOn
5. Retrieves Customer records for each pending item
6. Starts the IU processor, if needed, to add Inter/Intra rows to PENDING_DST
7. Inserts into PENDING_DST from PS_PG_PENDST_TAOn
8. Updates GROUP_CONTROL DST_BAL_STATUS=’I’ and unlocks group records
9. Updates PENDING_ITEM DST_BAL_ITEM=’I’ and unlocks pending item records

3. AR_POST
The AR_POST parallel job calls AR_POST1. The AR_POST1 AE Process contains a MAIN Section with only 1 POSTING Step and 2 Actions
that sets the partition and RP_ACTION to ‘P’ and then calls the AR_POSTING AE Process. The AR_POSTING AE is called twice:

1. The 1st time from AR_UPDATE to generate any Payment, Maintenance, Transfer, Draft or Direct Debit Groups

2. The 2nd time from AR_POST1 to post the generated group pending items

The AR_POSTING logic executed depends on RP_ACTION when the program is called. If the RP_ACTION = ‘W’ then AR_POSTING will
generate Worksheet groups and pending items. This was covered in the AR_UPDATE analysis section. If the RP_ACTION =’P’ then
AR_POSTING will post the generated groups pending items.

4. AR_UPDATE2
The AR_UPDATE2 AE consists of a MAIN calling the UPDATE2 Section. The UPDATE2 Section calls AR_POSTING, AR_REV_EST and
AR_JGEN as needed. The HISTORY Step calls AR_POSTING.UPDATE2 Section if the Run Control Parameters include any History flags. The
ARREVEST Step calls AR_REV_EST.MAIN Section if Commitment Control is switched ON for AR. The FS_JGEN Step calls AR_JGEN.MAIN
Section if the Run Control Process to GL flag is ‘Y’.

2 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

Verify if Online Recovery is Feasible

Step 1: Verify If Online Recovery is Feasible

Navigate to: Accounts Receivable > Receivables Update > Reset Process page.

If the process does not appear on the 'Process Reset' page, click 'Refresh' or Search (magnifying glass) to ensure the page has the latest updates. If
the process appears under the 'Not Successful Process Requests' list, then either or both of the 'Reset' and 'Restart' (yellow) buttons will be
highlighted. This verifies whether the process can be reset or restarted online (hence avoiding external SQL updates and ensuring data integrity). If
there is a need to log a GCS SR later on, ensure this page is attached to SR.

Verify Where the Abend Occured

Step 2: Verify Where the Abend Occured

1.1 Which process abended: AR_UPDATE, AR_PGG [AR_PGG_SERV], AR_POST [AR_POSTING] or AR_UPDATE2?

1.1.1 If the Abend occured in AR_UPDATE (1st process) then the recovery is relatively simple as no accounting entries were generated
(AR_PGG) and no posting/table updating occured (AR_POST). Once the reason for the abend is resolved (which naturally varies according to the
Abend message), the job can be Restarted or Reset online (if feasible).

1.1.2 If the Abend occcured in AR_PGG/AR_PGG_SERV, then application data records have been 'locked' by the abending process (all relevant
table records PROCESS_INSTANCE field updated with the running/abended Process Instance number) and will need to be unlocked (otherwise,
they cannot be used by any other process). This 'unlocking' is part of the 'Reset' online process (if feasible) and effectively does 2 main updates:

1. Updates PROCESS_INSTANCE field in all relevant data records to Zero (the value '0', to allow data records to be selected again)

3 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

2. Deletes all records from RP_PARALLEL (to allow groups to be re-selected again)

The exact SQLs used by the online 'Reset' process can be found by executing this SQL:

select * from PS_AR_RESETSQL_TBL where PRCSNAME='ARUPDATE'

1.1.3 If the Abend occcured in AR_POST/AR_POSTING, then application data records could have been updated (e.g. new ITEMs inserted, new
ITEM_DST accounting records added, CUST_DATA customer balances updated, etc.). If the database records updates were not Rolled Back
(reveresed or not Committed) then this would constitute a major effort as it could mean data corruption (e.g. Related parent-child tables are out-of-
synch, totals incorrect, etc.). The determination of the extent of the data corruption, if any, would be the first task in this (thankfully rare) situation.

1.1.4 If the Abend occured in AR_UPDATE2, then it would mean that database application records have bee updated but Customer History may
have not. Depending on the Abend, this could be recovered in the next run of ARUDPATE (as Customer History updates are Run Time Options
driven) but it would need in-depth investigation.

Verify Data Status After Abend

Step 3: Verify Data Status after ARUPDATE Abend

1. Verify any existing records locked by Process Instance = Abending Process Instance PI# .

(some of the tables below, e.g VAT, SUBCUST, DRAFT may not be relevant to your site):

SELECT * FROM PS_GROUP_CONTROL WHERE PROCESS_INSTANCE =’PI#’

SELECT * FROM PS_PENDING_ITEM WHERE PROCESS_INSTANCE =’PI#’
SELECT * FROM PS_PENDING_DST WHERE PROCESS_INSTANCE =’PI#’
SELECT * FROM PS_PENDING_VAT WHERE PROCESS_INSTANCE =’PI#’

SELECT * FROM PS_ITEM WHERE PROCESS_INSTANCE =’PI#’

SELECT * FROM PS_ITEM_ACTIVITY WHERE PROCESS_INSTANCE =’PI#’
SELECT * FROM PS_ITEM_ACT_VAT WHERE PROCESS_INSTANCE =’PI#’

4 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

SELECT * FROM PS_CUST_DATA WHERE PROCESS_INSTANCE =’PI#’

SELECT * FROM PS_CUST_HISTORY WHERE PROCESS_INSTANCE =’PI#’
SELECT * FROM PS_SUBCUST_DATA WHERE PROCESS_INSTANCE =’PI#’
SELECT * FROM PS_SUBCUST_HISTORY WHERE PROCESS_INSTANCE =’PI#’

SELECT * FROM PS_PAYMENT WHERE PROCESS_INSTANCE =’PI#’

SELECT * FROM PS_DRAFT_CONTROL WHERE PROCESS_INSTANCE =’PI#’

2. Verify the Abended Process selected Groups:

SELECT * FROM PS_RP_PARALLEL WHERE RUN_CNTL_ID =’%’ and OPRID = ‘%’

3. Verify the Abended PI# Run Control ID and TAO status and data:

(where '%' is your actual Run Control ID or Operator ID)

SELECT * FROM PS_AERUNCONTROL WHERE PROCESS_INSTANCE=’PI#’ AND RUN_CNTL_ID =’%’ and OPRID = ‘%’
SELECT * FROM PS_AETEMPTBLMGR WHERE PROCESS_INSTANCE=’PI#’ AND RUN_CNTL_ID =’%’ and OPRID = ‘%’

SELECT * FROM PS_PGG_SERVICE_AET WHERE RUN_CNTL_ID =’%’ and OPRID = ‘%’

SELECT * FROM PS_RP_POSTING_AET WHERE RUN_CNTL_ID =’%’ and OPRID = ‘%’'
SELECT * FROM PS_AR_UPDATE_AET WHERE RUN_CNTL_ID =’%’ and OPRID = ‘%’
SELECT * FROM PS_AE_REQUEST WHERE RUN_CNTL_ID =’%’ and OPRID = ‘%’'
SELECT * FROM PS_AE_REQUEST_OPT WHERE RUN_CNTL_ID =’%’ and OPRID = ‘%’

Recovery from Abends

Option 1 [Online Data Correction]

5 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

Recovery Option 1 [Online Data Correction/Isolation]

If the 'offending' source of data (e.g. Online Group, External Billing Group, Maintenance Worksheet, etc) has been identfied, then this specific data
source could either be corrected or 'isolated' using online pages. This allows further processing of ARUDPATE (Restart or Reset, below),
especially if this Abend occurs in a Production environment with tight nightly schedules.

Example 1 : Online Group 123 is identified as containing data corruption, hence, if feasbile using online navigation path, it's Posting Status would
be set to Do Not Post so it will be bypassed by the next run of ARUPDATE or it could just be completely deleted online and re-added again.

Example 2: If a missing setup is causing the abend (e.g. missing currency rate, missing Entry Type, missing Payment Terms code to calculate Due
Date, etc.) then the missing setup data could be added/updated online before the Reset or Restart of the abended process.

In summary: If the identified data causing the abend can be corrected online then this would always be the preferred approach, otherwise external
SQL updates may be necessary (see below).

Option 2 [Online Restart]

Recovery Option 2 [Online Restart]

If the online page Accounts Receivable > Receivables Update > Reset Process shows the abended process and allows (highlighted yellow button)
online Restart, and the reason for the abend is known and is transient (not permanent) and fixed then the ARUPDATE job can be Restarted
(resumed) online. This will execute ARUPDATE again starting from the abended step, but this time, as the cause of the abend has been resolved
externally, the step will hence execute correctly. Naturally, this assumes that the Abend cause has been identified and resolved (setup changed,
currency rate added, etc.). If the Abend cause has not been identified, then Restart is not an effective action as the job will simply abend again on
the exact same step.

Option 3 [Online Reset]

6 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

Recovery Option 3 [Online Reset]

If the online page Accounts Receivable > Receivables Update > Reset Process shows the abended process and allows (highlighted yellow button)
online Reset, the Abend cause needs more in-depth further investigation or does not allow Restart (missing setup that will adversely impact further
processing if job allowed to resume), and the database records need to be 'unlocked' then the online Reset will execute the SQL Updates of
PS_AR_RESETSQL_TBL and 'release' all relevant locked records. This implies the next run of ARUPDATE will start from a 'clean slate' and all
relevant records (online groups, worksheets, etc.), as per Run Time Options, will be re-selected for processing.

References to the above Restart/Reset processes are as follows: (different release levels may have different chapter numbers):

PeopleSoft Enterprise Receivables 9.0 PeopleBook > Chapter 29 Posting and Unposting Groups > Section Understanding How to Restart or Reset
Receivable Update. Here is the text:
"Accounts Receivable, Receivables Update, Reset Process, Process Reset
Access the Process Reset page. Enter your search criteria for the process instance and click Refresh.
In the Not Successful Process Requests grid, select the process instance that you want"

Option 4 [Offline SQL to Unlock Records]

Recovery Option 4 [Offline SQL to Unlock Records]

Usually, this Option should not be attempted if any of the Online Options are feasible. However, in some situations this could be the only feasible
option. In most instances, but NOT all, an AR_PGG_SERV abend could be recovered by unlocking the records locked by the abended process (or,
in other scenarios, another process impacting the abended process).

Locking simply means the records PROCESS_INSTANCE field has been set to the abended Process Instance Number. Step 3: Verify Data Status
after ARUPDATE Abend above would have identified the locked records and hence the SQL updates to unlock them would take the form of (using
Group_Control as an example, but you will need to repeat for all tables):

UPDATE PS_GROUP_CONTROL SET PROCESS_INSTANCE = '0' WHERE GROUP_BU ='Group Business U:nit' AND GROUP_ID
='Group ID' AND PROCESS_INSTANCE =’PI#’

7 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

After all relevant tables (see Step 3 Verify above) has been unlocked, the RP_PARALLEL table (where selected Groups are stored during each
process run) will need to be cleared, otherwise any group whose record still exists on this table will NOT be selected in the next run of
ARUPDATE. The SQL code will take this form:

DELETE FROM PS_RP_PARALLEL WHERE RUN_CNTL_ID =’%’ and OPRID = ‘%’

Option 5 [Offline SQL to Correct Data]

Recovery Option 5 [Offline SQL to Correct Data]

There could be instances where SQL updates to data tables (whether temporary _TAO or application) could resolve the abend and allow a Restart
(resume processing) from the Abended step. This scenario typically occurs in duplicate INSERT abends (where a record with an identical key to
the Source table record already exists on the Target table). The approach could be to remove the existing record from the Target table or the new
record from the Source table, depending which record is erroneous.

Naturally, the offline SQLs would depend on which tables are impacted and after careful examination of subsequent impact on processing.
Hence, any such SQLs must be tested on a COPY of Production database first (this, as previously stated, is advised for all external SQLs).

Example 1: An abend at AR_POSTING.DISTRIB.INSERT where INSERT INTO PS_ITEM_DST abends due to duplicate records on the
PS_RP_PNDDST_TAO4 source temporary table.

1. Examine if the PS_RP_PNDDST_TAO4 source table contained multiple duplicate records. A sample SQL would be to select rows with same
KEY field values:

SELECT DISTINCT BUSINESS_UNIT, CUST_ID, ITEM, ITEM_LINE,

ITEM_SEQ_NUM, LEDGER_GROUP, LEDGER, DST_SEQ_NUM
FROM PS_RP_PNDDST_TAO4 WHERE PROCESS_INSTANCE = 'PI#'

2. Examine if ITEM_DST already has records that are similar (same Key fields) as the incoming records from PS_RP_PNDDST_TAO4. A sample
SQL would be to select from both tables the records that share the same KEY field values:

8 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

SELECT A.BUSINESS_UNIT, A.CUST_ID, A.GROUP_ID, A.GROUP_BU,

A.ITEM, A.ITEM_LINE, A.ITEM_SEQ_NUM, A.LEDGER_GROUP, A.LEDGER, A.DST_SEQ_NUM
FROM PS_RP_PNDDST_TAO4 A, PS_ITEM_DST B
WHERE A.PROCESS_INSTANCE = 'PI#'
AND A.BUSINESS_UNIT=B.BUSINESS_UNIT
AND A.CUST_ID=B.CUST_ID
AND A.ITEM=B.ITEM
AND A.ITEM_LINE=B.ITEM_LINE
AND A.ITEM_SEQ_NUM=B.ITEM_SEQ_NUM
AND A.LEDGER=B.LEDGER
AND A.LEDGER_GROUP=B.LEDGER_GROUP
AND A.DST_SEQ_NUM=B.DST_SEQ_NUM

Based on the results above, and approach could be to Update the PROCESS_INSTANCE = '0' on PS_RP_PNDDST_TAO4 or Delete the
(duplicate) rows from PS_RP_PNDDST_TAO4 table and then Restart the process.

Example 2: An abend at AR_PGG_SERV.MNT_SUSP.INSCFBAL with error message "ORA-00947: not enough values" is traced to the online
ChartField Inheritance page for SetID+Receivables (SHARE + AR_BI) not displaying the correct number of ChartFields (say displaying 4 instead
of the defined 5). This is maniftested itself in the INSERT and SELECT not having corresponding number of fields ("not enough values"). The
resolution would be to SQL Delete the 'SHARE'+'AR_BI' records in PS_CF_INHERIT_TBL (the table 'behind' the online page) and then re-
creating the records (all 5 of them, in this example) again using the online ChartField Inheritance page.

Example 3: Abends can sometimes occur due to incorrect data values in pre-defined fields, such as special characters in Customer Names or lower-
case letters in Item IDs. This is usually shown by such error messages as "ORA-01756: quoted string not properly terminated" (or similar). The
recovery is to update the data with the correct values that correspond to the Field Type (e.g UPPERCASE).

Option 6 [External Table, Program or Run Control Fixes]

Recovery Option 6 [External Table, Program or Run Control Fixes]

9 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

In some cases, the resolution could be by external actions on impacted objects (tables, programs, etc.) that does NOT involved SQL updates to
data. This would include re-building tables, re-compiling code, etc.

Example 1: A rare abend at AR_POSTING.ITEM_UYT.PNDRULES with error message "ORA-00600: internal error code" (which does not have
any prior references on the GCS database) was due to corrupt temporary table PS_RP_TERMS_TAO. Rebuilding the table resolved the issue,
without any further SQL data manipulation.

Example 2: Abends referring to COBOL calls/references could be resolved by ensuring the COBOL directory is correct and/or re-compiling
COBOL source objects.

Example 3: A one-time event (e.g. 1st run after an upgrade) caused an abend due to execution of a process step that should not be executed for
upgraded data missing information needed by new setup. A possible course of action would be to inactivate the Step, upgrade, then re-activate the
Step for new data.

Example 4: ARUPDATE abended due to Deadlock, hence the process can be Restarted (once the Deadlock is resolved). However, this would need
manipulation of the Run Control status of the Parent and Child processes. For a detailed example refer to KM Note 865335.1

Option 7 [Add/Fix Run Control]

Recovery Option 7 [Add/Fix Run Control]

Sometimes running ARUPDATE with a New Run Control record (different ID/Name to the abended process Run Control) will resolve the issue if
the existing Run Control records have been corrupted in any way. However, if the existing abended job Run Control ID needs to be used again it
would mean deleting its records from the control tables. This is the list of tbles where the Run Control may exist and needs to be deleted (verify
first then delete if/as needed):

DELETE FROM PS_AERUNCONTROL WHERE OPRID = '<>' AND RUN_CNTL_ID = '<>'

DELETE FROM PS_AETEMPTBLMGR WHERE OPRID = '<>' AND RUN_CNTL_ID = '<>'

DELETE FROM PS_AE_REQUEST WHERE OPRID = '<>' AND RUN_CNTL_ID = '<>'

DELETE FROM PS_AE_REQUEST_OPT WHERE OPRID = '<>' AND RUN_CNTL_ID = '<>'
DELETE FROM PS_PGG_SERVICE_AET WHERE OPRID = '<>' AND RUN_CNTL_ID = '<>'

10 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

DELETE FROM PS_RP_POSTING_AET WHERE OPRID = '<>' AND RUN_CNTL_ID = '<>'

(+)DELETE FROM PS_PRCSRUNCNTL WHERE OPRID = '<>' AND RUN_CNTL_ID = '<>'

(+)DELETE FROM PS_POSTING_REQUEST WHERE OPRID = '<>' AND RUN_CNTL_ID = '<>'
('+' exclude if you wish to retain Run Control, but reset AR Update)
(* where '<>' is your actual OPRID and actual Run Control ID)

Option 8 [Cleanup of Temporary Tables]

Recovery Option 8 [Cleanup of Temporary Tables]

[Optional] Clean up the abended Process Instance TAO tables. The complete list exists on PS_AETEMPTBLMGR keyed by
the PROCESS_INSTANCE and RUN_CNTL_ID. Delete from all _TAO tables EXCEPT for PS_PGG_SERVICE_TAO, which should NEVER be
deleted. Refer to above Step 3 '3. Verify the Abended PI# Run Control ID and TAO'

SELECT * FROM PS_AETEMPTBLMGR

Option 9 [Apply Known Resolution]

Recovery Option 9 [Apply Known Resolution]

The abend could be the result of a code error, hence no other fix would ultimately work in preventing it in the future. In this situation the
abended process will be recovered by the relevant Option above, any corresponding changes to setup or data if/as needed, the Resolution (whether
standalone fix or delivered in a Bundle) applied and then the process re-run.

The 1st step is to search the My Oracle Support (MOS) Knowledge Management (KM) database using the abended Process.Section.Step as the
keyword. Review the retrieved search results set for relevance to the on-site specific scenario.

11 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

Example: Search using keyword AR_PGG_SERV.TRN_CNTL.TRNCNTL will yield KM Doc:

EAR8.9: ARUPDATE Abended at Step AR_PGG_SERV.TRN_CNTL.TRNCNTL (Doc ID 758463.1)

that refers to Update ID 739417 FMS Financials 8.9 Bundle #29, which contains the code fix of ICE 1809891000.

The above Recovery Options are by no means comprehensive enough to cover all possible scenarios and resolutions. However, hopefully they will
provide a framework within which collaborative on-site and GCS investigation and teamwork can proceed to determine Cause and propose a
Solution to ARUPDATE abend issues.

Examples of Abends

Unique Constraint Violation

This document describes the troubleshooting steps to resolve a unique constraint error in the AR_POSTING section of Receivables Update. Those
familiar with the ARUPDATE multiprocess job, know that AR_POSTING occurs after the AR_PGG step, and is where transactions are posted and
permanent tables are updated. Therefore, if an ABEND occurs at this or any later step (e.g. ARUPDATE2), one must investigate before
performing a reset, cancel or SQL update of data. This document shows how to do such an investigation, using a real-life example scenario where
a maintenance group is created to write-off some old transactions on a customer instance and an ABEND occurs at
AR_POSTING.DISTRIB.INSERT.

Unique constraint violations typically arise from SQL inserts into a record that already contains data with the same key field values as the data
being inserted. Errors can have multiple causes even when the Application Engine section is the same, so it is important to investigate each
occurrence and determine root cause.

12 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

Troubleshooting 8 Steps

Step 1
1) First, determine the exact SQL statement that results in the unique constraint violation. Do this before cancelling or restarting a job. For
ARUPDATE, the message log for the application engine process usually contains the SQL statement. For example, we find in AR_POSTING1 the
log ends with:

Return: 805 - ORA-00001: unique constraint (SYSADM.PS_ITEM_DST) violated Failed SQL stmt:INSERT INTO PS_ITEM_DST
(BUSINESS_UNIT, CUST_ID, ITEM, ITEM_LINE, ITEM_SEQ_NUM, LEDGER_GROUP, LEDGER, DST_SEQ_NUM,
PRIMARY_LEDGER, BUSINESS_UNIT_GL, ACCOUNT, ALTACCT, DEPTID, OPERATING_UNIT, PRODUCT, FUND_CODE,
CLASS_FLD, PROGRAM_CODE, BUDGET_REF, AFFILIATE, AFFILIATE_INTRA1, AFFILIATE_INTRA2, CHARTFIELD1,
CHARTFIELD2, CHARTFIELD3, BUSINESS_UNIT_PC, PROJECT_ID, ACTIVITY_ID, RESOURCE_TYPE, RESOURCE_CATEGORY,
RESOURCE_SUB_CAT, ANALYSIS_TYPE, BUDGET_DT, STATISTICS_CODE, MONETARY_AMOUNT, STATISTIC_AMOUNT,
JRNL_LN_REF, LINE_DESCR, JOURNAL_ID, JOURNAL_DATE, GL_DISTRIB_STATUS, PROCESS_INSTANCE, APPL_JRNL_ID,
ACCOUNTING_DT, FOREIGN_CURRENCY, FOREIGN_AMOUNT, RT_TYPE, RATE_MULT, RATE_DIV, CURRENCY_CD,
SYSTEM_DEFINED, FISCAL_YEAR, ACCOUNTING_PERIOD, DOC_TYPE, DOC_SEQ_NBR, DOC_SEQ_DATE, DOC_SEQ_STATUS,
MOVEMENT_FLAG, USER1, USER2, USER3, USER4, USER5, JOURNAL_LINE, GL_ACCTG_DT, VAT_APPLICABILITY,
VAT_TXN_TYPE_CD, TAX_CD_VAT, VAT_DST_ACCT_TYPE, ENTRY_EVENT, DEBIT_CREDIT, IN_PROCESS_FLG,
EE_PROC_STATUS, OPEN_ITEM_KEY, FED_DISTRIB_STATUS) SELECT BUSINESS_UNIT, CUST_ID, ITEM, ITEM_LINE,
ITEM_SEQ_NUM, LEDGER_GROUP, LEDGER, DST_SEQ_NUM, PRIMARY_LEDGER, BUSINESS_UNIT_GL, ACCOUNT, ALTACCT,
DEPTID, OPERATING_UNIT, PRODUCT, FUND_CODE, CLASS_FLD, PROGRAM_CODE, BUDGET_REF, AFFILIATE,
AFFILIATE_INTRA1, AFFILIATE_INTRA2, CHARTFIELD1, CHARTFIELD2, CHARTFIELD3, BUSINESS_UNIT_PC, PROJECT_ID,
ACTIVITY_ID, RESOURCE_TYPE, RESOURCE_CATEGORY, RESOURCE_SUB_CAT, ANALYSIS_TYPE, BUDGET_DT,
STATISTICS_CODE, MONETARY_AMOUNT, STATISTIC_AMOUNT, JRNL_LN_REF, LINE_DESCR, ' ', NULL, GL_DISTRIB_STATUS,
PROCESS_INSTANCE, APPL_JRNL_ID, ACCOUNTING_DT, FOREIGN_CURRENCY, FOREIGN_AMOUNT, RT_TYPE, RATE_MULT,
RATE_DIV, CURRENCY_CD, SYSTEM_DEFINED, 0, 0, DOC_TYPE, DOC_SEQ_NBR, DOC_SEQ_DATE, DOC_SEQ_STATUS,
MOVEMENT_FLAG, USER1, USER2, USER3, USER4, USER5, 0, GL_ACCTG_DT, VAT_APPLICABILITY, VAT_TXN_TYPE_CD,
TAX_CD_VAT, VAT_DST_ACCT_TYPE, ENTRY_EVENT, ' ', ' ', EE_PROC_STATUS, OPEN_ITEM_KEY, FED_DISTRIB_STATUS FROM
PS_RP_PNDDST_TAO6 WHERE PROCESS_INSTANCE = 428261 Process 428261 ABENDED at Step AR_POSTING.DISTRIB.INSERT
(SQL) -- RC = 805 (108,524) Process %s ABENDED at Step %s.%s.%s (Action %s) -- RC = %s PSAESRV completed service request at 04.50.44
2009-07-01

In cases where the message log is incomplete, a trace file will provide the information, if it is enabled. If AE trace is not enabled, the AE trace
option can be turned on in the Process Scheduler config file psprcs.cfg, or via PeopleTools, using the override parameters for the Process Name

13 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

AR_POSTING1. It may be too late to turn on trace after an abend occurs. One could quite plausibly generate the same error in a copy of the
problem instance, using the same test scenario and identify the sql statement there.

Step 2
2. Confirm the duplicate data condition exists by running the same SQL statement in SQLPlus or a query tool such as SQL Developer. Depending
on the commit points prior to where ARUPDATE abends, the data may still exist in the temporary tables. This is why one should not cancel or
restart the job until the investigation is completed. In the above example, the data is still present in the PS_RP_PNDDST_TAO6, [the table referred
to in the FROM clause] and abends on insert into the PS_ITEM_DST record.

Step 3
3. Identify the key fields for the table being inserted to, e.g. PS_ITEM_DST. This can be done in App Designer, by viewing the key fields of the
record or by a query against the PSKEYDEFN record:

SELECT * FROM pskeydefn WHERE recname = 'ITEM_DST' AND indexid = '_'

This returns 8 rows and identifies the following key fields:

BUSINESS_UNIT
CUST_D
ITEM
ITEM_LINE
ITEM_SEQ_NUM
LEDGER_GROUP
LEDGER
DST_SEQ_NUM

Step 4
4. Identify the rows from the PS_RP_PNDDST_TA06 table that have duplicate rows in the table being inserted to, by constructing a SQL select
statement using the same key fields:

SELECT *
FROM ps_rp_pnddst_tao6 t
WHERE EXISTS (SELECT 'X'
FROM ps_item_dst d
WHERE t.business_unit = d.business_unit
AND t.cust_id = d.cust_id

14 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

AND t.item = d.item

AND t.item_line = d.item_line
AND t.item_seq_num = d.item_seq_num
AND t.ledger_group = d.ledger_group
AND t.dst_seq_num = d.dst_seq_num)

Step 5
5. If rows are returned, one needs to examine the data returned from both tables. In this example, one item was returned by the above SQL. We
know that the system assigns the same ITEM identifier for two or more related items, such as a payment applied to an item. We also know that an
ITEM_SEQ_NUM is uniquely assigned to each related item. So we verify whether the ITEM_SEQ_NUM is already in use via this query, and we
confirm that it is indeed a duplicate.

SELECT *
FROM ps_rp_pnddst_tao6 p
WHERE business_unit = 'US001'
AND item_seq_num < (SELECT Max(item_seq_num)
FROM ps_item_dst d
WHERE d.business_unit = p.business_unit
AND d.cust_id = p.cust_id
AND d.item = p.item
AND d.item_line = p.item_line)

If NO rows are returned from Step 4, consider the possibility that the FROM table (e.g. ps_rp_pnddst_tao6) contains duplicate data within itself,
such as duplicate data with different process instance values. This could happen because the two tables have different key fields. To identify these
rows, use a query to group by the key fields of the table being inserted to, and run against the source or from table, for example:

SELECT count(*),
max(business_unit),max(cust_id),max(item),max(item_line),max(item_seq_num),max(ledger_group),max(dst_seq_num)
FROM ps_rp_pnddst_tao6
GROUP BY business_unit,cust_id,item,item_line,item_seq_num,ledger_group,dst_seq_num
HAVING count(*) > 1

Step 6
6. Important: Investigate the reason for the duplicate row. In this case, we determined the reason is due to an item having orphan distributions. i.e.

15 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

payment distributions that exist in ITEM_DST without corresponding rows in ITEM and ITEM_ACTIVITY. Since ITEM_SEQ_NUM is
sequentially assigned (Formula: MAX(ITEM_SEQ_NUM) + 1), based on the rows that exist for the item, it assigned a number that already exists
in the ITEM_DST record.

Step 7
7. A check with users indicates that this item is still outstanding and no payment has actually been made. It makes sense then to proceed with the
write-off as the item is still outstanding. Therefore, it was decided to delete orphan distributions from the PS_ITEM_DST record, and allow the
ARUPDATE process to complete. The investigation indicated exactly which distributions to delete, and these values were passed to the delete
statement, in this fashion:

DELETE FROM ps_item_dst d

WHERE d.item = '&item'
AND d.item_seq_num = 2
AND d.business_unit = '&BU'
AND d.appl_jrnl_id = 'AR_PAYMENT'
AND d.gl_distrib_status = 'I'

Step 8
8. Since this involved a manual SQL update, the procedure was performed first on a copy of the production database and then the results were
verified by the functional users. Once confirmed, the procedure was carried out in the Production database.

NOTE: The above steps illustrate one specific scenario and the SQL statements are not meant to be applied generically, even to similar issues with
the exact same error messages. This is due to the variations in each customer's environment, as well as the nature of the underlying data. However
the general approach and troubleshooting steps can most certainly be adapted for most unique constraint violation cases.

For more information about the different steps comprising ARUPDATE, see the PeopleSoft Receivables PeopleBook entry on Receivables Update
Multiprocess Job and other KM Documents linked.

QUEUED Status After Abend

16 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

Troubleshooting Steps

1. Hit the +(plus sign) on the left hand side of AR_PGG, and it will take you to the AR_PGG1 process which abended due to the deadlock (If the
abend was encountered in AR_POST1, follow appropriate steps by looking at the above ARUPDATE structure defined)

2. Go into the details for this process, and at this point in time (and if the abend is in its original state i.e. no backend SQL updates etc have been
tried so far), the HOLD request and CANCEL request radio-buttons should be available for selection. All other buttons should be grayed out.
Select the HOLD radio-button, and hit OK and navigate all the way to the process monitor.

3. Now notice that at the highest level in the ARUPDATE process, instead of saying QUEUED, it will change to NO SUCCESS.

4. Navigate into the highest level where it says NO SUCCESS, and go into the details section. Here restart request radio-button should be available
at this point in time.

5. Press the restart request radio-button from this level, and hit OK.

6. Now notice that instead of saying QUEUED, ARUPDATE process will change to Processing, and it will pass through all the child processes.

Notes:

1. The above solution has worked for many customers who had run into production critical issues, and by following the above steps they were able
to get ARUPDATE restarted.

2. As all the above steps documented above are done online, it is a reasonably safe option to try if you run into a similar production critical
situation where time is of essence. Please add a comment if this document assisted in resolving a similar issue.

3. If the above steps do not help resolve the Abended ARUPDATE jobs staying in QUEUED status, then please log a new SR with the PeopleSoft
Enterprise Support team for further troubleshooting.

You may also benefit from Docs 1068495.1 and 864741.1.

17 of 18 6/30/2018, 12:45 PM
Note: 1457104.1 https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=1966505209917&_afrW...

References

ARUPDATE Abends References:

Document 864741.1 EAR 8.8+: Troubleshooting ARUPDATE Abends

Document 855221.1 EAR 8.9+: Troubleshooting ARUPDATE AR_POSTING Step Unique Constraint Violation Errors
Document 865335.1 EAR 8.8+: Troubleshooting ARUPDATE Abended Job Staying in QUEUED Status Even After Restart
Document 846177.1 EAR 8.8+: Troubleshooting ARUPDATE Abended or Corrupted Run Control IDs
Document 1068495.1 EAR 8.8+: Recovery from an Unsuccessful Receivables Process FAQ [ ARUPDATE , etc.]
Document 1302171.1 EAR 8.8+: ARUPDATE Abends at Step AR_PGG_SERV.BAT_DST.NEXT_DST
Document 1305124.1 EAR 8.8+: ARUPDATE Abends at Step IU_PROCESSOR.U400.U400-010
Document 1305634.1 EAR 8.8+: ARUPDATE Abends at Step AR_PGG_SERV.MNT_SUSP.INSCFBAL
Document 1311485.1 EAR 8.8+: ARUPDATE Abends at Step AR_PGG_SERV.TRN_CNTL.TRNCNTL
Document 762467.1 EAR 8.8+: ARUPDATE Abends at Step AR_PGG_SERV.TRNCNTV

18 of 18 6/30/2018, 12:45 PM