Marketing Project Sunrise

Supporting Marketing Decision Analytics

PEC Resolution Guide for OPS

Data Scope: IBM Worldwide Sales and

And Marketing Data

Version: 0.2
Date: 15th September 2016

Page 1 of 27

Contents
Background:............................................................................................................ 4
Document Purpose:................................................................................................. 4
Overview................................................................................................................. 4
Monitoring Activities................................................................................................ 5
ACQUIRE METADATA.................................................................................................... 5
DATASTAGE DIRECTOR........................................................................................... 5
PEC QUEUE SCREEN............................................................................................... 5
Step Queue.......................................................................................................... 5
Task Queue........................................................................................................... 5
Process Queue...................................................................................................... 5
FIFO View............................................................................................................. 5
Failure View.......................................................................................................... 5
DB2TOP................................................................................................................... 5
NMON...................................................................................................................... 5
Standard Errors(SE)................................................................................................. 6
SE 1: 911 error..................................................................................................... 7
SE 2a: 438/NXXXX error....................................................................................... 8
SE 2b: 438/Z1002 error........................................................................................ 9
SE 2c: 438/PXXXX error...................................................................................... 10
SE 2d: 438/Z1000 error...................................................................................... 11
SE 2e: 438/Z1000 error...................................................................................... 12
SE 3: 803 Error................................................................................................... 13
SE 4:206 Error.................................................................................................... 14
SE5: Validation Error........................................................................................... 15
Process Specific..................................................................................................... 16
Hourly.................................................................................................................... 16
OTM.................................................................................................................... 16
TSIH.................................................................................................................... 16
X1P..................................................................................................................... 16
UCP.................................................................................................................... 16
Page 2 of 27

D AILY...................................................................................................................... 16
X1I...................................................................................................................... 16
SCN.................................................................................................................... 16
IDM..................................................................................................................... 16
TSID.................................................................................................................... 16
Weekly:................................................................................................................. 16
SRWPB................................................................................................................ 16
SOB.................................................................................................................... 16
MKARSP.............................................................................................................. 16
MKARCH............................................................................................................. 16
MO2.................................................................................................................... 16
LDR.................................................................................................................... 16
ADRWCP............................................................................................................. 16
DMWCP.............................................................................................................. 16
DMWB................................................................................................................ 16
MTG.................................................................................................................... 17
Database Shutdown:............................................................................................. 18
SRWHSE................................................................................................................ 18
SRDM.................................................................................................................... 18
ADR....................................................................................................................... 18
BLUEGRASS PEC:..................................................................................................... 18
APPENDIX.............................................................................................................. 19

Page 3 of 27

Background:
As the number of data sources integrated through Sunrise increases, new data movement processes
are implemented to handle the data such as Datastage jobs, DB2 procedures etc. to integrate and
transform data to a BI optimized format. Many of these processes are scheduled and/or managed
[semi-]manually. This increases not only the workload on the operations and architect teams, but also
makes the processing more prone to errors.
The data components within Sunrise are currently are grouped into the following semantic layers
1. Data Source Layer
a. 27 Sources
b. Average ~20 data areas per source
2. Sunrise Data Warehouse: The Sunrise Data Warehouse stores data within the following database
areas with varying retention periods as suggested by the name of the database area.
a. Staging database area
b. Load database area
c. Active database area
d. History database area
e. Delete database area
3. Sunrise Data Mart
a. Active database area
b. Active copy database area
4. Analyst Data Repository
Data is moved between these components using any of the following data flow processes
1. Acquire: Primarily done using datastage
2. Warehouse Data Management(WDM) Procedures
3. Plan M
4. Plan R
5. Build Processes

Document Purpose:
1.
2.
3.
4.
5.

Operations Monitoring Activities

PEC Errors
PEC resolution instructions for global errors.
Process and source specific PEC scenarios.
Database Event Related scenarios

Overview

Page 4 of 27

Monitoring Activities
One of the primary responsibilities of operations to diligently monitor the Sunrise
jobs, processes and servers. The primary monitoring activities expected from
operations times are listed but not limited to below.

ACQUIRE METADATA
DATASTAGE DIRECTOR
PEC QUEUE SCREEN

PEC is fast becoming the primary

Step Queue
Task Queue
Process Queue
Views

FIFO View(
long running queries
Failure View
Monitor, Communicate, Resolve
DB2TOP
Db2top as a text graphical interface helps the operation team monitor a variety of
db operational info. The following details on db2top need to be monitored for
Sunrise operational efficiency
Needed
1. Locks
a. Application status: Be alert for locked status on any of the applications
b. OS user
Good to monitor (order of priority)
2. Time InStatus
3. Check rows read and rows written
4. Everything else
Page 5 of 27

NMON
System utilization
With option C
Topas P
Ps ef | grep <application id>

Page 6 of 27

Standard Errors(SE)
This section contains issues that can arise across sources and processes. When one
of these is identified, irrespective of the process or source, PEC Operations can refer
to these instructions outlined below to provide timely and accurate resolution.
PEC Screens

Page 7 of 27

SE 1: 911 error
Description: This error occurs due to a deadlock occurring while PEC is trying to manipulate
contents of a database object.
Ex: ..

To address this error, the following actions must be performed in the order specified.

Page 8 of 27

SE 2a: 438/NXXXX error

Description: When load utility invoked through PEC for database loads fail because
of the database returning a negative return code we get this error. The XXXX within
the SQLSTATE shown the PEC failure message captures the negative sql code
returned from the database.

Report:
Please identify sql within the cursor configuration and if you identify any errors
please send the details of the error within the communication to the SMEs.

Page 9 of 27

SE 2b: 438/Z1002 error

Description: When the database load successfully executes and as a part of the load
some of the records on the target database are deleted the Z1002 sqlstate shows
up.

When this error occurs

1. Please navigate to the PEC step metadata and using the
reference of the step id navigate to the plugin
configuration of the step metadata.
2. Please review the target table and identify the primary
key of the target table
3. select statement in the create cursor and
* PXXXX: when cursor load returns a positive SQLCODE e.g. if SQLCODE 499 is
returned then this SQLSTATE will be P0499 (SQLSTATE codes are always 5 char long)
* NXXXX: when cursor load returns a negative SQLCODE e.g. if SQLCODE -104 is returned
then this SQLSTATE will be N0104 (SQLSTATE codes are always 5 char long)
* Z1000: execution was successful but there some records were skipped
* Z1001: execution was successful but there some records were rejected
* Z1002: execution was successful but there some records were deleted

Page 10 of 27

SE 2c: 438/PXXXX error

Description: When the database load successfully executes and as a part of the load
some of the records on the target database are deleted the Z1002 sqlstate shows
up.

Page 11 of 27

SE 2d: 438/Z1000 error

Description : When a successful load execution happens inspite of skipped rows, the
event of skipped rows is identified by the Z1000 sql state

Page 12 of 27

SE 2e: 438/Z1001 error

Description : When a successful load execution happens inspite of skipped rows, the
event of skipped rows is identified by the Z1000 sql state

Page 13 of 27

SE 3: 803 Error

Page 14 of 27

SE 4:206 Error

Page 15 of 27

SE5: Validation Error

Page 16 of 27

SE6: 438/99001 Error

Page 17 of 27

Process Specific
Hourly
OTM
TSIH
X1P
UCP
DAILY
X1I
SCN
IDM
TSID
Weekly:
SRWPB
SOB
MKARSP
MKARCH
MO2

Page 18 of 27

LDR
ADRWCP
ADR EIW Copy

Page 19 of 27

DMWCP
DMWB
MTG
ESTMETR

Add the clear stepruninfo tables

Page 20 of 27

Database Shutdown:
SRWHSE
SRDM
ADR
BLUEGRASS PEC:

Page 21 of 27

Wheres the Build

Thursday 9AM

Page 22 of 27

APPENDIX
PEC Screen Detail
Explain each field and usage within a PEC screen

Page 23 of 27

Purpose of this documentation:

In order to monitoring the DB2 activities on the database server (Sunrise data mart
SRDM) on development server (1s80-8), DB2 WLM is set up. With configuration
classes and work load on DB2 server, the system usage and queries are monitored
by development and architect team. Initially DB2 WLM will be using to determine
performance issue on Cognos reports through the BACCDEV ID.
In case future monitoring and work load management is needed on TEST/PROD
environments, this documentation will be used as guide/reference to set up on
those systems.

Pre requirements:

DB2 System have correct license in order to use service classes and work load
other than pre defined default one. In case you do not have correct licenses, you
will get following error:
CREATE SERVICE CLASS highlvl DISABLE
DB21034E The command was processed as an SQL statement because it was not a
valid Command Line Processor command. During SQL processing it returned:
SQL8029N A valid license key was not found for the requested function. The
current license key for this product does not allow the requested
functionality. Purchase the license key for this function from your IBM
representative or authorized dealer and update your license using the License
Center or the db2licm command line utility.
Following licenses are installed/configured on ls80-8 DB2 server:
ls80-8:/home/inst7$ db2licm -l
Product name:
"DB2 Enterprise Server Edition"
License type:
"CPU Option"
Expiry date:
"Permanent"
Product identifier:
"db2ese"
Version information:
"9.7"
Enforcement policy:
"Soft Stop"
Features:
DB2 Performance Optimization ESE: "Licensed"
DB2 Storage Optimization:
"Licensed"
DB2 Advanced Access Control:
"Licensed"
DB2 Geodetic Data Management:
"Licensed"
IBM Homogeneous Replication ESE: "Licensed"

Page 24 of 27

1. Using the filter options on the PEC queue screen set up a failure alert view at
the start of the shift and keep the view up at all times to monitor failures. The
same view can be used to setup failure alerts at task and process level and
you can toggle between the 3 different failure levels. Prioritize step failure
view since the failures show up first at the step level.

Note: Operations team can refer the operations guide via the BOX for
creation of the failure screen
2. If there is a step error (in WDMs involving a stored procedure), please clear
the ops control tables corresponding to the data area and process. For
example, if there is a failure in UCP MCT_Campaign data area please run the
below SQL to clear the ops control tables specific to that task/data area.
DELETE FROM ZCNTRL.SROPSACTIVEDATAAREA
WHERE SROADA_EXECPROCACR='UCP'
AND SROADA_EXECPROCDA='MCT_CAMPAIGN'

3. After clearing the ops control tables successfully, please capture the PEC
error message and include the error message in the operations notification
that is circulated everyday within the Sunrise team. For example the screen
below captures the 911 error

Page 25 of 27

4. After capturing the error screen please RESUME the step if any of the
following errors appear on the step queue. If any other error occurs
a. 911 error

b. ERRORCODE=-4222

Page 26 of 27

Note: Operations team can refer the operations guide available via the BOX
for resuming a step. I am attaching the step resume instructions for quick
reference.

Step Resume - Ops

Instructions.xls

Page 27 of 27