Академический Документы
Профессиональный Документы
Культура Документы
Project: FSR
Document: KT – Understanding document
Subject: Taxonomy
Taxonomy - Understanding
FSR
Version History:
Version Date By Changes
Contributors:
Name Role Location Remarks
Santhosh Eturi
Akshatha Hegde
BalajiGangadhar
Uthranarayani
Taxonomy - Understanding
FSR
CONTENTS
Taxonomy - Understanding
FSR
1 Overview:
FSR Framework uses database driven change control mechanism, wherein business users
enter the request on set of predefined tables and raise a change request via share point,
which will be communicated to ETL Team. Upon receiving the communication and approval
of change request, CR coordinator analyse the requirement and perform synch of the
requirement from work area to live area, then notify the ETL development team about the
new request. This document outlines the understanding of KT team.
2 Prerequisite:
• Share point access
• Access to the GII_OWNER@FEED_D.WORLD schema
3 Data objects:
Data objects pertain to Taxonomy can be classified as Master and Transaction
Master
• T_TAX_ENGINE_DASHBOARD
• T_TAX_ERROR_CATALOG
• T_TAX_STD_NAMING_ABBEV
Transaction
• T_TAX_TAX
• V_T_TAX_TAX_ACT
• T_TAX_WORK
• T_TAX_WORK_II
3.1.1 T_TAX_TAX
This table contains all the versions of the rules. The removed flag signifies that the rule has
been removed and the CHANGED_CONTROL field signifies the rule when the rule has been
added.The RULE_ENGINE_REG_REF field is the one which describes the component in
which the rule has to be applied.
The ETLGRKEY tells us the order in which the rules have to be applied for deriving the
OBJECT_BUSINESS_NAME field. There could be more then one ETLGERKEY which
denotes it belong to same group.
The (field processing order,object business name and rule_engine_reg_ref) fields gives the
order in which the output field should be derived for a particular component.
Taxonomy - Understanding
FSR
The ERROR_EXCEPTION field is the error code and the reference(error message) would be
provided in the T_TAX_ERROR_CATALOG table.
3.1.2 V_T_TAX_TAX_ACT
This view contains the latest version of the rule. To derive the mapping logic applied on a
field for a particular component; we need to filter the data based on the rule_engine_reg_ref
and object_business_name. Then the field should be sorted in the ascending order of the
field processing order.
3.1.3 T_TAX_ENGINE_DASHBOARD
This table contains all the engines which are referred in the rule_engine_reg_ref field of
T_TAX_TAX table. This table also specifies whether a particular engine is a derivation engine
or validation engine.
3.1.4 T_TAX_ERROR_CATALOG
This table contains all the error messages for the error_exception field in the T_TAX_TAX
table. This table contains also the details of error level (Document, Field, Logic and
Message). In addition, the region wise classification of the error is stored in this table.
3.1.5 T_TAX_WORK
This table stores all the rules which violate the integrity constraints, standard naming
conventions which can be referred from T_TAX_STD_NAMING_ABBEV and
T_TAX_ERROR_CATALOG table.
The ERROR_COMMENT field gives the exact reason why the particular rule has been
included in the T_TAX_WORK table like ‘INPUT/OUTPUT BUSINESS NAME not in standard
naming convention’ or the ‘error code mentioned by the business is not in the
T_TAX_ERROR_CATALOG table’.
Taxonomy - Understanding
FSR
The standard naming conventions table would contain only the names to be followed for
input and output business names and the error codes would be validated against the
ERROR CATALOG table.
3.1.6 T_TAX_WORK_II
This table contains all the rules which are present in the V_T_TAX_TAX_ACT (Minus) The
number of rules with removed flag of Y (Minus) number of records in the T_TAX_WORK.
3.1.7 T_TAX_STD_NAMING_ABBEV
This table contains the naming conventions to be followed for input and output business
names,
3.1.8 T_TAX_NULL_CHK_STR_COMP_VAL
3.1.9 T_TAX_CRS
3.1.10 T_TAX_TASK_WORK
Taxonomy - Understanding
FSR
3.1.11 V_T_TAX_WORK
Not sure whether this table is being used or not.
Taxonomy - Understanding
FSR
Q and A
1) If the initial rule has been changed, then will a new record will be inserted into the table? If
yes, then what is purpose of the field CHANGE_TEXT?
2) Will we have the same rule number if the rule has to be applied to 2 different components?
T_tax_tax may contain the same rulenumber,but the view contains only ruleno per
component. We can get query from t_tax_tax by grouping by ruleno and get the max of id.
5) What is ETLGRKEY?
ETL GROUP KEY
6) Is there a chance that OUTPUT_PHYSICAL_NAME used will be different for the same
rule (Eg 2770)
7) What does the ERROR_COMMENT column specify? Does this ERROR_COMMENT have
any relationship with ERROR_ID’s.
No this is independent field; it gives the reason of why field is in this table and not in the
model.
Taxonomy - Understanding
FSR
GFIS Tables
(ETL DB) Reference Data Engine
Pre-processing Engine
Validation Engine
Derivation Engine
T_GW_STP_
STARTUP
(GII DB)
Calculation Engine
ETL1 FSR Framework
Taxonomy - Understanding
FSR
The FSR framework process begins once the run for the individual feed is completed
and the data is available in the corresponding GFIS table in the ETL environment. The value
‘Y’ in DATA_PRESENCE_FLAG column in the T_GW_STP_STARTUP table indicates that
the data is available in the corresponding GFIS table and it is ready for further processing.
This source data from GFIS table is taken and it is directly loaded in the
T_GW_TMP_PROCESSED_DATA table in the GII environment with a STAGE_NO value ‘0’.
Every row that is populated in this T_GW_TMP_PROCESSED_DATA table will acquire a
unique RECORD_ID.
The records with STAGE_NO ‘0’ are taken for a particular PRC_PROCESS_ID and
FEED_ID and are allowed to traverse through the various engines present in the framework.
There are 9 (from the diagram above) processing engines in which each engine comprises of
many components.
The input record is propelled through each component and the respective output
record is also stored in the same T_GW_STP_STARTUP table, with the STAGE_NO
increased by 1 for every stage. In this process the PRC_PROCESS_ID, RECORD_ID and
FEED_ID remains the same for a particular record.
On completion of the first 8 (from the diagram above) processing engines the ‘Target
Load Entry’ engine, which is the final engine in FSR starts processing. This Target Load
Entry engine selects the records with the maximum STAGE_NO for a particular
PRC_PROCESS_ID and FEED_ID and makes it available for generating the target file.
The target file is then generated for the respective target systems depending on the
requisite file format.
Run always workflow runs continuously. The first session “s_WaitSession” in the workflow
internally polls startup table for the data_presence flag and triggers the processing if the flag
is set to ‘Y’ for Template, Interface and System defined in Parameter file else if the flag is set
to ‘N’ it waits for a period of time as defined in parameter file as wait_time and polls again
after the wait is over till it finds the data_presence flag is ‘Y’.
Taxonomy - Understanding
FSR
Once the data_presence flag is ‘Y’ in T_GW_STP_STARTUP table the next session
“s_StartProcess” starts and it first creates process id for the given System and Interface in
the table T_GW_PRC_PROCESS. It also creates a partition in temp table for the particular
process id.
The next session Creates Execution order list and Parameter file for all the subsequent child
processes.
“s_StartTasks” is a command task which invokes all child processes (Run once) through a
script which does the actual data loading.
Example-
$$SystemName=EP
$$InterfaceName=ACC
$$TemplateName=APAC_1
$$SleepTime=3
$$TimeOut=0
Internally the session invokes the procedure ‘P_WAIT_ON_FILE’ which takes System name,
Interface name and Template Name as an input and checks for the
DATA_PRESENCE_FLAG for the combination of input given to it till it finds flag value as ‘Y’
to trigger next session in the workflow.
Session ‘s_StartProcess’ makes an entry to the ‘T_GW_PRC_PROCESS ‘ table with the new
process_id to be used during entire processing of the current feed. The process_id is
derived using max (previous id+1). A partition in the temp table based on the process_id is
created. This session uses ‘F_GET_PROCESS_ID’ and ‘F_CREATE_TMP_PARTITION ‘ to
achieve the above mentioned task.
The next session ‘s_GenerateFiles’ generates dynamic Parameter File that will be used in
next shell script. File names must correspond to the names defined in Parameter file. This
task also generates an execution order list for all the tasks which will be executed for the
current system.
Taxonomy - Understanding
FSR
The last session ‘s_CloseProcessingSeq ‘ in the workflow is used for all post loading task i.e
– Prepares Source and Suspended records for erroneous records.
Copies content of the partition (in T_GW_TMP_PROCESSED_DATA) to a temporary table
and drop partition. Once the data is copied from this temporary table to the target table this
temporary table can be dropped. This calls a function ‘F_STORE_TMP_PARTITION_DATA’
for this purpose.
In case of failures in “wf_Strar WF_EP” or inside command task ‘s_StartTasks’, Tivoli worklet
will be executed and Tivoli alert will be raised. After successful completion of last session,
“wf_Start_WF_EP” will be restarted by scheduler.
4.1.2 Metadata
The metadata entry should be made in the following tables, before the ETL processing
starts. Once the entry is made in the respective tables the process kicks off manually or
automatically based on the process type.
The T_GW_STP_TEMPLATE table contains details about the location to where its
corresponding feeds are associated. An initial level entry is made in this table once feeds for
a particular location start arriving. Each and every template has a unique TEMPLATE_ID
which is generated internally by taking the maximum available TEMPLATE_ID + 1.
Taxonomy - Understanding
FSR
Sample Data:
The T_GW_STP_SYSTEM table contains the details of all the feeds corresponding to a
particular location. Each and every feed has a unique SYSTEM_ID which is generated
internally by taking the maximum available SYSTEM_ID + 1. A new entry should be made on
arrival of a new feed for any particular location.
Sample Data:
The T_GW_STP_INTERFACE table contains the details of all the different types of interface
details. Only one entry is made for a particular interface type. If any new interface is
encountered in any of the new feeds, then a fresh entry is made in this table, by taking the
maximum INTERFACE_ID + 1.
Sample Data:
The T_GW_STP_COMPONENT table contains the details about all the components that are
involved in the FSR architecture(i.e. business engines). The mapping level detail for every
component is stored in this table. Each and every component has a unique
COMPONENT_ID which is generated internally by taking the maximum available
COMPONENT_ID + 1. Hence a new entry should be made if any new component is added to
the existing architectural design.
Sample Data:
Taxonomy - Understanding
FSR
The T_GW_STP_TASK table contains the details of all the available workflows. If there are
multiple copies of the same workflow that is sourced on the same component, there should
be multiple entries in this task table. Every workflow is treated as an individual task. Each
and every task has a unique TASK_ID which is generated internally by taking the maximum
available TASK_ID + 1. Hence a new entry should be made if any new workflow is created.
Sample Data:
Sample Data:
Once the details are entered in the above tables, we move on to fill in the details for
T_GW_STP_PROCESS table. Here the SYSTEM, INTERFACE, TEMPLATE and
COMPONENT ID’s are entered, through which the engines are related for a particular
process. These 4 keys together form a key element, for which a unique PROCESS_ID is
given, by taking the maximum PROCESS_ID + 1.
Sample Data:
Sample Data:
Taxonomy - Understanding
FSR
Once the values are entered in the above mentioned tables, the V_GW_SYSTEM_SETUP
view gets updated with corresponding details. Unless and until data is entered in
T_GW_STP_TASK_INSTANCE table, the values will not be reflected in this view.
Sample Data:
Below data is generated from the details from the above tables.
EXECUTION_ORDE
R FOLDER_NAME TASK_NAME TASK_ID
1 KT wf_KT 76
We have set of tables T_GW_ERR_* for error handling. The error encountered is logged
separately to a system called PORTAL. Errors had been categorized as below
1 – Message
2 – Soft Error
3 – Hard Error
Based on the error category there are different ways of handling it .
Hard errors will stop processing and soft errors and messages will be posted to SAP.
Taxonomy - Understanding
FSR
no Hard error occurs the file will be transferred to SAP and the errors, and SAC will be sent
to PORTAL. SAC contains the data , which are defaulted in Automated feeds to hit the SAC-
Account in SAP. The inserted default values will than corrected manually in PORTAL-SAC
and sent again to SAP.
2) Etl2Portal_<2147>.err – This files contain only the Errors, not the data records. We will not
send any data files back to portal in PORTAL-MAN.
Once the file had been uploaded successfully to the portal, the PORTAL team will manually
correct all the record and the corrected file is available at the portal a JMS control message
is put at the message channel which will be used by the subscriber who subscribe for that
message. JMS Queues are used for this purpose. Two dedicated queues were created for
ETLPortal and PortalETL communications.
5.1.1 wf_PORTAL_CM_SUBSCRIBER
[GW_APAC_1.WF:wf_PORTAL_CM_SUBSCRIBER.ST:s_PORTAL_CM_SUBSCRIBER]
$$TemplateName=APAC_1
[GW_APAC_1.WF:wf_PORTAL_CM_SUBSCRIBER.WT:DBF_WL_TIV_WFLOWERROR.ST:
DBF_SESS_TIV_WFLOWERROR]
$$WFLOW='WF_PORTAL_CM_SUBSCRIBER'
To decide on to which workflow to run for the next we have following Request_type in the
T_GW_PORTAL_CONTROL_MESSAGE table:
1) FLA
2) MAN
3) NFE
5.1.2 wf_Start_PORTAL_MAN_1
Taxonomy - Understanding
FSR
Let us consider a scenario where the Request_type is MAN so one of the workflow
wf_Start_PORTAL_MAN_<1-5> will be executed based on the availability.
s_CreatePortalParamFile
This session will create parameter file for the rest of processing.
s_StartPortalInstance
The above command task is used to start the workflow required for PORTAL data load to
the SAP/BW. The Script is at $PM_HOME/Scripts/gw_core/gw_start_single_task.sh.
As parameter of this script has to be provided Workflow parameter name.
s_ClosePortalProcessing
This session checks the record count for the records processed based on the process id and
it also Check do we have erroneous records for the particular ProcessId. Any record in the
table T_GW_ERR_RECORD_HDR is counted as error.
This also updates the status flag in the status table with ‘C’ if the process completed
successfully or else the flag value will be ‘E’ in case of any error occurred .
5.1.3 wf_Start_PORTAL_FLA_1
Taxonomy - Understanding
FSR
Let us consider a scenario where the Request_type is FLA so one of the workflow
wf_Start_PORTAL_FLA_1 will be executed based on the availability.
s_CreatePortalParamFile
This session will create parameter file for the rest of processing.
s_StartPortalInstance
The above command task is used to start the workflow required for PORTAL data load to
the SAP/BW. The Script is at $PM_HOME/Scripts/gw_core/gw_start_single_task.sh.
As parameter of this script has to be provided Workflow parameter name.
s_ClosePortalProcessing
This session checks the record count for the records processed based on the process id and
it also Check do we have erroneous records for the particular ProcessId. Any record in the
table T_GW_ERR_RECORD_HDR is counted as error.
This also updates the status flag in the status table with ‘C’ if the process completed
successfully .
5.1.4 wf_Start_PORTAL_NFI_1
Let us consider a scenario where the Request_type is NFI so one of the workflow
wf_Start_PORTAL_NFI_1 will be executed based on the availability.
s_CreatePortalParamFile
This session will create parameter file for the rest of processing.
Taxonomy - Understanding
FSR
s_StartPortalInstance
The above command task is used to start the workflow required for PORTAL data load to
the SAP/BW. The Script is at $PM_HOME/Scripts/gw_core/gw_start_single_task.sh.
As parameter of this script has to be provided Workflow parameter name.
s_ClosePortalProcessing
This session checks the record count for the records processed based on the process id and
it also Check do we have erroneous records for the particular ProcessId. Any record in the
table T_GW_ERR_RECORD_HDR is counted as error.
This also updates the status flag in the status table with ‘C’ if the process completed
successfully or else the flag value will be ‘E’ in case of any error occurred
DBF_WL_TIV_WFLOWERROR
This workflow is used to handle any error which had occurred during current load session
followed by a mail alert sent to respective system.
5.1.5 s_PORTAL_PUBLISHER_SAC
The s_PORTAL_PUBLISHER_SAC publishes only the JMS Messages to Portal which refers
to the SAC-Error file
S_PORTAL_PUBLISHER publishes the JMS Message after a Protal run back to portal. In
case of errors, in this message will referenced to the according error-file
Taxonomy - Understanding
FSR
Example:-
The next session ‘s_GenerateFiles’ generates dynamic Parameter File that will be used in
next shell script. File names must correspond to the names defined in Parameter file.
This task also generates an execution order list for all the tasks which will be executed for
the current system.
Taxonomy - Understanding
FSR
In order to process a task it updates the status of the task in the table
T_GW_STP_TASK_RUN for the TASK_STATUS_FLAG to ‘Y’ so that until the current
execution is over no other process can run the same task. To achieve this script internally
calls the procedure P_SET_TASK_RUN_IND which in turn checks for the current status and
update the flag accordingly. To run a particular workflow there needs to be at least one task
corresponding to that workflow with the status ‘N’. Once there is a free task found the script
starts the workflow using pmcmd command. The status of the workflow will be monitored till
the end of its processing.
After the completion of the task the run indicator in the T_GW_STP_TASK_RUN table is
reset to ‘N’ using procedure ‘P_UNSET_TASK_RUN_IND’. Shell script also checks for
workflow execution status(pmcmd getworkflowdetails) and in case if execution status is not
“Succeeded” shell script will return non-zero error code.
The last session ‘s_CloseProcessingSeq ‘ in the workflow is used for all post loading task
i.e – Prepares Source and Suspended records for erroneous records.
Copies content of the partition (in T_GW_TMP_PROCESSED_DATA) to a temporary table
and drop partition. Once the data is copied from this temporary table to the target table this
temporary table can be dropped. This calls a function ‘F_STORE_TMP_PARTITION_DATA’
for this purpose.
The below listed ETL2 components are discussed. The purpose of the mappings is to
download the RDM data from the SAP Database and load the same to GFIS tables in ETL
environment. These updated GFIS tables are used as corrected reference data in ETL2
processing. This can be done in two procedures.
Taxonomy - Understanding
FSR
This mapping is used to load the data from the feed specific feed_id’s (240,241,242,243 and
239) to the constant feed_id (10021) into GFIS REFERENCE TABLES. So this mapping
merges several feed ids into one feed id in same GFIS tables.
m_LOAD_RDM_DELTA_RFC
In this mapping, we call an RFC function which creates the RFC delta files in the SAP server.
The generated delta files will be downloaded to ETL environment for further processing.
Delta generation RFC works in two modes – taking changes (deltas) generated since
previous call (in case of two consecutive RFC calls first call will return some data, but second
call will return no results) or taking all changes for given time period. Parameters used for
RFC call can be set in T_GW_STP_TASK_PARAMETERS table (which tables should be
processed and which mode is used – take only latest changes or changes for given period of
time).
m_LOAD_RDM_DELTA_UPD
The above delta files are used as source and loaded the data into GFIS REFERENCE
tables. Before loading the data we check for the data and create flag’s (Modified record or
new record manually inserted by operator in RDM). So we are updating GFIS tables with
changes received via delta files from SAP RDM. Received files have same structure as
original tables in SAP.
Please refer below for the GFIS REFERENCE tables which are referred above.
DBF_GFIS13_T_COUNTERPART_REF
DBF_GFIS13_T_CONTRACT_REF
Taxonomy - Understanding
FSR
DBF_GFIS13_T_INS_REF
DBF_GFIS13_T_INS_CROSS_REF
T_CON_ASIA_DATAWAREHOUSE
In this engine, ETL loads the data from the source to the T_GW_TMP_PROCESSED_DATA table with
stage_no=0. Please refer below for the mappings.
m_LOAD_GFIS_SOURCE_DATA_2_STAGE_0
Loads the source data from GFIS DDM table (DBF_GFIS13_T_DD_ACC_MOVE) to T_
GW_TMP_PROCESSED_DATA table with stage_no=0
m_B4.1.1_CORE_SL
Loads the source data from SAP table to T_GW_TMP_PROCESSED_DATA table with stage_no=0
m_B2.1.3_APAC_OFFBL
Loads the source data from SAP and GFIS CON table (DBF_GFIS13_T_CONTRACT_REF) to
T_GW_TMP_PROCESSED_DATA table with stage_no=0
m_B4.1.1_CORE_FLA
m_B4.1.1_CORE_NFE
m_B4.1.1_CORE_MAN
The above three mappings load the data from PORTAL to T_GW_TMP_PROCESSED_DATA table
with stage_no=0.
In this engine, ETL derives the business logic as per the rule mentioned in the Taxonomy. Please refer
below for the list of mappings in Derivation Engine.
Taxonomy - Understanding
FSR
Shortcut_to_m_B4.2.1.1_CORE_ALL
Shortcut_to_m_B4.2.1.2_CORE_ALL
Shortcut_to_m_B4.2.2.2_APAC_ALL
Shortcut_to_m_B4.3.1_CORE_ALL
Shortcut_to_m_B4.3.2_APAC_ALL
Shortcut_to_m_B4.4.1_APAC_ALL
Shortcut_to_m_B4.4.1_APAC_MAN
Shortcut_to_m_B4.5.1_APAC_ALL
Shortcut_to_m_B4.5.1_APAC_MAN
Shortcut_to_m_B4.5.2_APAC_ALL
Shortcut_to_m_B4.5.3_APAC_ALL
Shortcut_to_m_B4.5.4_APAC_DW_ACC
Below we have explained the flow of one of the Derivation Engine mapping.
Mapping: B4.3.1_CORE_ALL
Please query the database to get the set of rules to be implemented in the component
“B4.3.1_CORE_ALL’. In this component the GDS attributes are derived as per the Taxonomy rules.
T_GW_TMP_PROCESSED_DATA: This is the source in the mapping. We will retrieve the data from
this table with maximum stage_no.
In this engine, ETL compare the debit amount and credit amount based on the group by conditions
mentioned in the taxonomy rules and raise the error message if there is a mismatch. Totally there are
three mappings in Calculation Engine. Please refer below for the mappings.
Taxonomy - Understanding
FSR
m_B6.1_CORE_ALL
m_B6.1_CORE_MAN
m_B6.2_CORE_ALL
We are explaining one of the Calculation Engine mapping in this document. The other two mappings
also follow the same flow. Please confirm.
Mapping: m_B6.1_CORE_ALL
Please query the database to get the set of rules to be implemented in the component
“m_B6.1_CORE_ALL
T_GW_TMP_PROCESSED_DATA: This is the source in the mapping. We will retrieve the data from
this table with maximum stage_no.
T_GW_ERR_RECORD_HDR: In this table the record_id which is having error will be stored. It also
stores the PRC_PROCESS_ID and processing time (ITS)
Taxonomy - Understanding
FSR
Note: Record_id stored in the target is the first record_id of the whole group for a condition.
Questions:
1. Why is the batch_no been used in the groupby clause when it’s not given in the taxonomy
rule. What data will be stored in the batch_no column?
Joerg has confirmed that the batch_no column data will be same for each process.
2. In m_B6.1_CORE_ALL, the condition “epozflg <> 'R' or epozflg is null” is mentioned in the
source qualifier which is not given in taxonomy rule. Please check and confirm.
T_GW_TMP_PROCESSED_DATA:
In the Validation engines which are used in Portal workflow, is the SQL-Overwrite (especially
the where-clause) different from the other Engines. Here is because of the special handling
of the Reversal Rows in case of SAC-revalidation a additional clause added.
Taxonomy - Understanding
FSR
AND
(T_GW_TMP_PROCESSED_DATA.EPOZFLG <> 'R' OR
T_GW_TMP_PROCESSED_DATA.EPOZFLG is null)
This construct is needed, because if a SAC will be sent to portal it will automatically create
two rows. First row is the corrected row and second row is the row with the defaulted values
for. The second record hits the SAC in SAP and is the contra entry to the original entry
coming from automated feeds. This row is marked with a ‘R’ in attribute EPOZFLG an can be
filtered using this attribute.
This filter should only be in Validation engines. This construct is not longer needed in
calculation engines, because here are all rows needed for Balance check. So the B6… Portal
engines should not use it. I think at the moment the filter is used only in one B6… -engine.
But this engine is not used in Portal workflow so the filter will not affect anything. For the next
release we will remove this filter from all B6…- engines.
Mapping: B3.1.1_CORE_MAN
Please query the database to get the set of rules to be implemented in the component
“B3.1.1_CORE_MAN’
This component is a part of Validation Engine in which we perform mandatory for the below set of
columns.
POSTING DATE
DOCUMENT DATE
POSTING KEY
LINE ITEM TEXT
COMPANY CODE
MANAGEMENT AREA
TRANSACTION CURRENCY CODE
COUNTER
TRANSACTION CURRENCY AMOUNT
T_GW_TMP_PROCESSED_DATA: This is the source in the mapping. We will retrieve the data from
this table with maximum stage_no.
Taxonomy - Understanding
FSR
mplt_ERROR_HANDLING: In this mapplet we will implement the logic (Please refer Appendix for the
logic) and route the data to the Error tables. The inputs are ERROR_CODE, PRC_PROCESS_ID,
STAGE_NO, COMPONENT_ID, RECORD_ID and all the MSG_PARAM’s and MSG_VALUE’s
T_GW_ERR_RECORD_HDR: In this table the record_id which is having error will be stored. It also
stores the PRC_PROCESS_ID and processing time (ITS)
APPENDIX:
Intially we will filter the null error_code records. And the get the error_id for the
T_GW_ERR_CATALOG table by passing ERROR_CODE.
A function “PKG_GATEWAY.F_GET_MESSAGE_ID” is used to generate the message_id which will be
loaded into T_GW_ERR_RECORD_ASN table for all the error fields of a record_id.
Taxonomy - Understanding
FSR
Comment:
T_GW_TMP_PROCESSED_DATA:
In the Validation engines which are used in Portal workflow, is the SQL-Overwrite (especially
the where-clause) different from the other Engines. Here is because of the special handling
of the Reversal Rows in case of SAC-revalidation a additional clause added.
This construct is needed, because if a SAC will be sent to portal it will automatically create
two rows. First row is the corrected row and second row is the row with the defaulted values
for. The second record hits the SAC in SAP and is the contra entry to the original entry
coming from automated feeds. This row is marked with a ‘R’ in attribute EPOZFLG an can be
filtered using this attribute.
In this engine, ETL checks the threshold limit and default the SAC error columns if the threshold limit
value is less than the specified value for that specific system.
m_B8.2.4.1.ALL
In this mapping, the default values will be populated to SAC error columns. Initially during the
validation, the error record default values are inserted into T_GW_ERR_RECORD_DEFAULT. In this
engine, ETL look up into T_GW_ERR_RECORD_DEFAULT table and default value to those error
Taxonomy - Understanding
FSR
record_id’s before posting into SAP. In this process, the original source record and suspended error
record will be inserted into T_GW_ERR_RECORD_DTL with RECORD_TYPE_ID=1 and
RECORD_TYPE_ID=2 respectively.
Bojan: This is ok, but for further details please see TDD (Z:\80-
Feeds\GII_Gateway\Global\TDD\General\dbFeeds Technical Design v1.5.doc) Chapter: 14.3 SAC
Processing
NOTE: The look up’s taken on dbnexus table (DBF_MAP_T_MAPPING) has to be modified.
In mapping m_B8.6.4.1.ALL, look up LKP_MAP_THRESHOLD_PARAMETERS_DB_NEXUS the
VALIDTO_OVERRIDE and VALIDFROM_OVERRIDE should be compared with the effective date in
the condition tab.
8 Scripts
8.1.1 gw_start_seq_tasks.sh
This script is used to start the processing sequence as per the execution order list.
It takes a parameter file as an input which contains the details about the parameters that are
invoked in the session and workflow level.
The run status and the errors encountered are logged in every stage in a log file.
Taxonomy - Understanding
FSR
8.1.2 gw_start_single_task.sh
The script gives the status of the workflow (running or stopped or execution completed)
8.1.3 restart_workflow.sh
This script is used to restart a workflow. This script is used in places where a re-run should
be done.
1. Folder name
2. Workflow name
3. System ID
4. Interface ID
5. Template ID
6. PrcProcess ID
This script invokes gw_start_single_task.sh for a re-run of the same workflow, using the
existing parameters.
8.1.4 start_workflow.sh
It gets the server and port details from the Infa.txt file
Initially the wait period specified is taken into account and the workflow execution is kept on
hold till the specified period.
Once the sleep time is over, the execution of each and ever workflow specified in the file
starts.
Irrespective of the status of the previous workflow, all the workflow names specified in the file
is started.
Taxonomy - Understanding
FSR
8.1.5 move_file.sh
The script takes the remote server and remote file details from the parameter file, and moves
them to the specified target directory location.
8.1.6 portal_upload.sh
This script is used to upload the files generated to the specified destination directory, through
secure file transfer protocol.
The script gets the server and file details from the parameter file. Then it connects to the
destination directory through sftp, and copies the source files to the target location defined.
Once the files are copied, the file permissions are changed to 666 in the destination
directory.
8.1.7 attach_and_send_mail.sh
This script is used to send a file through email, to a list of user id’s defined in a group.
8.1.8 add_timestmp.sh
Taxonomy - Understanding
FSR
8.1.9 gw_push_tlr_bw.sh
This script is used to copy the TLR BW target files to the target (SAP) server. It also creates
a parameter file that contains a list of files that has been pushed to the SAP server.
1. Parameter file that contains the SAP server details, along with the directory and file
names that need to be pushed.
2. Input data file name, which needs to be renamed and pushed to the SAP server.
3. Input control file name, which needs to be renamed and pushed to the SAP server.
4. Output file name
9 RFC
The RFC functions are implemented in the SAP environment. These RFC functions are
called by the ETL Layer during various phases, by passing the required input parameters.
The usage of RFC in the Derivation, Validation and Target Load Ready (TLR) phases is
explained here.
RFC input/output parameters and how they are used in ETL is described in separate TDD
documents.
Taxonomy - Understanding
FSR
GET_WORKDAY contains only scalar parameters and is called only once, so single-stream mapping
is used.
The input parameters specified for a particular RFC function are sent to the corresponding
RFC function call. The output from RFC is send back from SAP to our server.
The input parameters that are being passed to this function are:
Return value to
T_GW_TMP_PROCESSED_DATA
Output from RFC table
E_DATE YYYYMMDD format
Taxonomy - Understanding
FSR
1. B3.4.1_APAC_ALL
2. B3.6.4.1.1
The SAP RFC and FSI validations are done in the SAP environment, through the RFC calls
from ETL layer. ETL always sends all available fields, but actual configuration of what and
how is checked is performed on SAP side. If any errors are detected in the data present in
SAP, then the error records are sent back. Input records and error records from SAP are
linked by unique ETL’s record_id (from T_GW_TMP_PROCESSED_DATA). Else no
message is returned from the SAP through this RFC, indicating that the data available in
SAP is correct. Each input field from ETL may generate zero (no errors) or several (one for
each error) return records from SAP.
The details about the error are recorded in the error tables by standard reusable mapplet
mplt_ERROR_HANDLING.
1. T_GW_ERR_RECORD_DEFAULT
2. T_GW_ERR_RECORD_HDR
3. T_GW_ERR_RECORD_ASN
4. T_GW_ERR_MSG_PARAMETERS
Query:
9.1.3 RULE_TEXT
These validations would be handled through an RFC Process.
FSI value will be derived based on GCoA and its possible GDS combinations.
Once the GDS values are derived ETL need to check against this C3REDA table and ensure
that a valid FSI value is possible to be derived.
Taxonomy - Understanding
FSR
I.e. GDS values for specific records are derived on previous processing stages (GDS
derivation engines) and sent to FSI RFC. Checks described in rule 3173 are performed
internally in SAP (check against C3REDA table; ensure that FSI values can be derived).
The SAP R/3 and BW TLR engine covers the following components.
1. B9.2_APAC_ALL_UPLOAD
2. B9.4_APAC_ALL_UPLOAD
3. B9.6_APAC_FLA_UPLOAD
4. B9.7_APAC_NFI_UPLOAD
5. B9.8_APAC_ALL_UPLOAD
6. B9.9_APAC_ALL_UPLOAD
Query: Can you please confirm whether we are using all the above mappings in the
processing?
Now we are using only B9.2 (for SAP BW) and B9.4 (for SAP R/3) upload mappings.
B9.6-B9.9 upload mappings are the copies of B9.2, only difference is in component name in
mappings’ parameters. These unique component names were intended for better readability
of processing information. But as we always have TLR creation mappings (like B9.6, B9.7)
prior to upload mapping and it’s listed in processing information, it’s possible to use only one
generic B9.2_upload mapping.
Output files are created by the TLR engines. These target files are renamed with a unique
name (by adding timestamp) and uploaded to the SAP server using shell scripts:
gw_push_tlr_sl.sh – for SAP SL (R/3) files, B9.4 mappings
gw_push_tlr_bw.sh – for SAP BW files, all other B9.* mappings
These shell-scripts are configured as post-session commands in TLR creation engines. Files
are uploaded to SAP servers using Unix sftp utility. Shell-scripts and sftp output is saved in
scriptlogs/gw_apac_1 directory.
Taxonomy - Understanding
FSR
Only data files are required for SAP R/3, but SAP BW requires uploading of paired control
and data files. Control file contains information about file name for data file, number of
records and some additional service information.
After uploading of TLR files to SAP server, notification about new TLR files is send to the
SAP server by the RFC function that is being called in the TLR UPLOAD Components.
Notifications for SAP BW are also sent to SAP R/3 and forwarded internally between two
SAP systems (as we don’t have Informatica license for connecting directly to SAP BW).
TLR submission status (SAP return code and text message) is returned by RFC and saved in
T_GW_PRC_FILES table.
Queries:
1. How will the errors encountered during transferring of the files through SFTP
command notified? Are we raising a Tivoli alert If the SFTP transfer is not successful?
No, we are not raising alerts if sftp is not successful. TLR submission status (also “file
not found” errors) is returned from SAP by TLR notification RFC. This status is saved
in T_GW_PRC_FILES table and in case if return code is not 0, error message will be
sent to Portal.
2. How do we know which RFC function is to be called at a specified point? Has it been
specified anywhere.
RFC notification mapping should be called exactly after TLR creation mapping. Which
RFC should be called for which file types – it’s mentioned in our “SAP_R3_TLR.doc”
TDD. Original specifications for processing of TLR files was provided by SAP team,
copy saved on shared drive.
Taxonomy - Understanding
FSR
The configuration file (system32/saprfc.ini) has to been updated with the connection
parameters provided by SAP in the local system (saprfc.ini sections can be simply copied
from UNIX server). This local configuration is used in design-time by Designer’s wizards.
Taxonomy - Understanding
FSR
4. Fill in the Connect string, Username, Password, Client and Language details as given
by SAP. Click on the Connect tab, to get connected to SAP database.
5. Set the Program mode to Stream
We have two program modes.
File Mode: If we select this then the file will be created in the SAP server which has
to be downloaded to our server and process the same. This mode allows shorter on-
line sessions with SAP. But it is not used currently, as this mode requires additional
sftp file transfer from SAP server and more complicated workflows.
Stream Mode: We can directly connect to the source table in the SAP and process
the data. This mode requires longer on-line sessions with SAP, but mappings and
workflows are simplier.
6. Set YYFGLV4 as development class ($TMP class can not be transported between
SAP systems)
7. Select the B1.6 mapping in the ‘list of mappings’ tab and click the ‘Generate Files’ tab.
This file can be used for manual checking of ABAP code if Designer reports some
errors during code installation to SAP.
8. Select Direct Installation
9. If ABAP code was not installed for new mapping, session will fail and log-file will
contain message about missing ABAP program
10. If ABAP code was not regenerated after mapping was changed, session will fail and
log-file will contain message about different timestamp for mapping and ABAP code.
Note:
1. If any change is done in the mapping level, then before refreshing in the workflow
level, the SAP R/3 code needs to be regenerated every time.
2. ABAP code can be directly installed only in DEV environment. Updated code should
be copied to UAT/PROD SAP systems via SAP transports (SAP functionality for
copying changes between DEV/UAT/PROD).
3. When copying mappings between repositories (dev/uat/prod), “Copy program
information” option should be checked. This will copy ABAP program information
together with mapping and ensure that mapping’s and ABAP program’s timestamps
are in sync.
4. The comments given in the Mappings -> Edit -> Comments tab should not be long.
(ABAP code lines are restricted to 72 characters). This is the reason why CVS header
variable is not saved in SAP source mappings.
Taxonomy - Understanding
FSR
The following are the steps to generate the mapping which calls the RFC mapping
1. SAP Function Modules (i.e. RFC) can be checked directly in SAP GUI, using se37
transaction (SAP “transaction” is a subprogram, which provides some functionality
like RFC execution). This subprogram allows checking of RFC parameters and direct
execution in SAP environment, what is useful for debugging purposes.
2. Open Informatica Power Centre Designer -> Mapping Designer
3. Select Mappings -> Generate SAP RFC/BAPI Mapping
4. Provide all the connection details (Connect String, User name, Password, Client,
Language) for SAP connectivity
5. Give the RFC function name in the Filter Criterion, preceded by a ‘%’ as a wildcard
character
6. Click “Connect” button
7. Select the required RFC from ‘Retrieved Functions’ and divert to the ‘Selected
Functions’
8. Go to the ‘Next’ option, and set the $$SAPCONNECTION to the configured
connection parameters in workflow Manager (fixed to FSR_SAP_R3 for our
mappings)
9. Once the mapping is created the input parameters will be passed to the RFC mapplet
as per the SAP Requirements and output parameters will be used for further
processing if required.
10. Created mapping will have virtual sources and targets for RFC input/output. These
virtual objects should be replaced with real sources/targets
11. Due to Informatica bug, repeated mapping generation can produce invalid mapping
with unconnected mapplets. To avoid this bug, previously generated RFC mapplets
and reusable transformations should be deleted prior to repeated mapping
generation.
Taxonomy - Understanding
FSR
Note:
Single Stream RFC is created when RFC has only scalar input/output parameters (i.e. only
one row of data is expected for input/output).
Multiple Stream RFC is created when RFC has table parameters (i.e. requires more than
one row for input or output). Same table parameter can be used both for input and output.
Data is converted to Informatica’s internal format in preparation pipeline and actual RFC call
is performed (using previously prepared data) in functional call pipeline. Intermediate files are
used for passing data between these two pipelines. Indirect Flat file source (which contains
list of actual data files) is used for feeding multiple prepared flat files to functional call
pipeline.
Steps need to be taken care when we use a Multiple Stream for RFC.
1. Target Load Plan (The RfcPreparedData pipeline should be executed before processing
the actual RFC mapplet. Otherwise it may happen that data prepared on previous run will be
used for new RFC call.
2. Parallel Processing. (If we use this session in different workflow’s and filenames are same
don’t process the workflow’s in parallel). Otherwise it may happen that data prepared by
another parallel run will be used for this RFC call.
The following are the steps to be taken care in the workflow level
1. Open Workflow Designer and configure the connection parameters.
Connections-> Application connection Browser
Please refer below for the existing select type.
Taxonomy - Understanding
FSR
Query: For the session configured on SAP REFERENCE data, go to mapping tab
For Source tables -> Readers -> SAP Streaming Reader
What is the difference between SAP Streaming Reader and SAP Staging Reader?
Streaming reader is used for Stream mode ABAP programs (when data is loaded from SAP
in on-line session). Staging reader is used for File mode programs (i.e. mode when file is
created on SAP server and then transferred to Informatica via ftp)
Note: The default value of the fields in the SAP RFC function should not be a pipe character
(‘|’) or hash character (‘#’). “Typing” option for RFC import/export parameters on SAP side
should be set to “TYPE” (“LIKE” parameters can be recognized incorrectly by Informatica
wizard, for example integer can be recognized as char(1) and incorrectly converted in run-
time):
“Typing” option for RFC table parameters on SAP side should be set to “LIKE”:
Taxonomy - Understanding
FSR
11 Logging Mechanism
During the execution of the workflow, the status details are logged in 2 levels.
1. Database level
2. Script level.
1. T_GW_PRC_FILES
2. T_GW_PRC_PROCESS
3. T_GW_PRC_STAGE
The T_GW_PRC_PROCESS table contains the details of the workflow status specified in
PROCESSING_STATUS column.
S – Started
C – Completed
E – Soft Error
H – Hard Error
The number of soft and hard errors will be recorded with its count in
ERRONEOUS_REC_COUNT column.
The T_GW_PRC_FILES table contains the status of the files that have been uploaded
through the PROCESSING_STATUS column.
The reason behind the failure of file uploading will be logged in SAP_ERR_MESSAGE
column.
Query: We could see few columns with a SAP_ERR_MESSAGE, ‘Data file not found’ even
when the PROCESSING_STATUS column has a value ‘Y’. Please clarify.
For PROCESSING_STATUS we are checking only error code returned from SAP. If this code
is 0, PROCESSING_STATUS = Y. Unconnected error code and error message - possibly it’s
bug on SAP side. Latest example on UAT was in April, but if it will be repeated, clarifications
should be requested from SAP team.
The T_GW_PRC_STAGE table contains the status of every session in the process in the
PROCESSING_STATUS column.
Taxonomy - Understanding
FSR
S – Started
C –Completed
The script level logging is done based on the PROCESS_ID, which is appended before the
file name details.
1. Server details
2. Archive directory
3. Current working directory (where the script log is currently located during processing)
4. Parameter filename along with its path
5. Workflow name
6. Command for start of workflow
7. Execution status details
Queries:
1. What are the possible values that the execution status might have in the script log?
Execution status mentioned in screenshot above is returned from pmcmd command.
Possible return codes are listed in Informatica help (Workflow Administration Guide
Chapter 23: Using pmcmd Using the Command Line Mode pmcmd Return Codes):
Code Description
For all commands, a return value of zero indicates that the command ran successfully.
You can issue these commands in the wait or nowait mode: starttask, startworkflow,
resumeworklet, resumeworkflow, aborttask, and abortworkflow. If you issue a
0 command in the wait mode, a return value of zero indicates the command ran
successfully. If you issue a command in the nowait mode, a return value of zero
indicates that the request was successfully transmitted to the PowerCenter Server,
and it acknowledged the request.
Taxonomy - Understanding
FSR
2 The specified task name, workflow name, or folder name does not exist.
8 You do not have the appropriate permissions or privileges to perform this task.
9 The connection to the PowerCenter Server timed out while sending the request.
The PowerCenter Server cannot start recovery because the session or workflow is
12 scheduled, suspending, waiting for an event, waiting, initializing, aborting, stopping,
disabled, or running.
The PowerCenter Server found the parameter file, but it did not have the initial values
18
for the session parameters, such as $input or $output.
The PowerCenter Server cannot start the session in recovery mode because the
19
workflow is configured to run continuously.
A repository error has occurred. Please make sure that the Repository Server and the
20
database are running and the number of connections to the database is not exceeded.
24 Out of memory.
25 Command is cancelled.
Taxonomy - Understanding
FSR
The possible errors that can occur during the processing can be due to
1. Informatica Errors
2. Database errors
3. Connectivity errors
The occurrence of the error is logged in the script logs that can be found in
$PM_HOME/scriptlogs/<feed_specific_dir_name>
The workflow or session logs can also be viewed through the Workflow Monitor.
Note that if processing was restarted, previous log-files will be overwritten on each run.
Occurrence of any errors discontinues executing the normal flow and it takes the Tivoli alert
notification path.
Failure of any task in the workflow will lead to execution of the Tivoli worklet path.
Taxonomy - Understanding
FSR
The DBF_WL_TIV_WFLOWERROR worklet is used to raise Tivoli alerts. The ‘Always on’
(parent) workflows are configured in such a way that all the tasks in the flow are connected to
this Tivoli worklet during a failure condition. Hence whenever a session gets failed, it stops
continuing in the regular flow, and takes the Tivoli path.
1. DBF_MAP_TIV_WFLOWERROR (Session)
2. DBF_CMD_LOG (Command Task)
Location: $PMRootDir\PrmFiles\dbf_wflow.prm
It is also possible to use ‘Always on’ workflows’ parameter files for setting $$WFLOW
variable (instead of dbf_wflow.prm).
Location: $PMRootDir\PrmFiles\gw_apac_1\Start_*.prm
Sample Entry:
[<Folder_Name>.WF:<Workflow_Name>.WT:DBF_WL_TIV_WFOLWERROR.ST:DBF_SESS
_TIV_WFLOWERROR]
$$WFLOW='<Workflow_Name>'
Also, metadata describing the workflow details need to be entered in the following tables.
1. DBF_T_WORKFLOW
2. DBF_T_MODULE_WORKFLOW
For inserting metadata in Tivoli configuration tables, it’s also possible to use stored
procedure:
DECLARE
BEGIN
FEED_OWNER.DBF_PR_INSERT_METADATA('GW_APAC_1', 'FSR ETL-Portal', 'wf_Start_WF_PORTAL_FLA');
FEED_OWNER.DBF_PR_INSERT_METADATA('GW_APAC_1', 'FSR ETL-Portal', 'wf_Start_WF_PORTAL_MAN');
Taxonomy - Understanding
FSR
This alert message contains the details about the failed parent workflow name. The session
level details in the workflow that got failed can be checked through the script log file or in
workflow monitor. In addition to this Tivoli alert, a failure notification will be sent through mail,
by the em_FailureNotification reusable email task, to the intended recipients. List of
recipients is configured in em_FailureNotification parameters.
em_FailureNotification will be triggered twice – on session level (as ‘On Failure E-Mail’ in
failed session) and on parent workflow level (as next task after Tivoli worklet).
The child workflows that are started through the gw_start_seq_tasks.sh (ETL1/ETL2
processing) or gw_start_single_task.sh (PORTAL processing), will return a non-zero value in
case of errors. Command task will return non-zero values and this will in turn raise a Tivoli
alert, with the parent workflow name.
Taxonomy - Understanding