Ds Loop With TX

DataStage: Loop With Transformer – MyDatahack https://www.mydatahack.
com/datastage-loop-with-transformer/
Home (https://www.mydatahack.com) / ETL (https://www.mydatahack.com/category/etl/) /

DataStage: Loop With Transformer
DataStage: Loop With Transformer Search

Posted: October 2, 2017 | Category : DataStage (https://www.mydatahack.com
/category/etl/datastage/), ETL (https://www.mydatahack.com/category/etl/) | Tags:
DataStage (https://www.mydatahack.com/tag/datastage/), ETL Categories
(https://www.mydatahack.com/tag/etl/), Loop (https://www.mydatahack.com
/tag/loop/), Transformer (https://www.mydatahack.com/tag/transformer/)
Data Engineering 49
(https://www.mydatahack.com
The Transformer stage has the built-in looping functionality where you can use /category/data-
Stage Variables and Loop Conditions to construct looping logics. In this post, we engineering/)
will present 3 different examples.
Data Science 12
Ranking
/category/data-science/)
Aggregation
Vertical Pivot ETL 31
Before going into the examples, here are the useful variables for loop
/category/etl/)
construction.
Others 33
@ITERATION – System Variable to set the while loop condition. It starts (https://www.mydatahack.com
from 1 and increments by one as the loop progresses. Once while loop /category/others/)
breaks, it gets back to 1.
SaveInputRecord() – Utility function to save the current input row in the
cache and return the count of records currently in the cache. Each input row
in a group is saved until while loop breaks.
GetSavedInputRecords() – Output SaveInputRecord() value every time it
loops. The output value can become ranking.
LastRowInGroup() – According to the grouping, True indicates the last row
in the group. Recent
Comments
Example 1: Ranking
In this example, we will add the rank columns that indicate the descending order
mydatahack on
of the transaction date per customer.
Introduction to Dense
Layers for Deep
Input
Learning with Keras
/introduction-to-
dense-net-with-
keras/#comment-5)
haramoz on
Introduction to Dense
1 of 8 18-11-2018, 09:22:48
DataStage: Loop With Transformer – MyDatahack https://www.mydatahack.com/datastage-loop-with-transformer/
Layers for Deep

Learning with Keras
/introduction-to-
dense-net-with-
keras/#comment-4)
mydatahack on
Salesforce API with
Simple Salesforce
For Python
/salesforce-api-with-
Output simple-salesforce-
for-python
/#comment-3)
LeMaurien19 on
Salesforce API with
Simple Salesforce
For Python
/salesforce-api-with-
simple-salesforce-
for-python
/#comment-1)
Steps
Recent Posts
How to Restore
AdventureWorks to
SQL Server Express
/how-to-restore-
adventureworks-to-sql-
server-express/)
How to Connect to
Locally Installed SQL
Server Express with
JDBC
(1) Before the transformer stage, we need to sort the input data by Cust_Id in (https://www.mydatahack.com
ascending order and Trans_Date in descending order. /how-to-connect-to-
locally-installed-sql-
server-express-with-
jdbc/)
How to Enable User

Login with SQL Server
2 of 8 18-11-2018, 09:22:48
Express
/how-to-enable-user-
login-with-sql-server-
express/)
How to Solve ‘Missing

Authentication Token
Error’ with API Gateway
Custom Domain
/how-to-solve-missing-
authentication-token-
error-with-api-gateway-
custom-domain/)
Using EditorConfig and

ESLint in Node.js
(2) Set Hash as the inputs partition in the Sort Stage. Set Cust_Id as the hash key. Project
/using-editorconfig-and-
eslint-in-node-js-
project/)
Archives
November 2018
/2018/11/)
October 2018
(3) Set the inputs partition in the Transformer stage to ‘same’. Make sure the (https://www.mydatahack.com
perform sort is unticked. /2018/10/)
September 2018
/2018/09/)
August 2018
/2018/08/)
July 2018
/2018/07/)
June 2018
/2018/06/)
(4) Set Stage Variables as below May 2018
3 of 8 18-11-2018, 09:22:48
svInput = SaveInputRecord() (https://www.mydatahack.com

/2018/05/)
svLastrowingroup = LastRowInGroup(Cust_Id)
April 2018
svBreakloop = If svLastrowingroup then svInput Else 0
/2018/04/)
(5) Set Loop Condition as Loop While: @ITERATION <= svBreakloop March 2018
(6) Set Loop Variable as LoopVar = GetSavedInputRecord(). This LoopVar will /2018/03/)
become the Rank column. February 2018
/2018/02/)
January 2018
/2018/01/)
December 2017
/2017/12/)
November 2017
/2017/11/)
October 2017
/2017/10/)
September 2017
/2017/09/)
August 2017
Example 2: Aggregation (https://www.mydatahack.com
/2017/08/)
We will use the same data in Example 1 and create a column that has the total
purchased unit count per customer.
Input
Same as Example 1
Output
4 of 8 18-11-2018, 09:22:48
Steps
(1) As in the example 1, we need to sort the input data by Cust_Id and set hash
partition.
(2) Set Stage Variable as below. The order of the variables have to be kept. The
key idea here is Stage Variables are executed from top to bottom. At the last row
in the group, svSumUnits gets reset after the total unit is calucated as
svTotalUnits.
svInput = SaveInputRecord()
svLastRowInGroup = LastRowInGroup(lnk_trans.Cust_Id)
svTotalUnits = if svLastRowInGroup then svSumUnits + lnk_trans.Units
Else 0
svSumUnits = if svLastRowInGroup then 0 Else svSumUnits +
lnk_trans.Units
svBreakLoop = if svLastRowInGroup then svInput Else 0
(3) Set the while loop condition as Loop While: @ITERATION <=
svBreakLoop. You need to set LoopVar= GetSavedInputRecord(). Using
SaveInputRecord() without GetSavedInputRecord() results in a compilation error.
(4) The stage variable svTotalUnits becomes the total units.
5 of 8 18-11-2018, 09:22:48
Example 3: Vertical Pivot
In the final example, we use the while loop in the transformer stage to perform
vertical pivoting operation. Input records show employers and 3 columns that list
their skill set. We will vertically transpose this table to list employers’ skillset in a
single column as in the output.
Input
Output
6 of 8 18-11-2018, 09:22:48
Steps
(1) Set the while loop condition as @ITERATION <= 3.
(2) Set Loop Variable as below. Each iteration will add empl_id, and skill column 1,
2 or 3 according to the iteration variable (@ITERATION).
loopVar: If @ITERATION = 1 Then lnk_trans.Skill_1 Else If @ITERATION = 2

Then lnk_trans.Skill_2 Else lnk_trans.Skill_3
7 of 8 18-11-2018, 09:22:48
SIMILAR NEWS
/running-jobs-with- (https://www.mydatahack.com(https://www.mydatahack.com
ETL /how-to-rename-mapping-job-
informatica-cloud-rest-api/) /tips-and-troubleshooting-for- ETL
uploading-csv-to-database-in-in-informatica-cloud/)
ETL
Running Job… talend/)
How To Rena…
Informatica Cloud offers Tips and Tro…
Mappings are where all
REST API for us to
There will be time when the magic happens in
interact with the
you want to upload a Informatica Cloud.
platform
big csv file (with many When I started using it,
programmatically. At
rows and hundreds of it took me a while to
columns) to a relational
© 2017
MyDataHack.com
8 of 8 18-11-2018, 09:22:48

Ds Loop With TX

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Ds Loop With TX

Загружено:

Авторское право:

Доступные форматы

DataStage: Loop With Transformer – MyDatahack https://www.mydatahack.

Home (https://www.mydatahack.com) / ETL (https://www.mydatahack.com/category/etl/) /

DataStage: Loop With Transformer Search

Layers for Deep

How to Enable User

How to Solve ‘Missing

Using EditorConfig and

svInput = SaveInputRecord() (https://www.mydatahack.com

(4) The stage variable svTotalUnits becomes the total units.

Example 3: Vertical Pivot

(1) Set the while loop condition as @ITERATION <= 3.

loopVar: If @ITERATION = 1 Then lnk_trans.Skill_1 Else If @ITERATION = 2

Вам также может понравиться