Вы находитесь на странице: 1из 26

Advanced Topics in ODI-EE: Hands on Lab

Advanced Topics in Oracle Data


Integrator: Hands-on Lab

Advanced Topics in ODI-EE: Hands on Lab


Page 1 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

Supported Version: Oracle Data Integrator 11.1.1.3


Objectives:
This lab will highlight to Developers, DBAs and Architects some of the best practices for
implementing techniques to improve performance for real-time data warehousing using Oracle
Data Integrator Enterprise Edition (ODI-EE).
In this lab participants will walk through the steps to enable change data capture on a source
relational table, then deploy and manage these implementations to load and transform records to
a relational target.
Prerequisite:
Basic knowledge of Oracle Data Integrator, Installed Oracle Data Integrator software and Oracle
Database instance.

S308461 - Using ODI-EE and Oracle BI for Real-Time Insight:


Hands on Lab

Advanced Topics in ODI-EE: Hands on Lab


Page 2 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

Table of Contents
Part 1: Configuring a Model for CDC ........................................................................... 4
Part 2: Creating a Package to Set Up CDC ................................................................. 8
Part 3: Creating the Integration Flows to Consume the Changes ............................... 13
Part 4: Run the CDC Set Up and Consumption Processes in Production................... 22
Summary .................................................................................................................. 26

Advanced Topics in ODI-EE: Hands on Lab


Page 3 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

Part 1: Configuring a Model for CDC


In this exercise you will configure an existing source Oracle data model to support ODI CDC
feature. The CDC mode used will be Consistent-Set CDC. Oracle Log Miner will be used to track
the changes.
1. Double-click the icon Reset_ODI.bat from the desktop. This will reset the files for all Handson Labs to the initial state.
Warning: If you have custom ODI projects in the repository on this machine, they will be
deleted.

2.
3.

Open the Oracle Data Integrator Studio using the Start Menu > Oracle > Oracle Data Integrator > ODI
Studio menu.
Go to the Designer Navigator and press Connect to Repository

4.

Select HOL Advanced from the drop-down list of logins and press OK.

5.

Open the Designer navigator menu and make sure that the Default Execution Context is set to the
Development context (not Production).

Advanced Topics in ODI-EE: Hands on Lab


Page 4 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

6.

Open the Models accordion in the Designer navigator by clicking on the label.

7. Right-click on the Oracle CDC Source model and select Changed Data Capture > Add to
CDC.
Click Yes to add all the tables to the CDC.

8. Double-click on the Oracle CDC Source model.


The editor for this model opens.

Advanced Topics in ODI-EE: Hands on Lab


Page 5 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

9. Go to the Journalizing tab.


10. Select the Consistent Set journalizing mode, click OK to close the popup window then the
JKM Oracle 10g Consistent (Log Miner) knowledge module. Set the
AUTO_CONFIGURATION KM option to Yes.

11. Go to the Journalized Tables tab, select the SRC_REGION table and click on the Remove
from CDC

button.

12. Click on the Reorganize button


to order the tables according to their foreign-keys.
Press yes on all confirmation dialogs.
The datastores are automatically organized as shown below.

Advanced Topics in ODI-EE: Hands on Lab


Page 6 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

Note: Consistent Set Journalizing tracks changes to a group of the model's datastores, taking into
account the referential integrity between these datastores. The group of datastores journalized in
this mode is called a Consistent Set.
13. Click Save(

) on the toolbar to save the Model.

Advanced Topics in ODI-EE: Hands on Lab


Page 7 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

Part 2: Creating a Package to Set Up CDC


In this exercise you will create a package that sets up the CDC infrastructure in a given context.
1. In the Designer navigator, Click on the Projects accordion.

2. Expand the Advanced Topics in ODI-EE project and then expand the First Folder folder.

3. Select the Packages node, right-click and select New Package.


A package editor opens.
4. Enter the following package name: Tutorial Part 2 - CDC Setup
5. Go to the Diagram tab.

6. Drag and drop into the diagram the Oracle CDC Source data model from the Designers
Models tree view. A new step appears in the diagram, named after your data model.

Advanced Topics in ODI-EE: Hands on Lab


Page 8 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

Advanced Topics in ODI-EE: Hands on Lab


Page 9 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

7. Click on this step. In the Properties panel:

Select Journalizing Model in the Type drop-down list

Click the Start and Add Subscribers checkboxes.


Enter CONSUMER_1 in the Subscribers field, and then click Add.
Enter CONSUMER_2 in the Subscribers field, and then click Add.

8. Click Save(

) on the toolbar to save the package.

9. Click on the Execute(

) button on the toolbar.

10. In the Execution window, select Development for the context and then press OK.

Advanced Topics in ODI-EE: Hands on Lab


Page 10 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

11. Press OK when the Session Started window appears.


12. Open the Operator navigator.

13. In the Operator, select the Session List tab, and expand the All Executions node.

Advanced Topics in ODI-EE: Hands on Lab


Page 11 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

14. Check that the last session ran correctly. You can review the steps and tasks that have
activated the CDC process.

Advanced Topics in ODI-EE: Hands on Lab


Page 12 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

Part 3: Creating the Integration Flows to Consume the Changes


In this exercise you will create the integration flows that will consume the changes captured using
CDC.
1. In the Designer navigator, open the Projects accordion.
2. Expand the Advanced Topics in ODI-EE project and then expand the First Folder folder.
3. Expand the Procedures node.
4. Right-click on the CDC Changes procedure and select Execute.

5. In the Execution window, select Development for the context and then press OK.

6. Press OK when the Session Started window appears.


7. The procedure did the following:
1 insert in SRC_CITY.
1 insert in SRC_CUSTOMER.
1 update in SRC_CITY.
2 updates in SRC_CUSTOMER.

Advanced Topics in ODI-EE: Hands on Lab


Page 13 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

8. Expand the Interfaces node and double-click on Load SRC_CITY CDC.

9. Go to the Mapping tab.


10. Click on the source datastore SRC_CITY and check the Journalized Data only checkbox.
This ensures that ODI will load only the changed data (latest inserts, updates or deletes) from
the source instead of the entire source dataset.

11. Click on the filter


and modify the filter expression from JRN_SUBSCRIBER =
'SUNOPSIS' to JRN_SUBSCRIBER = 'CONSUMER_1'

Advanced Topics in ODI-EE: Hands on Lab


Page 14 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

12. Right-click on the filter

and select Data to review the changed data.

13. Save and close the interface.


14. Double-click on Load SRC_CUSTOMER CDC in the Interfaces node.
15. Go to the Mapping tab.
16. Click on the source datastore SRC_CUSTOMER and check the Journalized Data only
checkbox.

Advanced Topics in ODI-EE: Hands on Lab


Page 15 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

17. Click on the filter


and modify the filter expression from JRN_SUBSCRIBER =
'SUNOPSIS' to JRN_SUBSCRIBER = 'CONSUMER_1'.

18. Right-click on the filter

and select Data to review the changed data.

14. Click on the Models tab in the tree view.

Advanced Topics in ODI-EE: Hands on Lab


Page 16 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

19. Expand the Flat Files Sources model.

20. Expand the Flat Files Sources model and drag and drop the SRC_SALES_PERSON.txt
datastore from the Designers Models tree view into the source panel of the interface.

21. Create a join between the SRC_CUSTOMER table and the SRC_SALES_PERSON.txt file.
To create it, drag and drop the SALES_PERS_ID column from the SRC_CUSTOMER table
into the SALES_ID column from the SRC_SALES_PERSON.txt file.
Note: The join expression is:
SRC_CUSTOMER.SALES_PERS_ID = SRC.SALES_ID

Advanced Topics in ODI-EE: Hands on Lab


Page 17 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

22. We now need to map the remaining target column:


- SALES_PERS: map FIRST_NAME and LAST_NAME from the SRC_SALES_PERS.txt
file into it. The two source columns should be concatenated with an extra space between
them then we use the LTRM and RTRIM to remove any whitespaces.
Note: The mapping expression is
LTRIM(RTRIM(SRC.FIRST_NAME)) || ' ' || LTRIM(RTRIM(SRC.LAST_NAME))
23. Save and close the interface.
24. Expand the Advanced Topics in ODI-EE project and then expand the First Folder folder.
25. Select the Packages node, right-click and select New Package.

A package editor opens.


26. Enter the following package name: Tutorial Part 3 - Load CDC Data
27. Go to the Diagram tab.
28. Drag and drop into the diagram the Oracle CDC Source data model from the Designers
Models tree view. A new step appears in the diagram, named after your data model.

29. Click this step. In the Properties panel:


Select Journalizing Model in the Type drop-down list.

Click the Extend Window and Lock Subscribers checkboxes.

Advanced Topics in ODI-EE: Hands on Lab


Page 18 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

Enter CONSUMER_1 in the Subscribers field, and then click Add.

30. Drag and drop the Load SRC_CITY CDC and Load SRC_CUSTOMER CDC interfaces from
the Designers Projects tree view.

31. Drag and drop into the diagram the Oracle CDC Source data model from the Designers
Models tree view. A new step appears in the diagram, named after your data model.

32. Click this step. In the Properties panel:


Select Journalizing Model in the Type drop-down list.
Click the Purge Journal and Unlock Subscribers checkboxes.

Enter CONSUMER_1 in the Subscribers field, and then click Add.

33. Link the package steps together using green arrows.

34. Click Save(

) on the toolbar to save the package.

Advanced Topics in ODI-EE: Hands on Lab


Page 19 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

35. Click on the Execute(

) button on the toolbar.

36. In the Execution window, select Development for the context and then press OK.
37. Press OK when the Session Started window appears.
38. Open the Operator navigator.

39. In the Operator, select the Session List tab, and expand the All Executions node.
40. Check that the last session ran correctly. There should be 2 inserts and 3 updates.

We can see that the changed data for CONSUMER_1 has been propagated correctly.
41. Open the Models navigator and expand the Oracle CDC Source data model.
42. Right-click on SRC_CITY, select Changed Data Capture > Journal Data

Advanced Topics in ODI-EE: Hands on Lab


Page 20 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

Since we used CONSUMER_1 in our package, the changes for CONSUMER_2 have not been
propagated. Having multiple subscribers enables us to execute multiple processes loading the
same changed data without any interference.

Advanced Topics in ODI-EE: Hands on Lab


Page 21 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

Part 4: Run the CDC Set Up and Consumption Processes in Production


In this exercise you will run in a production environment the two packages you have created to
set up CDC, and then consume the changes captured by the CDC process.
ODI is already configured to connect to the production environment. A context called Production
has been created and it uses our Oracle production schema (CUST_DW_PROD) instead of the
development schema (CUST_DW_DEV).
1. In the Designer navigator, open the Projects accordion.
2. Expand the Advanced Topics in ODI-EE project and then expand the First Folder folder.
3. Expand the Packages node.

4. Right-click on Tutorial Part 2 - CDC Setup and select Generate Scenario. Press OK in the
New Scenario dialog.
Note: A scenario is a frozen version of an ODI component, it is used to put that component
into production.
In this lab we have simplified the deployment process and we can simply run our scenarios in
production using the Production context.
Typically we would have exported a scenario from the development repository and then
imported it into the production repository.
5. Expand the package and the Scenarios node then right-click on
TUTORIAL_PART_2__CDC_SETUP and select Execute.

Advanced Topics in ODI-EE: Hands on Lab


Page 22 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

6. In the Execution window, set the Context to Production and press OK.

7. Press OK when the Session Started window appears.


8. Go to Operator and make sure the session ran successfully.
9. In Designer, go to the Projects tree view and expand the Procedures node in the Advanced
Topics in ODI / First Folder folder.
10. Right-click on the CDC Changes procedure and select Execute.

11. In the Execution window, set the Context to Production and press OK.

12. Press OK when the Session Started window appears.

Advanced Topics in ODI-EE: Hands on Lab


Page 23 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

13. Click on the Operator

button in the toolbar.

14. In the Operator, select the Session List tab, and expand the All Executions node.
15. Validate that the procedure ran correctly, 2 inserts and 3 updates should have been
performed.

16. Under Packages, right-click on Tutorial Part 3 Load CDC Data and select Generate
Scenario then click OK.
17. Expand the package and the Scenarios node then right-click on
TUTORIAL_PART_3__LOAD_CDC_DATA and select Execute.

18. In the Execution window, set the Context to Production and press OK.

Advanced Topics in ODI-EE: Hands on Lab


Page 24 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

19. Press OK when the Session Started window appears.


20. Open the Operator navigator and select the Session List tab, and expand the All
Executions node.
21. Validate that the last session ran correctly.

Advanced Topics in ODI-EE: Hands on Lab


Page 25 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved

Advanced Topics in ODI-EE: Hands on Lab

Summary
At the end of these exercises the participants understand how to:
- Enable Change Data Capture.
- Design Integration Flows to consume the changes captured by the
CDC process.
- Run their processes in a Development environment as well as in a
Production environment.
Questions & Clarifications:
If you have any comments or need additional information, please communicate through the
Oracle Data Integrator forum at: http://forums.oracle.com/forums/forum.jspa?forumID=374

Advanced Topics in ODI-EE: Hands on Lab


Page 26 of 26
7/9/2010 6:08:00 PM (version 1.0)
Oracle, All rights reserved