Вы находитесь на странице: 1из 45
<Insert Picture Here>
<Insert Picture Here>
<Insert Picture Here> Oracle Data Integrator – Solution Overview Nguyen Tuan Khang , khang.nguyen@oracle.com
<Insert Picture Here> Oracle Data Integrator – Solution Overview Nguyen Tuan Khang , khang.nguyen@oracle.com

Oracle Data Integrator – Solution Overview

Nguyen Tuan Khang, khang.nguyen@oracle.com

Senior Solutions Consultant Fusion Middleware Oracle Vietnam

Why Data Integration? NEED… InformationInformation HowHow andand WhereWhere youyou WantWant ItIt Business Intelligence

Why Data Integration?

NEED…

InformationInformation HowHow andand WhereWhere youyou WantWant ItIt Business Intelligence Corporate Performance
InformationInformation HowHow andand WhereWhere youyou WantWant ItIt
Business Intelligence
Corporate Performance
Management
Business Process
Management
Business Activity
Monitoring
DataData IntegrationIntegration Migration Data Master Data Data Federation Real Time Warehousing Management
DataData IntegrationIntegration
Migration
Data
Master Data
Data
Federation
Real Time
Warehousing
Management
Synchronization
Messaging
- - - - - -
- - - - - -
HAVE…
DataData inin DisparateDisparate SourcesSources --- --- --- --- --- --- --- --- ------ --- ------
DataData inin DisparateDisparate SourcesSources
---
---
---
---
---
---
---
---
------
---
------
- - -
------
---
---
---
Legacy
ERP
CRM
Best-of-breed Applications
--- --- --- --- --- ------ --- ------ - - - ------ --- --- --- Legacy
3 Pillars of Data Integration Async cnyS 4

3 Pillars of Data Integration

Async cnyS
Async
cnyS
3 Pillars of Data Integration Async cnyS 4

4

Enterprise Information Integration The Traditional Approach Source Load Transform Extract Target Data Warehouse

Enterprise Information Integration

The Traditional Approach

Source Load Transform Extract Target Data Warehouse Applications
Source
Load
Transform
Extract
Target Data
Warehouse
Applications

ETL “processes” often use batch processing approaches

Example: Customer nightly batch runs can take > 24 hours!

“Services” that operate on data are not easily reusable in other contexts

ETL “Services” and “Processes” are insecure and hard to monitor (i.e. no SLA)

reusable in other contexts • ETL “Services” and “Processes” are insecure and hard to monitor (i.e.
reusable in other contexts • ETL “Services” and “Processes” are insecure and hard to monitor (i.e.
Challenges In Data Integration 1. 2. 3. 4. CHALLENGE Increasing data volumes; decreasing batch windows

Challenges

In Data Integration

1.

2.

3.

4.

CHALLENGE

Increasing data volumes; decreasing batch windows

Non-integrated integration

Complexity, manual effort of conventional ETL design

Lack of knowledge capture

batch windows Non-integrated integration Complexity, manual effort of conventional ETL design Lack of knowledge capture 6
Oracle Data Integrator Based on Technology from Data Movement and Transformation from Multiple Sources to

Oracle Data Integrator

Based on Technology from

Oracle Data Integrator Based on Technology from Data Movement and Transformation from Multiple Sources to Heterogeneous

Data Movement and Transformation from Multiple Sources to Heterogeneous Targets

 

B E N E F I T

D I F F E R E N T I A T O R

1

Best Performance

Heterogeneous “E-LT”

2

Productivity

Declarative Design

3

Real-time Integration

Declarative CDC

4

Hot-Pluggable

Knowledge Modules

5

Future Proof

The Chosen Integration Technology of Oracle Fusion

4 Hot-Pluggable Knowledge Modules 5 Future Proof The Chosen Integration Technology of Oracle Fusion 7
4 Hot-Pluggable Knowledge Modules 5 Future Proof The Chosen Integration Technology of Oracle Fusion 7
Typical Considerations for ODI • High volume data synchronization • more than 20MB/min • Heterogeneous

Typical Considerations for ODI

High volume data synchronization

more than 20MB/min

Heterogeneous data sources

DB2/AS400, Oracle, Excel, File, SQL, BAM…

Capture new data changes regardless of data sources

CDC using Native Journal, LogMiner or Trigger…

Real-time data synchronization

Easy to implement the solution without changing your current IT infrastructure

No separate server required

• Easy to implement the solution without changing your current IT infrastructure • No separate server
Challenges & Emerging Solutions In Data Integration   CHALLENGE EMERGING SOLUTION 1. Increasing data

Challenges & Emerging Solutions

In Data Integration

 

CHALLENGE

EMERGING SOLUTION

1.

Increasing data volumes; decreasing batch windows

Shift from E-T-L to E-LT

2.

Non-integrated integration

Convergence of integration solutions

3.

Complexity, manual effort of conventional ETL design

Shift from custom coding to declarative design

4.

Lack of knowledge capture

Shift to pattern-driven development

from custom coding to declarative design 4. Lack of knowledge capture Shift to pattern-driven development 9
from custom coding to declarative design 4. Lack of knowledge capture Shift to pattern-driven development 9
1 1
1
1

E-LT Architecture

High Performance

Transform in Separate ETL Server

Proprietary Engine

Poor Performance

High Costs

Transform in Existing RDBMS

Leverage Resources

Efficient

High Performance

Benefits

Optimal Performance & Scalability Easier to Manage & Lower Cost

Conventional ETL Architecture

Extract Transform Load
Extract
Transform
Load

Eg. Informatica, IBM Datastage

Next Generation Architecture ““E-LTE-LT”” Transform Transform Extract Load Oracle Data Integrator
Next Generation Architecture
““E-LTE-LT””
Transform
Transform
Extract
Load
Oracle Data Integrator
Next Generation Architecture ““E-LTE-LT”” Transform Transform Extract Load Oracle Data Integrator 10
1 1
1
1

Traditional E-T-L

Technical Detail

Need one powerful server for Transform Server and for its staging data tables

High total cost for maintenance

It is not flexible when we add more source and target data sources

Require coding

Bad performance (more I/O among staging tables and source/target)

Conventional ETL Architecture Transform Extract Server Load S1 Target 1 S2 ETL DB S3 Repository
Conventional ETL Architecture
Transform
Extract
Server
Load
S1
Target 1
S2
ETL DB
S3
Repository
----
----
----
----
Staging tables
1 1
1
1

Next General Architecture: E-LT

Technical Detail

Leverage resources for transformation for high performance, less I/O, and license

Design data flow by pre-defined templates, open for all types of data sources (drag & drop)

Capture changes data for near real-time data synchronization

No coding required

E-LT Architecture

synchronization • No coding required E-LT Architecture Target 1 Transform Staging tables S1 Extract Load ODI
Target 1 Transform Staging tables
Target 1
Transform
Staging tables
S1
S1

Extract

Load

Target 1 Transform Staging tables S1 Extract Load ODI Agent S2 S3 ---- ---- ---- ----
ODI Agent
ODI Agent

ODI Agent

ODI Agent
S2
S2

S3

---- ---- ---- ----
----
----
----
----
S1 Extract Load ODI Agent S2 S3 ---- ---- ---- ---- For scheduling and real-time monitoring

For scheduling and real-time monitoring changes only

ODI Designer

S2 S3 ---- ---- ---- ---- For scheduling and real-time monitoring changes only ODI Designer No

No need at production

12

2 2
2
2

Active Integration

Batch, Event-based, and Service-oriented Integration

Evolve from Batch to Near Real-time Warehousing on Common Platform

Unify the Silos of Data Integration

Data Integrity on the Fly

Services Plug into Oracle SOA Suite

Benefits

Oracle Data Integrator

EventEvent ConductorConductor ServiceService ConductorConductor Event-oriented Service-oriented Integration
EventEvent ConductorConductor
ServiceService ConductorConductor
Event-oriented
Service-oriented
Integration
Integration
Metadata
Declarative Design
Data-oriented Integration
DataData ConductorConductor

Enables real-time data warehousing and operational data hubs Services plug into Oracle SOA Suite for comprehensive integration

data warehousing and operational data hubs Services plug into Oracle SOA Suite for comprehensive integration 13
data warehousing and operational data hubs Services plug into Oracle SOA Suite for comprehensive integration 13
3 3
3
3

Declarative Design

Developer Productivity

Specify ETL Data Flow Graph

Developer must define every step of Complex ETL Flow Logic

Traditional approach requires specialized ETL skills

And significant development and maintenance efforts

Declarative Set-based Design

Simplifies the number of steps

Automatically generates the Data Flow whatever the sources and target DB

Benefits

Significantly reduce the learning curve

Shorter implementation times

Streamline access to non-IT pros

Conventional ETL Design

Conventional ETL Design
ODI Declarative Design 11 22 Define Automatically What Generate You Want Dataflow Define How: Built-in
ODI Declarative Design
11
22
Define
Automatically
What
Generate
You Want
Dataflow
Define How: Built-in Templates
Declarative Design 11 22 Define Automatically What Generate You Want Dataflow Define How: Built-in Templates 14
Declarative Design 11 22 Define Automatically What Generate You Want Dataflow Define How: Built-in Templates 14
4 4
4
4

Pluggable Data Integration Architecture

Hot-Pluggable: Modular, Flexible, Extensible

Pluggable Architecture

Reverse Engineer Metadata Journalize Read from CDC Source Load From Sources to Staging Check Constraints
Reverse
Engineer Metadata
Journalize
Read from CDC
Source
Load
From Sources to
Staging
Check
Constraints before
Load
Integrate
Transform and Move
to Targets
Service
Expose Data and
Transformation
Services
Reverse
WS WS WS
Staging Tables
Load
Integrate
Services
CDC
Target Tables
Check
Journalize
Sources

Benefits

Tailor to existing best practices Ease administration work

Depend on the specific data source, we will select right pre-defined coding module (Knowledge Module) -> Hot-Pluggable Support all types of data sources (DB2/AS400, Oracle, Excel, File…)

Reduce cost of ownership

-> Hot-Pluggable Support all types of data sources (DB2/AS400, Oracle, Excel, File…) Reduce cost of ownership
-> Hot-Pluggable Support all types of data sources (DB2/AS400, Oracle, Excel, File…) Reduce cost of ownership
4 4
4
4

Knowledge Modules

Hot-Pluggable: Modular, Flexible, Extensible

Pluggable Knowledge Modules Architecture

Reverse Engineer Metadata Journalize Read from CDC Source Load From Sources to Staging Check Constraints
Reverse
Engineer Metadata
Journalize
Read from CDC
Source
Load
From Sources to
Staging
Check
Constraints before
Load
Integrate
Transform and Move
to Targets
Service
Expose Data and
Transformation
Services
Reverse
WS WS WS
Staging Tables
Load
Integrate
Services
CDC
Target Tables
Check
Journalize
Sources
Error Tables

Sample out-of-the-box Knowledge Modules

SQL Server Oracle Check MS TPump/ Oracle Web SAP/R3 Log Miner JMS Queues Oracle Merge
SQL Server
Oracle
Check MS
TPump/
Oracle Web
SAP/R3
Log Miner
JMS Queues
Oracle Merge
Triggers
DBLink
Excel
Multiload
Services
Oracle
Check
Siebel EIM
DB2 Web
Siebel
DB2 Journals
DB2 Exp/Imp
Type II SCD
SQL*Loader
Sybase
Schema
Services

Benefits

Tailor to existing best practices

Ease administration work

Reduce cost of ownership

Schema Services Benefits Tailor to existing best practices Ease administration work Reduce cost of ownership 16
Schema Services Benefits Tailor to existing best practices Ease administration work Reduce cost of ownership 16
4 4
4
4

KMs: Truly Heterogeneous

Generic SQL DB

Oracle DB 9i

Oracle DB 10g

Oracle DB 10g XE

IBM DB2/400

IBM DB2/UDB

IBM Informix SE

IBM LDAP Server

MS SQL Server 2000

MS SQL Server 2005

MS SQL Server 2005 SE

MS Office Access 2000

MS Office Excel 2000

MS Active Directory

Sybase ASA 8.x & 9.x

Sybase IQ 12.x

Sonic MQ v7.0

Teradata V2R5.x

Teradata V2R6.x

Netezza Performance Server 2.2.1

Hyperion Essbase

PostgresSQL 8.1

MySQL 4.0

MySQL 5.0

Oracle BI Suite 10g

Oracle BAM 10g

Oracle Internet Directory 9i

OpenLDAP 2.3

Siebel CRM 7.8

JD Edwards

PeopleSoft

SAP R/3

Oracle EBusiness Suite

Oracle AQ 10g

Oracle SOA Suite

Oracle ESB 10g

SalesForce.com App Exchange

Any JMS Standard Implementation

Out-of-Box Knowledge Modules
Out-of-Box
Knowledge
Modules
Oracle ESB 10g • SalesForce.com App Exchange • Any JMS Standard Implementation Out-of-Box Knowledge Modules 17
Popular Usage Scenarios 18

Popular Usage Scenarios

Popular Usage Scenarios 18

Operational

Analytics

Operational Analytics E-LT for Data Warehouse Create Data Warehouse for Business Intelligence Populate Warehouse with High

E-LT for Data Warehouse

Create Data Warehouse for Business Intelligence Populate Warehouse with High Performance ODI

Load Transform Capture Changes
Load
Transform
Capture Changes
Incremental Update Data Integrity
Incremental
Update
Data Integrity
Aggregate Export
Aggregate
Export
Cube ---- ---- Cube ---- Data Warehouse ---- Cube
Cube
----
----
Cube
----
Data Warehouse
----
Cube

Metadata

Data Transformation

Data Warehousing

Heterogeneous sources and targets Incremental load Slowly changing dimensions Data integrity and consistency Changed data capture Data lineage

load Slowly changing dimensions Data integrity and consistency Changed data capture Data lineage 19
ODI for Master Data Management Common Data Quality, and Middleware Services Master Data Management Telco

ODI for Master Data Management

Common Data Quality, and Middleware Services

Master Data Management Telco Telco Energy Energy Banking Banking Retail Retail Mfr Mfr Industry ….
Master Data Management
Telco Telco
Energy Energy
Banking Banking
Retail Retail
Mfr Mfr
Industry
…. …. Solutions
MDM
Applications
Customer Customer Supplier Supplier Employee Employee
Product Product
Asset Asset
…. ….
Fusion Middleware Foundation
Oracle Data Integrator
Golden
Master
E-LT
Records
E-LT Agent
Metadata
Other
Oracle
Siebel
PeopleSoft
SAP/R3
Sources
EBS
CRM

Solutions & Applications

Vertical Driven Data Object Centric Application Focus

Middleware Foundation

Process Orchestration Business Intelligence Registry & Policies Data Integration & Quality

Oracle Data Integrator

Batch & Real-time Integration Data Quality & Profiling Transformation & Data Routing

Data Integrator Batch & Real-time Integration Data Quality & Profiling Transformation & Data Routing 20
ODI Enhances Oracle BI Populate Warehouse with High Performance ODI Oracle BI Suite EE Interactive

ODI Enhances Oracle BI

Populate Warehouse with High Performance ODI

Oracle BI Suite EE Interactive Answers Publisher Delivers Dashboards Oracle BI Presentation Server Oracle BI
Oracle BI Suite EE
Interactive
Answers
Publisher
Delivers
Dashboards
Oracle BI Presentation Server
Oracle BI Server
Oracle BI Enterprise Data Warehouse
Oracle BI
Enterprise Data
Warehouse

Bulk E-LT

Oracle Data Integrator E-LT E-LT Agent Metadata
Oracle Data Integrator
E-LT
E-LT Agent
Metadata
PeopleSoft

PeopleSoft

Data Integrator E-LT E-LT Agent Metadata PeopleSoft SAP/R3 Oracle EBS Oracle Business Intelligence Suite EE:
SAP/R3

SAP/R3

SAP/R3
Integrator E-LT E-LT Agent Metadata PeopleSoft SAP/R3 Oracle EBS Oracle Business Intelligence Suite EE:
Oracle EBS

Oracle

EBS

Oracle Business Intelligence Suite EE:

Simplified Business Model View Advanced Calculation & Integration Engine Intelligent Request Generation Optimized Data Access

Oracle Data Integrator:

Populate Enterprise Data Warehouse Optimized Performance for Load and Transform Extensible Pre-packaged E-LT Content

Other Sources
Other
Sources
Siebel CRM

Siebel

CRM

Optimized Performance for Load and Transform Extensible Pre-packaged E-LT Content Other Sources Siebel CRM 21
ODI Enhances Oracle SOA Suite Add Bulk Data Transformation to BPEL Process Oracle SOA Suite

ODI Enhances Oracle SOA Suite

Add Bulk Data Transformation to BPEL Process

Oracle SOA Suite Business Activity BPEL Process Manager Monitoring Web Services Manager Declarative Rules Engine
Oracle SOA Suite
Business Activity
BPEL Process Manager
Monitoring
Web Services
Manager
Declarative Rules
Engine
Enterprise Service
Bus
Oracle Data Integrator
E-LT
E-LT Agent
Metadata
Bulk Data
Processing

Oracle SOA Suite:

BPEL Process Manager for Business Process Orchestration

Oracle Data Integrator:

Efficient Bulk Data Processing as Part of Business Process Interact via Data Services and Transformation Services

Efficient Bulk Data Processing as Part of Business Process Interact via Data Services and Transformation Services
ODI with BAM Populate BAM with ETL Data Efficiently Oracle SOA Suite Business Activity Monitoring

ODI with BAM

Populate BAM with ETL Data Efficiently

Oracle SOA Suite Business Activity Monitoring Event Monitoring Web Applications BPEL Process Manager Web Services
Oracle SOA Suite
Business Activity Monitoring
Event Monitoring Web Applications
BPEL Process
Manager
Web Services
Manager
Business Rules
Engine
Event Engine
Report Cache
Enterprise Service
Bus
Active Data Cache
Oracle Data Integrator
Bulk and
Real-Time
Agent
Metadata
Data Processing
Message
Queues
Data
CDC
Warehouse
PeopleSoft
SAP/R3

Oracle SOA Suite

Business Activity Monitoring for Real-time Business Insight Message-based, event- driven, memory-resident architecture

Oracle Data Integrator

High Performance Loading of BAM’s Active Data Cache Pre-built and Integrated via Knowledge Modules BAM Java APIs Exposed through “Interface” Like Any Other Target

Sample Combined Use Cases

Monitor Together Events and the Aggregate Implications of Events

Like Any Other Target Sample Combined Use Cases Monitor Together Events and the Aggregate Implications of
Integration with SOA/BI/Fusion Resolve All Integration Challenges Oracle BPA and Human Workflow Invoke Oracle BI

Integration with SOA/BI/Fusion

Resolve All Integration Challenges

Oracle BPA and Human Workflow

All Integration Challenges Oracle BPA and Human Workflow Invoke Oracle BI Dashboards, Reporting, Analysis,

Invoke

Oracle BI

Challenges Oracle BPA and Human Workflow Invoke Oracle BI Dashboards, Reporting, Analysis, Publishing Invoke Invoke
Challenges Oracle BPA and Human Workflow Invoke Oracle BI Dashboards, Reporting, Analysis, Publishing Invoke Invoke
Dashboards, Reporting, Analysis, Publishing
Dashboards, Reporting,
Analysis, Publishing

Invoke

Invoke

Oracle Data Integrator Oracle BAM Transformation Data Services Services Invoke E-LT Agent Metadata Repository
Oracle Data Integrator
Oracle BAM
Transformation
Data Services
Services
Invoke
E-LT Agent
Metadata
Repository
Active
Knowledge
Data Cache
Modules

BPEL Process

Manager

Knowledge Data Cache Modules BPEL Process Manager Invoke Invoke WSDL Generate Data Service as High speed

Invoke

Invoke

WSDL
WSDL
Cache Modules BPEL Process Manager Invoke Invoke WSDL Generate Data Service as High speed High speed
Cache Modules BPEL Process Manager Invoke Invoke WSDL Generate Data Service as High speed High speed

Generate Data

Service as

High speed

High speed

Services

Data Source

Batch ELT

JMS ELT

XML
XML
Services Data Source Batch ELT JMS ELT XML Oracle BI Enterprise Data Warehouse Oracle JMS CDC
Oracle BI Enterprise Data Warehouse

Oracle BI

Enterprise Data

Warehouse

Oracle JMS
Oracle JMS

CDC based

ELT

Data Source Batch ELT JMS ELT XML Oracle BI Enterprise Data Warehouse Oracle JMS CDC based
CDC

CDC

Data Source Batch ELT JMS ELT XML Oracle BI Enterprise Data Warehouse Oracle JMS CDC based
Data Source Batch ELT JMS ELT XML Oracle BI Enterprise Data Warehouse Oracle JMS CDC based
Data Source Batch ELT JMS ELT XML Oracle BI Enterprise Data Warehouse Oracle JMS CDC based
Performance 25

Performance

Performance 25
ODI vs. ESB Recommended Considered Can use 26

ODI vs. ESB

ODI vs. ESB Recommended Considered Can use 26

Recommended

Considered

Can use

ODI vs. ESB Recommended Considered Can use 26
ODI vs. ESB Recommended Considered Can use 26
Performance Report Source and Target: 2 dual core CPU, 12GB RAM 27

Performance Report

Performance Report Source and Target: 2 dual core CPU, 12GB RAM 27

Source and Target: 2 dual core CPU, 12GB RAM

27
27
ODI with ESB Data Latency Batch (over 2 hours) Asynchronous Synchronous (immediate) Oracle Data Integrator

ODI with ESB

Data

Latency

Batch (over 2 hours)

ODI with ESB Data Latency Batch (over 2 hours) Asynchronous Synchronous (immediate) Oracle Data Integrator Oracle

Asynchronous

ODI with ESB Data Latency Batch (over 2 hours) Asynchronous Synchronous (immediate) Oracle Data Integrator Oracle
ODI with ESB Data Latency Batch (over 2 hours) Asynchronous Synchronous (immediate) Oracle Data Integrator Oracle

Synchronous

(immediate)

Batch (over 2 hours) Asynchronous Synchronous (immediate) Oracle Data Integrator Oracle Enterprise Service Bus
Oracle Data Integrator Oracle Enterprise Service Bus Real-life Scenarios
Oracle Data Integrator
Oracle Enterprise Service Bus
Real-life
Scenarios

Message by

Mini Batches

Message
Message

Large Volume (over 1M)

Service Bus Real-life Scenarios Message by Mini Batches Message Large Volume (over 1M) Data Volume Processing

Data Volume Processing

Service Bus Real-life Scenarios Message by Mini Batches Message Large Volume (over 1M) Data Volume Processing
Service Bus Real-life Scenarios Message by Mini Batches Message Large Volume (over 1M) Data Volume Processing
Understanding Performance Choices When you need to transform data at large size Depends on whether

Understanding Performance Choices

When you need to transform data at large size

Depends on whether an intermediary XML format is useful for other processing (use ESB), or
Depends on whether an
intermediary XML format
is useful for other
processing (use ESB),
or if joining File data to
tabular RDB data is
required (use ODI)
Depends on ho much cross-referencing among the data values and rows is required during transformation
Depends on ho much
cross-referencing
among the data values
and rows is required
during transformation –
the more there is, the
faster ODI will perform
relative to ESB
If the source and target are both XML, and there is no cross-referencing of data
If the source and target
are both XML, and there
is no cross-referencing
of data among rows,
then a streaming-type or
parallel-engine-type
approach might scale

Less than 10MB

(target)

XML

File

DB

(source)

XML

 

ESB

ESB

ESB

 

File

 

ESB

ESB

depends

 

DB

 

ESB

depends

ODI

Between 10-50MB

(target)

XML

File

DB

(source)

XML

 

depends

depends

ODI

 

File

 

depends

ODI

ODI

 

DB

 

ODI

ODI

ODI

Greater than 50MB

(target)

XML

File

DB

(source)

XML

 

depends

ODI

ODI

 

File

 

ODI

ODI

ODI

 

DB

 

ODI

ODI

ODI

ODI ODI ODI   DB   ODI ODI ODI *caveat – always benchmark if you are
ODI ODI ODI   DB   ODI ODI ODI *caveat – always benchmark if you are

*caveat – always benchmark if you are unsure and require best possible results

29

Topology 1 – Oracle to Oracle Vietnamese Customer PoC Hardware: Quad Core/4 GB RAM Oracle

Topology 1 – Oracle to Oracle

Vietnamese Customer PoC

Hardware: Quad Core/4 GB RAM

Oracle 10.2+/Linux
Oracle 10.2+/Linux

Oracle 10.2+/Linux

Data Synchronization
Data Synchronization

ODI Designer

Oracle 10.2+/Win Repositories Agent
Oracle 10.2+/Win
Repositories
Agent

Hardware: Dual Core/2 GB RAM

Oracle 10.2+/Linux Data Synchronization ODI Designer Oracle 10.2+/Win Repositories Agent Hardware: Dual Core/2 GB RAM
Performance Results • 100k rows, 15 fields • Load: LKM DBLink 3s • Real-time synchronization

Performance Results

100k rows, 15 fields

Load: LKM DBLink 3s

Real-time synchronization (JKM DBLink)

Update 65k: 13s

Delete 30k: 8s

1.2m rows, 8 fields (about 120 bytes/row)

Load: LKM DBLink 24s, JDBC 4.5 minutes

Real-time synchronization (JKM DBLink)

Update 5000 rows, 8s

Delete 5000 rows, 8s

24s, JDBC 4.5 minutes • Real-time synchronization (JKM DBLink) • Update 5000 rows, 8s • Delete
Real-time Synchronization with CDC CPU Usage • Without CDC: CPU 10%, 1s-1.5s • Enable CDC

Real-time Synchronization with CDC

CPU Usage

Without CDC: CPU 10%, 1s-1.5s

Enable CDC (LogMiner) and Use AgentScheduler

CPU 2%, 1s-1.5s

Scenario with 1.2m rows

Update 3900 rows, CPU 23%, 2s

Delete 3900 rows, CPU 21%, 2s

• CPU 2%, 1s-1.5s • Scenario with 1.2m rows • Update 3900 rows, CPU 23%, 2s
Summary 35

Summary

Summary 35
Oracle Data Integrator Data Movement and Transformation from Multiple Sources to Heterogeneous Targets   B

Oracle Data Integrator

Data Movement and Transformation from Multiple Sources to Heterogeneous Targets

 

B E N E F I T

D I F F E R E N T I A T O R

1

Best Performance

Heterogeneous “E-LT”

2

Productivity

Declarative Design

 

Real-time

 

3

Integration

Declarative CDC

4

Hot-Pluggable

Knowledge Modules

5

Future Proof

The Chosen Integration Technology of Oracle Fusion

4 Hot-Pluggable Knowledge Modules 5 Future Proof The Chosen Integration Technology of Oracle Fusion 36
4 Hot-Pluggable Knowledge Modules 5 Future Proof The Chosen Integration Technology of Oracle Fusion 36
Reference Customers 37

Reference Customers

Reference Customers 37
Customer: Solution: Technology: Overstock.com High-Volume Real-Time Data Transformation Oracle Data Integrator,
Customer:
Solution:
Technology:
Overstock.com
High-Volume Real-Time Data Transformation
Oracle Data Integrator, Oracle 9i & 10g RAC,
Dell Linux, IBM AIX, Teradata 8-node 54000
Oracle Data Integrator Solution:
“Having access to key business metrics in real-time is no
longer a fantasy.”
“Oracle Data Integrator is helping us
turn our data into gold”
“Data Integrator allows us to perform data
transformations using the power of our Teradata
Enterprise Warehousing platform. […] With Oracle,
over 300 users are now able to have access to their
relevant data in real-time, hourly, daily, or weekly
depending upon their needs.”
• Found a way to ensure that Teradata data warehouse
was constantly updated.
• Even highly complex transformations are
automated within the
• Supporting several terabytes of data stored in the
enterprise warehouse, and millions of daily transactions
“In short, Oracle Data Integrator give us the ability to make
better decisions and better manage our bottom line.”

Business Problem:

Solution Architecture:

Wanted to enable sales, finance, marketing and merchandising teams to have access to near real-time data so that they could make timely, more intelligent business decisions.

Wanted to know at any point in time if company performance is meeting the target metrics.

Needed a data integration product that could handle our high-volume loading and transformation requirements in near real time.

Data Sources, Targets, and Platforms

Oracle 9i RAC & 10g RAC

GoldenGate TDM Transactional Management

Data Integration Architecture

Teradata 8-node 54000

Platforms:

IBM AIX, Dell Linux

Oracle Data Integrator: 100% Java architecture, high-performance E- LT transformations, business-rules driven transformation design tool, automatic load script generation

>1.2M SKU’s, > 5M daily transactions, >300 users, deployable for both batch and real-time use cases, leverages power of Teradata engine for improved speed of data transformation

Company: Overstock.com

Overstock.com, Inc. (NASDAQ: OSTK) operates as an online retailer offering bed-and-bath goods, furniture, watches, jewelry, electronics, sporting goods, and designer accessories.

Product: Oracle Data Integrator Contact: Miranda Nash Email: miranda.nash@oracle.com

and designer accessories. Product : Oracle Data Integrator Contact : Miranda Nash Email : miranda.nash@oracle.com 38
Customer: Solution: Technology: Sabre Holdings High-Volume Real-Time Data Transformation Oracle Data Integrator,
Customer:
Solution:
Technology:
Sabre Holdings
High-Volume Real-Time Data Transformation
Oracle Data Integrator, Oracle DB, MQ
sources, Teradata Data Warehouse target
Oracle Data Integrator Solution:
“We needed a data integration tool
that would reduce our
dependency on manual coding of
E-LT scripts and leverage the
power of our Teradata Warehouse
for data transformation.”
• E-LT architecture maximizes performance and
leverages existing investment in Teradata
infrastructure
• Lower development and maintenance costs for
E-LT driven by declarative design tools
• Bottom Line: Integrated travel industry data in
consolidated view enables Sabre to better serve
their customers and travel suppliers

Business Problem:

Solution Architecture:

Data Sources, Targets, and Platforms

High costs associated with Data Warehouse loading from new sources

Large Teradata Data Warehouse requires top performance for loading data in near-real time

Integrated views of data require complex transformations, expensive to maintain

Oracle RDBMS

Teradata Data Warehouse

Flat Files

Various other sources over MQ

Data Integration Architecture

Oracle Data Integrator: 100% Java architecture, high-performance E- LT transformations, business-rules driven transformation design tool, automatic load script generation

Company: Sabre Holdings

For more than 40 years, Sabre Holdings (NYSE: TSG) has transformed the airline industry through technological advancement, the Company offers a portfolio of travel marketing, distribution and technology solutions.

Product: Oracle Data Integrator Contact: Miranda Nash Email: miranda.nash@oracle.com

and technology solutions. Product : Oracle Data Integrator Contact : Miranda Nash Email : miranda.nash@oracle.com 39
Customer: Solution: Technology: DHL High-Volume Real-Time Data Transformation Oracle Data Integrator, Oracle
Customer:
Solution:
Technology:
DHL
High-Volume Real-Time Data Transformation
Oracle Data Integrator, Oracle RDBMS’s,
Teradata Data Warehouse, Cobol Flat Files…
Oracle Data Integrator Solution:
• With Oracle Data Integrator, every batch that used
to last one hour now lasts seconds
“Solution completely meets our
needs.” […] Oracle Data Integrator
was developed by ETL developers,
who really know and understand
ETL concerns and pains, and how
to do things better.”
• Reducing window time is critical to adding more
functionality
• Running mini-batches more often results in more
customer services and more revenue
• Using the RDBMS as an engine for data
transformation simplifies the administrative workload

Business Problem:

Solution Architecture:

Data Sources, Targets, and Platforms

24/7 business cannot be compromised by long ETL batches (via an ETL Tool)

Every daily load cannot last more than one hour

When the volume of data doubles, execution time triples

Data Integration was the bottleneck in providing more services

Oracle RDBMS

Teradata Data Warehouse

Flat Files

Platforms:

Linux, Cobol

Data Integration Architecture

Oracle Data Integrator: 100% Java architecture, high-performance E- LT transformations, business-rules driven transformation design tool, automatic load script generation

2.5 terabytes loaded every 15 minutes from 8 major data sources >50 events, >5 shipments and > piece/parcel records per day

Company: DHL

For more than 35 years, DHL has built the world's premier global delivery network by trailblazing express shipping in one country after another. Over 220 countries and territories later, DHL is the global market leader of the international express and logistics industry.

Product: Oracle Data Integrator Contact: Miranda Nash Email: miranda.nash@oracle.com

and logistics industry. Product : Oracle Data Integrator Contact : Miranda Nash Email : miranda.nash@oracle.com 40
Customer: Solution: Technology: iBasis High-Volume Real-Time Data Transformation Oracle Data Integrator, Oracle 10g,
Customer:
Solution:
Technology:
iBasis
High-Volume Real-Time Data Transformation
Oracle Data Integrator, Oracle 10g, Netezza
PowerCenter NPS8350 Warehouse Appliance
Oracle Data Integrator Solution:
“The first thing that struck us
was the speed with which we
ramped up our ETL
developments with Oracle
Data Integrator.”
"Given the massive volumes of data we need to
process every day, getting timely data in the data
warehouse requires high performance loading
processes. Using Oracle Data Integrator’s set of
Knowledge Modules for Netezza, we are able to
take advantage of the massively parallel processing
capabilities of Netezza and to reduce load times
significantly. […] as our goal is to go more and
more toward real-time, it will be easy for us to
change the latency of these flows – without having
to redevelop them."

Business Problem:

Solution Architecture:

Data warehouse had become obsolete and could not respond to the growing requirements of management, sales, and operational centers

Needed more accurate and timely data

Replaced entire Data Warehouse infrastructure

Needed a data integration that would provide the scalability and performance they needed to aggregate, transform, and load their data

Data Sources, Targets, and Platforms

Oracle RDBMS

Netezza PowerCenter NPS8350

Flat Files

Applications (future):

Call Billing, Network Monitoring

Data Integration Architecture

Oracle Data Integrator: 100% Java architecture, high-performance E- LT transformations, business-rules driven transformation design tool, automatic load script generation

4.5TB data warehouse, > 8 billion records, company processes >150 million transactions per day

Company: iBasis

Founded in 1996, iBasis (NASDAQ: IBAS) is one of the largest carriers of international voice traffic in the world and a leading provider of prepaid calling services.

Product: Oracle Data Integrator Contact: Miranda Nash Email: miranda.nash@oracle.com

prepaid calling services. Product : Oracle Data Integrator Contact : Miranda Nash Email : miranda.nash@oracle.com 41
Analysts Coverage 42

Analysts Coverage

Analysts Coverage 42
Gartner “Sunopsis (Oracle) has made strides in building market awareness beyond its base in Europe.

Gartner

“Sunopsis (Oracle) has made strides in building market awareness beyond its base in Europe. Sunopsis has a range of capabilities, spanning ETL and real-time messaging, and an architecture that enables distribution of transformation workload across data sources and targets.”

Ted Friedman, Bill Gassman, “Magic Quadrant for Extraction, Transformation and Loading, 1H05”, May 11, 2005

Ted Friedman, Bill Gassman, “Magic Quadrant for Extraction, Transformation and Loading, 1H05”, May 11, 2005 43
Bloor Research “While there are many relatively young vendors within the ETL market, Sunopsis has

Bloor Research

“While there are many relatively young vendors within the ETL market, Sunopsis has undoubtedly made the biggest impression, both in terms of the users that it has gained and in the way that its approach has influenced the market.”

Philip Howard, “Bullseye Report - Extract, Transform & Load”, March 28, 2006

has influenced the market.” Philip Howard, “Bullseye Report - Extract, Transform & Load”, March 28, 2006
Gartner By purchasing Sunopsis, Oracle has acquired a server-independent and platform-independent data integration tool,

Gartner

By purchasing Sunopsis, Oracle has acquired a server-independent and platform-independent data integration tool, which will be renamed Oracle Data Integrator (ODI). OFM and Oracle Applications customers will welcome the addition of the ODI's database independence. In particular, the acquisition could provide needed new momentum for Fusion Middleware. Fusion Middleware customers have heterogeneous IT environments, as do former PeopleSoft, Siebel Systems and JD Edwards customers, who have an ongoing requirement for integration with non- Oracle systems. The acquisition will provide OFM with a data integration tool that is capable of deploying small-grained data services within a service-oriented architecture (SOA) environment. This capability could have a positive influence on Fusion Middleware - if Oracle leverages the Sunopsis philosophy.

Mark A. Beyer, Ted Friedman “Sunopsis Data Integration May Fuel Oracle Fusion Middleware” October 23, 2006

Mark A. Beyer, Ted Friedman “Sunopsis Data Integration May Fuel Oracle Fusion Middleware” October 23, 2006
Forrester Research “Oracle has recognized that its customers require diverse data integration features without having

Forrester Research

“Oracle has recognized that its customers require diverse data integration features without having to integrate and manage products from many vendors. Integrating Sunopsis’ heterogeneous extract, load, transform (ELT) and event-driven CDC capabilities within its middleware offerings is a great start.”

Rob Karel “Oracle Makes Serious Move In Data Heterogeneity by Acquiring Sunopsis” October 29, 2006

great start.” Rob Karel “Oracle Makes Serious Move In Data Heterogeneity by Acquiring Sunopsis” October 29,

46