F1439567061 Bryson 158

T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.
com
Stewart Bryson, Managing Director, Rittman Mead America
RMOUG Training Days, Denver, February 2011
Real-Time DW with OBIEE and Oracle Database
Monday, January 24, 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Who am I?
Stewart Bryson, Managing Director, Rittman
Mead America
Oracle BI/DW Architect and Delivery Specialist
Oracle Database and BI Stack since 1996
Regular Community Speaker
Writer for Rittman Mead Blog:
http://www.rittmanmead.com/blog
Developer of Transcend Framework
Email : stewart.bryson@rittmanmead.com
Twitter : @stewartbryson
Who is Rittman Mead?
Oracle BI&DW Experts
Consulting, Training and Support
Global Company, Offices in the US, UK and India
Clients in the UK, USA, Europe, Middle-East
Voted UKOUG BI Partner of the Year 2008
16 Sessions at Oracle Open World 2010
Twitter : @rittmanmead
Rittman Mead Blog
Longest-running Oracle (BI) Blog
OBIEE, OWB, BI, data warehousing, Hyperion, etc.
Step-by-step demonstrations
Test cases
Product evaluations
Authors include
Mark Rittman (ACE Director)
Venkat Janakiraman (ACE)
Stewart Bryson
Borkur Steingrimsson
Peter Scott
Jon Mead
Rittman Mead Oracle BI Training Days 2011
Running in Atlanta, Brussels, Johannesburg and Bangalore, Winter 2011
Three days of intense OBIEE 11gR1 hands-on training
Led by the Rittman Mead 11g Beta Testing team
Limited places still available: http://www.rittmanmead.com/trainingdayswinter2011
What will this session answer?
What do I mean by real-time data warehousing?
Zero latency, or just reduced latency?
Do I really mean right-time data warehousing?
What can I use to reduce latency?
What does OBIEE offer?
What does the Oracle Database (and OWB)
offer?
Can I blend these different offerings?
Of course you can!
Its the name of the presentation
What do I mean by real-time?
Traditional data warehouses
Daily to weekly batch-load cycles
Key-metrics are always delayed
Information may not be actionable
Real-time data warehouses
Up to the minute or hour reporting
Key-metrics are moving
Information is actionable
Business Case
We want to be able to segment our reporting
between real-time needs and static reports
Most users will want real-time numbers
delivered with dashboards
Some users (financial-minded) dont want there
numbers to change each time they run a report
An interval will always exist between the occurrence of a measurable
event and our ability to process that event as a reportable fact
An interval will always exist between the occurrence of a measurable
event and our ability to process that event as a reportable fact
GETTING REAL
Peter Scott says companies have to be sure of the need before they insist on real-time business
intelligence.
My car does real-time computing (or at least hope the engine management, traction control and ABS systems are real time!).
My mobile phone is real time, even some of my home entertainment gadgets are real time but what about business
intelligence systems?
n the early days of B (or was it called decision support back then?) reporting data for the current month was an achievement.
As techniques and technologies evolved, companies moved to report on the previous day but were still reliant on an out-of-
hours batch process to move data from the source systems and store and aggregate it in their data warehouses and data
marts.
But with 24-hour business days and the need to report across multiple time-zones, the traditional batch window is being
squeezed into near non-existence. New data extraction and load paradigms have also been developed to trickle-feed reporting
systems. But are these really real time and does it really matter?
n reality, there is always going to be a degree of latency between an event happening and it being recorded on a transactional
system, propagated to the reporting system, loaded, and finally aggregated on a data warehouse before being ready to query.
Add to that the delay between the user issuing a query that accesses this new nugget of information and the result being
returned to the user, and you find a discernible lag between event and observation.
True, there are things you can do to mitigate this lag but there will always be a couple of steps in the process where reducing
one time period increases another for example, you can shorten report execution time by aggregating the data to better
answer the query, but this step adds more time to the data load and aggregate phase.
Furthermore, different types of incoming data can have differing latencies. For example, a new transaction may have a shorter
latency than a change to a regional grouping of stores that may need to be propagated through many layers of historical
aggregation tables (see Figure 1).
The division between data acquisition and data presentation presents a key question that any company needs to answer when
contemplating moving to a real-time B system: are you just interested in providing, as rapidly as possible, information based
on already acquired data, or do you also need to report on newly added factual information?
For example, a worker in a mobile phone company's call
centre may like to see information about 'how good a
customer is' and the likelihood that someone within that
customer's demographic would switch to another
supplier.
This information needs to be presented quickly (whilst they
are speaking to them) and preferably within the context of the
line-of-business application that manages the customer
interaction. However it is unlikely that any of the metrics used
by the call centre agent for customer profitability and
predicted churn would be influenced by events of the last few
hours.
Conversely, a system that is using embedded B functionality
to alert to potential fraud (stock markets and credit card
companies are possible example users) may well need to
know about up-to-the-moment activity.
www.evaIuationcentre.com
FIGURE 1: BI latency
Recording Propagation Aggregation Query
Event Report
Real-time query latency
Latency versus Query Performance
OLTP
Reporting
Federated
OLTP/EDW
Reporting
EDW with
Realtime
Component
Standard
EDW
OLTP
Reporting
Federated
OLTP/EDW
Reporting
EDW with
Realtime
Component
Standard
EDW
Latency
Query Performance
OLTP
Reporting
Federated
OLTP/EDW
Reporting
EDW with
Realtime
Component
Standard
EDW
Latency
Query Performance
Oracle Next-Generation Reference Architecture
Federated OLTP/EDW Reporting
Layering in OLTP data to provide the real-time
component
General approach is to use a BI tool such as
OBIEE
No real database or modeling involvement
Classic Batch Load Star Schema: Process Flow
Classic Batch Load Star Schema: Fact Table Load
Federation with OBIEE
OBIEE Metadata and Data Layers
Relational, OLAP, File and
Application Data Sources
Physical Data Sources, such
as relational databases,
OLAP Cubes and application
source adapters
A Semantic Model made up
of Metrics, dimensions,
hierarchies and calculations,
derived from...
source adapters
Enterprise Semantic Model
The Web Catalog contains
definitions of reports, alerts
dashboards, KPIs,
based on...
A Semantic Model made up
of Metrics, dimensions,
hierarchies and calculations,
derived from...
source adapters
Web Catalog
Enterprise Semantic Model
Flow of Data Through the Three-Layer Semantic Model
Flow of Data Through the Three-Layer Semantic Model
Simplification of the Data Model
Integration of Disparate DataSources
Addition of Business Logic and Calculations
Logical Table Sources Map to Physical Models
WITH
SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2,
T43901.CUST_LAST_NAME as c3,
T43971.SAL_AMT as c4
from
GCBC_CRM.CUSTOMERS T43901,
GCBC_POS.POS_TRANS T43971,
GCBC_POS.POS_TRANS_HEADER T43978
where ( T43901.CUST_ID = T43978.CUST_ID and T43971.TRANS_ID = T43978.TRANS_ID and TO_DATE
('2010-09-18 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T43978.TRANS_DATE )
union all
select T44042.CUSTOMER_FIRST_NAME as c2,
T44042.CUSTOMER_LAST_NAME as c3,
T44105.AMOUNT as c4
from
GCBC_EDW.CUSTOMER_DIM T44042,
GCBC_EDW.SALES_FACT T44105
where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY ) )),
SAWITH1 AS (select sum(D3.c4) as c1,
D3.c2 as c2,
D3.c3 as c3
from
SAWITH0 D3
group by D3.c2, D3.c3)
select distinct 0 as c1,
D2.c2 as c2,
D2.c3 as c3,
D2.c1 as c4
from
SAWITH1 D2
WITH
from
union all
T44105.AMOUNT as c4
from
D3.c2 as c2,
D3.c3 as c3
from
SAWITH0 D3
D2.c2 as c2,
D2.c3 as c3,
D2.c1 as c4
from
SAWITH1 D2
WITH
from
union all
T44105.AMOUNT as c4
from
D3.c2 as c2,
D3.c3 as c3
from
SAWITH0 D3
D2.c2 as c2,
D2.c3 as c3,
D2.c1 as c4
from
SAWITH1 D2
WITH
SAWITH0 AS (select sum(T44105.AMOUNT) as c1,
T44042.CUSTOMER_FIRST_NAME as c2,
T45416.CALENDAR_YEAR as c4
from
GCBC_EDW.DATE_DIM T45416,
where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T45416.DATE_KEY and
T45416.SQL_DATE = TO_DATE('2003-01-09 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') )
group by T44042.CUSTOMER_FIRST_NAME, T44042.CUSTOMER_LAST_NAME, T45416.CALENDAR_YEAR)
D1.c2 as c2,
D1.c3 as c3,
D1.c4 as c4,
D1.c1 as c5
from
SAWITH0 D1
WITH
SAWITH0 AS (select sum(T44105.AMOUNT) as c1,
T44042.CUSTOMER_FIRST_NAME as c2,
T45416.CALENDAR_YEAR as c4
from
GCBC_EDW.DATE_DIM T45416,
where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T45416.DATE_KEY and
T45416.SQL_DATE = TO_DATE('2003-01-09 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') )
group by T44042.CUSTOMER_FIRST_NAME, T44042.CUSTOMER_LAST_NAME, T45416.CALENDAR_YEAR)
D1.c2 as c2,
D1.c3 as c3,
D1.c4 as c4,
D1.c1 as c5
from
SAWITH0 D1
The BI Server understands from the Fragmentation
content how to optimize the query
Only one LTS is needed to provide the answer
Logical Table Sources Respect the Physical Model
Fragmentation with OLTP and EDW
Demonstration
Strengths
Low latency
No complex modeling or ETL
Incorporated into any standard dimensional
EDW
Weaknesses
Performance
Difficult to conform multiple systems
EDW with a Real-Time Component
Modeling complimentary real-time structures in an
EDW
Use functionality of the Oracle Database and OWB
Complex ETL and Database interaction
Requirement: Identifying and
Capturing Change
Requirement: Change Data Capture
Identify
Change
Oracle CDC
OWB Tickle-feed
mappings
Trigger-based
capture
Recording
modification dates
MINUS Command
Identify
Change
Oracle CDC
OWB Tickle-feed
mappings
Trigger-based
capture
Recording
modification dates
MINUS Command
Propagate
Change
Oracle CDC
OWB Tickle-feed
mappings
Message Queues
Views over
database links
File movements
Identify
Change
Oracle CDC
OWB Tickle-feed
mappings
Trigger-based
capture
Recording
modification dates
MINUS Command
Propagate
Change
Oracle CDC
OWB Tickle-feed
mappings
Message Queues
Views over
database links
File movements
Apply Change
to Target
OWB Mappings
and Process Flows
Other ETL Tools
Custom ETL code
Custom transaction
code
Identify
Change
Oracle CDC
OWB Tickle-feed
mappings
Trigger-based
capture
Recording
modification dates
MINUS Command
Propagate
Change
Oracle CDC
OWB Tickle-feed
mappings
Message Queues
Views over
database links
File movements
Apply Change
to Target
OWB Mappings
and Process Flows
Other ETL Tools
Custom ETL code
Custom transaction
code
Oracle Asynchronous Change Data Capture
Hot Log
Online redo logs mined locally
Change tables are updated and stored locally
Hot Log Distributed
Online redo logs are mined locally
Change tables are updated and stored remotely
Autolog Online
Online redo logs are mined remotely
Autolog Archive
Archived redo logs are mined remotely
Hot Log
Online redo logs mined locally
Change tables are updated and stored locally
L
a
t
e
n
c
y
Hot Log Distributed
Online redo logs are mined locally
Autolog Online
Online redo logs are mined remotely
Autolog Archive
Archived redo logs are mined remotely
Identify
Change
Oracle CDC
OWB Tickle-feed
mappings
Trigger-based
capture
Recording
modification dates
MINUS Command
Propagate
Change
Oracle CDC
OWB Tickle-feed
mappings
Message Queues
Views over
database links
File movements
Apply Change
to Target
OWB Mappings
and Process Flows
Other ETL Tools
Custom ETL code
Custom transaction
code
Identify
Change
Oracle CDC
OWB Tickle-feed
mappings
Trigger-based
capture
Recording
modification dates
MINUS Command
Propagate
Change
Oracle CDC
OWB Tickle-feed
mappings
Message Queues
Views over
database links
File movements
Apply Change
to Target
OWB Mappings
and Process Flows
Other ETL Tools
Custom ETL code
Custom transaction
code
Identify
Change
Propagate
Change
Apply Change
to Target
Identify
Change
Propagate
Change
Apply Change
to Target
Trickle-feed mappings allow the development of Streams
Capture and Apply processes as continuously running
mappings.
The Real-Time Partition
Table
Partition
Table
Partition
Table
Partition
Fact Table
Table
Partition
Table
Partition
Table
Partition
Fact Table
Physical
Table
Fact Real-Time
Partition
Table
Partition
Table
Partition
Table
Partition
Fact Table
Physical
Table
Fact Real-Time
Partition
Table
Partition
Table
Partition
Table
Partition
Fact Table
Physical
Table
Fact Real-Time
Partition
Physical
Table
All ETL processing of fact records are directed at
the Real-Time Partition using Micro-Batch
No indexes, small data set
Partition Exchange at the end of the day
Physical
Table
Table
Partition
Table
Partition
Table
Partition
Physical
Table
Table
Partition
Table
Partition
Table
Partition
Physical
Table
Table
Partition
Early-Arriving Facts
Grab the most recent dimension record when that
entity already exists
Use a default Not Known Yet record when they
dont
Prior to the partition exchange, we run our late-
arriving dimension loads
We update the Real-Time Partition with the correct
surrogate keys
OWB generates an efficient MERGE
Small data sets in the Real-Time Partition
What if Type 2 dimension changes havent arrived yet?
Real-Time Integration: Fact Table RT Load
Real-Time Integration: Fact Table RT Load
Real-Time Integration: Correct Dimensions
Real-Time Integration: Correction Process Flow
Plugging it together with OBIEE (Non-Fragmentation)
Using standard database
functionality (UNION ALL)
Could be done with a view
Using Surrogate Keys in the
RT Partition
OBIEE doesnt have to
manage different LTSs
RT Partition
RT Partition
EDW with a Real-Time Component
Demonstration
Summary
Latency versus performance and scalability
Latency and performance versus degree of
difficulty
Different variations on the two test cases
Stewart Bryson, Managing Director, Rittman Mead America
RMOUG Training Days, Denver, February 2011
Real-Time DW with OBIEE and Oracle Database

F1439567061 Bryson 158

Загружено:

Сведения о документе

Исходное описание:

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

F1439567061 Bryson 158

Загружено:

Авторское право:

Доступные форматы

T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.

Вам также может понравиться