Академический Документы
Профессиональный Документы
Культура Документы
Agenda
Data warehousing –
overview
Raw
Data
[Barry Devlin]
What are the users saying...
A process of
Information
transforming data into
information and
making it available to
users in a timely
enough manner to
make a difference
Data
Evolution
60’s: Batch reports
hard to find and analyze information
inflexible and expensive, reprogram every new
request
70’s: Terminal-based DSS and EIS (executive
information systems)
still inflexible, not integrated with desktop tools
80’s: Desktop data access and analysis tools
query tools, spreadsheets, GUIs
easier to use, but only access operational databases
90’s till now: Data warehousing with
integrated OLAP engines and tools, real time
DW
Data Warehouse
A data warehouse is a
subject-oriented
integrated
time-varying
non-volatile
Accessible
collection of data that is used primarily in
organizational decision making.
-- Bill Inmon, Building the Data Warehouse 1996
Explorers, Farmers and Tourists
Relational
Databases
Optimized Loader
Extraction
ERP
Systems Cleansing
Data Warehouse
Engine Analyze
Purchased Query
Data
Legacy
Data Metadata Repository
Data Mining works with Warehouse
Data
Data Warehousing
provides the Enterprise
with a memory
Function
Missing data: Decision support requires historical data, which
Operational database do not typically maintain.
Application-Orientation Subject-Orientation
Operational Data
Database Warehouse
Credit
Loans Customer
Card
Vendor
Trust Product
Savings Activity
OLTP vs Data Warehouse
The Data
Warehouse helps
to “optimize” the
business
Why Now?
Data is being produced
DW Architecture
Loading, refreshing
Structuring/Modeling
L
One,
company-
wide
T warehouse
T
E
T
E Simpler data access
Single ETL for
enterprise data warehouse Dependent data marts
(EDW) loaded from EDW
The ETL Process
Capture
Transform
Load
Record-level: Field-level:
Selection – data partitioning single-field – from one field to one field
Joining – data combining multi-field – from many fields to one, or
Aggregation – data summarization one field to many
Steps in data reconciliation (continued)
Minimizes risk.
External
Data
SOURCE DATA
Operational Data
Staging Area
External
Data
SOURCE DATA
Operational Data
Data Warehouse
Data Marts
Staging Area
Data Warehouse
External
Data
SOURCE DATA
Staging Area
Operational Data
Data Marts
Database organization
must look like business
must be recognizable by business user
approachable by business user
Must be simple
Schema Types
Star Schema
Fact Constellation Schema
Snowflake schema
42
Conceptual Modeling of
Data Warehouses
Modeling data warehouses: dimensions &
measures
Star schema: A fact table in the middle connected to a
set of dimension tables
Snowflake schema: A refinement of star schema where
some dimensional hierarchy is normalized into a set of
smaller dimension tables, forming a shape similar to
snowflake
Fact constellations: Multiple fact tables share dimension
tables, viewed as a collection of stars, therefore called
galaxy schema or fact constellation
Dimension Tables
Dimension tables
Define business in terms already
familiar to users
Wide rows with lots of descriptive text
Small tables (about a million rows)
Joined to fact table by a foreign key
heavily indexed
typical dimensions
time periods, geographic region (markets,
cities), products, customers, salesperson,
etc.
44
Fact Table
Central table
mostly raw numeric items
narrow rows, a few columns at most
large number of rows (millions to a
billion)
Access via dimensions
45
Example of Star Schema
time
time_key item
day item_key
day_of_the_week Sales Fact Table item_name
month brand
quarter time_key type
year supplier_type
item_key
branch_key
branch location
location_key
branch_key location_key
branch_name units_sold street
branch_type city
dollars_sold province_or_street
country
avg_sales
Measures
Example of Snowflake Schema
time
time_key item
day item_key supplier
day_of_the_week Sales Fact Table item_name supplier_key
month brand supplier_type
quarter time_key type
year item_key supplier_key
branch_key
location
branch location_key
location_key
branch_key
units_sold street
branch_name
city_key city
branch_type
dollars_sold
city_key
avg_sales city
province_or_street
Measures country
Example of Fact Constellation
time
time_key item Shipping Fact Table
day item_key
day_of_the_week Sales Fact Table item_name time_key
month brand
quarter time_key type item_key
year supplier_type shipper_key
item_key
branch_key from_location
Business data
business terms and definitions
ownership of data
charging policies
operational metadata
data lineage: history of migrated data and
sequence of transformations applied
currency of data: active, archived, purged
monitoring information: warehouse usage
statistics, error reports, audit trails.
The BI/DW Lifecycle
Source:
http://www.atre.com/navigator/#3
The BI/DW Lifecycle…
Source: http://www.atre.com
Popular BI/DW Suites & Tools
Oracle • IBM
LDMs & Database Logical Data Model & IBM DB2 Database
Oracle Warehouse Builder
Oracle Discoverer & Oracle Reporting DB2 Cube Views
BI Beans & JOLAP API ETL Ascential DataStage
Microsoft Full Suites DB2 Alphabox
Database • Cognos BI Suite
SQL Server Analysis Services • BusinessObjects & Crystal
SQL Server Reporting Services
SQL Server Integration Services • Microstrategy
• Actuate
Reporting
• SAS 9 – the BI Platform • Hyperion/Brio (Acquired by
Tools
Hyperion)
Logical Data Model & SAS Database
SAS ETL
BI and Reporting
SAS Data Mining
• Informatica • SAP BW
• Ab initio • Peoplesoft EPM
• Any Database SQL Language or any other • Embarcadero Suite
Programming Language • Erwin
• Teradata Databases • Cognos Performance
Specialized
• Redbrick Apps
ETL Tools Tools
• Hyperion Essbase • Planning &
• Oracle Express Server Budgeting
56