13-09-2019

Data Warehouse Data Mining

Rahul Sachdeva

Syllabus

1
 13-09-2019

Syllabus

Data Warehouse and OLAP

• Why data warehouse

• What’s data warehouse
• What’s multi-dimensional data model
• What’s difference between OLAP and
OLTP

CSE601 4

2
 13-09-2019

The Traditional Research Approach

Query-driven (lazy, on-demand)
Clients

Integration System Metadata

...
Wrapper Wrappe Wrapper
r

...
Source Source Source

Disadvantages of Query-Driven
Approach

• Delay in query processing

− Slow or unavailable information sources
− Complex filtering and integration
• Inefficient and potentially expensive for
frequent queries
• Competes with local processing at sources
• Hasn’t caught on in industry

3
 13-09-2019

From DBMS to Decision Support

• DBMSs widely used to maintain transactional data

• Attempts to use of these data for analysis,
exploration, identification of trends etc. has led to
Decision Support Systems.
• Rapid Growth since mid 70’s
• DBMSs vendors have answered this trend by
adding new features to existing products
• Rarely enough

DBs for Decision Support

• Trend towards Data Warehousing

• Data Warehousing – consolidation of data
from several databases which are in turn
maintained by individual business units
along with historical and summary
information

4
 13-09-2019

Characteristics of TPSs

Characteristic OLTP

Typical operation Update

Level of analytical requirements Low

Screens Unchanging

Amount of data per transaction Small

Data level Detailed

Age of data Current

Orientation Records

CSE601 10

10

5
 13-09-2019

CSE601 11

11

Relational Database Theory

• Relational database modeling process –

normalization, relations or tables are progressively
decomposed into smaller relations to a point
where all attributes in a relation are very tightly
coupled with the primary key of the relation.
− First normal form: data items are atomic,
− Second normal form: attributes fully depend on primary
key,
− Third normal form: all non-key attributes are
completely independent of each other.

CSE601 12

12

6
 13-09-2019

Student University Tables

matricN fName lName gender year super Course
um reg visor course credit
121212 Mary Hill F 200 1234 code value
3 c1 120
232323 Steve Gray M 200 1234 c3 60
5 c5 60
123456 Jimm Smith M 200 1111 Enrolled
y 0 course student
code Num
Staff staff first last gender c1 121212
Num Name Name
c3 121212
1234 Jane Smith F
c3 123456
2323 Tom Green M
c1 232323
1111 Jim Brow M
CSE601
n Etc etc Etc etc 13

13

Relation Database Theory, cont’d

• The process of normalization generally

breaks a table into many independent tables.
• A normalized database yields a flexible
model, making it easy to maintain dynamic
relationships between business entities.
• A relational database system is effective
and efficient for operational databases – a
lot of updates (aiming at optimizing update
performance).

CSE601 14

14

7
 13-09-2019

Problems

• A fully normalized data model can perform

very inefficiently for queries.
• Historical data are usually large with static
relationships:
− Unnecessary joins may take unacceptably long
time
• Historical data are diverse

CSE601 15

15

Problem: Heterogeneous Information

Sources
“Heterogeneities are everywhere”
Personal
Databases

World
Scientific Databases
Wide
Web
Digital Libraries

Different interfaces
Different data representations
Duplicate and inconsistent information
CSE601 16

16

8
 13-09-2019

Problem: Data Management in Large

Enterprises
• Vertical fragmentation of informational systems
(vertical stove pipes)
• Result of application (user)-driven development of
operational systems
Sales Planning Suppliers Num. Control
Stock Mngmt Debt Mngmt Inventory
... ... ...

Sales Administration Finance Manufacturing ...

17

Goal: Unified Access to Data

Integration System

World
Wide
Personal
Web
Digital Libraries Scientific Databases Databases

• Collects and combines information

• Provides integrated view, uniform user interface
• Supports sharing
CSE601 18

18

9
 13-09-2019

The Traditional Research Approach

• Query-driven (lazy, on-demand)
Clients

Integration System Metadata

...
Wrapper Wrapper Wrapper

...
Source Source Source

CSE601 19

19

Disadvantages of Query-Driven
Approach

 Delay in query processing

 Slow or unavailable information sources
 Complex filtering and integration
 Inefficient and potentially expensive for
frequent queries
 Competes with local processing at sources
 Hasn’t caught on in industry

CSE601 20

20

10
 13-09-2019

The Warehousing Approach

• Information Clients
integrated in
advance Data
Warehouse
• Stored in wh for
direct querying
Integration System Metadata
and analysis
...
Extractor/ Extractor/ Extractor/
Monitor Monitor Monitor

...
Source Source Source
CSE601 21

21

Advantages of Warehousing Approach

• High query performance
− But not necessarily most current information
• Doesn’t interfere with local processing at sources
− Complex queries at warehouse
− OLTP at information sources
• Information copied at warehouse
− Can modify, annotate, summarize, restructure, etc.
− Can store historical information
− Security, no auditing
• Has caught on in industry
CSE601 22

22

11
 13-09-2019

Not Either-Or Decision

• Query-driven approach still better for

− Rapidly changing information
− Rapidly changing information sources
− Truly vast amounts of data from large numbers
of sources
− Clients with unpredictable needs

CSE601 23

23

A Data Warehouse is...

• Stored collection of diverse data

− A solution to data integration problem
− Single repository of information
• Subject-oriented
− Organized by subject, not by application
− Used for analysis, data mining, etc.
• Optimized differently from transaction-
oriented db
• User interface aimed at executive
24

12
 13-09-2019

A Data Warehouse is...

(continued)
• Large volume of data (Gb, Tb)
• Non-volatile
− Historical
− Time attributes are important
• Updates infrequent
• May be append-only
• Examples
− All transactions ever at WalMart
− Complete client histories at insurance firm
25 − Stockbroker financial information and portfolios

Summary
Business Business Information
Information Guide Interface

Data
Data Warehouse
Warehouse
Catalog
Data Warehouse
Population

Enterprise
Modeling
Operational Systems

26

13
 13-09-2019

Warehouse is a Specialized DB

Standard DB Warehouse
• Mostly updates Mostly reads
• Many small transactions Queries are long and complex
• Mb - Gb of data Gb - Tb of data
• Current snapshot History
• Index/hash on p.k. Lots of scans
• Raw data Summarized, reconciled data
• Thousands of users (e.g., clerical Hundreds of users (e.g.,
users) decision-makers, analysts)

27

Warehousing and Industry

• Warehousing is big business

− $2 billion in 1995
− $3.5 billion in early 1997
− Predicted: $8 billion in 1998 [Metagroup]
• WalMart has largest warehouse
− 900-CPU, 2,700 disk, 23 TB Teradata system
− ~7TB in warehouse
− 40-50GB per day

28

14
 13-09-2019

Types of Data

• Business Data - represents meaning

− Real-time data (ultimate source of all business
data)
− Reconciled data
− Derived data
• Metadata - describes meaning
− Build-time metadata
− Control metadata
− Usage metadata
• Data as a product* - intrinsic meaning
29 − Produced and stored for its own intrinsic value

MIS and Decision Support

Ad hoc access
Production
platforms

Operational reports Decision makers

• MIS systems provided business data

• Reports were developed on request
• Reports provided little analysis capability
• no personal ad hoc access to data

30

15
 13-09-2019

Analyzing Data from Operational

Systems
• Data structures are complex
• Systems are designed for high performance and
ERP throughput
• Data is not meaningfully represented
• Data is dispersed
• TPS systems unsuitable for intensive queries

Production
platforms

Operational reports

31

Data Extract Processing

Operational systemsExtracts Decision makers

• End user computing offloaded from the

operational environment
• User’s own data

32

16
 13-09-2019

Management Issues

Operational systems Extracts Decision makers

Extract explosion
• Duplicated effort
• Multiple technologies
• Obsolete reports
• No metadata
33

Data Quality Issues

• No common time basis

• Different calculation algorithms
• Different levels of extraction
• Different levels of granularity
• Different data field names
• Different data field meanings
• Missing information
• No data correction rules
• No drill-down capability

34

17
 13-09-2019

What is a Data Warehouse?

A Practitioners Viewpoint

“A data warehouse is simply a single,

complete, and consistent store of data
obtained from a variety of sources and made
available to end users in a way they can
understand and use it in a business context.”
-- Barry Devlin, IBM Consultant

CSE601 35

35

What is a Data Warehouse?

An Alternative Viewpoint

“A DW is a
− subject-oriented,
− integrated,
− time-varying,
− non-volatile
collection of data that is used primarily in
organizational decision making.”
-- W.H. Inmon, Building the Data Warehouse, 1992

CSE601 36

36

18
 13-09-2019

The Data Warehouse

• The Data Warehouse is an integrated,
subject-oriented, time-variant, non-volatile
database that provides support for decision
making.

− Integrated
• The Data Warehouse is a centralized, consolidated
database that integrates data retrieved from the
entire organization.

− Subject-Oriented
• The Data Warehouse data is arranged and optimized
to provide answers to questions coming from
diverse functional areas within a company.

37

The Data Warehouse

− Time Variant
• The Warehouse data represent the flow of data
through time. It can even contain projected data.

− Non-Volatile
• Once data enter the Data Warehouse, they are
never removed.
• The Data Warehouse is always growing.

38

19
 13-09-2019

A Data Warehouse is...

• Stored collection of diverse data
− A solution to data integration problem
− Single repository of information
• Subject-oriented
− Organized by subject, not by application
− Used for analysis, data mining, etc.
• Optimized differently from transaction-
oriented db
• User interface aimed at executive

CSE601 39

39

… Cont’d
• Large volume of data (Gb, Tb)
• Non-volatile
− Historical
− Time attributes are important
• Updates infrequent
• May be append-only
• Examples
− All transactions ever at Sainsbury’s
− Complete client histories at insurance firm
− LSE financial information and portfolios
CSE601 40

40

20
 13-09-2019

Generic Warehouse Architecture

Client Client
Query & Analysis

Design Phase Loading

Warehouse Metadata
Maintenance
Integrator Optimization

Extractor/ Extractor/ Extractor/

Monitor Monitor Monitor

...
CSE601 41

41

Data Warehouse Architectures:

Conceptual View
Operational Informational

• Single-layer
systems systems

− Every data element is stored once only “Real-time data”

− Virtual warehouse

• Two-layer Operational Informational

− Real-time + derived data

systems systems

− Most commonly used approach in

Derived Data
industry today
Real-time data

CSE601 42

42

21
 13-09-2019

Three-layer Architecture:
Conceptual View
• Transformation of real-time data to derived
data really requires two steps
Operational Informational
systems systems

View level
“Particular informational
Derived Data
needs”

Reconciled Data
Physical Implementation
of the Data Warehouse

Real-time data

CSE601 43

43

DW vs DM

CSE601 44

44

22
 13-09-2019

CSE601 45

45

CSE601 46

46

23
 13-09-2019

CSE601 47

47

Data Warehousing: Two Distinct

Issues
(1) How to get information into warehouse
“Data warehousing”
(2) What to do with data once it’s in
warehouse
“Warehouse DBMS”
• Both rich research areas
• Industry has focused on (2)

CSE601 48

48

24
 13-09-2019

Issues in Data Warehousing

• Warehouse Design
• Extraction
− Wrappers, monitors (change detectors)
• Integration
− Cleansing & merging
• Warehousing specification & Maintenance
• Optimizations
• Miscellaneous (e.g., evolution)

CSE601 49

49

OLTP vs. OLAP

• OLTP: On Line Transaction Processing

− Describes processing at operational sites
• OLAP: On Line Analytical Processing
− Describes processing at warehouse

CSE601 50

50

25
 13-09-2019

Issues (1)

• Warehouse uses relational data model or

multi-dimensional data model (e.g., data
cube)
• On the other hand, source types
− Relational, OO, hierarchical, legacy
− Semistructured: flat file, WWW
• How do we get the data out?

51

Issues (2)

• Warehouse must be kept current in light of

changes to underlying sources
• How do we detect updates in sources?

52

26
 13-09-2019

Wrapper

Converts data and queries from one data model to

another
Data Queries Data
Model Model
A Data B

Extends query capabilities for sources with

limited capabilities

Queries Wrapper Source

53

Wrapper Generation

• Solution 1: Hard code for each source

• Solution 2: Automatic wrapper generation
Wrapper
Wrapper Definition
Generator

54

27
 13-09-2019

Routine When...
• Many tools for dealing with “standard situations”
− Standard sources with full/many capabilities
• e.g., most commercial DBMSs, all ODBC-compliant
sources
− Standard interactions
• e.g., pass-through queries, extraction from rel. tables,
replication
− Cooperative sources or sources under our control
• Tools
− Replication tools, ODBC, report writers, third-party
“wrappers”

55

Not So Routine When...

• “Non-standard situations”
− Unstructured or semistructured sources with
little or no explicit schema
− Uncooperative sources
− Sources with limited capabilities (e.g., legacy
sources, WWW)
• Few commercial tools
• Mostly research

56

28
 13-09-2019

Data Transformations

• Convert data to uniform format

− Byte ordering, string termination
− Internal layout
• Remove, add & reorder attributes
− Add key
− Add data to get history
• Sort tuples

57

Monitors

• Goal: Detect changes of interest and

propagate to integrator
• How?
− Triggers
− Replication server
− Compare query results
− Compare snapshots/dumps

58

29
 13-09-2019

Data Integration

• Receive data (changes) from multiple wrappers/monitors and

integrate into warehouse
• Rule-based
• Actions
− Resolve inconsistencies
− Eliminate duplicates
− Integrate into warehouse (may not be empty)
− Summarize data
− Fetch more data from sources (wh updates)
− etc.

59

Data Cleansing

• Find (& remove) duplicate tuples

− e.g., Jane Doe vs. Jane Q. Doe
• Detect inconsistent, wrong data
− Attribute values that don’t match
• Patch missing, unreadable data
• Notify sources of errors found

60

30
 13-09-2019

Warehouse is a Specialized DB
Standard DB (OLTP)
• Mostly updates
• Many small transactions
• Mb - Gb of data
• Current snapshot
• Index/hash on p.k.
• Raw data
• Thousands of users (e.g.,
clerical users)
Warehouse (OLAP)
• Mostly reads
• Queries are long and complex
• Gb - Tb of data
• History
• Lots of scans
• Summarized, reconciled data
• Hundreds of users (e.g.,
decision-makers, analysts)

CSE601 61

61

Data Warehouse – Blend of

Technologies

CSE601 62

62

31
 13-09-2019

Decision Support
• Information technology to help the
knowledge worker (executive, manager,
analyst) make faster & better decisions
− “What were the sales volumes by region and product category for
the last year?”
− “How did the share price of comp. manufacturers correlate with
quarterly profits over the past 10 years?”
− “Which orders should we fill to maximize revenues?”

• On-line analytical processing (OLAP) is an

element of decision support systems (DSS)

CSE601 63

63

Three-Tier Decision Support Systems

• Warehouse database server
− Almost always a relational DBMS, rarely flat files
• OLAP servers
− Relational OLAP (ROLAP): extended relational DBMS that
maps operations on multidimensional data to standard
relational operators
− Multidimensional OLAP (MOLAP): special-purpose server
that directly implements multidimensional data and operations
• Clients
− Query and reporting tools
− Analysis tools
− Data mining tools
CSE601 64

64

32
 13-09-2019

The Complete Decision Support

System
Information Sources Data Warehouse OLAP Servers Clients
Server (Tier 2) (Tier 3)
(Tier 1)
e.g., MOLAP
Semistructured Analysis
Sources
Data
Warehouse serve

extract Query/Reporting
transform
load serve
refresh
etc. e.g., ROLAP
Operational
DB’s Data Mining
serve

Data Marts
CSE601 65

65

Data Warehouse vs. Data Marts

• Enterprise warehouse: collects all information about
subjects (customers,products,sales,assets,
personnel) that span the entire organization
− Requires extensive business modeling (may take years to design
and build)
• Data Marts: Departmental subsets that focus on selected
subjects
− Marketing data mart: customer, product, sales
− Faster roll out, but complex integration in the long run
• Virtual warehouse: views over operational dbs
− Materialize sel. summary views for efficient query processing
− Easy to build but require excess capability on operat. db servers
CSE601 66

66

33
 13-09-2019

OLAP for Decision Support

• OLAP = Online Analytical Processing
• Support (almost) ad-hoc querying for business analyst
• Think in terms of spreadsheets
− View sales data by geography, time, or product
• Extend spreadsheet analysis model to work with
warehouse data
− Large data sets
− Semantically enriched to understand business terms
− Combine interactive queries with reporting functions
• Multidimensional view of data is the foundation of
OLAP
− Data model, operations, etc.
CSE601 67

67

Approaches to OLAP Servers

• Relational DBMS as Warehouse Servers
• Two possibilities for OLAP servers
• (1) Relational OLAP (ROLAP)
− Relational and specialized relational DBMS to
store and manage warehouse data
− OLAP middleware to support missing pieces
• (2) Multidimensional OLAP (MOLAP)
− Array-based storage structures
− Direct access to array data structures

CSE601 68

68

34
 13-09-2019

OLAP Server: Query Engine

Requirements
• Aggregates (maintenance and querying)
− Decide what to precompute and when
• Query language to support
multidimensional operations
− Standard SQL falls short
• Scalable query processing
− Data intensive and data selective queries

CSE601 69

69

35