Академический Документы
Профессиональный Документы
Культура Документы
Lecture-1 Lecture-2 Lecture-3 Motivation: Why data mining? What is data mining? Data Mining: On what kind of data? Lecture-4 Lecture-5 Data mining functionality Classification of data mining systems Lecture-6 Major issues in data mining
Lecture-1
Lecture-1 Motivation
Lecture-1 Motivation
Lecture-1 Motivation
1990s-present:
Lecture-1 Motivation
Lecture-1 Motivation
Data Cleaning
Data Integration Databases
Choosing the mining algorithms Data mining: search for patterns of interest Pattern evaluation and knowledge presentation
Pattern evaluation
Knowledge-base
Filtering
Databases
Data Warehouse
Making Decisions
Data Presentation Visualization Techniques Data Mining Information Discovery Data Exploration Statistical Analysis, Querying and Reporting
End User
Data Warehouses / Data Marts OLAP, MDA Data Sources Paper, Files, Information Providers, Database Systems, OLTP
Lecture-2 What is Data Mining?
DBA
Object-oriented and object-relational databases Spatial databases Time-series data and temporal data Text databases and multimedia databases Heterogeneous and legacy databases WWW
Lecture-3 Data Mining: On What kind of data?
Data can be associated with classes or concepts Ex. AllElectronics store classes of items for sale include computer and printers. Description of class or concept called class/concept description.
Data characterization
Data discrimination
Multi-dimensional vs. single-dimensional association age(X, 20..29) ^ income(X, 20..29K) => buys(X, PC) [support = 2%, confidence = 60%] contains(T, computer) => contains(x, software) [support=1%, confidence=75%]
Lecture-4 Data Mining Funcionalities
Finding models (functions) that describe and distinguish data classes or concepts for predict the class whose label is unknown E.g., classify countries based on climate, or classify cars based on gas mileage Models: decision-tree, classification rules (ifthen), neural network Prediction: Predict some unknown or
Analyze class-labeled data objects, clustering analyze data objects without consulting a known class label. Clustering based on the principle: maximizing the intra-class similarity and minimizing the interclass similarity
Outlier: a data object that does not comply with the general behavior of the model of the data It can be considered as noise or exception but is quite useful in fraud detection, rare events analysis
Trend and deviation: regression analysis Sequential pattern mining, periodicity analysis Similarity-based analysis
Information Science
Data Mining
MachineLearning
Visualization
Other Disciplines
Efficiency and scalability of data mining algorithms Parallel, distributed and incremental mining methods
Handling relational and complex types of data Mining information from heterogeneous databases and global information systems (WWW)
Lecture-7
What is Data Warehouse?
Data WarehouseSubjectOriented
Organized around major subjects, such as customer, product, sales. Focusing on the modeling and analysis of data for decision makers, not on daily operations or transaction processing. Provide a simple and concise view around particular subject issues by excluding data that are not useful in the decision support process.
Data WarehouseIntegrated
Constructed by integrating multiple, heterogeneous data sources
Ensure consistency in naming conventions, encoding structures, attribute measures, etc. among different data sources
E.g., Hotel price: currency, tax, breakfast covered, etc.
Data warehouse data: provide information from a historical perspective (e.g., past 5-10 years)
Contains an element of time, explicitly or implicitly But the key of operational data may or may not contain time element.
Data WarehouseNon-Volatile
A physically separate store of data transformed
from the operational environment. Operational update of data does not occur in the data warehouse environment.
Requires only two operations in data accessing: initial loading of data and access of data.
User and system orientation: customer vs. market Data contents: current, detailed vs. historical, consolidated Database design: ER + application vs. star + subject View: current, local vs. evolutionary, integrated Access patterns: update vs. read-only but complex queries
DBMS tuned for OLTP: access methods, indexing, concurrency control, recovery Warehousetuned for OLAP: complex OLAP queries, multidimensional view, consolidation.
missing data: Decision support requires historical data which operational DBs do not typically maintain data consolidation: DS requires consolidation (aggregation, summarization) of data from heterogeneous sources data quality: different sources typically use inconsistent data representations, codes and formats which have to be reconciled
Lecture-8
0-D(apex) cuboid
1-D cuboids
time,item
time,location
item,location item,supplier
location,supplier
time,supplier time,item,location
2-D cuboids
time,location,supplier
3-D cuboids
item,location,supplier
time,item,supplier
4-D(base) cuboid
time, item, location, supplier
Star schema: A fact table in the middle connected to a set of dimension tables
similar to snowflake
Fact constellations: Multiple fact tables share dimension tables, viewed as a collection of stars,
item
Sales Fact Table time_key item_key branch_key
item_key item_name brand type supplier_type
branch
branch_key branch_name branch_type
location
location_key street city province_or_street country
location_key units_sold
dollars_sold
avg_sales
Measures
time
supplier
supplier_key supplier_type
time_key
item_key branch_key
branch
branch_key branch_name branch_type
location
location_key street city_key
location_key
units_sold dollars_sold avg_sales Measures
city
city_key city province_or_street country
time
time_key day day_of_the_week month quarter year
item_key
shipper_key
from_location
location
location_key street city province_or_street country
branch
branch_key branch_name branch_type
First time as cube definition define dimension <dimension_name> as <dimension_name_first_time> in cube <cube_name_first_time>
define dimension time as (time_key, day, day_of_week, month, quarter, year) define dimension item as (item_key, item_name, brand, type, supplier(supplier_key, supplier_type))
algebraic: if it can be computed by an algebraic function with M arguments (where M is a bounded integer), each of which is obtained by applying a distributive aggregate function.
E.g., avg(), min_N(), standard_deviation().
country
Germany
...
Spain
Canada
...
Mexico
city office
Frankfurt
...
Toronto
M. Wind
Multidimensional Data
Sales volume as a function of product, month, and region Dimensions: Product, Location, Time
Hierarchical summarization paths Industry Region Year
Product
Product
City Office
Month
Date
3Qtr 4Qtr
Country
date
product,country
country
1-D cuboids
date, country
product,date
OLAP Operations
Roll up (drill-up): summarize data
by climbing up hierarchy or by dimension reduction from higher level summary to lower level summary or detailed data, or introducing new dimensions project and select
OLAP Operations
Pivot (rotate):
Other operations
Lecture-9
Data warehouse architecture
Top-down view
allows selection of the relevant information necessary for the data warehouse
Top-down: Starts with overall design and planning (mature) Bottom-up: Starts with experiments and prototypes (rapid) Waterfall: structured and systematic analysis at each step before proceeding to the next Spiral: rapid generation of increasingly functional systems, short turn around time, quick turn around
Choose a business process to model, e.g., orders, invoices, etc. Choose the grain (atomic level of data) of the business process Choose the dimensions that will apply to each fact table record Choose the measure that will populate each fact table record
Multi-Tiered Architecture
Monitor & Integrator OLAP Server
other
Metadata
sources
Operational Extract Transform Load Refresh
DBs
Data Warehouse
Serve
Data Marts
Data Sources
Data Storage
Metadata Repository
Meta data is the data defining warehouse objects. It has the following kinds
Operational meta-data
data lineage (history of migrated data and transformation path), currency of data (active, archived, or purged), monitoring information (warehouse usage statistics, error reports, audit trails)
The algorithms used for summarization The mapping from operational environment to the data warehouse Data related to system performance
warehouse schema, view and derived data definitions
Business data
business terms and definitions, ownership of data, charging policies
Data Mart
Data Mart
Model refinement
Model refinement
Use relational or extended-relational DBMS to store and manage warehouse data and OLAP middle ware to support missing pieces Include optimization of DBMS backend, implementation of aggregation navigation logic, and additional tools and services greater scalability
Array-based multidimensional storage engine (sparse matrix techniques) fast indexing to pre-computed summarized data
The bottom-most cuboid is the base cuboid The top-most cuboid (apex) contains only one cell How many cuboids in an n-dimensional cube with L levels?
n T ( Li 1) i 1
Materialize every (cuboid) (full materialization), none (no materialization), or some (partial materialization) Selection of which cuboids to materialize
Based on size, sharing, access frequency, etc.
Cube Operation
Cube definition and computation in DMQL
define cube sales[item, city, year]: sum(sales_in_dollars) compute cube sales
Transform it into a SQL-like language (with a new operator cube by, introduced by Gray et al.96)
SELECT item, city, year, SUM (amount)
FROM SALES
CUBE BY item, city, year
(city) Need compute the following Group-Bys
()
(item)
(year)
(date, product, customer), (date,product),(date, customer), (product, customer), (city, item) (city, year) (date), (product), (customer) ()
(city, item, year)
(item, year)
ROLAP-based cubing algorithms (Agarwal et al96) Array-based cubing algorithm (Zhao et al97) Bottom-up computation method (Bayer & Ramarkrishnan99)
Sorting, hashing, and grouping operations are applied to the dimension attributes in order to reorder and cluster related tuples Grouping is performed on some sub aggregates as a partial grouping step
Aggregates may be computed from previously computed aggregates, rather than from the base fact table
b3
b2
9
5 1
b1
b0
a0
a1
a2
a3
Multi-Way Array Aggregation for Cube Computation Method: the planes should be sorted and computed according to their size in ascending order.
Idea: keep the smallest plane in the main memory, fetch and compute only one chunk at a time for the largest plane
Limitation of the method: computing well only for a small number of dimensions
If there are a large number of dimensions, bottom-up computation and iceberg cube computation methods can be explored
Index on Region
Index on Type
Type RecIDAsia Europe America RecID Retail Dealer Retail 1 1 0 1 1 0 0 Dealer 2 2 0 1 0 1 0 Dealer 3 1 0 0 3 0 1 Retail 4 0 0 1 4 1 0 0 1 0 5 0 1 Dealer 5
It materializes relational join in JI file and speeds up relational join a rather costly operation
In data warehouses, join index relates the values of the dimensions of a start schema to rows in the fact table.
E.g. fact table: Sales and two dimensions city and product A join index on city maintains for each distinct city a list of R-IDs of the tuples recording the Sales in the city Join indices can span multiple dimensions
transform drill, roll, etc. into corresponding SQL and/or OLAP operations, e.g, dice = selection + projection
Determine to which materialized cuboid(s) the relevant operations should be applied. Exploring indexing structures and compressed vs. dense array structures in MOLAP
Information processing
supports querying, basic statistical analysis, and reporting using crosstabs, tables, charts and graphs
Analytical processing multidimensional analysis of data warehouse data supports basic OLAP operations, slice-dice, drilling, pivoting
Data mining knowledge discovery from hidden patterns supports associations, constructing analytical models, performing classification and prediction, and presenting the mining results using visualization tools.
High quality of data in data warehouses DW contains integrated, consistent, cleaned data Available information processing structure surrounding data warehouses ODBC, OLEDB, Web accessing, service facilities, reporting and OLAP tools OLAP-based exploratory data analysis mining with drilling, dicing, pivoting, etc. On-line selection of data mining functions integration and swapping of multiple mining functions, algorithms, and tasks.
Architecture of OLAM
An OLAM Architecture
Mining query
User GUI API
Mining result
OLAM Engine
Data Cube API
OLAP Engine
Layer3
OLAP/OLAM
Layer2
MDDB
Meta Data
Filtering&Integration
MDDB
Database API
Data cleaning
Filtering