Академический Документы
Профессиональный Документы
Культура Документы
Introduction
Motivation: Why data mining? What is data mining? Data Mining: On what kind of data? Data mining functionality Are all the patterns interesting? Classification of data mining systems Data Mining Task Primitives
1990s:
2000s
Alternative names
Target marketing, customer relationship management (CRM), market basket analysis, cross selling, market segmentation
Risk analysis and management
Cross-market analysisFind associations/co-relations between product sales, & predict based on such association Customer profilingWhat types of customers buy what products (clustering or classification) Customer requirement analysis Identify the best products for different customers
Resource planning summarize and compare the resources and spending monitor competitors and market directions group customers into classes and a class-based pricing procedure set pricing strategy in a highly competitive market
8
Competition
Retail industry
Anti-terrorism
9
Data Mining
Data Cleaning
Data Integration Databases
10
11
Decision Making
Data Presentation Visualization Techniques Data Mining Information Discovery
End User
Data Exploration Statistical Summary, Querying, and Reporting Data Preprocessing/Integration, Data Warehouses Data Sources Paper, Files, Web documents, Scientific experiments, Database Systems
DBA
12
Statistics
Machine Learning
Pattern Recognition
Data Mining
Visualization
Algorithm
Other Disciplines
13
Knowledge to be mined
Techniques utilized
Applications adapted
15
Multimedia database
Text databases The World-Wide Web
17
E.g., classify countries based on (climate), or classify cars based on (gas mileage)
Predict some unknown or missing numerical values
18
interesting
Suggested approach: Human-centered, query-based, focused mining A pattern is interesting if it is easily understood by humans, valid on Interestingness measures
20
Can a data mining system find all the interesting patterns? Do we need to find all of the interesting patterns?
Heuristic vs. exhaustive search Association vs. classification vs. clustering Can a data mining system find only the interesting patterns? Approaches First general all the patterns and then filter out the uninteresting ones Generate only the interesting patternsmining query optimization
21
Finding all the patterns autonomously in a database?unrealistic because the patterns could be too many but uninteresting
User directs what to be mined
Users must be provided with a set of primitives to be used to communicate with the data mining system
Incorporating these primitives in a data mining query language More flexible user interaction Foundation for design of graphical user interface Standardization of data mining industry and practice
22
23
24
Interactive mining multi-level knowledge Necessity of mining knowledge and patterns at different levels of abstraction by drilling/rolling, pivoting, slicing/dicing, etc.
Integration of multiple mining functions Characterized classification, first clustering and then association
25
Loose coupling
Fetching data from DB/DW Provide efficient implement a few data mining primitives in a DB/DW system, e.g., sorting, indexing, aggregation, histogram analysis, multiway join, precomputation of some stat functions Semi-tight couplingenhanced DM performance
DM is smoothly integrated into a DB/DW system, mining query is optimized based on mining query, indexing, query processing methods, etc.
26
Database
Integration of the discovered knowledge with existing one: knowledge fusion User interaction Data mining query languages and ad-hoc mining
Applications and social impacts Domain-specific data mining & invisible data mining Protection of data security, integrity, and privacy
28
Basics
Itemset: a set of items
E.g., acm={a, c, m}
Support of itemsets
Sup(acm)=3
Items bought
f, a, c, d, g, I, m, p a, b, c, f, l,m, o b, f, h, j, o b, c, k, s, p a, f, c, e, l, p, m, n
200 Given min_sup=3, 300 acm is a frequent 400 pattern 500 Frequent pattern mining: find all frequent patterns in a database
30
32
Apriori-based Mining
Generate length (k+1) candidate itemsets from length k frequent itemsets, and Test the candidates against DB
33
Apriori Algorithm
A level-wise, candidate-generation-and-test approach (Agrawal & Srikant 1994)
Data base D
TID 10 20 30 40 Items a, c, d b, c, e a, b, c, e b, e
1-candidates
Scan D
Itemset a b c d e Sup 2 3 3 1 3
Freq 1-itemsets
Itemset a b c e Sup 2 3 3 3
2-candidates
Itemset ab ac ae bc be ce
Min_sup=2 3-candidates
Scan D
Itemset bce
Freq 2-itemsets
Itemset ac bc be ce Sup 2 2 3 2
Counting
Itemset ab ac ae bc be ce Sup 1 2 1 2 3 2
Scan D
Freq 3-itemsets
Itemset bce Sup 2
34
return k Lk;
Ck+1 = candidates generated from Lk; for each transaction t in database do increment the count of all candidates in Ck+1 that are contained in t Lk+1 = candidates in Ck+1 with min_support
35
36
Step 2: pruning
INSERT INTO Ck SELECT p.item1, p.item2, , p.itemk-1, q.itemk-1 FROM Lk-1 p, Lk-1 q WHERE p.item1=q.item1, , p.itemk-2=q.itemk-2, p.itemk-1 < q.itemk-1
For each itemset c in Ck do For each (k-1)-subsets s of c do if (s is not in Lk-1) then delete c from Ck
37
Example of Candidate-generation
L3={abc, abd, acd, ace, bcd} Self-joining: L3*L3
abcd from abc and abd acde from acd and ace
Pruning:
acde is removed because ade is not in L3
C4={abcd}
38
Method:
Candidate itemsets are stored in a hash-tree Leaf node of hash-tree contains a list of itemsets and counts Interior node contains a hash table Subset function: finds all the candidates contained in a transaction
39
40
Summary
Data mining: Discovering interesting patterns from large amounts of data A natural evolution of database technology, in great demand, with wide applications A KDD process includes data cleaning, data integration, data selection, transformation, data mining, pattern evaluation, and knowledge presentation Mining can be performed in a variety of information repositories Data mining functionalities: characterization, discrimination, association, classification, clustering, outlier and trend analysis, etc. Data mining systems and architectures Major issues in data mining
41
1995-1998 International Conferences on Knowledge Discovery in Databases and Data Mining (KDD95-98) Journal of Data Mining and Knowledge Discovery (1997) ACM SIGKDD conferences since 1998 and SIGKDD Explorations
Journals
43