Advanced Querying and Information Retrival

Chapter18:DataAnalysisandMining
DatabaseSystemConcepts
Silberschatz,KorthandSudarshan Seewww.dbbook.comforconditionsonreuse
Chapter18:DataAnalysisandMining
s DecisionSupportSystems s DataAnalysisandOLAP s DataWarehousing s DataMining
DatabaseSystemConcepts5thEdition,Aug26,2005
18.<number>
Silberschatz,KorthandSudarshan
DecisionSupportSystems
s Decisionsupportsystemsareusedtomakebusinessdecisions,often
basedondatacollectedbyonlinetransactionprocessingsystems.
q q q
s Examplesofbusinessdecisions:
Whatitemstostock? Whatinsurancepremiumtochange? Towhomtosendadvertisements? Retailsalestransactiondetails Customerprofiles(income,age,gender,etc.)
s Examplesofdatausedformakingdecisions
q q
18.<number>
DecisionSupportSystems:Overview
s DataanalysistasksaresimplifiedbyspecializedtoolsandSQL
extensions
q
Exampletasks Foreachproductcategoryandeachregion,whatwerethetotal salesinthelastquarterandhowdotheycomparewiththesame quarterlastyear Asabove,foreachproductcategoryandeachcustomercategory
s Statisticalanalysispackages(e.g.,:S++)canbeinterfacedwith
databases
q
Statisticalanalysisisalargefield,butnotcoveredhere
s Dataminingseekstodiscoverknowledgeautomaticallyintheformof
statisticalrulesandpatternsfromlargedatabases.
s Adatawarehousearchivesinformationgatheredfrommultiplesources,
andstoresitunderaunifiedschema,atasinglesite.
q q
Importantforlargebusinessesthatgeneratedatafrommultiple divisions,possiblyatmultiplesites Datamayalsobepurchasedexternally

18.<number> Silberschatz,KorthandSudarshan
DataAnalysisandOLAP
s OnlineAnalyticalProcessing(OLAP)
q
Interactiveanalysisofdata,allowingdatatobesummarizedand viewedindifferentwaysinanonlinefashion(withnegligibledelay)
s Datathatcanbemodeledasdimensionattributesandmeasure
attributesarecalledmultidimensionaldata.
q
Measureattributes

measuresomevalue canbeaggregatedupon e.g.theattributenumberofthesalesrelation definethedimensionsonwhichmeasureattributes(or aggregatesthereof)areviewed e.g.theattributesitem_name,color,andsizeofthesales relation
Dimensionattributes
18.<number>
CrossTabulationofsalesbyitemname andcolor
s Thetableaboveisanexampleofacrosstabulation(crosstab),also
referredtoasapivottable.
q q q q
Valuesforoneofthedimensionattributesformtherowheaders Valuesforanotherdimensionattributeformthecolumnheaders Otherdimensionattributesarelistedontop Valuesinindividualcellsare(aggregatesof)thevaluesofthe dimensionattributesthatspecifythecell.

RelationalRepresentationofCrosstabs
s Crosstabscanberepresented
asrelations
s Weusethevalueallisusedto
representaggregates
s TheSQL:1999standard
actuallyusesnullvaluesin placeofalldespiteconfusion withregularnullvalues
18.<number>
DataCube
s Adatacubeisamultidimensionalgeneralizationofacrosstab s Canhavendimensions;weshow3below s Crosstabscanbeusedasviewsonadatacube
18.<number>
OnlineAnalyticalProcessing
s Pivoting:changingthedimensionsusedinacrosstabiscalled s Slicing:creatingacrosstabforfixedvaluesonly
q
Sometimescalleddicing,particularlywhenvaluesformultiple dimensionsarefixed.
s Rollup:movingfromfinergranularitydatatoacoarsergranularity s Drilldown:Theoppositeoperationthatofmovingfromcoarser
granularitydatatofinergranularitydata
18.<number>
HierarchiesonDimensions
s Hierarchyondimensionattributes:letsdimensionstobeviewed
atdifferentlevelsofdetail
5 E.g.thedimensionDateTimecanbeusedtoaggregatebyhourof day,date,dayofweek,month,quarteroryear
18.<number>
CrossTabulationWithHierarchy
s Crosstabscanbeeasilyextendedtodealwithhierarchies
5 Candrilldownorrolluponahierarchy
18.<number>
OLAPImplementation
s TheearliestOLAPsystemsusedmultidimensionalarraysinmemoryto
storedatacubes,andarereferredtoasmultidimensionalOLAP (MOLAP)systems. relationalOLAP(ROLAP)systems
s OLAPimplementationsusingonlyrelationaldatabasefeaturesarecalled s Hybridsystems,whichstoresomesummariesinmemoryandstorethe
basedataandothersummariesinarelationaldatabase,arecalled hybridOLAP(HOLAP)systems.
18.<number>
OLAPImplementation(Cont.)
s EarlyOLAPsystemsprecomputedallpossibleaggregatesinorderto
provideonlineresponse
q
Spaceandtimerequirementsfordoingsocanbeveryhigh 2ncombinationsofgroupby Itsufficestoprecomputesomeaggregates,andcomputeotherson demandfromoneoftheprecomputedaggregates
Cancomputeaggregateon(itemname,color)fromanaggregate on(itemname,color,size) Forallbutafewnondecomposableaggregatessuchas median
ischeaperthancomputingitfromscratch s Severaloptimizationsavailableforcomputingmultipleaggregates
q q
Cancomputeaggregateon(itemname,color)fromanaggregateon (itemname,color,size) Cancomputeaggregateson(itemname,color,size), (itemname,color)and(itemname)usingasinglesorting ofthebasedata

ExtendedAggregationinSQL:1999
s Thecubeoperationcomputesunionofgroupbysoneverysubsetofthe
specifiedattributes
s E.g.considerthequery
selectitemname,color,size,sum(number) fromsales groupbycube(itemname,color,size) Thiscomputestheunionofeightdifferentgroupingsofthesalesrelation: {(itemname,color,size),(itemname,color), (itemname,size),(color,size), (itemname),(color), (size),()} where()denotesanemptygroupbylist.

s Foreachgrouping,theresultcontainsthenullvalue
forattributesnotpresentinthegrouping.
18.<number>
ExtendedAggregation(Cont.)
s Relationalrepresentationofcrosstabthatwesawearlier,butwithnullin
placeofall,canbecomputedby
selectitemname,color,sum(number) fromsales groupbycube(itemname,color)

s Thefunctiongrouping()canbeappliedonanattribute
q
Returns1ifthevalueisanullvaluerepresentingall,andreturns0inall othercases.
selectitemname,color,size,sum(number), grouping(itemname)asitemnameflag, grouping(color)ascolorflag, grouping(size)assizeflag, fromsales groupbycube(itemname,color,size)

s Canusethefunctiondecode()intheselectclausetoreplace
suchnullsbyavaluesuchasall
q
E.g.replaceitemnameinfirstqueryby decode(grouping(itemname),1,all,itemname)
s Therollupconstructgeneratesuniononeveryprefixofspecifiedlistof
attributes
s E.g.
selectitemname,color,size,sum(number) fromsales groupbyrollup(itemname,color,size) Generatesunionoffourgroupings: {(itemname,color,size),(itemname,color),(itemname),()}

s Rollupcanbeusedtogenerateaggregatesatmultiplelevelsofa
hierarchy.
s E.g.,supposetableitemcategory(itemname,category)givesthe
categoryofeachitem.Then
selectcategory,itemname,sum(number) fromsales,itemcategory wheresales.itemname=itemcategory.itemname groupbyrollup(category,itemname) wouldgiveahierarchicalsummarybyitemnameandbycategory.
18.<number>
s Multiplerollupsandcubescanbeusedinasinglegroupbyclause
q
Eachgeneratessetofgroupbylists,crossproductofsetsgivesoverall setofgroupbylists
s E.g.,
selectitemname,color,size,sum(number) fromsales groupbyrollup(itemname),rollup(color,size) generatesthegroupings {itemname,()}X{(color,size),(color),()} ={(itemname,color,size),(itemname,color),(itemname), (color,size),(color),()}
18.<number>
Ranking
s Rankingisdoneinconjunctionwithanorderbyspecification. s Givenarelationstudentmarks(studentid,marks)findtherankofeach
student.
selectstudentid,rank()over(orderbymarksdesc)assrank fromstudentmarks
s Anextraorderbyclauseisneededtogettheminsortedorder
selectstudentid,rank()over(orderbymarksdesc)assrank fromstudentmarks orderbysrank

s Rankingmayleavegaps:e.g.if2studentshavethesametopmark,both
haverank1,andthenextrankis3
q
dense_rankdoesnotleavegaps,sonextdenserankwouldbe2
18.<number>
Ranking(Cont.)
s Rankingcanbedonewithinpartitionofthedata. s Findtherankofstudentswithineachsection.
selectstudentid,section, rank()over(partitionbysectionorderbymarksdesc) assecrank fromstudentmarks,studentsection wherestudentmarks.studentid=studentsection.studentid orderbysection,secrank

s Multiplerankclausescanoccurinasingleselectclause s Rankingisdoneafterapplyinggroupbyclause/aggregation
18.<number>
Ranking(Cont.)
s Otherrankingfunctions:
q q
percent_rank(withinpartition,ifpartitioningisdone) cume_dist(cumulativedistribution)
fractionoftupleswithprecedingvalues
row_number(nondeterministicinpresenceofduplicates)
s SQL:1999permitstheusertospecifynullsfirstornullslast
selectstudentid, rank()over(orderbymarksdescnullslast)assrank fromstudentmarks
18.<number>
Ranking(Cont.)
s Foragivenconstantn,therankingthefunctionntile(n)takesthe
tuplesineachpartitioninthespecifiedorder,anddividestheminton bucketswithequalnumbersoftuples. selectthreetile,sum(salary) from( selectsalary,ntile(3)over(orderbysalary)asthreetile fromemployee)ass groupbythreetile
s E.g.:
18.<number>
Windowing
s Usedtosmoothoutrandomvariations. s E.g.:movingaverage:Givensalesvaluesforeachdate,calculateforeach
datetheaverageofthesalesonthatday,thepreviousday,andthenext day
q
s WindowspecificationinSQL:
Givenrelationsales(date,value)
selectdate,sum(value)over (orderbydatebetweenrows1precedingand1following) fromsales

s Examplesofotherwindowspecifications:
q q q
betweenrowsunboundedprecedingandcurrent rowsunboundedpreceding rangebetween10precedingandcurrentrow
Allrowswithvaluesbetweencurrentrowvalue10tocurrentvalue Notincludingcurrentrow
rangeinterval10daypreceding
Windowing(Cont.)
s Candowindowingwithinpartitions s E.g.Givenarelationtransaction(accountnumber,datetime,value),
wherevalueispositiveforadepositandnegativeforawithdrawal
q
Findtotalbalanceofeachaccountaftereachtransactiononthe account selectaccountnumber,datetime, sum(value)over (partitionbyaccountnumber orderbydatetime rowsunboundedpreceding) asbalance fromtransaction orderbyaccountnumber,datetime
18.<number>
DataWarehousing
s Datasourcesoftenstoreonlycurrentdata,nothistoricaldata s Corporatedecisionmakingrequiresaunifiedviewofallorganizational
data,includinghistoricaldata
s Adatawarehouseisarepository(archive)ofinformationgathered
q q
frommultiplesources,storedunderaunifiedschema,atasinglesite Greatlysimplifiesquerying,permitsstudyofhistoricaltrends Shiftsdecisionsupportqueryloadawayfromtransaction processingsystems
18.<number>
DataWarehousing
18.<number>
DesignIssues
s Whenandhowtogatherdata
q
Sourcedrivenarchitecture:datasourcestransmitnewinformation towarehouse,eithercontinuouslyorperiodically(e.g.atnight) Destinationdrivenarchitecture:warehouseperiodicallyrequests newinformationfromdatasources Keepingwarehouseexactlysynchronizedwithdatasources(e.g. usingtwophasecommit)istooexpensive

UsuallyOKtohaveslightlyoutofdatedataatwarehouse Data/updatesareperiodicallydownloadedformonline transactionprocessing(OLTP)systems.
s Whatschematouse
q
Schemaintegration
18.<number>
MoreWarehouseDesignIssues
s Datacleansing
q q
E.g.correctmistakesinaddresses(misspellings,zipcodeerrors) Mergeaddresslistsfromdifferentsourcesandpurgeduplicates Warehouseschemamaybea(materialized)viewofschemafrom datasources Rawdatamaybetoolargetostoreonline Aggregatevalues(totals/subtotals)oftensuffice Queriesonrawdatacanoftenbetransformedbyqueryoptimizer touseaggregatevalues
s Howtopropagateupdates
q
s Whatdatatosummarize
q q q
18.<number>
WarehouseSchemas
s Dimensionvaluesareusuallyencodedusingsmallintegersand
mappedtofullvaluesviadimensiontables
q
s Resultantschemaiscalledastarschema
Morecomplicatedschemastructures

Snowflakeschema:multiplelevelsofdimensiontables Constellation:multiplefacttables
18.<number>
DataWarehouseSchema
18.<number>
DataMining
s Dataminingistheprocessofsemiautomaticallyanalyzinglarge
databasestofindusefulpatterns
q
s Predictionbasedonpasthistory
Predictifacreditcardapplicantposesagoodcreditrisk,basedon someattributes(income,jobtype,age,..)andpasthistory Predictifapatternofphonecallingcardusageislikelytobe fraudulent Classification
s Someexamplesofpredictionmechanisms:
q
Givenanewitemwhoseclassisunknown,predicttowhichclass itbelongs Givenasetofmappingsforanunknownfunction,predictthe functionresultforanewparametervalue
Regressionformulae
18.<number>
DataMining(Cont.)
s DescriptivePatterns
q
Associations
Findbooksthatareoftenboughtbysimilarcustomers.Ifa newsuchcustomerbuysonesuchbook,suggesttheothers too. E.g.associationbetweenexposuretochemicalXandcancer, E.g.typhoidcaseswereclusteredinanareasurroundinga contaminatedwell Detectionofclustersremainsimportantindetectingepidemics
Associationsmaybeusedasafirststepindetectingcausation
Clusters
18.<number>
ClassificationRules
s Classificationruleshelpassignnewobjectstoclasses.
q
E.g.,givenanewautomobileinsuranceapplicant,shouldheorshe beclassifiedaslowrisk,mediumriskorhighrisk?
s Classificationrulesforaboveexamplecoulduseavarietyofdata,such
aseducationallevel,salary,age,etc.
q
personP,P.degree=mastersandP.income>75,000 P.credit=excellent personP,P.degree=bachelorsand (P.income25,000andP.income75,000) P.credit=good
s Rulesarenotnecessarilyexact:theremaybesomemisclassifications s Classificationrulescanbeshowncompactlyasadecisiontree.
18.<number>
DecisionTree
18.<number>
ConstructionofDecisionTrees
s Trainingset:adatasampleinwhichtheclassificationisalready
known.
q
s Greedytopdowngenerationofdecisiontrees.
Eachinternalnodeofthetreepartitionsthedataintogroups basedonapartitioningattribute,andapartitioningcondition forthenode Leafnode:
all(ormost)oftheitemsatthenodebelongtothesameclass, or allattributeshavebeenconsidered,andnofurtherpartitioning ispossible.
18.<number>
BestSplits
s Pickbestattributesandconditionsonwhichtopartition s ThepurityofasetSoftraininginstancescanbemeasuredquantitativelyin
severalways.
q
Notation:numberofclasses=k,numberofinstances=|S|, fractionofinstancesinclassi=pi.
k
s TheGinimeasureofpurityisdefinedas
Gini(S)=1 p2i
i1
q q
Whenallinstancesareinasingleclass,theGinivalueis0 Itreachesitsmaximum(of11/k)ifeachclassthesamenumberof instances.
18.<number>
BestSplits(Cont.)
s Anothermeasureofpurityistheentropymeasure,whichisdefinedas
entropy(S)= pilog2pi
i1
s WhenasetSissplitintomultiplesetsSi,I=1,2,,r,wecanmeasurethe
purityoftheresultantsetofsetsas: purity(S1,S2,..,Sr)=
r
| S i|
i=1 |S|
purity(Si)
s TheinformationgainduetoparticularsplitofSintoSi,i=1,2,.,r
Informationgain(S,{S1,S2,.,Sr)=purity(S)purity(S1,S2,Sr)
18.<number>
BestSplits(Cont.)
s Measureofcostofasplit:
Informationcontent(S,{S1,S2,..,Sr}))=
r |S | i
i1 |S|
log2
|S i | |S |
s Informationgainratio=Informationgain(S,{S1,S2,,Sr})
Informationcontent(S,{S1,S2,..,Sr})
s Thebestsplitistheonethatgivesthemaximuminformationgainratio
18.<number>
FindingBestSplits
s Categoricalattributes(withnomeaningfulorder):
q q
Multiwaysplit,onechildforeachvalue Binarysplit:tryallpossiblebreakupofvaluesintotwosets,and pickthebest Binarysplit:
s Continuousvaluedattributes(canbesortedinameaningfulorder)
q
Sortvalues,tryeachasasplitpoint E.g.ifvaluesare1,10,15,25,splitat1,10,15 Pickthevaluethatgivesbestsplit Aseriesofbinarysplitsonthesameattributehasroughly equivalenteffect
Multiwaysplit:
18.<number>
DecisionTreeConstructionAlgorithm
ProcedureGrowTree(S) Partition(S); ProcedurePartition(S) if(purity(S)>por|S|<s)then return; foreachattributeA evaluatesplitsonattributeA; Usebestsplitfound(acrossallattributes)topartition SintoS1,S2,.,Sr, fori=1,2,..,r Partition(Si);
18.<number>
OtherTypesofClassifiers
s Neuralnetclassifiersarestudiedinartificialintelligenceandarenotcovered
here
s BayesianclassifiersuseBayestheorem,whichsays
p(cj|d)=p(d|cj)p(cj) p(d) where p(cj|d)=probabilityofinstancedbeinginclasscj, p(d|cj)=probabilityofgeneratinginstancedgivenclasscj, p(cj)=probabilityofoccurrenceofclasscj,and p(d)=probabilityofinstancedoccuring
18.<number>
NaveBayesianClassifiers
s Bayesianclassifiersrequire
q q q
computationofp(d|cj) precomputationofp(cj) p(d)canbeignoredsinceitisthesameforallclasses
s Tosimplifythetask,naveBayesianclassifiersassumeattributes
haveindependentdistributions,andtherebyestimate p(d|cj)=p(d1|cj)*p(d2|cj)*.*(p(dn|cj)
q
Eachofthep(di|cj)canbeestimatedfromahistogramondi valuesforeachclasscj
thehistogramiscomputedfromthetraininginstances
Histogramsonmultipleattributesaremoreexpensivetocompute andstore
18.<number>
Regression
s Regressiondealswiththepredictionofavalue,ratherthanaclass.
q
Givenvaluesforasetofvariables,X1,X2,,Xn,wewishtopredictthe valueofavariableY. Y=a0+a1*X1+a2*X2++an*Xn
s Onewayistoinfercoefficientsa0,a1,a1,,ansuchthat s Findingsuchalinearpolynomialiscalledlinearregression.
q
Ingeneral,theprocessoffindingacurvethatfitsthedataisalsocalled curvefitting. becauseofnoiseinthedata,or becausetherelationshipisnotexactlyapolynomial
s Thefitmayonlybeapproximate
q q
s Regressionaimstofindcoefficientsthatgivethebestpossiblefit.
18.<number>
AssociationRules
s Retailshopsareofteninterestedinassociationsbetweendifferentitems
thatpeoplebuy.
q q
Someonewhobuysbreadisquitelikelyalsotobuymilk ApersonwhoboughtthebookDatabaseSystemConceptsisquite likelyalsotobuythebookOperatingSystemConcepts. E.g.whenacustomerbuysaparticularbook,anonlineshopmay suggestassociatedbooks.
s Associationsinformationcanbeusedinseveralways.
q
s Associationrules:
breadmilkDBConcepts,OSConceptsNetworks
q q
Lefthandside:antecedent,righthandside:consequent Anassociationrulemusthaveanassociatedpopulation;the populationconsistsofasetofinstances
E.g.eachtransaction(sale)atashopisaninstance,andtheset ofalltransactionsisthepopulation
18.<number>
AssociationRules(Cont.)
s Ruleshaveanassociatedsupport,aswellasanassociatedconfidence. s Supportisameasureofwhatfractionofthepopulationsatisfiesboththe
antecedentandtheconsequentoftherule.
q
E.g.supposeonly0.001percentofallpurchasesincludemilkand screwdrivers.Thesupportfortheruleismilkscrewdriversislow.
s Confidenceisameasureofhowoftentheconsequentistruewhenthe
antecedentistrue.
q
E.g.therulebreadmilkhasaconfidenceof80percentif80 percentofthepurchasesthatincludebreadalsoincludemilk.
18.<number>
FindingAssociationRules
s s
Wearegenerallyonlyinterestedinassociationruleswithreasonably highsupport(e.g.supportof2%orgreater) Navealgorithm

1. 2.
Considerallpossiblesetsofrelevantitems. Foreachsetfinditssupport(i.e.counthowmanytransactions purchaseallitemsintheset).

5
Largeitemsets:setswithsufficientlyhighsupport FromitemsetAgeneratetheruleA{b}bforeachbA. Supportofrule=support(A). Confidenceofrule=support(A)/support(A{b})
3.
Uselargeitemsetstogenerateassociationrules.
1.
18.<number>
FindingSupport
s Determinesupportofitemsetsviaasinglepassonsetoftransactions
q
Largeitemsets:setswithahighcountattheendofthepass
s Ifmemorynotenoughtoholdallcountsforallitemsetsusemultiplepasses,
consideringonlysomeitemsetsineachpass.
s Optimization:Onceanitemsetiseliminatedbecauseitscount(support)istoo
smallnoneofitssupersetsneedstobeconsidered.
q
s Theaprioritechniquetofindlargeitemsets:
Pass1:countsupportofallsetswithjust1item.Eliminatethoseitems withlowsupport Passi:candidates:everysetofiitemssuchthatallitsi1itemsubsets arelarge

Countsupportofallcandidates Stopiftherearenocandidates
18.<number>
OtherTypesofAssociations
s Basicassociationruleshaveseverallimitations s Deviationsfromtheexpectedprobabilityaremoreinteresting
q q
E.g.ifmanypeoplepurchasebread,andmanypeoplepurchasecereal, quiteafewwouldbeexpectedtopurchaseboth Weareinterestedinpositiveaswellasnegativecorrelationsbetween setsofitems

Positivecorrelation:cooccurrenceishigherthanpredicted Negativecorrelation:cooccurrenceislowerthanpredicted
s Sequenceassociations/correlations
q
E.g.wheneverbondsgoup,stockpricesgodownin2days E.g.deviationfromasteadygrowth E.g.salesofwinterweargodowninsummer

s Deviationsfromtemporalpatterns
q q
Notsurprising,partofaknownpattern. Lookfordeviationfromvaluepredictedusingpastpatterns
Clustering
s Clustering:Intuitively,findingclustersofpointsinthegivendatasuchthat
similarpointslieinthesamecluster
q
s Canbeformalizedusingdistancemetricsinseveralways
Grouppointsintoksets(foragivenk)suchthattheaveragedistance ofpointsfromthecentroidoftheirassignedgroupisminimized
Centroid:pointdefinedbytakingaverageofcoordinatesineach dimension.
Anothermetric:minimizeaveragedistancebetweeneverypairof pointsinacluster Dataminingsystemsaimatclusteringtechniquesthatcanhandlevery largedatasets E.g.theBirchclusteringalgorithm(moreshortly)
s Hasbeenstudiedextensivelyinstatistics,butonsmalldatasets
q
18.<number>
HierarchicalClustering
s Examplefrombiologicalclassification
q
(thewordclassificationheredoesnotmeanapredictionmechanism)
chordata mammaliareptilia leopardshumanssnakescrocodiles

s Otherexamples:Internetdirectorysystems(e.g.Yahoo,moreonthislater) s Agglomerativeclusteringalgorithms
q
Buildsmallclusters,thenclustersmallclustersintobiggerclusters,and soon Startwithallitemsinasinglecluster,repeatedlyrefine(break)clusters intosmallerones
s Divisiveclusteringalgorithms
q
18.<number>
ClusteringAlgorithms
s Clusteringalgorithmshavebeendesignedtohandleverylarge
datasets
q
s E.g.theBirchalgorithm
Mainidea:useaninmemoryRtreetostorepointsthatarebeing clustered InsertpointsoneatatimeintotheRtree,merginganewpoint withanexistingclusterifislessthansomedistanceaway Iftherearemoreleafnodesthanfitinmemory,mergeexisting clustersthatareclosetoeachother Attheendoffirstpasswegetalargenumberofclustersatthe leavesoftheRtree
Mergeclusterstoreducethenumberofclusters
18.<number>
CollaborativeFiltering
s Goal:predictwhatmovies/books/apersonmaybeinterestedin,on
thebasisof
q q q
Pastpreferencesoftheperson Otherpeoplewithsimilarpastpreferences Thepreferencesofsuchpeopleforanewmovie/book/ Clusterpeopleonthebasisofpreferencesformovies Thenclustermoviesonthebasisofbeinglikedbythesame clustersofpeople Againclusterpeoplebasedontheirpreferencesfor(thenewly createdclustersof)movies Repeatabovetillequilibrium
s Oneapproachbasedonrepeatedclustering
q q q q
s Aboveproblemisaninstanceofcollaborativefiltering,whereusers
collaborateinthetaskoffilteringinformationtofindinformationof interest
18.<number>
OtherTypesofMining
s Textmining:applicationofdataminingtotextualdocuments
q q q
clusterWebpagestofindrelatedpages clusterpagesauserhasvisitedtoorganizetheirvisithistory classifyWebpagesautomaticallyintoaWebdirectory
s Datavisualizationsystemshelpusersexaminelargevolumesofdata
anddetectpatternsvisually
q
Canvisuallyencodelargeamountsofinformationonasingle screen Humansareverygoodadetectingvisualpatterns
18.<number>

Advanced Querying and Information Retrival

Загружено:

Сведения о документе

Исходное описание:

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Advanced Querying and Information Retrival

Загружено:

Авторское право:

Доступные форматы

Chapter18:DataAnalysisandMining

Whatitemstostock? Whatinsurancepremiumtochange? Towhomtosendadvertisements? Retailsalestransactiondetails Customerprofiles(income,age,gender,etc.)

Exampletasks Foreachproductcategoryandeachregion,whatwerethetotal salesinthelastquarterandhowdotheycomparewiththesame quarterlastyear Asabove,foreachproductcategoryandeachcustomercategory

Importantforlargebusinessesthatgeneratedatafrommultiple divisions,possiblyatmultiplesites Datamayalsobepurchasedexternally

measuresomevalue canbeaggregatedupon e.g.theattributenumberofthesalesrelation definethedimensionsonwhichmeasureattributes(or aggregatesthereof)areviewed e.g.theattributesitem_name,color,andsizeofthesales relation

Valuesforoneofthedimensionattributesformtherowheaders Valuesforanotherdimensionattributeformthecolumnheaders Otherdimensionattributesarelistedontop Valuesinindividualcellsare(aggregatesof)thevaluesofthe dimensionattributesthatspecifythecell.

actuallyusesnullvaluesin placeofalldespiteconfusion withregularnullvalues

storedatacubes,andarereferredtoasmultidimensionalOLAP (MOLAP)systems. relationalOLAP(ROLAP)systems

Spaceandtimerequirementsfordoingsocanbeveryhigh 2ncombinationsofgroupby Itsufficestoprecomputesomeaggregates,andcomputeotherson demandfromoneoftheprecomputedaggregates

Cancomputeaggregateon(itemname,color)fromanaggregate on(itemname,color,size) Forallbutafewnondecomposableaggregatessuchas median

Cancomputeaggregateon(itemname,color)fromanaggregateon (itemname,color,size) Cancomputeaggregateson(itemname,color,size), (itemname,color)and(itemname)usingasinglesorting ofthebasedata

selectitemname,color,size,sum(number) fromsales groupbycube(itemname,color,size) Thiscomputestheunionofeightdifferentgroupingsofthesalesrelation: {(itemname,color,size),(itemname,color), (itemname,size),(color,size), (itemname),(color), (size),()} where()denotesanemptygroupbylist.

selectitemname,color,sum(number) fromsales groupbycube(itemname,color)

selectitemname,color,size,sum(number), grouping(itemname)asitemnameflag, grouping(color)ascolorflag, grouping(size)assizeflag, fromsales groupbycube(itemname,color,size)

selectitemname,color,size,sum(number) fromsales groupbyrollup(itemname,color,size) Generatesunionoffourgroupings: {(itemname,color,size),(itemname,color),(itemname),()}

selectcategory,itemname,sum(number) fromsales,itemcategory wheresales.itemname=itemcategory.itemname groupbyrollup(category,itemname) wouldgiveahierarchicalsummarybyitemnameandbycategory.

selectitemname,color,size,sum(number) fromsales groupbyrollup(itemname),rollup(color,size) generatesthegroupings {itemname,()}X{(color,size),(color),()} ={(itemname,color,size),(itemname,color),(itemname), (color,size),(color),()}

selectstudentid,rank()over(orderbymarksdesc)assrank fromstudentmarks orderbysrank

selectstudentid,section, rank()over(partitionbysectionorderbymarksdesc) assecrank fromstudentmarks,studentsection wherestudentmarks.studentid=studentsection.studentid orderbysection,secrank

selectstudentid, rank()over(orderbymarksdescnullslast)assrank fromstudentmarks

tuplesineachpartitioninthespecifiedorder,anddividestheminton bucketswithequalnumbersoftuples. selectthreetile,sum(salary) from( selectsalary,ntile(3)over(orderbysalary)asthreetile fromemployee)ass groupbythreetile

selectdate,sum(value)over (orderbydatebetweenrows1precedingand1following) fromsales

betweenrowsunboundedprecedingandcurrent rowsunboundedpreceding rangebetween10precedingandcurrentrow

frommultiplesources,storedunderaunifiedschema,atasinglesite Greatlysimplifiesquerying,permitsstudyofhistoricaltrends Shiftsdecisionsupportqueryloadawayfromtransaction processingsystems

Sourcedrivenarchitecture:datasourcestransmitnewinformation towarehouse,eithercontinuouslyorperiodically(e.g.atnight) Destinationdrivenarchitecture:warehouseperiodicallyrequests newinformationfromdatasources Keepingwarehouseexactlysynchronizedwithdatasources(e.g. usingtwophasecommit)istooexpensive

UsuallyOKtohaveslightlyoutofdatedataatwarehouse Data/updatesareperiodicallydownloadedformonline transactionprocessing(OLTP)systems.

Predictifacreditcardapplicantposesagoodcreditrisk,basedon someattributes(income,jobtype,age,..)andpasthistory Predictifapatternofphonecallingcardusageislikelytobe fraudulent Classification

Givenanewitemwhoseclassisunknown,predicttowhichclass itbelongs Givenasetofmappingsforanunknownfunction,predictthe functionresultforanewparametervalue

Findbooksthatareoftenboughtbysimilarcustomers.Ifa newsuchcustomerbuysonesuchbook,suggesttheothers too. E.g.associationbetweenexposuretochemicalXandcancer, E.g.typhoidcaseswereclusteredinanareasurroundinga contaminatedwell Detectionofclustersremainsimportantindetectingepidemics

personP,P.degree=mastersandP.income>75,000 P.credit=excellent personP,P.degree=bachelorsand (P.income25,000andP.income75,000) P.credit=good

Eachinternalnodeofthetreepartitionsthedataintogroups basedonapartitioningattribute,andapartitioningcondition forthenode Leafnode:

all(ormost)oftheitemsatthenodebelongtothesameclass, or allattributeshavebeenconsidered,andnofurtherpartitioning ispossible.

Whenallinstancesareinasingleclass,theGinivalueis0 Itreachesitsmaximum(of11/k)ifeachclassthesamenumberof instances.

Multiwaysplit,onechildforeachvalue Binarysplit:tryallpossiblebreakupofvaluesintotwosets,and pickthebest Binarysplit:

Sortvalues,tryeachasasplitpoint E.g.ifvaluesare1,10,15,25,splitat1,10,15 Pickthevaluethatgivesbestsplit Aseriesofbinarysplitsonthesameattributehasroughly equivalenteffect

p(cj|d)=p(d|cj)p(cj) p(d) where p(cj|d)=probabilityofinstancedbeinginclasscj, p(d|cj)=probabilityofgeneratinginstancedgivenclasscj, p(cj)=probabilityofoccurrenceofclasscj,and p(d)=probabilityofinstancedoccuring

computationofp(d|cj) precomputationofp(cj) p(d)canbeignoredsinceitisthesameforallclasses

Givenvaluesforasetofvariables,X1,X2,,Xn,wewishtopredictthe valueofavariableY. Y=a0+a1*X1+a2*X2++an*Xn

Ingeneral,theprocessoffindingacurvethatfitsthedataisalsocalled curvefitting. becauseofnoiseinthedata,or becausetherelationshipisnotexactlyapolynomial

Someonewhobuysbreadisquitelikelyalsotobuymilk ApersonwhoboughtthebookDatabaseSystemConceptsisquite likelyalsotobuythebookOperatingSystemConcepts. E.g.whenacustomerbuysaparticularbook,anonlineshopmay suggestassociatedbooks.

Lefthandside:antecedent,righthandside:consequent Anassociationrulemusthaveanassociatedpopulation;the populationconsistsofasetofinstances

Wearegenerallyonlyinterestedinassociationruleswithreasonably highsupport(e.g.supportof2%orgreater) Navealgorithm

Considerallpossiblesetsofrelevantitems. Foreachsetfinditssupport(i.e.counthowmanytransactions purchaseallitemsintheset).

Largeitemsets:setswithsufficientlyhighsupport FromitemsetAgeneratetheruleA{b}bforeachbA. Supportofrule=support(A). Confidenceofrule=support(A)/support(A{b})

Pass1:countsupportofallsetswithjust1item.Eliminatethoseitems withlowsupport Passi:candidates:everysetofiitemssuchthatallitsi1itemsubsets arelarge

E.g.ifmanypeoplepurchasebread,andmanypeoplepurchasecereal, quiteafewwouldbeexpectedtopurchaseboth Weareinterestedinpositiveaswellasnegativecorrelationsbetween setsofitems

E.g.wheneverbondsgoup,stockpricesgodownin2days E.g.deviationfromasteadygrowth E.g.salesofwinterweargodowninsummer

Anothermetric:minimizeaveragedistancebetweeneverypairof pointsinacluster Dataminingsystemsaimatclusteringtechniquesthatcanhandlevery largedatasets E.g.theBirchclusteringalgorithm(moreshortly)

chordata mammaliareptilia leopardshumanssnakescrocodiles

Buildsmallclusters,thenclustersmallclustersintobiggerclusters,and soon Startwithallitemsinasinglecluster,repeatedlyrefine(break)clusters intosmallerones

clusterWebpagestofindrelatedpages clusterpagesauserhasvisitedtoorganizetheirvisithistory classifyWebpagesautomaticallyintoaWebdirectory

Canvisuallyencodelargeamountsofinformationonasingle screen Humansareverygoodadetectingvisualpatterns

Вам также может понравиться

Givenvaluesforasetofvariables,X1,X2,,Xn,wewishtopredictthe valueofavariableY. Y=a0+a1X1+a2X2++an*Xn