DataShopExerciseforPSLCSummerSchool,July,2009

KenKoedinger

Thisexercisehasthefollowinggoals:
1) FamiliarizeyouwiththedatasetusedinCen,Koedinger,&Junker(2006)1.
2) GiveyouanexperienceinusingthePittsburghScienceofLearningCenters
DataShop,arepositoryoflearningdataopenfordatamining.
3) Giveyouabriefexperienceindoingsomediggingintothisdataset.
Inthisexercise,youfirstneedtologintoDataShopandreviewsomehelppages(partA
below),thengettotheGeometryArea(199697)dataset(partBbelow),andthen
answersomequestionsaboutthatdataset(partCbelow).
PartA.LoggingintoDatashopandreviewinghelppages
1) Gotohttp://www.pslcdatashop.org
2) Loginyouneedtoregisterifthisisyourfirsttime
3) ClickonHelpintheupperleftcorner.ClickonDocumentationHomeatthe
bottomofthepagethatappears.
4) Clickon4.Glossary
5) Scrolldown,clickonandwatchVideo:SolvingMakingCans,Question1.This
videoshowsanexampleof(amorerecentversionof)thetutorusedintheCenpaper
andinthedatasetyouwilllookatinPartsBandC.Notethatthisproblemwas
calledPogsinthe199697dataset,notMakingCans.
6) ReadtherestoftheGlossarypagesandbesuretolookattheTablesofdata.Which
rowsinTable1correspondwiththeactionsshowninthevideo?
Whatquestionsdoyouhave?Pleaseask!
PartB.GettingtotheGeometryArea(199697)dataset
1) ClickBacktoDataShop(orgetyourselftowhereyouwereafterstep2inpartA).
2) UnderPublicDatasets,clickonGeometryArea(199697),whichisunderthe
headingGeometryCourseabouthalfwaydownthepage.ClickonLearningCurve
atthetop.
3) OnthetopleftintheSamplessection,clickonAllData.Youmayhavetowaita
moment,butyoushouldseealearningcurveappear.
4) FindtheKCModelssectionontheleftside,thirdpaneldown.Pickadifferent
knowledgecomponentmodel(e.g.,Original)fromthePrimarymenuandinspect
thelearningcurvethatappears.Theredsolidlineshowsthedataandthebluedotted
lineshowspredictionsbasedontheKCmodel(morelater).Alsonoticethe
observationtablebelowthegraph.TrysomeotherKCmodels(e.g.,Geometry,
DecomposeArith,etc.)andtrytomakesenseofthedifferencesinthenumberof
ThispapercanbefoundbyfollowingstepsA1,A2,andB2andthenclickingonthe
DatasetInfotab(topleft)andthenPapersandFiles(secondfromleft).Finally,click
onthelinkinthePapercolumn.
1

5)
6)
7)

8)

observationsperopportunity,inthesmoothnessofthecurves,andtheclosenessof
fitofmodel(blue)anddata(red).Thesedifferentmodelsarejustsomeofthestates
inthehugespaceofmanypossibleknowledgecomponentmodelsthattheLFA
algorithmdescribedintheCenpapercansearchthrough.
Scrolldownandnotethatyoucanclickonindividualknowledgecomponentlearning
curvesandtheyarebroughtupintothelargedisplay.
IntheKCModelspanelsetthePrimarymodeltoOriginal.
ClickonLFAvaluessubtab(belowErrorReport)toseethebestfitting
parametersforthecurrentKnowledgeComponentmodel.TheStudentandKC
interceptvaluesarejustlikethevaluesyouwouldgetfromanItemResponseTheory
model(thoughKCreplacesitem).Thekeydifferenceistheadditionoftheslope
parameterontheKCs,whichdetermineshowfastthelearningcurvegoesdown.
ClickbackonLineGraphtoseethelearningcurvesagain.

PartC.Dosomedatamining!
Writeshortanswerstoasmanyofthefollowingquestionsasyoucan.
LearningCurvetool
1) Whydoesthenumberofobservationsperopportunitychangewhenyouswitchfrom
oneKCmodeltoanother?Hint:ThereisonlyoneKCintheGeometryKCmodel
andfifteenintheOriginalmodel.
2) Forwhichoftheknowledgecomponentmodelsdoesthelearningcurveappeartogo
down?
3) WhydotheKCmodelsforsomecurvesnotgodown?Whyistherenoapparent
reductioninerrorrateforsomeoftheseKCmodels?Hint:Itisnotbecausethere
isntlearningstudentslearnedfromthisunit,butonlysomeKCmodelsrevealit.
Forthefollowingquestions,lookatindividualknowledgecomponentlearningcurvesfor
theOriginalKCmodel.
4) WhatisoneKCthatshowsarelativelysmoothdroppingcurve?
5) WhatisoneKCthatincreasesinerrorrate?Whymightthiscurvebegoingup?
6) DoyouseeanyopportunitiesfortutorredesignbylookingattheseindividualKC
learningcurves?
7) IsthereaKCwithacurvethathadalowerrorratefromthestartandyetstudents
receivedlotsofpracticeonthatKC?
8) IsthereaKCwithacurvethathadahigherrorrateatthestart,butstudentsreceived
toofewpracticeopportunitiessuchthattheerrorratewasstillhighattheendof
practice?
9) WhatisanexampleofaKCwheretheLFAmodel(theblueline)doesnotseemtofit
thelearningcurvedata(theredpoints)verywell?Whatmightbegoingon?
QuestionsaboutPerformanceProfilertool(clickonPerformanceProfilerattopof
page)

1) Whichproblemdidstudentshavethemostdifficultywith?Howdidyoufigureit
out?Onlycountproblemsthatareasonablenumberofstudentsattempted.Youcan
putthemouseoverbarstogetrelevantinformation.Betteryetyoucansetthe
minimumnumberofstudents,sayto5,inthethenavigationarea.
2) Whichproblemiseasiest?
3) Whichstudentmadethefewesterrors?(Donotcountstudentswhoonlysolvedafew
problems.)Toseedataonstudents,clickonProblem(whichappearssidewaysonthe
leftaxisofthegraph)andselectStudentinthepopupmenuthatappears.
4) Whichstudentmadethemosterrors?(Donotcountstudentswhoonlysolvedafew
problems.)
5) MakesureyouhaveOriginalselectedintheKCModelspanelontheleft.Forwhat
knowledgecomponentwastheoverallerrorratethehighest?(Asinquestion3,you
canclickontheleftaxisProblemorStudentandchangeittoKnowledge
Component.)
6) ChangetheleftaxisbacktoProblemandchangethetopaxisfromErrorRate(%)to
ResidualErrorRate%(PredictedActual).Thisvariableshowsthedifference
betweentheLFAmodelspredictionandtheactualdata.GotothePerformance
ProfilerpanelonleftandchangeSortByfromErrorRatetoResiduals.Forwhich
problemdoestheOriginalKCmodelpredictamuchlowererrorratethanthe
observed?(Hint:Thatis,whichproblemhasthebiggestnegativeresidual?As
before,remembertoignoreproblemswithtoofewdatapoints.)
7) ForwhichproblemdoestheOriginalKCmodelpredictamuchhigherratethan
observed?(Hint:Thatis,whichproblemhasthebiggestpositiveresidual?)
8) Fortheproblemidentifiedinquestion6,whataretheknowledgecomponentsthatare
accountingforthepoorfitonthisproblem?Tofindoutwhatknowledgecomponents
gowithaproblem,clickonErrorReportatthetopandthenselecttheProbleminthe
panelonthelowerleft.Youwillseeadisplayoftheerrorrateoneachstepinthat
problemandtheknowledgecomponent(s)associatedwitheachstep.
9) Fortheproblemidentifiedinquestion7,whataretheknowledgecomponentsthatare
accountingforthepoorfitonthisproblem?
QuestionstoaddressusingExportStepRollupTable.
1) WhydoesthelearningcurvefortheGeometryKCmodelhavebigupwardblipsin
theerrorrateatsomeopportunitieslike28,38,and42?Forinstance,between
opportunity27and28,theerrorratejumpsupfrom8%to50%.Toanswerthis
question,clickonStudentStepRollupunderneathLearningCurve,checkKnowledge
ComponentsintheStepRolluppanelontheleftandchangetheKnowledge
ComponentmodeltoGeometry.ThenclickonExportStepRollup.Loadthefileyou
getintoExcel.Inspectthesteps,forinstance,withopportunitynumbers27and28
usingExcelsAutoFilteror,betteryet,apivottable.
a. Togetalessononpivottables,gobacktothelearnlab.orgmainpage,click
EnablingTechnologies,clickMeetings,andviewtheDocumentsandVideo
ofPresentation.