TheMathematicsofGamification

AtFoursquare,wemaintainadatabaseof60millionvenues.Andliketheworlditrepresents,our databaseiseverchanging,withusersfromallovertheworldsubmittingupdatesoneverything fromthehoursofarestauranttotheaddressofanewbarbershop.Tomaintaintheaccuracyof ourvenuedatabase,thesechangesarevoteduponbyourloyalSuperusers(SUs)whovigilantly maintainawatchfuleyeoverourdatafortheircityorneighborhood. Likemanyexistingcrowdsourceddatasets(Quora,StackOverflow,AmazonReviews),we assignuserspointsorvotesbasedontheirtenure,reputation,andtheactionstheytake. Superuserslikepointsandgamification.Itrewardsdiligent,hardworkingSUs(whicharethe majority)andpunishesthefewmaliciousbadplayers.Butdatascientistslikeprobabilitiesand guarantees.Wereinterestedinmakingstatementslike,weare99%confidentthateachentryis correct.Howdoweallocatepointstousersinawaythatrewardsthemforbehaviorbutallows ustomakeguaranteesabouttheaccuracyofourdatabase? AtFoursquare,wehaveasimple,firstprinciplesbasedmethodofresolvingproposedvenue attributeupdates.WecangaugeeachSuperusersvotingaccuracybasedontheirperformance onhoneypots(proposedupdateswithknownanswerswhicharedeliberatelyinsertedintothe updatesqueue).Measuringperformanceandusingtheseprobabilitiescorrectlyisthekeytohow weassignpointstoaSuperusersvote. TheMath Letsmakethismoreconcretewithsomemath.LetH0denotethetruestateoftheworld,either

1or1,whichwecaninterpretasaproposedupdatebeingtrueorfalse,respectively.Wedo
notobservethisbutweknowH0=1withaprioriprobabilityp0.User1votesH1(again,either1 or1,representingyayornay)withindependentprobabilityp1ofagreeingwiththetruthH0 and(1p1)ofdisagreeing.BayesRulethengivesus

P(H0=1|H1=1)=P(H1=1|H0=1)P(H0=1)P(H1=1)=p0p1p0p1+(1p0)(1p1)=01 01+1
wherewehavewrittenthesolutionintermsofthelikelihoodratiok=(pk)givenby

(p)=p1p1()=1+.
Thenwehavethat

P(H0=1|H1=1)=1((p0)(p1)).
Infact,itiseasytoseethatinthegeneralcase,

P(H0=1|H1=h1)=1((p0)(p1)h1).
Multiplicationishardsowewilldefinethelogitorloglikelihoodfunction

logit:(0,1)(,)
givenby

logit(p)=log((p))=log(p1p)logit1()=e1+e.
Thenwehave

P(H0=1|H1=h1)=logit1(logit(p0)+h1logit(p1)).
Continuing,assumethatafteruser1caststheirvote,user2votesH2withanindependent probabilityp2ofbeingcorrect(i.e.agreeingwithH0).Wecanthinkoftheposteriorprobability

P(H0=1|H1=h1)asournewpriorandinductivelyrepeattheaboveBayesiananalysistoobtain P(H0=1|H1=h1,H2=h2)=logit1(logit(P(H0=1|H1=h1))+h2logit(p2))=logit1(logit (p0)+h1logit(p1)+h2logit(p2)).

Infact,ifwehavenvotesH1,,Hn,thenwehave

P(H0=1|H1=h1,,Hn=hn)=logit1(logit(p0)+k=1nhklogit(pk)).(1)

TheSolution Theaboveequationsuggeststhatweshouldassignskpointsorvotestouserkbasedon

sk=logit(pk).(2)
Wecanaddupalltheyayvotesandsubtractallthenayvotestoobtainascoreforthe update.Thisscorecaneasilybeinterpretedasaprobabilitythattheupdateiscorrect.Wecan setacertaintythresholdp(e.g.p=99%)asathresholdforadesiredaccuracyofthisedit.

Then,weacceptaproposededitassoonas

logit(p0)+k=1nhklogit(pk)logit(p)(3)
andrejectitassoonas

logit(p0)+k=1nhklogit(pk)logit(p).(4)
Inotherwords,ifwetaket=logit(p)tothethepointsthresholdands0=logit(p0)tobethe pointsallocatedtoanewproposededit,then(3)and(4)become

s0+k=1nhkskt
and

s0+k=1nhkskt,
whichareexactlytheequationsforvotingyouwouldexpect.Butnow,theyrederivedfrommath! TheBenefits Efficient,datadrivenguaranteesaboutdatabaseaccuracy.Bychoosingthepoints basedonausersaccuracy,wecanintelligentlyaccruecertaintyaboutaproposed updateandstopthevotingprocessassoonasthemathguaranteestherequired certainty. Stillusingpoints,justsmartaboutcalculatingthem.Byrelatingausersaccuracyand thecertaintythresholdneededtoacceptaproposedupdatetoanadditivepoint system(2),wecanstillgiveauserthepointsthattheylike.Thisalsomakesiteasy totakeasystemofadhocpointsandconvertitovertoasmartersystembasedon empiricalevidence. Scalableandeasilyextensible.Theparametersareautomaticallytrainedandcan adapttochangesinthebehavioroftheuserbase.Nomorelongmeetingsdebating howmanypointstogranttoanarrowusecase. Sofar,wevetakenaveryusercentricviewofpk(thisistheaccuracyofuserk).But wecangowellbeyondthat.Forexample,pkcouldbetheaccuracyofuserksvote giventhattheyhavebeentothevenuethreetimesbeforeandworknearby.These clausescanbearbitrarilycomplicatedandestimatedfroma(logistic)regressionof

thehoneypotperformance.Thepointisthatthesechangeswillbebasedondataand notsubjectivejudgmentsofhowmanypointsauserorsituationshouldget. Somepracticalconsiderations: Inpractice,wemightwantadifferentthresholdforaccepting(3)versusrejecting(4) aproposededit. Fornotationalsimplicity,wehaveassumedthatafalsepositivesandfalsenegatives inuserksvotingaccuracyhavethesameprobabilitypk.Ingeneral,thisisnotthe case.Weleaveittothereadertofigurethemathofthegeneralcase. Userslikeintegerpoints.Wehavetoroundsktothenearestinteger.Becausewe canmultiplylinearequationslike(3)and(4)byapositiveconstant,wecanset

sk=[logit(pk)]where[]istheroundingfunctionandisalargepositive
constant.Alargewillpreventthelossoffidelity. Weveexplainedhowtoobtainp1,p2,fromhoneypotsbuthowdoweobtainp0, theaccuracyofnewlyproposedupdates.Onewayistousetheabovetobootstrap thoseaccuraciesfromvoting:wecanusethisvotingtechniquetoinfertheaccuracy ofproposalsbylookingatwhatfractionofproposedupdatesareaccepted! BayesianSmoothing.Weassumearelativelylowaccuracypriorfortheaccuracyof individuals.Thisisapessimisticassumptionthatkeepsnew,untestedusersfrom havingtoomuchinfluence.Italsorewardsusersforlendingtheirjudgmentand castingvotesaslongasthosearemoreaccuratethanourpessimisticprior.Of course,wealsoincreasethelikelihoodofshowingnewSuperUsershoneypotsto givethemachancetoprovethemselves. MichaelLi DataScientist @tianhuil
PostedinFoursquareEngineeringBlog