Академический Документы
Профессиональный Документы
Культура Документы
Started a company to make iOS apps Ended up building A/B testing tools Twitter acquired our IP All open sourced github.com/clutchio
Show of hands
Show of hands
Who does web stu? Keep your hand up if youve done A/B testing on the web.
Show of hands
Show of hands
Who does iOS or Android apps? Keep your hand up if youve done A/B testing on these apps.
Show users dierent variations of your app Track those users and their actions Analyze their actions to nd the most eective variation
Now Mobile
However...
How does this ab() function work on mobile? What can our latency be on this call? Can it talk to a database? What happens when you are oine?
Download manifest of all A/B tests on launch Client must be smart enough to make its own decision, immediately, oine
Goal Tracking
Goal Tracking
goal_reached(user, phone-signup)
Goal Tracking
Must have capability on frontend and backend Frontend example: registration page completed successfully Backend example: e-mail veried successfully
Store all results in a local phone database Upload all of the information periodically If possible, upload everything when the app is quit
Note: Complications
May receive some data twice -- have to query to double-check rst May receive very old data -- working set of data now very wide How long is too long to wait for data to come back?
So...Minimum Requirements
Download manifest up-front Make weighted random decisions oine Track goals and store progress in a local db Peg users to the same bucket during the session Periodically upload progress
Each release might take weeks or even months A/B testing at this time scale is frustrating Can anything be done to improve it?
Instead of a simple boolean if-else, pass back data instead Now you can change the tests on the server Still need to think ahead, but this can add lots of exibility
Note
Remember, this decision still needs to be made instantly, and o-line So all of this data now becomes part of the manifest Data must be kept compact and cant store e.g. a lot of binary
We care about two things: Which variation is winning? How condent are we about it?
Can easily calculate a ratio for each variation: How many people have seen this variation? How many of those people have reached the goal?
ABBA Example
http://www.thumbtack.com/labs/abba/
Questions?