Case study: How We Lost $7,500 on Mobile App A/B Tests But Learned How to Do Them

How Classic A/B Tests Work, and What’s Wrong With Them

Conventional A/B testing is a go-to method when it comes to making some changes to a product. It’s quite straightforward: we divide users into groups and assign one of the options to each group. Then we define the metric by which we assess the results and then, following the final values, we choose the best option.

Part 1. A/B Test

One of our customers had a hookup dating service named Sweet.

4 paywalls in the test. The last one in the image is the paywall used in the app before the test.

Multi-Armed Bandit

Evgeny employed the method that is the cornerstone of the Multi-Armed Bandit. Remember those slot machines in a casino? Every time you pull the lever, you win — or do not win — some money. You have a limited number of attempts. According to the law of probability, some machines can give you more money than others. And you want to win more, right? If you knew in advance which machines can yield more winnings, you would probably hit a jackpot.

Part 2. Multi-Armed Bandit Comes Into Play

Developing the A/B testing service, we complemented it with the Thompson Sampling algorithm that can solve the problem of the multi-armed bandit. Such a bandit will be a flawless version of Evgeny: it will switch distribution more frequently and take more factors into account. We decided to verify its consistency on real historic data of the Sweet app, so we exported and fed them to the algorithm to see how it would redistribute traffic flows between alternatives.

Traffic Distribution by Alternative

More about

So far, the algorithm employs AppsFlyer (Enterprise or Business rate) and Amplitude; more trackers are coming soon.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store

Proba is a tool for A/B testing in mobile apps. Carry out experiments faster, and at a better price — using the mobile app product hypothesis testing tool.