10 mistakes in A/B testing of mobile apps

1. Improper user segmentation

What to do?

2. Running tests without defining and detailing the problem/hypothesis

What to do?

  • Focus on the problem you would like to solve. This way, you will avoid having too many variants to test.
  • If it is extremely important to include several variants to test, use statistical corrections.

3. Ignoring external factors

  • Food delivery and taxi services, etc. have more orders in bad weather;
  • Dating apps are used more actively on Fridays and Saturdays;
  • Flower e-commerce can usually be seen blowing up during holidays.

What to do?

4. Neglecting statistical data

What to do?

5. Testing hypotheses on a small amount of traffic

What to do?

6. Premature test termination

What to do?

7. Insubstantial difference between the variants being tested

What to do?

8. Using conversion as the only metric of the experiment

What to do?

9. Using classic experiments only

What to do?

10. Using testing tools unsuitable for the test requirements

  • A free pricing plan for a quick start;
  • Classic A/B testing with optimization for any metrics using the standard frequency approach methodology (the traffic is distributed equally into every variant, and the conclusions are made based on confidence intervals);
  • Accelerated experiments with optimization for conversion using Bayesian statistics and multi-armed bandits (the traffic is distributed proportionally to the probabilities of a certain variant being superior at every step);
  • Accelerated experiment with optimization for financial metrics using Bayesian statistics (the traffic is distributed equally, and the best variant is determined based on the probability of its superiority).

What to do?

--

--

--

proba.ai is a tool for A/B testing in mobile apps. Carry out experiments faster, and at a better price — using the mobile app product hypothesis testing tool.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How to Find Images for PowerPoint — minus copyright worries — with Pixabay.

Top 10 Modern Logo Design Trends

Tutorial: how to draw a fantasy village map

How The SMRT In-Train Screen Should Look

Breaking Down Metaverse

Fitelit Harga

Ironhack Challenge 1: Design Thinking

Best Practices for Creating a Responsive Free-Form Dossier

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Proba

Proba

proba.ai is a tool for A/B testing in mobile apps. Carry out experiments faster, and at a better price — using the mobile app product hypothesis testing tool.

More from Medium

Understanding A/B Testing and the Statistical logics involved

Before you start an A/B test

T-tests — when to use which?

Statistical Test for Multi-Group Experiment