Minimum detectable effect for subscription app A/B tests
The minimum detectable effect (MDE) is the smallest change your A/B test can tell apart from chance. It shrinks as traffic and weeks grow. If the change you're testing is likely smaller than your MDE, the test will most likely end "can't tell", however good the idea.
Typical MDEs for subscription apps
Paywall, judged on money per paywall view (2.75% of views pay, weekly $6.99 and annual $39.99 half and half), at 95% confidence and 80% power:
| Views a week | 2 wk | 4 wk | 6 wk | 8 wk |
|---|---|---|---|---|
| 10,000 | +28.9% | +20.5% | +16.7% | +14.5% |
| 25,000 | +18.3% | +12.9% | +10.6% | +9.1% |
| 50,000 | +12.9% | +9.1% | +7.5% | +6.5% |
| 100,000 | +9.1% | +6.5% | +5.3% | +4.6% |
| 250,000 | +5.8% | +4.1% | +3.3% | +2.9% |
Onboarding, judged on trial starts (5.8% of new users start a trial):
| Users a week | 2 wk | 4 wk | 6 wk | 8 wk |
|---|---|---|---|---|
| 5,000 | +22.6% | +16.0% | +13.0% | +11.3% |
| 10,000 | +16.0% | +11.3% | +9.2% | +8.0% |
| 20,000 | +11.3% | +8.0% | +6.5% | +5.6% |
| 50,000 | +7.1% | +5.0% | +4.1% | +3.6% |
| 100,000 | +5.0% | +3.6% | +2.9% | +2.5% |
How it's worked out
Per-version sample size n = 2·(z₁₋α/₂ + z₁₋β)²·CV² / MDE², solved for the MDE a given number of people can see. For conversion CV² = (1 − p)/p; for money per view it also counts the spread of prices people buy. Three versions use a Bonferroni correction. The planner shows the MDE for 2, 4, 6 and 8 weeks next to published test ideas, so you can see which ones your test is big enough to catch.
What to do when your MDE is high
- Test a bolder change: change what the screen shows or the offer, not just the words.
- Run longer, in whole weeks.
- Put more traffic into the test, or test two versions instead of three.
- Decide on a metric that answers faster (paid conversion or trial starts) when the change doesn't touch prices.