How long should a paywall A/B test run?
Long enough for the change you expect to be bigger than the noise, in whole weeks, plus the longest free trial and about a week for refunds before you decide. For most subscription apps that is 4 to 8 weeks of traffic, and the smaller your traffic, the bolder the change has to be.
What a test can catch, by traffic and weeks
The smallest change in money per paywall view an A/B test can tell apart from chance (95% confidence, 80% power), for a paywall where 2.75% of views end in a payment and buyers split half and half between weekly $6.99 and annual $39.99:
| Views a week | 2 wk | 4 wk | 6 wk | 8 wk |
|---|---|---|---|---|
| 10,000 | +28.9% | +20.5% | +16.7% | +14.5% |
| 25,000 | +18.3% | +12.9% | +10.6% | +9.1% |
| 50,000 | +12.9% | +9.1% | +7.5% | +6.5% |
| 100,000 | +9.1% | +6.5% | +5.3% | +4.6% |
| 250,000 | +5.8% | +4.1% | +3.3% | +2.9% |
Judged on paid conversion instead, the same paywall catches a little less: 50,000 views a week catch +7.5% in 4 weeks (+9.1% on money per view). Your own numbers change these figures; the planner works them out for your traffic, conversion, prices and trial.
How to pick the length
- Start from what you expect the change to do. A headline or button tweak rarely moves revenue by more than a few percent. Making the annual plan the hero has moved revenue per user by +25% to +40% in Adapty's A/B test library. If your test only catches +9%, test the bolder change.
- Run whole weeks. Weekdays and weekends convert differently; a test that stops mid-week reads that as an effect.
- Add the trial and the refunds. People who start a trial on the last day pay only when it ends, and refunds arrive in the days after. With a 3-day trial, decide 3 + 7 days after the traffic stops.
- Don't stop early on a good day. Peeking and stopping the moment B looks ahead turns noise into wins. Pick the length first, then wait.
Judge on revenue or on conversion?
Prices, plans or trial changes: money per paywall view, because conversion can fall while revenue rises. Copy and layout changes: paid conversion answers faster.
Plan your paywall test