The A/B testing casino
Before you ship a green A/B result, or add it to the year’s total, it helps to know what the p-value on it can and can’t tell you.
Contents4 sections
An A/B test dashboard that calls a win “95% significant” isn’t saying there’s a 95% chance the change works. That chance isn’t on the card at all, because it depends on how often your ideas work. At the rates big companies publish, about 1 in 4 green wins is noise, and the real wins are smaller than the dashboard shows.
An A/B test shows a change to a random half of users and compares a metric, such as conversion, with the other half. Say checkout button v2 won. Conversion is up 4.1%, p = 0.03. The figure tests 1,000 ideas like it. Change how many of them really work and watch the share of fake winners move.
1,000 ideas, one significance bar
Formula (Kohavi et al.), not real experimentsCheckout button v2
+4.1% conversion · p = 0.03 · statistically significant (95%)
An example result, not real data.
What’s the chance this win is fake?
The card can’t say. It depends on how many of your ideas work. Pick a success rate below, and the figure tests 1,000 ideas like this one.
real winner fake winner (noise)
of your winners are fake
Berman et al., 2,101 real experiments: 27% fake at 95% confidence.
What does p = 0.03 say?
A p-value of 0.03 means that if the change did nothing, a result at least this good would show up 3% of the time. The significance level, written α (alpha), is the bar you set before the test. “Significant at 95%” only means the p-value came in under 0.05.
The phrase “chance it’s fake” hides 2 different numbers. The false positive rate is how often a dead idea, one that does nothing, still shows green. That is α. The false positive risk is the share of green wins that are dead ideas. It’s the one you care about, and the card doesn’t show it. Kohavi, Deng and Vermeer call the 5% reading “a very common misunderstanding”.
Why are about 1 in 4 wins noise?
A casino doesn’t need to win every bet. A small built-in advantage, the house edge, wins over thousands of them. Testing has an edge like that, and it works against you when most ideas don’t work.
The false positive risk depends on 2 numbers the card leaves out. The base rate is the share of tested ideas that really work. Power is the chance that a real win of the planned size shows up green. Write π for the base rate. The fake share is then the dead ideas that got through, divided by all the green results:
fake share = α(1-π) / (α(1-π) + power·π)
Kohavi and his co-authors use α = 0.025. A two-sided test at 0.05 raises false alarms in both directions, 2.5% each way. A team that ships only improvements acts only on the upward ones. Their Table 2 uses success rates that companies have published, at 80% power.
| Success rate | False positive risk | Where it comes from |
|---|---|---|
| 33% | 5.9% | Microsoft |
| 20% | 11.1% | Avinash Kaushik |
| 15% | 15.0% | Bing |
| 10% | 22.0% | Booking.com, Google Ads, Netflix |
| 8% | 26.4% | Airbnb Search |
Berman and colleagues studied 2,101 real experiments on Optimizely, a commercial testing platform. They put the share of false wins at the 95% level at 27%.
A win that only just clears the bar is worse. The 22% averages over every winner, including ones far past the bar. A win right at p = 0.03 barely made it, so the figure gives it about a 51% chance of being fake.1. The 51% assumes each idea either does nothing or has exactly the effect the test was sized for. Across success rates of 8 to 15% and power of 50 to 90%, the same calculation gives 35% to 66%.
Peeking, which means checking a test while it runs and stopping the first time it shows green, raises the false positive rate itself. Evan Miller showed that checking after every observation can push a 5% false positive rate to 26.1%. A sequential test sets a higher bar for early looks, so the rate stays where you planned it. That fixes α but not the base rate, so about 1 in 4 wins is still noise.
Why do shipped wins add up to more than you got?
Real wins have a second problem. Each result is the true effect plus some noise. If you keep only the results that cleared the bar, you keep the ones that noise pushed up. So a real winner’s measured size is too big. This is called the winner’s curse.
Gelman and Carlin measure it with the exaggeration ratio, which is how many times too big a real winner’s measured effect is. At 80% power it’s 1.12. Below 50% power it grows fast. The next figure simulates a year of 500 experiments at 50% power.
Your wins don’t add up: one year, 500 experiments
Simulation, seed 7, not real experimentsDrag power down to 20%, then press "Shrink the estimates" to pull each measured lift back toward what the whole year makes likely.
The year’s total is off by more than the exaggeration ratio, because it also counts the shipped changes that did nothing. Over 2,000 simulated years, the dashboard total is about 1.4 times the truth at 80% power and 2 times at 50%. At 20% power it’s over 4 times. No test here peeks, so the gap comes from selection alone.
A holdout measures the real total. It keeps a slice of users away from every shipped change, so you can measure their combined effect directly. It takes a quarter or more and needs millions of users. The cheaper fix is shrinkage, also called empirical Bayes. You estimate how true effects are spread across all your tests, then pull each result toward what that spread makes likely. “Shrink the estimates” in the figure does this to the simulated year. Lee and Shen described this bias at Airbnb in 2018.
Published wins shrink the same way when someone tests them again. A 2023 paper reported that rounded buttons raised click-through by 55%. Kohavi and colleagues reran it with more than 2,000 times the users and found an effect about 100 times smaller.
What should a results card show?
None of this means you should test less. Azevedo and colleagues found that Bing’s experiments have fat tails, meaning rare huge wins among many small effects. Under those conditions, testing many ideas with smaller samples is the better strategy. The mistake is treating an estimate as a fact, for example by adding up wins for a finance review. Test loosely to find ideas, then replicate or shrink before you believe a size.
The card is where that goes wrong, because most people only ever see the card. I’d put the false positive risk next to the p-value, worked out from the team’s own past success rate. I’d show the shrunk estimate beside the raw one, and only the shrunk one would go into slides. Both numbers are cheap to compute, and both are already in the papers above.