A/B Test Calculator
Check whether the difference between two variants is more than chance.
Verdict
B is better, with 95% confidence
If A and B really converted equally, a gap this large in either direction would show up in about 2.8% of tests. That is below the 5% this confidence level allows.
- Conversion rate A
- 5.00%
- 95% interval: 4.59% to 5.44%
- Conversion rate B
- 5.70%
- 95% interval: 5.26% to 6.17%
- Uplift of B over A
- +14%
- +0.7 percentage points
- p-value
- 0.0278
- z = 2.20, two-sided
How the result is calculated
The calculator runs a two-proportion z-test. It assumes for a moment that A and B convert equally well, pools everything into one conversion rate, and then measures how far apart the two observed rates are in standard errors.
| Step | Formula | 500 of 10,000 against 570 of 10,000 |
|---|---|---|
| Conversion rates | conversions ÷ visitors | 5.00% and 5.70% |
Pooled rate p | all conversions ÷ all visitors | 1,070 ÷ 20,000 = 5.35% |
| Standard error | √(p × (1 − p) × (1/nA + 1/nB)) | 0.00318 |
| z | (rate B − rate A) ÷ standard error | 0.0070 ÷ 0.00318 = 2.20 |
| p-value | share of the normal curve beyond ±2.20 | 0.0278 |
0.0278 is below 0.05, so this result is significant at 95% confidence. It is above 0.01, so at 99% it is not.
The interval under each conversion rate is a Wilson score interval, always two-sided at the confidence level you picked. It is used here instead of the simpler "rate ± 1.96 standard errors" because it stays sensible for small samples and for rates close to zero.
What the p-value means
The p-value is the chance of seeing a difference at least this large if A and B in truth perform the same. It is not the chance that B is better, and "95% confidence" does not mean the uplift is 95% certain to be real.
Decide the sample size before you start
Use the sample size tab first, then let the test run until every variant has had that many visitors. The 5% error rate of a 95% test only holds if you judge the result once. Checking along the way and stopping the moment the result turns significant is called peeking, and it hands chance several attempts at crossing the line: with five evenly spaced looks, about 14% of tests between two identical pages produce a "winner", and with ten looks about 19%.
The planner uses the standard formula for comparing two proportions:
n = (zc × √(2 × m × (1 − m)) + zp × √v)² ÷ d²
Here m is the average of the current rate p1 and the improved rate p2, v is p1 × (1 − p1) + p2 × (1 − p2) and d is the difference p2 − p1. zc follows from the confidence level (1.96 for 95%, two-sided) and zp from the power (0.84 for 80%). A 5% baseline with a 10% relative uplift, so 5% against 5.5%, needs 31,234 visitors per variant at those settings. Halving the uplift you want to detect roughly quadruples that number.
One-sided or two-sided
A two-sided test asks whether A and B differ, in either direction. A one-sided test asks only whether B beats A. It reaches significance with less data, but it can never tell you that B is worse. Choose before the test starts: picking the one-sided test after seeing which version is ahead doubles the error rate you thought you had.
Significant is not the same as important
Significance says a difference is unlikely to be chance alone. It says nothing about its size. With enough traffic a lift from 5.00% to 5.05% becomes significant, and it may still not be worth the work of rolling out. Look at the uplift and the intervals, and at what the difference is worth in money.
The reverse also holds. "No clear winner" does not prove that A and B are equal; the test may simply have been too small to see the difference.
Where this test falls short
The z-test leans on a normal approximation that gets rough when a variant has only a handful of conversions, which is why small samples get a warning. It also assumes each visitor is counted once and was assigned to a version at random. Comparing three or more versions, or judging the same test on several metrics, raises the chance of a false winner, and this calculator does not correct for that.
Each tab on a page of its own
More marketing
All tools