- Home
- Tools
- Marketing
- A/B Test Calculator
- Statistical Significance Calculator
Statistical Significance Calculator
Enter visitors and conversions for two versions to see whether the difference between them is more than chance, with the p-value.
Verdict
B is better, with 95% confidence
If A and B really converted equally, a gap this large in either direction would show up in about 2.8% of tests. That is below the 5% this confidence level allows.
- Conversion rate A
- 5.00%
- 95% interval: 4.59% to 5.44%
- Conversion rate B
- 5.70%
- 95% interval: 5.26% to 6.17%
- Uplift of B over A
- +14%
- +0.7 percentage points
- p-value
- 0.0278
- z = 2.20, two-sided
What "significant" means here
The calculator asks one question. If A and B really converted equally, how often would chance alone produce a difference at least as large as the one you measured? That frequency is the p-value. When it falls below the threshold that goes with your confidence level, the result is called statistically significant.
| Confidence level | Significant when p is below | z needed, two-sided | z needed, one-sided |
|---|---|---|---|
| 90% | 0.10 | 1.645 | 1.282 |
| 95% | 0.05 | 1.960 | 1.645 |
| 99% | 0.01 | 2.576 | 2.326 |
The z-value, shown under the p-value, is the difference between the two conversion rates counted in standard errors.
What the p-value does and does not say
A p-value of 0.03 says that a gap this size would be unusual if there were no real difference. It does not say that B has a 97% chance of being better, how large the difference really is, or whether it matters to your business. And a result that is not significant is no proof that A and B are equal. Often the test was simply too small.
The same uplift on more and on less data
In every row B converts 14% better than A, 5.7% against 5.0%. Only the number of visitors differs.
| Visitors per variant | Conversions A | Conversions B | p-value | At 95% |
|---|---|---|---|---|
| 1,000 | 50 | 57 | 0.4867 | No clear winner |
| 2,000 | 100 | 114 | 0.3253 | No clear winner |
| 5,000 | 250 | 285 | 0.1199 | No clear winner |
| 10,000 | 500 | 570 | 0.0278 | B is better |
| 20,000 | 1,000 | 1,140 | 0.0019 | B is better |
The uplift never changed. What changed is how sure you can be that it is no accident. The sample size calculator tells you beforehand how many visitors a test needs.
Do not stop at the first significant result
At 95% confidence, 5% of tests between two identical pages still show a winner. That holds only if you judge the test once, at the end. Look repeatedly and stop the moment the result turns significant, and the share rises: to about 14% with five evenly spaced looks, and to about 19% with ten.
The A/B test calculator page explains the test behind both tabs, the formulas and where they fall short.