Skip to the content

Statistical Significance Calculator

Enter visitors and conversions for two versions to see whether the difference between them is more than chance, with the p-value.

A, the control
B, the variant
Confidence
Test

Verdict

B is better, with 95% confidence

If A and B really converted equally, a gap this large in either direction would show up in about 2.8% of tests. That is below the 5% this confidence level allows.

Conversion rate A
5.00%
95% interval: 4.59% to 5.44%
Conversion rate B
5.70%
95% interval: 5.26% to 6.17%
Uplift of B over A
+14%
+0.7 percentage points
p-value
0.0278
z = 2.20, two-sided
Share this tool:TwitterFacebookLinkedInWhatsAppEmail

What "significant" means here

The calculator asks one question. If A and B really converted equally, how often would chance alone produce a difference at least as large as the one you measured? That frequency is the p-value. When it falls below the threshold that goes with your confidence level, the result is called statistically significant.

Confidence levelSignificant when p is belowz needed, two-sidedz needed, one-sided
90%0.101.6451.282
95%0.051.9601.645
99%0.012.5762.326

The z-value, shown under the p-value, is the difference between the two conversion rates counted in standard errors.

What the p-value does and does not say

A p-value of 0.03 says that a gap this size would be unusual if there were no real difference. It does not say that B has a 97% chance of being better, how large the difference really is, or whether it matters to your business. And a result that is not significant is no proof that A and B are equal. Often the test was simply too small.

The same uplift on more and on less data

In every row B converts 14% better than A, 5.7% against 5.0%. Only the number of visitors differs.

Visitors per variantConversions AConversions Bp-valueAt 95%
1,00050570.4867No clear winner
2,0001001140.3253No clear winner
5,0002502850.1199No clear winner
10,0005005700.0278B is better
20,0001,0001,1400.0019B is better

The uplift never changed. What changed is how sure you can be that it is no accident. The sample size calculator tells you beforehand how many visitors a test needs.

Do not stop at the first significant result

At 95% confidence, 5% of tests between two identical pages still show a winner. That holds only if you judge the test once, at the end. Look repeatedly and stop the moment the result turns significant, and the share rises: to about 14% with five evenly spaced looks, and to about 19% with ten.

The A/B test calculator page explains the test behind both tabs, the formulas and where they fall short.