Skip to the content

A/B Test Calculator

Check whether the difference between two variants is more than chance.

A, the control
B, the variant
Confidence
Test

Verdict

B is better, with 95% confidence

If A and B really converted equally, a gap this large in either direction would show up in about 2.8% of tests. That is below the 5% this confidence level allows.

Conversion rate A
5.00%
95% interval: 4.59% to 5.44%
Conversion rate B
5.70%
95% interval: 5.26% to 6.17%
Uplift of B over A
+14%
+0.7 percentage points
p-value
0.0278
z = 2.20, two-sided
Share this tool:TwitterFacebookLinkedInWhatsAppEmail

How the result is calculated

The calculator runs a two-proportion z-test. It assumes for a moment that A and B convert equally well, pools everything into one conversion rate, and then measures how far apart the two observed rates are in standard errors.

StepFormula500 of 10,000 against 570 of 10,000
Conversion ratesconversions ÷ visitors5.00% and 5.70%
Pooled rate pall conversions ÷ all visitors1,070 ÷ 20,000 = 5.35%
Standard error√(p × (1 − p) × (1/nA + 1/nB))0.00318
z(rate B − rate A) ÷ standard error0.0070 ÷ 0.00318 = 2.20
p-valueshare of the normal curve beyond ±2.200.0278

0.0278 is below 0.05, so this result is significant at 95% confidence. It is above 0.01, so at 99% it is not.

The interval under each conversion rate is a Wilson score interval, always two-sided at the confidence level you picked. It is used here instead of the simpler "rate ± 1.96 standard errors" because it stays sensible for small samples and for rates close to zero.

What the p-value means

The p-value is the chance of seeing a difference at least this large if A and B in truth perform the same. It is not the chance that B is better, and "95% confidence" does not mean the uplift is 95% certain to be real.

Decide the sample size before you start

Use the sample size tab first, then let the test run until every variant has had that many visitors. The 5% error rate of a 95% test only holds if you judge the result once. Checking along the way and stopping the moment the result turns significant is called peeking, and it hands chance several attempts at crossing the line: with five evenly spaced looks, about 14% of tests between two identical pages produce a "winner", and with ten looks about 19%.

The planner uses the standard formula for comparing two proportions:

n = (zc × √(2 × m × (1 − m)) + zp × √v)² ÷ d²

Here m is the average of the current rate p1 and the improved rate p2, v is p1 × (1 − p1) + p2 × (1 − p2) and d is the difference p2 − p1. zc follows from the confidence level (1.96 for 95%, two-sided) and zp from the power (0.84 for 80%). A 5% baseline with a 10% relative uplift, so 5% against 5.5%, needs 31,234 visitors per variant at those settings. Halving the uplift you want to detect roughly quadruples that number.

One-sided or two-sided

A two-sided test asks whether A and B differ, in either direction. A one-sided test asks only whether B beats A. It reaches significance with less data, but it can never tell you that B is worse. Choose before the test starts: picking the one-sided test after seeing which version is ahead doubles the error rate you thought you had.

Significant is not the same as important

Significance says a difference is unlikely to be chance alone. It says nothing about its size. With enough traffic a lift from 5.00% to 5.05% becomes significant, and it may still not be worth the work of rolling out. Look at the uplift and the intervals, and at what the difference is worth in money.

The reverse also holds. "No clear winner" does not prove that A and B are equal; the test may simply have been too small to see the difference.

Where this test falls short

The z-test leans on a normal approximation that gets rough when a variant has only a handful of conversions, which is why small samples get a warning. It also assumes each visitor is counted once and was assigned to a version at random. Comparing three or more versions, or judging the same test on several metrics, raises the chance of a false winner, and this calculator does not correct for that.