CFTROU
Shop
Portfolio

PORTFOLIO · EXPERIMENTATION

A/B Test Significance Calculator

Compare two conversion rates, test statistical significance, and estimate the value of a wider rollout.

Email subject line test

Control subject line

Rate: 3.00%

New subject line

Rate: 3.50%

Inputs

Confidence levelHow strict to be before calling a winner. 95% means you are willing to accept up to a 5% chance of treating a fluke as a real win. Pick 99% if a wrong call is costly, or 90% if you want to move faster on weaker evidence.
Minimum meaningful lift (%)The smallest improvement that would actually be worth the cost and risk of shipping. A real but tiny lift might not justify the engineering effort. Set this to whatever size of gain would make rolling out the change worthwhile.
Revenue per click ($, optional)What one conversion is worth in dollars. Used only to translate the lift into the estimated revenue impact below. Leave it blank to skip that estimate.
Expected monthly traffic (optional)Left blank, the revenue estimate below covers only this test's sample. Enter your expected visitors per month after a full rollout to see the cost of not making the change, projected over the next 6 months.
How this is calculated

rateA = conversionsA / visitorsA, rateB = conversionsB / visitorsB

pooledP = (conversionsA + conversionsB) / (visitorsA + visitorsB)

pooledSE = sqrt(pooledP × (1 − pooledP) × (1/visitorsA + 1/visitorsB))

z = (rateB − rateA) / pooledSE → two-tailed p-value from the standard normal CDF

unpooledSE = sqrt(rateA(1−rateA)/visitorsA + rateB(1−rateB)/visitorsB), used for the confidence interval, since the CI doesn't assume the two rates are equal

Minimum detectable effect: the smallest relative lift this sample size could catch with 80% power at the chosen confidence level. It flags an "inconclusive" result instead of mislabeling it "no effect."