Variant A (Control)10.00%
Variant B12.50%
Confidence level
RESULT
+25.00%
Trending — keep testing
RATE A
10.00%
RATE B
12.50%
P-VALUE
0.0526
95% CI for difference: [-0.03%, 5.03%]
TEST SUMMARY
Head-to-head
Variant A10.00%
Variant B12.50%
Z-SCORE
1.938
α
0.05
Rules of thumb
- Decide sample size before the test. Peeking early inflates false positives.
- Aim for at least a few hundred conversions per variant for a reliable read.
- Run whole business cycles — weekends and weekdays behave differently.
Combine it with lift, confidence intervals, and product judgment before shipping a change.
One test at a time
Change a single variable per experiment so you can attribute the lift to something real.
Mind the effect size
Huge samples make tiny differences “significant”. Ask if the lift is worth the ship risk.
Watch for novelty
Big early lifts often fade. Let the test run long enough for behavior to normalize.
