A/B testing explanations often emphasize mechanics such as p-values, confidence intervals and z-tests. Product decisions also require assumptions, effect size, uncertainty and business context. This post covers both: how to run the test correctly and the mistakes that make correct tests produce wrong decisions.
What Statistical Significance Actually Means
A result is statistically significant when the observed statistic would be sufficiently unusual under the chosen null model and test assumptions.
The Setup: A/B Test for Onboarding
Hypothesis: a new onboarding flow (Version B) improves activation rate compared to the current flow (Version A).
SELECT
variant,
COUNT(DISTINCT e.user_id) AS total_users,
COUNT(DISTINCT p.user_id) AS converted,
ROUND(
COUNT(DISTINCT p.user_id)::DECIMAL /
COUNT(DISTINCT e.user_id), 4
) AS conversion_rate
FROM experiments e
LEFT JOIN purchases p ON e.user_id = p.user_id
GROUP BY variant;
-- Result:
-- A: 32.4% (3240 / 10000)
-- B: 36.5% (3650 / 10000)Running the Z-Test in Python
from scipy import stats
import numpy as np
n_a, n_b = 10000, 10000
conv_a, conv_b = 3240, 3650
p_a = conv_a / n_a # 0.324
p_b = conv_b / n_b # 0.365
p_pool = (conv_a + conv_b) / (n_a + n_b)
se_null = np.sqrt(p_pool * (1 - p_pool) * (1/n_a + 1/n_b))
se_diff = np.sqrt(p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b)
z = (p_b - p_a) / se_null
p_value = 1 - stats.norm.cdf(z)
diff = p_b - p_a
margin = 1.96 * se_diff
ci_lower = diff - margin
ci_upper = diff + margin
print(f"Conversion A: {p_a:.1%}")
print(f"Conversion B: {p_b:.1%}")
print(f"Lift: {diff:.1%}")
print(f"P-value: {p_value:.4f}")
print(f"95% CI: [{ci_lower:.1%}, {ci_upper:.1%}]")
print(f"Significant: {p_value < 0.05}")
# Output:
# Conversion A: 32.4%
# Conversion B: 36.5%
# Lift: 4.1%
# P-value: 0.0000
# 95% CI: [2.7%, 5.5%]
# Significant: TrueSample Size — Calculate It Before You Start
Running a test without calculating required sample size first is one of the most common errors. You either stop too early (false positives) or run too long (wasted time).
from statsmodels.stats.power import NormalIndPower
baseline_rate = 0.324 # current conversion rate
mde = 0.03 # minimum lift you care about
alpha = 0.05
power = 0.80
analysis = NormalIndPower()
n = analysis.solve_power(
effect_size=(mde / np.sqrt(baseline_rate * (1 - baseline_rate))),
alpha=alpha,
power=power,
alternative='larger'
)
print(f"Required per variant: {int(n):,}")
print(f"Total required: {int(n * 2):,}")If expected traffic cannot reach the planned sample in a useful period, reconsider the detectable effect, design, decision cost or whether experimentation is the right method.
Four Mistakes That Produce Wrong Results
- Peeking — checking results daily and stopping when p < 0.05. This inflates your false positive rate. Use fixed sample sizes or sequential testing methods.
- Multiple comparisons — testing 10 metrics and declaring victory when one crosses 0.05. At alpha = 0.05, you expect one false positive every 20 tests by chance. Apply Bonferroni correction or define one primary metric upfront.
- Novelty effect — new users behave differently because the change is new, not because it's better. Choose a duration that covers relevant product cycles and assess novelty or learning effects; no single duration fits every product.
- Network effects — in social products, users in A and B influence each other. Standard A/B testing assumes independence. When interference is material, consider cluster randomization or another design that models the dependency.
What to Report to Stakeholders
Never just report a p-value. Report:
- Baseline and treatment conversion rates
- Absolute lift (4.1%) and relative lift (12.7%)
- 95% confidence interval — shows the range, not just the point estimate
- Sample sizes and test duration
- Business impact — if 4.1% lift holds at full rollout, what does that mean in revenue?
- Your recommendation — ship, do not ship, or run a follow-up test
Stakeholders make decisions. Give them what they need to decide, not a statistics lecture.