Why Most A/B Tests Are Run Wrong
Running an A/B test feels simple: split traffic, measure conversion, pick the winner. In practice, most teams make at least two of the following errors — and those errors produce false positives that get shipped and later fail to replicate in production.
Scroll horizontally to inspect the complete figure.
Skipping Step 2 can leave an experiment underpowered or needlessly large, making its conclusion harder to trust.
Step 01
Define the Hypothesis Precisely
A vague hypothesis produces a vague result. "The new onboarding will perform better" is not a hypothesis. A testable hypothesis has three parts:
- Treatment: what exactly is changing (new onboarding flow — 3 steps instead of 7)
- Metric: what you are measuring (7-day activation rate — user completes first core action)
- Direction: what you expect (Version B increases 7-day activation by at least 3 percentage points)
The MDE (Minimum Detectable Effect) — that 3pp number — is not arbitrary. It should be driven by business impact. If a 1pp improvement is worth shipping, set MDE to 1pp. If only 5pp+ justifies the engineering cost, set it there. MDE determines how many users you need.
Step 02
Calculate Sample Size Before You Start
This is the step most teams skip. The consequences: either stopping too early (false positives) or running too long (wasted resources).
from statsmodels.stats.power import NormalIndPower
import numpy as np
baseline_rate = 0.32 # current 7-day activation rate
mde = 0.03 # minimum lift worth shipping (3pp)
alpha = 0.05 # acceptable false positive rate
power = 0.80 # probability of catching a real effect
effect_size = mde / np.sqrt(baseline_rate * (1 - baseline_rate))
analysis = NormalIndPower()
n_per_variant = analysis.solve_power(
effect_size=effect_size,
alpha=alpha,
power=power,
alternative='larger'
)
print(f"Users needed per variant: {int(n_per_variant):,}")
print(f"Total users needed: {int(n_per_variant * 2):,}")
# Output: ~5,600 per variant, ~11,200 totalScroll horizontally to inspect the complete figure.
Step 03
Randomization and SRM Check
Sample Ratio Mismatch (SRM) is when the actual split differs from the intended split. If you intended 50/50 but got 52/48, your randomization is broken and results are invalid. Always run a chi-squared test on the split before analyzing outcomes.
from scipy.stats import chi2_contingency
import numpy as np
observed_a = 5240 # actual users in control
observed_b = 4760 # actual users in treatment
total = observed_a + observed_b
expected = total / 2 # 50/50 split intended
chi2, p_value, _, _ = chi2_contingency([
[observed_a, expected],
[observed_b, expected]
])
print(f"SRM p-value: {p_value:.4f}")
if p_value < 0.01:
print("SRM DETECTED — do not analyze results")
else:
print("No SRM — proceed with analysis")Step 04
The 6 Most Common Experiment Mistakes
Checking results daily and stopping at p < 0.05 inflates false positive rate to ~26% at alpha=0.05
Without MDE, you can't calculate sample size. You're guessing when to stop.
Testing 10 metrics at alpha=0.05 expects 0.5 false positives by chance alone. Apply Bonferroni.
New UX gets engagement spike. Run for 2+ weeks to see steady-state behavior.
Unequal splits invalidate the statistical test entirely.
Social products: A/B users influence each other. Use cluster randomization.
Step 05
Analyzing and Reporting Results
from scipy import stats
import numpy as np
n_a, n_b = 5600, 5600
conv_a, conv_b = 1792, 2016 # 32% vs 36%
p_a = conv_a / n_a
p_b = conv_b / n_b
p_pool = (conv_a + conv_b) / (n_a + n_b)
se = np.sqrt(p_pool * (1 - p_pool) * (1/n_a + 1/n_b))
z = (p_b - p_a) / se
p_value = 1 - stats.norm.cdf(z)
diff = p_b - p_a
ci_lower, ci_upper = diff - 1.96*se, diff + 1.96*se
print(f"Control: {p_a:.1%}")
print(f"Treatment: {p_b:.1%}")
print(f"Lift: {diff:.1%} ({diff/p_a:.1%} relative)")
print(f"95% CI: [{ci_lower:.1%}, {ci_upper:.1%}]")
print(f"P-value: {p_value:.4f}")
print(f"Decision: {'Ship' if p_value < 0.05 and diff >= 0.03 else 'Do not ship'}")Decision
Decision Framework
- Significant and above MDE: consider shipping after guardrail, quality and operational review
- Significant but below MDE: weigh the estimate and interval against engineering cost and strategic value
- Not significant: evidence is insufficient for the planned decision rule; review power, uncertainty and whether another test is useful
- Negative result: the treatment hurt — investigate why before discarding
Negative results are as valuable as positive ones. They tell you what not to build next. A team that treats every p < 0.05 as a shipping rule risks acting on effects that are too small or too uncertain to matter.
Summary
Key Takeaways
- Define the decision-relevant effect before running because it shapes sample-size and interpretation choices
- Calculate sample size before starting instead of relying on an unsupported guess
- Check for SRM before analyzing outcomes — a broken split invalidates the math
- Statistical significance and practical significance are different — you need both
- Peeking is the most common error — commit to a sample size and wait