A/B Testing for ML Models - Interactive Visualization

A/B testing is the gold standard for deciding whether a new model is actually better than the current one in production. Two groups of users receive predictions from Model A (control) and Model B (treatment). The z-test for proportions determines whether the observed difference in conversion rates could have occurred by chance. A p-value below 0.05 means: if the null hypothesis were true (no real difference), there is less than 5% probability of observing a difference this large. Effect size, sample size, and significance threshold all interact - this demo makes those relationships visible and interactive.

• p-value interpretation: p=0.03 means 3% chance of observing this uplift if there were no real model difference
• Confidence interval: the 95% CI shows the plausible range of the true conversion rate difference
• Statistical power: with small samples, even real improvements are undetectable - minimum sample calculator shown
• Two-tailed test: tests for any difference (B better OR worse than A) - the safe default for model experiments
• One-tailed test: only tests if B is better - use only when regression is impossible (rare in practice)
• Multiple testing problem: running many A/B tests simultaneously inflates false positive rate - use Bonferroni correction

Part of the EngineersOfAI Interactive 3D - free interactive visualizations covering every major concept in machine learning and AI engineering. Hover any element for a plain-English explanation. No code required.