What is A/B Testing?
A/B testing (split testing) is a method of comparing two versions of a product experience to determine which performs better.
A/B testing (split testing) is a method of comparing two versions of a product experience to determine which performs better. Users are randomly assigned to version A (control) or version B (variant), and a predefined metric is measured to determine the winner.
A/B testing requires statistical rigor: sufficient sample size (use a sample size calculator), appropriate test duration (typically 1-4 weeks), clearly defined success metrics, and statistical significance (p < 0.05 is the standard threshold).
Common A/B testing mistakes: stopping tests too early, testing too many variants simultaneously, choosing vanity metrics as success criteria, not accounting for novelty effects, and running tests on segments too small for statistical significance.
For product decisions, A/B tests are the gold standard of evidence. But they're not always appropriate — features with low traffic can't reach significance, and strategic decisions shouldn't be A/B tested (you don't A/B test your company's mission).
Why It Matters
A/B testing provides causal evidence that a change improves outcomes, unlike observational analytics that show correlation. It removes opinion from product decisions and replaces it with data.
Frequently Asked Questions
What is A/B testing?
A/B testing compares two versions of a product experience by randomly assigning users to each version and measuring which performs better on a predefined metric.
How long should an A/B test run?
Until statistical significance is reached — typically 1-4 weeks depending on traffic volume. Use a sample size calculator before starting. Never stop a test early because results look good.
Related Terms
Need Expert Help?
Richard Ewing is a Product Economist and AI Capital Auditor. He helps companies translate technical complexity into financial clarity.
Book Advisory Call →