Step by Step
A
Random assignment — eliminating selection bias
Users must be randomly assigned to version A (current) or version B (new), typically a 50/50 split, ensuring the two groups are comparable and any outcome difference can be attributed to the change itself, not pre-existing group differences.
Example: randomly assigning 50% of live website visitors to the current checkout flow (A) and 50% to a redesigned checkout flow (B).
B
Single change — isolating the cause
A/B tests should isolate a single change between versions A and B — changing multiple things simultaneously makes it impossible to know which specific change caused any observed difference.
Example: testing only a new checkout button color, rather than simultaneously changing the button color, page layout, and pricing all at once.
⚖
The right metric — business outcome, not just ML metric
A/B tests should measure the actual business metric that matters (conversion rate, revenue, retention) rather than just an internal ML metric (like model accuracy), since the ultimate goal is real-world impact.
Example: measuring actual purchase conversion rate rather than just measuring how confident the recommendation model's predictions are.
⏱
Never stop early — wait for the full pre-specified duration
A classic mistake is stopping a test as soon as a statistically significant result (p<0.05) first appears, rather than waiting for the full pre-specified test duration — this practice inflates false positive rates significantly.
Example: stopping a two-week planned test after just three days because early results happened to show p<0.05, when that early significance may well disappear or reverse if the test had continued to completion.
Applied Walkthrough
1
A team runs an A/B test comparing a new recommendation algorithm (B) against the current one (A), planning for a two-week test duration.
2
After just three days, they notice the new algorithm B shows a statistically significant improvement (p<0.05) and are tempted to stop the test early and declare victory.
3
Ask: is stopping early a sound practice? No — stopping as soon as p<0.05 first appears, rather than waiting for the full pre-specified duration, significantly inflates the false positive rate.
4
The correct approach is to commit to the full pre-specified two-week duration regardless of early results, ensuring the final conclusion is statistically trustworthy rather than a potentially spurious early blip.
Exam Application
Exams test whether you understand the core principles of a valid A/B test (random assignment, single isolated change, sufficient sample size, pre-specified duration) and specifically why stopping early when significance first appears is a serious, commonly tested methodological error.
⚠ Common Trap
The most common trap is stopping a test the moment it reaches statistical significance (p<0.05), rather than committing to the full pre-specified duration decided before the test began. This practice, sometimes called "peeking," significantly inflates the risk of a false positive conclusion.
✓ Quick Self-Check
1. Why is random assignment essential in an A/B test?
It eliminates selection bias, ensuring the two groups are comparable and any outcome difference can be attributed to the change itself.
Tap to reveal / hide
2. Why should an A/B test isolate a single change between versions?
To isolate the cause — testing multiple simultaneous changes makes it impossible to know which specific change drove any observed difference.
Tap to reveal / hide
3. Should A/B tests measure ML metrics or business metrics?
Business metrics (like conversion or revenue), since those reflect the actual real-world impact that matters.
Tap to reveal / hide
4. What is the common mistake regarding when to stop an A/B test?
Stopping as soon as p<0.05 first appears, rather than waiting for the full pre-specified duration — this inflates the false positive rate.
Tap to reveal / hide
5. What is the risk of stopping a test early upon seeing significance?
A significantly inflated false positive rate — the early result may be a spurious blip that would disappear with the full planned duration.
Tap to reveal / hide