What is A/B Testing?

A/B testing is an experimentation method that compares two (or more) variants of a feature by showing each to a randomly assigned segment of real users and measuring which performs better on a chosen metric, used to make data-driven product decisions.

Start a Free Mock Interview

Free to start · 7-day trial on paid plans

IN DEPTH

In depth.

A/B testing is about product impact, not bug-finding. You split live traffic, version A (control) to some users, version B (variant) to others, at random, then measure a metric (conversion, click-through, retention) to see whether the change actually helps. Statistical significance determines whether the difference is real or noise. It is a core practice at data-driven companies for deciding what to build and keep.

This differs from QA "testing," which verifies correctness before release. But QA has a real and often underappreciated role in A/B testing: validating the experimentation system itself. Are users correctly and randomly assigned to variants? Is assignment sticky (a user stays in the same variant)? Are events tracked accurately so the metrics are trustworthy? Do both variants actually work (no broken variant skewing results)? A bug in assignment or tracking can invalidate an entire experiment, so testing the experiment platform is quality-critical.

For QA engineers, A/B testing also complicates regular testing: feature flags and variants multiply the states an app can be in, so test coverage must account for which combinations are live. Understanding both the experimentation concept and QA's validation role is increasingly expected.

WHY IT MATTERS

Why interviewers ask about this.

A/B testing comes up for roles at experimentation-heavy companies. Distinguishing it from QA testing, and articulating what QA validates (assignment, stickiness, tracking accuracy, both variants working), shows you understand quality in a modern, experiment-driven product organization.

EXAMPLE

Example scenario.

A team A/B tests a new checkout button color. QA validates the experiment before launch: users are randomly and stickily assigned, both variants render and function correctly, and conversion events fire accurately for each variant. The test then runs until it reaches significance, and a tracking bug QA caught would otherwise have made the winning variant look like a loser.

TIP

Interview tip.

Define A/B testing as comparing variants with randomly assigned real users to measure impact on a metric, distinct from correctness-focused QA testing. Then highlight QA's role: validating assignment, stickiness, accurate event tracking, and that both variants work, because experiment-platform bugs invalidate results.

FAQ

Frequently asked questions.

What is the difference between A/B testing and QA testing?

QA testing verifies correctness before release, does the feature work as intended? A/B testing measures product impact after release, which variant performs better with real users? They are complementary: QA ensures both variants work; A/B testing decides which to keep based on data.

What does QA validate in an A/B test?

That users are correctly and randomly assigned to variants, that assignment is sticky (a user stays in one variant), that analytics events are tracked accurately so metrics are trustworthy, and that both variants actually function. Bugs in assignment or tracking can invalidate the entire experiment.