Why mistakes are so easy to make
Most A/B testing errors are not technical. They come from wanting an answer sooner than the data can give one. A test produces a number whether or not that number means anything, so the discipline is in knowing when to trust it. The pitfalls below are the ones that most often turn a noisy result into a confident, wrong decision.
Peeking and stopping early
Checking results every day is natural. Acting on them is the problem. Early in a test, numbers swing widely because few people have been counted. If you stop the moment one version leads, you will often stop on a chance gap that would have faded.
Imagine a test where version B is ahead after two days with 4 conversions to 2. That is a doubling on paper, yet the totals are tiny. A few more conversions for A would erase it. Decide the run length and the sample size in advance, and judge the result only when you reach them. If you must look, treat what you see as provisional.
Ending the test too soon in the calendar
Even with enough visitors, a test that covers only a few days can mislead. Behaviour changes across the week, and around paydays, holidays, promotions and sales. A common guideline is to run for at least one or two full weeks so each weekday appears evenly, but the right length depends on your traffic and business cycle.
Calling a winner without enough data
Small samples are unreliable. A handful of conversions either way proves little, and it is tempting to read meaning into a 20 percent difference that rests on a few people. Understand how statistical significance works, and ask whether the gap is big relative to the noise, not just whether it exists.
Testing too many things at once
Changing a headline, an image and a form in one version tells you whether the bundle worked, but not why. Running many tests at the same time on the same page can also interfere with each other. A related temptation is to compare many variants at once. The more versions you compare, the more likely one looks good by luck. Keep tests focused, and consider multivariate testing only when traffic is high.
Watching too many metrics
If you track twenty metrics, one will probably move by chance. Choose a primary metric before you begin and treat other numbers as supporting context. Picking whichever metric looks best after the fact is a form of fooling yourself, even when it is unintentional.
Choosing the wrong metric
Clicks are easy to count but may not reflect value. A variant might win more clicks and produce fewer purchases, for example if the wording attracts the wrong audience. Where possible, measure the outcome closest to the business goal, and check for side effects such as increased refunds or unsubscribes.
Starting without a hypothesis
Testing random ideas leaves you with results but no understanding. A written test hypothesis states what you changed, what you expect and why. Even if the variant loses, the reason attached to the prediction tells you what to try next.
Uneven or contaminated groups
For the comparison to be fair, groups must be similar and treated alike:
- Assign visitors randomly rather than by day or by source.
- Run both versions at the same time, never one after the other.
- Keep returning visitors on the same version.
- Watch for outside changes mid-test, such as a new ad campaign sending a different audience.
Ignoring segments, or over-slicing them
Sometimes a change helps one group and hurts another, such as mobile and desktop visitors. Looking at segments can reveal this. But slicing the data into many small pieces after the fact creates more chances for flukes. Plan the segments you care about in advance, and treat unexpected findings as ideas for new tests rather than proof.
Not recording what you learned
Teams often run a test, act on it and forget the details. Months later someone repeats the same experiment. A shared testing log of ideas, hypotheses, dates, results and decisions prevents this and helps results accumulate.
Treating one result as permanent
An audience, a season or a competitor can change. A winner is the best evidence you had at the time, not a law. Revisit important results occasionally, especially after big changes to traffic sources or the offer.
A short checklist
- One clear goal and one primary metric, chosen in advance.
- A written hypothesis.
- A planned sample size and duration.
- Random, simultaneous, stable groups.
- No early stopping.
- A written record of the outcome.
Following these habits will not guarantee a winner, and most tests are inconclusive. They do make it far more likely that the decisions you take from your tests are based on something real.