Why the numbers matter
A test compares two groups of visitors. Every visitor is a bit random: some convert, some do not, for reasons you cannot see. With a large group those random effects average out. With a small group they dominate. That is why sample size is the first thing to think about, and why small samples are unreliable.
Here is a simple illustration with invented numbers. Say a page converts about 3 percent of visitors. If 100 people see a version, you would expect about 3 conversions. If a different version gets 4, that is a 33 percent relative increase (1 more on a base of 3), yet it is one person. Run the same version again and it could easily produce 2 or 5. The gap tells you almost nothing.
What drives the traffic you need
Several factors determine how many visitors each version needs:
- Your baseline conversion rate. Lower rates mean fewer conversions per visitor, so you need more visitors to see enough of them.
- The size of the change you want to detect. Spotting a big improvement takes fewer visitors than spotting a small one. Detecting a lift of a few percent relative typically needs far more traffic than detecting a lift of half.
- How sure you want to be. Demanding stronger evidence requires more data. This relates to statistical significance and the chance of being fooled by luck.
- The number of versions. Each extra variant splits your traffic further, which is why multivariate testing is hungry for visitors.
The counts that matter are conversions, not just visitors. A page with 10,000 visitors and 20 conversions has less to learn from than one with 2,000 visitors and 200.
Working out a number
Most testing tools and many free online calculators will estimate the required sample for you. You usually enter your current conversion rate and the smallest improvement you would care about. Different tools use different methods and assumptions, so treat the result as a planning estimate, not a precise rule. Check the tool’s own documentation to see what it assumes.
Then divide by your weekly traffic to see how long the test would run. If the answer is many months, the test is probably not practical in that form. Tests that drag on can also be disrupted by changes in season, promotions or site updates.
A quick sanity check
If you expect fewer than a few dozen conversions per version across the whole test, the result will probably be too noisy to act on. That is not a precise threshold, just a signal to be cautious. Rare outcomes, such as purchases on a small shop, accumulate slowly.
Options for low-traffic sites
If your numbers are small, you still have choices:
- Test bigger changes. A radical change in offer, layout or message is easier to detect than a button colour. Large effects need fewer visitors.
- Test where traffic is highest. The homepage or main landing page usually has more visitors than a deep product page.
- Use an earlier step as the metric. A click or form start happens more often than a purchase, so it accumulates faster. The risk is that it may not predict the final result, so treat it as a clue.
- Run for longer. A planned run of several weeks is often better than a rushed one.
- Combine pages cautiously. Testing the same change across several similar pages can pool the results, though pages may behave differently.
- Use other evidence. Customer feedback, short surveys, session recordings, and reading support questions can show where visitors struggle without needing statistical volume.
- Make the change and watch carefully. A before-and-after comparison is weaker than a test because other things change over time, but a clear, well-reasoned improvement can still be worth making. Be honest in your notes about how much confidence it deserves.
Paid traffic and email
If you buy ads or send email, you control some of your traffic. An email list of a few thousand can support subject line testing, but only for fairly large differences. Be careful about buying traffic just to run a test: visitors from a different source may not behave like your usual audience.
Do not stop at the first sign
Whatever the number, avoid checking daily and ending the test when a winner appears. Decide the sample size and duration in advance and stick to them. That habit is one of the most important ways to avoid the pitfalls of everyday A/B testing.
A realistic mindset
Not every site can run formal experiments, and that is fine. Testing is a tool for reducing guesswork, not a requirement. If traffic is limited, focus on sound basics, collect qualitative feedback, and test only the changes that could matter. Outcomes vary, and a modest, honest test is better than an impressive-looking result built on too little data.