What an A/B test is
An A/B test is a controlled experiment. You take one thing you want to improve, such as a sign-up form or an email, and create two versions that differ in one deliberate way. Version A is the control, usually what you already have. Version B is the variant, with the change you want to try.
Visitors are split at random between the two versions, and you measure the same outcome for both groups. Because the groups are treated the same in every other respect and run at the same time, a difference in the outcome is more likely to come from your change than from something else, such as the season, a news story or a different mix of visitors.
That is the whole idea. Everything else in this guide is about doing it carefully enough that the answer can be trusted.
Why bother with a test
Most marketing decisions are made on opinion: the new headline “feels” stronger, the green button “looks” better. Opinions are useful for generating ideas, but they are poor at predicting what real visitors will do. A test replaces a debate with evidence from your own audience.
Testing also protects you from a quiet risk: comparing this month to last month. If sign-ups rise after a redesign, you cannot tell whether the redesign helped or whether something else changed. A test runs both versions side by side, which removes much of that doubt. Tests do not guarantee improvement, and many ideas turn out to make no measurable difference. That is still a useful result.
Step 1: Pick one goal and one metric
Decide what success means before you start. Choose a single primary metric tied to a real business outcome, for example the share of visitors who complete a form, click a call to action, or buy. Clicks are easy to count, but a click is not always a good outcome, so choose the measure closest to what you actually want.
You can watch secondary metrics as a safety check, such as refund requests or unsubscribes, but decide in advance which one is the main judge. Picking the winner after seeing the data makes it easy to fool yourself.
Step 2: Write a hypothesis
A hypothesis is a specific prediction with a reason: “If we change X, then Y will change, because Z.” A clear test hypothesis keeps the experiment focused and makes the result easier to learn from, even when the variant loses. There is more on this in the guide to writing one.
Step 3: Change one thing
If you change the headline, the image and the button all at once, you will not know which change caused the result. For beginners, testing a single element is the simplest path. Testing several elements at once is possible, but it needs far more traffic and is covered under multivariate testing.
Good candidates are the things visitors notice first: headline writing, the wording of a call to action, the length of a form, or the main image on a landing page.
Step 4: Plan how long and how big
Before launching, estimate how many visitors each version needs. Small numbers produce noisy results, so a handful of conversions either way proves very little. The sample size you need depends on your current conversion rate and on how small a difference you care about detecting. Decide the run length in advance, and make it cover at least one or two full weeks so that weekday and weekend behaviour are both included.
Step 5: Run it cleanly
- Split traffic randomly and, ideally, evenly.
- Run both versions at the same time.
- Keep each visitor on the same version if they return.
- Do not change the test, or other things that affect it, midway.
- Avoid checking the numbers daily and stopping the moment one side looks ahead. Early gaps often shrink or reverse.
Step 6: Read the result honestly
Here is a simple illustration with made-up numbers. Suppose 1,000 visitors see version A and 30 sign up, which is 3.0 percent. Another 1,000 see version B and 36 sign up, which is 3.6 percent. Version B has six more sign-ups, a relative increase of 20 percent (6 divided by 30).
That looks like a win, but with so few sign-ups, chance alone could easily produce a gap of six. This is where statistical significance comes in: it is a way of judging how likely it is that a difference this size appeared by luck. A result that could plausibly be chance should be treated as “not proven” rather than as a winner.
Step 7: Decide and record
There are usually three outcomes: the variant wins clearly, the control wins, or there is no meaningful difference. All three teach you something. If the variant wins, roll it out and keep monitoring. If not, keep what you have and use the learning for the next idea.
Write down what you tested, why, what happened and what you decided. A simple testing log stops you repeating old experiments and turns scattered results into knowledge about your audience.
Things to keep in mind
- Choose where to test by potential impact and effort. Test prioritization helps you avoid spending weeks on tiny details.
- Low-traffic sites can still learn from testing, but may need bigger changes, longer runs or other methods such as customer feedback.
- Tests that collect data from visitors, or involve email addresses, are subject to privacy and consent rules that vary by country and change over time. Check the rules that apply to you; this guide is general information, not legal advice.
- Results describe your audience at one moment. A winner today may not stay one forever, and outcomes vary from business to business.
Start small: one page, one change, one clear metric, and a written hypothesis. Done carefully, even a modest programme of tests builds a reliable picture of what your visitors respond to, and it fits naturally alongside broader conversion rate optimization work and regular review of your marketing analytics.