The question it tries to answer
When you compare two versions of a page or email, the numbers almost never come out identical. One will be a little ahead. The question is whether that gap reflects a real difference in how people respond, or just the normal randomness of who happened to see which version.
Statistical significance is a way of putting a rough measure on that. It does not tell you that a variant is good, or that a result is important. It tells you how surprising your result would be if there were actually no difference between the versions.
An everyday picture: flipping a coin
Flip a fair coin ten times. You will not always get exactly five heads. Getting six or seven is common, and nobody would conclude the coin is rigged. Now suppose you flip it a thousand times and get 700 heads. That would be very hard to explain by chance, and you would suspect something is off.
Testing works the same way. A small gap on few visitors is the “seven heads out of ten” situation: unremarkable. A consistent gap across a large number of visitors is the “700 out of a thousand” situation: harder to put down to luck.
Key terms in plain language
- Null hypothesis. The assumption that the two versions perform the same. Tests ask whether the data make that assumption hard to believe.
- P-value. Roughly, how likely you would be to see a gap at least this large if the versions truly performed the same. A small p-value suggests the gap is unlikely to be pure chance. It is not the probability that your variant is better, which is a common misreading.
- Significance level. The threshold you pick in advance for “unlikely enough”. A common convention is 5 percent, but it is a convention, not a law of nature.
- Confidence level. The flip side of the same threshold. A 95 percent confidence level corresponds to a 5 percent significance level. It does not mean you are 95 percent sure the variant is better.
- Confidence interval. A range of plausible values for the true difference. A wide range tells you the data are imprecise even if the headline gap looks big.
Your testing tool will usually report some of these. Check its documentation to see which method it uses, since tools differ.
Why early results mislead
When a test has just started, only a few people have been counted, so the numbers jump around. A variant can lead by a wide margin in the first few days and then drift back to level. If you check often and stop as soon as the numbers cross a threshold, you give chance many opportunities to fool you. This is the mistake known as peeking, one of the most common testing pitfalls.
The fix is dull but effective: decide your sample size and run length first, wait until you reach them, then read the result.
Significant does not mean important
With very large audiences, tiny differences can be statistically significant while being too small to matter commercially. The reverse also happens: a big, valuable difference may fail to reach significance because the sample was small. Always look at the size of the change, and ask whether it is worth acting on, as well as whether it is likely to be real.
Not significant does not mean no effect
If a result is not significant, the honest reading is usually “we cannot tell”. The variant might truly make no difference, or it might help or hurt a little and your test lacked the traffic to show it. This is common on small websites, and it is why checking how much traffic you need matters before you begin.
Many comparisons, more flukes
If you look at twenty metrics or segments, you should expect one or two to look significant by luck alone at a 5 percent threshold. Choose one primary metric in advance. Treat surprises found by digging afterward as ideas for new tests, not as conclusions.
A worked illustration
Imagine 5,000 visitors per version. Version A converts 150 people (3.0 percent) and version B converts 165 (3.3 percent). B has 15 more conversions, a relative lift of 10 percent (15 divided by 150). Many people would call that a win. Yet a gap of 15 on a base of 150 is within the range that ordinary variation can produce, so a tool would likely report it as not clearly significant. The sensible conclusion is “promising but unproven”, perhaps worth a longer test, not a firm decision. These numbers are invented for illustration.
Using the idea sensibly
- Set the threshold and the sample size before you start.
- Do not stop early because a result looks good.
- Report the size of the difference and the uncertainty, not just “significant”.
- Be cautious with surprising winners and repeat important tests.
- Remember that no threshold guarantees a result will hold in future. Outcomes vary.
You do not need to do the mathematics by hand. What matters is knowing what the label means: a measure of how well your data rules out luck, not a guarantee of success. It is one input to a decision, alongside your test hypothesis, common sense about your customers, and the cost of being wrong.