Why a log is worth the effort
Running a test is only half of the work. If the details stay in someone’s head, or in a tool that gets cancelled, they disappear. Months later a new colleague proposes the same idea, runs it again, and the business pays for the same lesson twice.
A testing log fixes that. It is a single place where every experiment is described, what happened is recorded, and the decision is explained. Over time it turns individual results into a body of knowledge about what your audience does and does not respond to. It can also protect against selective memory, because teams tend to remember the wins and forget the inconclusive tests.
Where to keep it
Keep the log simple enough that people actually use it. A shared spreadsheet or a shared document is enough for most small businesses. What matters is that it is:
- Shared, so everyone who runs or requests tests can read it.
- Searchable, so you can look up “headline” or “checkout”.
- Owned by someone, even if everyone contributes.
- Stored somewhere stable and backed up.
If you use a customer relationship management system or a project tool, you can keep it there, but do not let tool setup delay the habit.
What to record
A sensible record has fields covering the whole life of the experiment.
Before the test
- A short name and a date.
- The page, email or channel involved.
- The test hypothesis: what you changed, what you expect and why.
- The primary metric, and any secondary metrics you will watch for harm.
- The planned sample size and run length.
- Screenshots or copies of the control and variant.
After the test
- Actual start and end dates.
- Number of visitors and conversions per version.
- The result in plain words: variant better, control better, or no clear difference.
- How confident you are, including any statistical significance measure your tool reported and its caveats.
- Anything unusual during the test, such as a promotion, an outage or a traffic spike.
- The decision: roll out, keep the control, or run a follow-up.
- What you learned, in one or two sentences.
- Ideas for next tests this result suggests.
Record the losses and the draws
Inconclusive and negative results are valuable. They tell you what not to waste time on, and they correct overconfidence. Write them down with the same care as the wins. If the sample was small, say so, since a “no difference” from a tiny test may only mean the test could not detect one.
Write for a stranger
The best entries make sense to someone who joins the team in a year. Avoid shorthand such as “V2 beat V1”. Say what V2 was. Include the audience too, for example “new visitors from search, mobile and desktop combined”, because a result may not transfer to a different group.
Tag entries so patterns appear
Add a few tags to each row: the element tested (headline, call to action, form, image), the channel (landing page, email, ad), and the audience segment. After a dozen or so tests, you can filter and look for patterns, such as whether shorter forms consistently help or whether benefit-led wording tends to do better. Treat patterns as hints to test further, not as rules, since each result came from a particular audience at a particular time.
Make it part of the routine
- Add the log entry as a required step before a test is launched, so hypotheses are written before results are seen.
- Hold a short review at regular intervals, say monthly or quarterly, to read recent entries and decide next steps.
- Use the log to feed test prioritization, so the ideas queue sits next to the results.
- When a result changes how you work, such as a new standard form length, record that decision separately so it is easy to find.
Respect data limits
Logs should not contain personal data about individual visitors. Record aggregate numbers only. If your testing involves personal information, consent and privacy rules vary by country and change over time, so check the rules that apply to you; this is general information, not legal advice.
Revisit old results
Audiences and markets change. Mark important findings with a date, and consider rerunning key tests after a year or after a major shift in your traffic or offer. The log makes it easy to see what is old and what is worth checking again.
A log will not make your tests win more often. What it does is make sure that every test, whatever the outcome, leaves something behind that the next one can build on.