A/B testing – what it is, how it works and when to skip it

Definition

A/B testing means showing two variants of a piece of content to different segments of an audience and measuring which one better meets a defined goal – judged on statistical significance rather than gut feeling.

What is A/B testing?

A/B testing – also known as an AB test or split testing – is a method in which two versions of a page, form or email are shown to different segments of your traffic at the same time. Both variants are measured against the same goal, such as the number of leads or completed purchases, and a winner is declared only once the difference is statistically significant. The method replaces opinion and guesswork with data from real visitors.

How does A/B testing work?

Variant A is usually the original (the control) and variant B is the changed version (the challenger). Traffic is split randomly between the variants until enough data has been collected to draw a statistically sound conclusion. What separates A/B testing from an educated guess is the requirement for statistical significance: the result must not be explainable by chance.

How to formulate a testable hypothesis

An A/B test is only as good as its hypothesis. A testable hypothesis is built in four steps:

• Data source – Start with what you know: analytics, heatmaps or session recordings. Where do visitors leave the page? Where do they get stuck?
• Insight – Put into words what the data suggests. For example: "a large share of mobile visitors never scroll down to the form."
• Hypothesis – Write it in the format "If we change X, we expect Y, because Z". For example: "If we move the form higher up the page, we expect more leads, because more visitors will see it."
• Success metric – Define exactly which metric decides the test before it starts, and stick to it.

A good hypothesis needs two things at once: an understanding of the business and what drives conversion, and the technical understanding to know what can be measured correctly. Knowing the tool on its own measures nothing – the tool does not know which question is worth asking.

Statistical significance – what it means in practice

Statistical significance indicates how likely it is that the observed difference is due to the variant and not to chance. The standard in CRO is a 95% confidence level, meaning there is less than a 5% probability that the result is down to chance.

To get there, you need:

• A large enough sample – You need around 400 conversions per variant to demonstrate a 20% improvement. The common rule of thumb of 100 conversions is only enough to detect differences of around 40%. How much data you actually need depends on your baseline conversion rate and how small an effect you want to be able to detect. Work out the sample size in advance, for example with Evan Miller's sample size calculator.
• A long enough run time – At least 1–2 full weeks, to capture how behaviour varies across the days of the week.
• A clear hypothesis – Before the test starts, define what effect you expect and the significance level at which you will stop.

Also watch out for the peeking problem. A typical scenario: the test starts on Monday, and every morning someone opens the testing tool's report for a quick look. On Thursday, variant B suddenly shows 97% confidence – the test is stopped and the winner declared. The problem: if you look every day, you give chance plenty of opportunities to pass itself off as a win, and the risk of false positives rises sharply. Modern testing platforms deal with this through sequential testing methods, which let you follow the results as they come in without compromising validity. If your tool does not support that, decide the run time in advance and let the test run its course.

How long should an A/B test run?

An A/B test should run for at least two full business cycles – for most websites that means two whole calendar weeks – and at most about six weeks. Shorter tests miss differences between days of the week. Longer tests suffer from cookies expiring and seasonal effects creeping into the data. Decide the run time based on the sample size you calculated in advance, not on when the result happens to look good.

Example: what an A/B test can look like in practice

A worked example with made-up figures: a B2B website has a landing page with 8,000 visits a month and a baseline conversion rate of 2% – around 160 leads a month. Heatmaps show that many visitors start filling in the contact form but give up halfway. The hypothesis: "If we shorten the form from seven fields to three, we expect more submitted forms, because it lowers the barrier to finishing."

The team wants to be able to detect an improvement from a 2% to a 3% conversion rate. A sample size calculator shows that this requires roughly 3,900 visitors per variant. With traffic split evenly – 4,000 visits per variant per month – the test needs to run for about four weeks.

After four weeks, the original has 79 conversions (2.0%) and the shortened variant 114 (2.9%). The difference is statistically significant at the 95% level, so the new variant is rolled out. Had the lift been smaller – say from 2.0% to 2.2% – the same test would have needed around 80,000 visitors per variant, which is impractical at this level of traffic. That is why you always calculate the sample size before the test, not after.

If you want to put numbers on your own starting point, you can calculate your conversion rate with our calculator.

A/B vs multivariate testing – when to choose which?

An A/B test changes one thing at a time. A multivariate test tries combinations of several elements at once – for example headline, image and CTA text in different combinations. It needs considerably more traffic, but it reveals interaction effects that an A/B test never can.

As a rule of thumb, stick with A/B testing until you have enough data on how each element performs on its own. Multivariate testing is for websites with high traffic volumes and a mature testing culture.

What should you A/B test?

The elements that typically have the biggest impact on conversion rate, in descending order:

• The offer – What you offer and how it is worded: positioning, price, terms
• The headline – The first impression that decides whether the visitor reads on
• CTA – The text, colour, placement and size of the call-to-action button
• Forms – The number of fields, their order and their labels
• Images and video – The choice of visual content, especially hero images
• Social proof – The placement and design of testimonials and reviews

The same logic applies to email: subject lines and send times are tested in exactly the same way, but with shorter cycles and a lower traffic threshold, because each send is a naturally self-contained experiment.

Avoid testing cosmetic changes without a clear business hypothesis.

Tools for A/B testing

• Optimizely – An enterprise platform with an advanced statistics engine and strong integrations with CMSs and data tools.
• Wingify (VWO and AB Tasty) – VWO and AB Tasty merged in 2026 and now operate under the name Wingify. VWO is known for built-in heatmaps, session recording and multivariate testing, AB Tasty for strong personalisation features and good support for European markets.

For small and medium-sized businesses, price is often the deciding factor: Optimizely is priced at enterprise level, while VWO and AB Tasty are more affordable ways in. And if your website has little traffic, the tool is rarely the bottleneck anyway – the problem is sample size, not the platform.

Note that Google Optimize was shut down in September 2023 and is no longer an option.

Does A/B testing affect your SEO?

A/B testing done correctly does not harm your rankings. Google Search Central sets out four guidelines:

• No cloaking – Do not show Googlebot different content from what users see. Let the testing tool treat the search engine's crawler like any other visitor.
• Use rel="canonical" – If the variant has its own URL, its canonical should point to the original so the search engine understands which version is the real one.
• Use 302, not 301 – In redirect tests, the redirect should be temporary (302). A permanent 301 signals that the original page has been replaced.
• Only run the test for as long as necessary – A test left running long after you have reached a conclusion can be seen as an attempt to mislead the search engine.

When should you NOT A/B test?

A/B testing requires traffic, and what counts is conversions, not visitors. To demonstrate a 20% improvement you need around 400 conversions per variant – in the region of 1,000 conversions a month in the flow you are testing. Most websites belonging to small and medium-sized Swedish companies are nowhere near that. That is no failure – A/B testing is just the wrong tool for the stage you are at.

What to do instead:

• Heatmaps – Show where visitors click and how far they scroll. Hotjar is well established; Microsoft Clarity is free.
• Session recordings – Reveal where individual visitors get stuck, hesitate or give up.
• User testing – Ask around five people to complete specific tasks on your website. That alone goes a long way towards finding the most serious problems.
• Customer interviews – Surface objections and questions that no quantitative data will show.

With low traffic, qualitative analysis often delivers more improvement per hour invested than A/B testing – and the insights become ready-made hypotheses for the day you have enough traffic to test them. We go through how that investment pays for itself in what a CRO agency costs and how to calculate ROI.