What Is A/B Testing?
A/B testing (also called split testing) is a method of comparing two versions of a web page, email, ad, or other marketing asset to determine which one performs better. You show version A to one group of users and version B to another group, measure the results, and adopt whichever version delivers the better outcome.
The concept is simple, but the execution requires discipline. Done correctly, A/B testing replaces opinions and assumptions with data. Instead of debating whether a green button or a blue button will generate more clicks, you test both and let the results decide.
For businesses investing in conversion rate optimisation, A/B testing is the foundational methodology. It is how you systematically improve the performance of your website, landing pages, and marketing campaigns — one evidence-based change at a time.
Why A/B Testing Matters
Every element of your website influences whether a visitor converts — your headlines, button text, page layout, images, form length, pricing display, social proof placement, and dozens of other factors. The problem is that you cannot reliably predict which variation of these elements will perform best.
Marketing intuition is valuable, but it is wrong more often than most marketers admit. Studies have shown that even experienced marketers correctly predict the winning variation in A/B tests only about 50% of the time — essentially a coin flip. Without testing, you are making decisions that directly affect revenue based on guesswork.
A/B testing eliminates this uncertainty by providing statistically valid evidence of what actually works. Even small improvements compound significantly over time. A 5% increase in conversion rate on a page that generates £100,000 in annual revenue is worth £5,000 per year — from a single test.
Setting Up Your First A/B Test
Step 1: Choose a Tool
You need an A/B testing platform to manage the experiment. Options range from free to enterprise-grade:
- Google Optimize successor tools: Several platforms have filled the gap left by Google Optimize's shutdown, including free tiers from Statsig, GrowthBook, and similar tools.
- VWO (Visual Website Optimizer): A popular mid-range option with a visual editor for creating variations without code.
- Optimizely: Enterprise-grade testing platform with advanced targeting and statistical analysis.
- Custom implementation: If you have development resources, you can build simple A/B tests using feature flags or server-side splits.
For your first test, choose a tool with a visual editor that lets you create variations without code changes. This reduces the barrier to getting started.
Step 2: Identify What to Test
The best first A/B test targets a high-impact element on a high-traffic page. You want quick, meaningful results to build confidence in the process.
High-impact elements to test first:
- Headlines: The first thing visitors read. Even small wording changes can significantly affect engagement and conversion.
- Call-to-action buttons: Button text, colour, size, and placement all influence click-through rates.
- Hero images or videos: The visual that dominates the top of your page sets the tone for the entire visit.
- Form length: Reducing form fields from 7 to 4 can dramatically increase completion rates (or it might reduce lead quality — you will not know until you test).
- Social proof placement: Moving testimonials or client logos higher on the page can increase trust and conversion.
Avoid testing trivial changes for your first experiment. Testing whether "Submit" performs better than "Submit Form" is unlikely to produce a statistically significant result. Test meaningful differences that have a plausible mechanism for affecting user behaviour.
Step 3: Form a Hypothesis
Every A/B test should start with a clear hypothesis. A hypothesis is not "let's see what happens" — it is a specific, testable statement about what you expect and why.
Weak hypothesis: "Changing the button colour might improve conversions."
Strong hypothesis: "Changing the CTA button from grey to green will increase click-through rate by at least 10% because the current grey button does not stand out against the page background, and green creates stronger visual contrast."
A strong hypothesis has three components:
- What you are changing: The specific element being modified.
- What you expect to happen: The measurable outcome you predict.
- Why you expect it: The reasoning behind your prediction.
The "why" is important because it helps you learn regardless of the result. If your hypothesis is wrong, understanding why you expected a different outcome helps you refine your thinking for future tests.
Step 4: Calculate Your Required Sample Size
One of the most common A/B testing mistakes is ending tests too early. To achieve statistically valid results, you need a sufficient sample size — and that number is usually larger than people expect.
The required sample size depends on three factors:
- Your current conversion rate: Lower conversion rates require larger samples.
- The minimum detectable effect (MDE): The smallest improvement you want to be able to detect. Smaller effects require larger samples.
- Statistical significance level: Typically set at 95%, meaning you want to be 95% confident the result is real and not due to random chance.
Use a sample size calculator (available free online) to determine how many visitors each variation needs before you can draw conclusions. For many websites, this means running the test for 2–4 weeks to accumulate enough data.
Critical rule: Decide your sample size before the test starts and do not look at results until you reach it. Peeking at results mid-test and stopping early when they look good is a form of statistical fraud that produces unreliable conclusions.
Step 5: Set Up the Experiment
Configure your test in your chosen tool:
- Create the variation: Build version B with the change you are testing.
- Split traffic evenly: Allocate 50% of traffic to version A (control) and 50% to version B (variation).
- Define the conversion goal: Specify exactly what action counts as a conversion — button click, form submission, purchase, or whatever metric your hypothesis addresses.
- Set the test duration: Based on your sample size calculation, determine how long the test needs to run.
- Exclude internal traffic: Ensure your own team's visits do not contaminate the data.
Step 6: Launch and Wait
Launch the test and resist the temptation to check results constantly. Seriously. Looking at results before reaching your target sample size leads to premature conclusions. Set a calendar reminder for when the test should have enough data and check then.
During the test, do not make other changes to the page being tested. Changing other elements mid-test invalidates the results because you can no longer attribute differences to your tested variation alone.
Analysing Your Results
Statistical Significance
When your test reaches the target sample size, evaluate the results:
- If one variation performs better with 95% or higher statistical significance: You have a winner. Implement the winning variation.
- If the difference is not statistically significant: The test is inconclusive. This does not mean the variations perform identically — it means you do not have enough evidence to confidently declare a winner.
- If the original performs better: The test succeeded in preventing you from making a harmful change. This is a valuable outcome.
Interpreting the Data
Look beyond the headline conversion rate:
- Segment by device: A variation might win on desktop but lose on mobile, or vice versa.
- Check secondary metrics: A variation that increases button clicks but decreases overall revenue has not actually improved performance.
- Consider the magnitude: A statistically significant 0.1% improvement is real but may not be worth implementing if it requires ongoing maintenance or complicates your codebase.
Documenting Results
Record every test you run — the hypothesis, the variations, the results, and what you learned. This documentation becomes invaluable over time. It prevents you from re-running tests you have already conducted, helps you identify patterns in what works, and builds institutional knowledge about your audience's preferences.
What to Test Next
After your first successful A/B test, build a testing programme that systematically improves your conversion funnel.
Prioritisation Framework
Use the ICE framework to prioritise test ideas:
- Impact: How much could this test improve the metric if the variation wins?
- Confidence: How confident are you that the variation will outperform the control?
- Ease: How easy is the test to implement?
Score each factor from 1 to 10, multiply them together, and prioritise tests with the highest ICE scores.
Testing Across the Funnel
Do not limit testing to a single page. Test across your entire conversion funnel:
- Landing pages: Headlines, hero sections, value propositions, social proof.
- Product/service pages: Descriptions, images, pricing display, feature presentation.
- Checkout/form pages: Form length, field order, progress indicators, trust signals.
- Email campaigns: Subject lines, send times, content layout, CTA placement.
- PPC ads: Ad copy variations, keyword-to-ad alignment, extension usage.
Each stage of the funnel offers opportunities for improvement, and gains at each stage compound multiplicatively.
Advanced Testing Methods
Once you are comfortable with basic A/B tests, explore more sophisticated methods:
- Multivariate testing: Test multiple elements simultaneously to understand interactions between changes. Requires significantly more traffic than simple A/B tests.
- A/B/n testing: Test more than two variations simultaneously. Useful when you have several strong hypotheses for the same element.
- Sequential testing: Statistical methods that allow you to check results at multiple points during the test without inflating false positive rates.
- Personalisation experiments: Test whether personalised experiences outperform generic ones — a natural extension of A/B testing into AI-powered personalisation.
Common A/B Testing Mistakes
Stopping Tests Too Early
The most frequent mistake. If a test shows a 20% improvement after 100 visitors, it is tempting to declare victory and implement the change. But with only 100 visitors, the result is almost certainly noise, not signal. Always reach your predetermined sample size.
Testing Too Many Things at Once
Changing five elements simultaneously makes it impossible to know which change drove the result. Test one element at a time in standard A/B tests. If you want to test multiple elements, use multivariate testing with appropriate sample size calculations.
Ignoring Practical Significance
A test result can be statistically significant but practically meaningless. A 0.02% improvement in conversion rate, even if statistically real, is not worth the effort of implementation for most businesses. Set a minimum meaningful improvement threshold before running the test.
Not Testing Long Enough
Some conversion actions have weekly patterns — B2B forms might convert differently on weekdays versus weekends. Run tests for complete weekly cycles (multiples of 7 days) to avoid day-of-week bias. A minimum of two full weeks is usually advisable.
Failing to Act on Results
Testing without implementing winning variations is wasted effort. Create a clear process for implementing test results promptly. If organisational barriers prevent implementation, address those barriers — they are undermining your optimisation programme.
Building a Testing Culture
The long-term value of A/B testing comes not from individual tests but from building a systematic testing programme. This means:
- Running tests continuously, not sporadically.
- Sharing results across the organisation to build data-driven decision making.
- Celebrating "losing" tests as valuable learning opportunities.
- Investing in analytics and tracking infrastructure that supports accurate measurement.
- Setting aside development resources for implementing test results.
Start Your First Test
A/B testing is one of the highest-return-on-investment activities in digital marketing. Every test you run either improves your conversion rate or prevents you from making a harmful change. Either outcome is valuable.
Pick a high-traffic page, form a hypothesis about one element, calculate your sample size, and launch your first experiment. The data will tell you what your users actually want — which is almost certainly different from what you think they want.
If you want help building a conversion rate optimisation programme backed by rigorous A/B testing, talk to our CRO team. At Dynamically, we help businesses across the UK turn more of their existing traffic into customers — through systematic testing, analytics, and evidence-based optimisation.
