A marketing team runs its first website test. They change the "Buy now" button from grey to orange, split traffic between the two versions, and check the results every morning. By day three the orange button is winning by 40%. They declare victory, switch every visitor to the new version, and announce the result in the team meeting. A month later, conversions look exactly the same as they did before anyone touched the button.
Nothing was wrong with the tool, and orange isn't a bad colour. The test simply failed before it started: it had no real problem behind it, no clear hypothesis, far too few visitors, and it stopped the moment the numbers looked exciting. Most first tests fail in some version of this way. The good news is that CRO for beginners isn't about advanced statistics. It's about doing the steps in the right order, and accepting one uncomfortable truth early: your traffic decides what kind of test you can run, not your list of ideas.
This guide covers website A/B testing and the research that should come before it. Related guides cover ad creative testing, email A/B testing and testing landing pages for paid traffic.
What CRO actually is
Conversion rate optimization, or CRO, is the practice of increasing the share of visitors who take a valuable action: buying, signing up, booking a demo, submitting a form. It is often treated as another name for A/B testing, but testing is only the last step. The real work is finding out where and why people who already want what you offer give up, and removing the confusion, doubt and friction that stop them.
That framing matters for beginners, because it changes where you start. Not with a list of things to try, but with evidence about what is going wrong. Before any of it, make sure your conversions are tracked accurately, since a test measured with broken tracking is worse than no test; the setup is covered in setting up conversion tracking you can trust. Consent banners can also thin out your data, a problem worth understanding through cookie consent without killing your analytics.
Step 1: find the real problem
Start with your analytics. Map the steps between arriving and converting, and find the biggest drop: a product page most people leave, a form most people abandon, a checkout step where many stop. The largest leak on a high-traffic page is usually where your first test belongs, and the approach in auditing your funnel end to end gives you a structured way to find it.
Analytics tells you where people drop out, but not why. For that, watch session recordings and heatmaps of that page, which many tools now provide free; run a one-question survey asking visitors what stopped them; and read support tickets, live chat logs and sales call notes for the questions people keep asking. Patterns appear quickly. Visitors scroll past the price looking for delivery costs. They click an image expecting it to enlarge. They hesitate on a form field that asks for a phone number. Each of those is a real problem you can fix.
TrueReview
Automated review-request and reputation platform to collect, monitor and respond to customer reviews.
Best for: Automated Google review generation & reputation management
Step 2: write a proper hypothesis
A hypothesis turns an observation into a testable idea, and it stops you from testing random changes. A simple template works well: because we saw [evidence], we believe that [change] for [audience] will increase [metric], because [reason].
For example: "Because recordings show visitors scrolling up and down the product page looking for delivery information, and support gets frequent questions about delivery times, we believe that showing delivery cost and dates next to the price for all visitors will increase add-to-basket rate, because it removes a major source of doubt." Compare that with "let's try an orange button." The first tells you exactly what to build, what to measure and what you'll have learned whichever way the result goes. For product pages specifically, the principles in product page optimisation are a rich source of well-evidenced hypotheses.
Step 3: check whether you can test at all
This is the step beginners skip, and it's the one that decides everything else. An A/B test compares two versions by splitting visitors between them, and it needs enough visitors and conversions in each version to tell a real improvement from random noise. Many small sites simply don't have enough.
A page converts at 3%, and you want to detect a 20% relative improvement, from 3% to 3.6%:
→ About 13,900 visitors per version, roughly 27,800 in total
→ That's around 417 conversions in the original version, more in the variant if the improvement is real
How long that takes (daily figures are total visitors to the page, before the 50/50 split):
→ At 3,000 visitors a day: about 9 days, so run 2 full weeks
→ At 1,000 visitors a day: about 28 days, so 4 weeks
→ At 500 visitors a day: about 56 days, so 8 weeks
Want to detect a smaller 10% improvement? You need roughly four times as many visitors: about 53,200 per version.
The rule of thumb: you need a few hundred conversions in each version before a test can detect a realistic improvement. Use a free sample-size calculator with your own numbers before you build anything.
If your page can't reach that sample in a reasonable time, a classic A/B test will struggle to give you a trustworthy answer. What counts as reasonable depends on the sample you need, your traffic and how stable conditions are: the longer a test runs, the more cookie deletion, returning visitors and seasonal swings can blur the result. That's not a failure; it just means a different approach suits your stage, as the table below shows.
| Your situation | What works best | What to avoid |
|---|---|---|
| Low traffic A few conversions a week |
Research-led fixes, measured before and after with care | A/B tests that can never finish |
| Moderate traffic A few hundred conversions a month |
A/B tests of bold changes on your busiest pages | Testing small tweaks |
| High traffic Thousands of conversions a month |
A regular testing programme, including smaller changes | Running overlapping tests on the same visitors |
The practical lesson for most beginners: test bold changes, not tiny ones. A rewritten headline and value proposition, a redesigned form or a restructured product page can plausibly move conversion by 20% or more, which is detectable with modest traffic. A slightly different shade of button almost never can.
The rule Your traffic decides what kind of test you can run, not your list of ideas. Most first tests fail before they start: no real problem, no hypothesis, too few visitors, and stopped the moment the numbers looked exciting.
Step 4: build it carefully
Choose a testing tool. Google Optimize was retired in 2023, so most teams now use a dedicated platform or the experimentation features in their analytics or website tools, and several offer free tiers suitable for a first test. Build the new version, then check it thoroughly on phones, tablets and desktop, in different browsers, and with a real test conversion. Make sure the original page doesn't flash briefly before the new one loads, which can bias results.
Decide in writing, before launch, three things: your one primary metric, such as completed purchases rather than clicks; the sample size and duration you calculated; and what result would make you adopt the change. Writing these down in advance is the single best protection against fooling yourself later.
Step 5: run it without interfering
Once the test is live, don't make decisions based on interim results. Run it in full weeks, because behaviour differs between weekdays and weekends. Don't stop early because one version looks ahead; early results swing wildly, and checking repeatedly then stopping at the first exciting moment is exactly how the orange-button team fooled itself. Do keep monitoring the test's health, though: broken pages, conversions that stop recording, a traffic split drifting away from 50/50, or errors on particular devices. If you find a serious bug, fix it and restart the test rather than quietly editing a version mid-test, and avoid starting or ending tests across major sales events or holidays unless the test is about that period.
Step 6: read the results honestly
When the planned sample is reached, look at your primary metric. Ask two questions: is the difference statistically reliable, meaning unlikely to be random noise, and is it practically meaningful, meaning big enough to matter for the business? A statistically significant 0.5% lift may not be worth the effort to implement; a promising 15% lift that falls short of significance may deserve a retest with more traffic.
Two further checks catch common problems. First, check that the traffic split is close to the 50/50 allocation you intended. The visitor counts won't always be exactly equal because normal random variation is expected, but a substantial or statistically unusual imbalance should be investigated with a sample-ratio mismatch check. If you find a genuine allocation or tracking problem, don't trust the experiment's results until it has been resolved. Second, be wary of slicing results into many segments after the fact. If you look at enough segments, one will always appear to win by chance.
Step 7: decide, document and move on
Adopt the winner, keep the original, or plan a follow-up, and write down what you tested, why, what happened and what you learned. Losing and inconclusive tests are not wasted: a clear "this didn't matter" saves you from building on a wrong assumption, and a documented history of tests becomes one of the most valuable things a marketing team owns. Many of the strongest ideas for later tests come from what earlier ones revealed.
Your next tests will usually move to the next biggest leak. For many sites that means pages close to purchase, where the principles in landing page design and conversion-focused web design apply, and checkout, covered in reducing cart abandonment. If your site gets visitors from AI assistants, their pages are worth testing separately, as covered in landing pages for visitors sent by AI. Teams that want the research, design and build handled together often bring in a website design and development partner.
The beginner mistakes, in one place
- Testing ideas instead of problems. Start from evidence of where and why visitors drop out.
- Skipping the traffic check. Calculate the sample size before building anything.
- Testing tiny changes. With modest traffic, only bold changes produce detectable differences.
- Stopping early. Decide the duration in advance and run full weeks.
- Measuring the wrong thing. Clicks are not conversions; choose the metric that matters to the business.
- Hunting through segments. Decide in advance which segments you'll examine.
The short version
Testing is the last step of CRO, not the first. Before your next test, work through this checklist:
- Make sure conversions are tracked accurately.
- Find the biggest drop-off on a high-traffic page, then find out why it happens.
- Write a hypothesis that links evidence, change, audience, metric and reason.
- Check your traffic can support a test before you build anything.
- With low traffic, make evidenced fixes and measure carefully; with moderate traffic, test bold changes.
- Decide your metric, sample size and duration in writing, then run full weeks without acting on interim results.
- Read results for reliability and practical value, check the traffic split, and document every test, including the losers.
Want more of your visitors to convert?
We help brands find what's stopping visitors and build pages that convert better.
Explore Website Design →Frequently asked questions
What is conversion rate optimization?
Conversion rate optimization (CRO) is the practice of increasing the share of website visitors who take a valuable action, such as buying, signing up or booking a demo. A/B testing is only one part of it: good CRO starts with research into where and why visitors drop out, then removes the friction and doubt that stop them.
How much traffic do you need to run an A/B test?
Usually more than people expect. It depends on your conversion rate and the size of improvement you want to detect, but a useful rule of thumb is a few hundred conversions in each version. If a page can't get there in a reasonable time, research-led fixes measured before and after are a better use of effort.
How long should an A/B test run?
Until it reaches the sample size you calculated before launch, in whole weeks so both weekday and weekend behaviour are included. There's no universal maximum: the right duration depends on the sample you need, your traffic and how stable conditions are. Don't stop early because one version looks ahead.
What should my first CRO test be?
A bold change aimed at a real, evidenced problem on a busy page close to conversion, such as an unclear headline, a confusing form or missing delivery information, rather than a button colour. If the page doesn't have enough traffic to test, make the fix and measure it carefully instead.
