A/B Testing: How to Run Tests That Actually Win

A/B testing means showing two versions of something (a headline, a button, a page) to similar visitors and measuring which one gets more of the action you care about. Done well, it replaces opinions with evidence. Done badly, it produces confident wrong answers. Here is how to run tests that hold up.

What A/B testing is (and what it is not)

A/B testing (also called split testing) shows two versions of the same thing to comparable visitors at the same time and measures which version produces more of a chosen outcome: purchases, quote requests, sign-ups, clicks. Version A is usually your current page (the “control”); version B contains one deliberate change (the “variation”).

It is not the same as comparing this month with last month. Seasons, ad budgets, news and a dozen other factors change from one period to the next, so a before-and-after comparison can make a change look like a winner or a loser for reasons that have nothing to do with it. A proper test splits traffic at the same time, so everything except the change is shared.

A related technique, multivariate testing, changes several elements at once and measures the effect of each combination. It needs far more traffic, so for most small and mid-sized sites, simple A/B tests are the right tool.

Step 1: Get your measurement right before you test

A test is only as good as the number it optimizes. Before changing anything, confirm that you can count the outcome reliably:

• Define one primary metric. For a service business this is usually a completed form, a click on a WhatsApp button or a booked call. Pick one. Secondary metrics (scroll depth, time on page) are for context, not for declaring winners.
• Check that the event fires correctly. Submit the form yourself on desktop and mobile and confirm it appears in your analytics. Broken tracking is the most common reason a test “shows no difference”.
• Know your baseline. If the page converts 2% today, you need to know that so you can judge whether 2.4% is meaningful.
• Exclude noise. Filter out your own team’s visits and obvious bot traffic where your tool allows it.

Step 2: Write a hypothesis, not a hunch

A hypothesis has three parts: the change, the expected effect and the reason. A reliable template is: “Because we observed [evidence], we believe that changing [element] to [variation] will improve [metric] for [audience].”

Swipe the table sideways to see all columns.

The evidence column matters. It comes from analytics, session recordings, customer calls, support messages or heatmaps. Tests built on evidence win more often than tests built on taste.

Step 3: Decide what to test first

You will always have more ideas than traffic. Rank them with a simple score from 1 to 5 on three questions: Impact (how big could the lift be?), Confidence (how strong is the evidence?) and Ease (how little work is it?). Multiply or add the scores and start from the top.

• Offer and value proposition: “free audit” vs. “free template”, price framing, guarantees. These often move results most.
• Headline and first screen: what the visitor reads in the first three seconds.
• Call to action: wording, position, and number of competing actions on the page.
• Forms: number of fields, order, and the promise next to the button.
• Trust elements: testimonials, logos, certifications, real photos.
• Layout and length: short vs. long pages, order of sections.

Skip tests whose best case would be a tiny change on a low-traffic page; they take months and rarely change decisions.

Step 4: Run it for the right amount of time

Two things decide when a test can end: time and data.

• Time: run at least one full week, and preferably two, so each weekday appears equally. Behavior on Monday morning differs from Saturday night.
• Data: you need enough conversions in each version. There is no magic number: the smaller the difference you want to detect, the more data you need. Use your testing tool’s calculator or a sample-size calculator, and decide the stopping point before launch.

The most damaging habit is “peeking”: checking daily and stopping as soon as one version looks ahead. Early gaps are often noise and frequently shrink or reverse. If a test ends without a clear winner, that is a result too: it means the change did not matter much, and you can keep the simpler version.

If your page gets only a few hundred visits a month, classic testing is slow. Test bolder changes (a different offer, a restructured page), combine evidence from several sources, and treat results as directional rather than proven.

A worked example (illustrative numbers)

Imagine a local services company whose landing page gets 3,000 visits a month from ads and converts 2.0% into quote requests. The team observes that visitors rarely reach the pricing explanation. Hypothesis: adding a three-step “how it works” section above the form will raise requests.

They split traffic 50/50 for three weeks. Suppose version B converts at 2.6%. The team checks that the result holds across mobile and desktop, that lead quality (share of leads that became proposals) did not drop, and that the gap is larger than normal week-to-week variation. Only then do they roll it out. They log the result and move to the next hypothesis: the form length.

These numbers are only an example to show the process. Your own baseline and uplift will differ.

Protect your search rankings while testing

Google’s documentation on website testing gives clear rules. In summary:

• Do not cloak. Showing Googlebot different content or URLs than users see violates spam policies, whether or not you are running a test.
• Use rel="canonical". If your variations live on separate URLs, point them to the original with a canonical link.
• Use 302 redirects (temporary) when you send users to a variation, not 301.
• End the test. After you have enough data, update the site with the winner and remove the test scripts and alternate URLs. Tests that run unnecessarily long can be treated as deceptive.

For small content variations, Google notes that it may not matter much if variations get crawled or indexed. The guidance mainly protects you from big mistakes.

Seven mistakes that ruin tests

1. Changing several things at once and not knowing which one worked.
2. Stopping early because a version “looks ahead”.
3. Testing without a hypothesis or a clear primary metric.
4. Running overlapping tests on the same page, so results contaminate each other.
5. Ignoring lead quality: more leads that never buy are not a win.
6. Not segmenting by device when the audience splits heavily between mobile and desktop.
7. Never recording what was learned, so the same test is repeated a year later.

Build a simple testing routine

Keep a shared log with one row per test: date range, page, hypothesis, variations, primary metric, sample size, result, decision and next step. Aim for a steady rhythm, such as one meaningful test at a time per key page, rather than many rushed ones. After ten tests, the log becomes a map of what your audience responds to, which is more valuable than any single win.

Testing works best when it sits inside a broader plan to attract and convert people; see our guide on how to get more customers, and if you want help setting up tracking and tests for your site, talk to the MetaEX team.