A/B Testing Inquiries: Choosing Variables, Sizing Samples, and Reading Significance
1. Choosing variables: test the high-impact ones first, not colors
Many teams start A/B testing with “should the button be red or blue?” That’s backwards. Button color usually moves conversion by less than 5%, while rewriting one line of your above-the-fold headline can move it by 30% or more. For a B2B site with limited traffic, you have to spend your visits on the variables that matter most.
Judging variable priority is simple: if this variable changes, will buyers reconsider whether to keep reading? If yes, it’s high priority. If it just looks different but doesn’t affect their decision, it’s low priority. Your above-the-fold headline, main CTA button copy, and form field count directly shape the conversion decision, so they come first.
| Priority | Variable type | Expected impact on conversion | When to test |
|---|---|---|---|
| High | Above-the-fold headline and subheadline | 20%-40% | First priority |
| High | Main CTA button copy and placement | 15%-30% | Test after the headline |
| High | Form field count (5 vs 8) | 15%-25% | Core landing page variable |
| Medium | Placement and number of trust elements | 5%-15% | Test after the big variables |
| Medium | Where the case studies section appears | 5%-10% | Test when you have traffic to spare |
| Low | Button color, font tweaks, spacing | 0%-5% | Only touch with plenty of traffic |
Form-specific testing gets its own article, see Form A/B testing, which focuses on form fields and layout. This article covers landing page test design more broadly. Before you test, use inquiry heatmap analysis to pinpoint where you’re actually losing people, then decide what to test. Don’t start testing off a guess.
2. Calculating sample size: how many visits do you need?
This is the step in A/B testing that gets skipped most often, and it’s the most dangerous one. Plenty of people launch a test today, see group B ahead by a few percentage points three days later, and declare B the winner. That conclusion is wrong about 90% of the time, because the sample is too small and random noise alone can manufacture a fake “significant difference.”
Sample size depends on three numbers: your baseline conversion rate (where you are now), the smallest lift you want to detect (say a 20% relative lift), and your significance level (usually 95%, i.e. p under 0.05). Inquiry conversion rates on B2B standalone sites typically run between 1% and 5%; the table below is a quick reference for that range.
| Baseline conversion rate | Lift you want to detect | Visits needed per group | Roughly how long (10k monthly visits) |
|---|---|---|---|
| 1% | 20% (to 1.2%) | About 25,000 per group | 50+ days |
| 2% | 20% (to 2.4%) | About 12,000 per group | 25+ days |
| 5% | 20% (to 6%) | About 4,500 per group | 9+ days |
| 10% | 20% (to 12%) | About 2,000 per group | 4+ days |
| 2% | 50% (to 3%) | About 3,500 per group | 7+ days |
Note the last row: if the lift you want to detect is large (say 2% to 3%), the sample size drops a lot. Conversely, if you want to detect something like “the difference between a red and a blue button” — under 5% — the sample size becomes unrealistic. That’s why low-traffic sites shouldn’t test small variables: you could run for six months and still never reach a statistically significant conclusion.
Another rule of thumb is at least 100 conversions per group. If group A gets 5,000 visits, at a 2% conversion rate that’s only 100 conversions, and group B also gets 100. At that point, a few conversions of difference between the two groups makes the p-value meaningless. This is also why low-traffic sites are better off with qualitative methods (user interviews, heatmaps) than with strict A/B tests. For the broader methodology of experiment design, see the hypothesis and attribution approach in GEO experiment design.
3. Statistical significance: how to tell if you really won
Once you’ve run the preset sample size, calculate the p-value with a chi-square or Z-test. The standard is p under 0.05, i.e. 95% confidence, before you call the difference significant. If p falls between 0.05 and 0.1, that’s only a “trend” and you can’t draw a conclusion; if p is above 0.1, the difference between the two groups is most likely random.
Another common trap is “peeking bias”: halfway through the test you can’t resist a look, see group B ahead, and stop the test early. Every peek raises the false-positive rate — peek five times in a row and it can climb from 5% to over 20%. The right approach is to set the sample size and test duration before you start, don’t look in between, and read the results once at the end.
| p-value range | Statistical meaning | Decision |
|---|---|---|
| p < 0.01 | Very strong significance | Ship the new version with confidence |
| 0.01 ≤ p < 0.05 | Significant | You can ship, keep monitoring |
| 0.05 ≤ p < 0.1 | Trend, not significant | No conclusion — extend the test or drop it |
| p ≥ 0.1 | No difference | The two groups are basically the same; keep the original |
Example: an industrial site changed its hero headline from “Professional Sensor Manufacturer” to “Pressure Sensors with ±0.1% Accuracy for OEM Equipment Makers,” and calculated that each group would need 12,000 visits before launch. By day 10, the data showed group B’s conversion rate had risen from 2% to 2.8%, with a p-value of 0.03.
Don’t celebrate yet. First confirm: did each group actually reach 12,000 visits? If not, keep running until it does. If it has, and the p-value is still below 0.05, then ship the new version.
4. Test discipline: change only one variable at a time
Here’s the counterintuitive call in this article:testing three variables at once looks three times more efficient, but in reality it tells you nothing. Say you change the headline, the button color, and the form length all at once, and group B’s conversion rate jumps 30%. You have no idea which change did it. Maybe the headline was better. Maybe the shorter form helped. Maybe traffic just happened to fluctuate. Next time you want to optimize something else, you have no reusable conclusion to build on.
The other benefit of testing one variable at a time is that conclusions compound: this month you test the headline and win; next month you test the button and win; after six months you have a proven best version. Test multiple variables at once, and six months later you’re still going in circles.
| Discipline requirements | What not to do | The right way |
|---|---|---|
| One variable | Change headline + button + form at once | Change one thing at a time, leave the rest untouched |
| Set the sample size in advance | Stop when the data looks about right | Calculate before launch, stop only when you hit the number |
| No peeking mid-test | Check the data once a day | Read the results once, after the test period ends |
| Split traffic evenly | Run only half the time on weekdays | Cover at least one full week |
| Filter out abnormal traffic | Count internal test traffic | Filter internal IPs and bot traffic |
The test period should cover at least one full week. B2B buyers behave very differently on weekdays and weekends: midweek it’s procurement managers doing research, while on weekends it may be casual browsers. Run Monday to Friday only, and you miss the weekend traffic pattern; run only a weekend, and the sample doesn’t represent weekdays. The shortest recommended test period is 7 days, ideally 14, to smooth out day-of-week fluctuations.
5. Alternatives for low-traffic sites
Many B2B standalone sites get only 3,000 to 5,000 visits a month. At the sample sizes above, one test would take three months, which simply isn’t realistic. Forcing an A/B test at that point is a waste of time. Switch methods instead.
| Monthly visits | Recommended method | Not recommended |
|---|---|---|
| Under 5,000 | 5-8 user interviews, heatmaps, session recordings | Rigorous A/B testing (won’t reach significance) |
| 5,000-20,000 | Single-variable tests, testing big lifts (30%+) | Testing small differences (within 10%) |
| 20,000-100,000 | Run regular A/B tests, 1-2 per month | Running 3+ tests at once |
| Over 100,000 | Multivariate testing, split-traffic tests | — |
With low traffic, the right move is qualitative research instead of quantitative testing: interview 5 to 8 real buyers for 20 minutes each and ask them “what are you looking for on this page, and why didn’t you fill out the form?”; then use a screen recording tool to watch session replays of a few dozen real visitors. At low traffic, these methods give you more reliable answers than A/B tests.
For performance baselines, see Core Web Vitals. Fix hard problems like load speed first, then worry about fine-grained testing.
FAQ
How many variables should an A/B test cover at once?
How long do I need to run it, and how much traffic is enough?
How do I know an A/B test result is a real win?
Does changing the button color to red really increase conversions?
My site only gets a few thousand visits a month. Should I run A/B tests?
What traffic should you filter out of an A/B test?
Final thoughts
A/B testing sounds like a scientific method, but what it really comes down to is execution discipline. Most teams can’t produce reliable test conclusions not because they don’t understand statistics, but because they’re too impatient — eager to see results, eager to declare a winner, eager to test several variables at once. Keep the discipline: change only one variable per test, run until you hit the required sample size, and read the results only when the test is done. Over six months, you’ll build a reusable set of optimization experience.
Three steps to get started: first, rank variables by impact and start by testing your above-the-fold headline; before launching, use a sample size calculator to determine how many visits each group needs and how many days to run; during the test, don’t peek — calculate the p-value when the test ends, ship the change if it’s significant, and keep the original if it isn’t. If your site gets fewer than 5,000 monthly visits, don’t touch A/B testing yet — use user interviews and heatmaps instead.
One line worth remembering: the most expensive cost of A/B testing isn’t the tool fee — it’s running tests for six months and finding nothing. Test only one variable at a time, and run until you hit the required sample size before drawing conclusions. These two rules matter more than any tool.