GEO Experiment Design: Hypotheses, Variables, and Attribution

The Four Stages of a GEO Experiment: Hypothesis, Variable Control, Run and Sampling, and Attribution

GEO Experiment Design: Hypotheses, Variables, and Attribution

Bottom line first: GEO (generative engine optimization) can’t be “publish the article and wait for results” — it has to be a test you can attribute results to. A valid GEO experiment has to meet three conditions: a fixed set of 30 to 50 questions as your yardstick, only one variable changed at a time, and a control period of at least 2 weeks as baseline. To call something effective, you need two hard thresholds: a net lift of ≥10 percentage points and a full 6-to-8-week run. Otherwise every “citation rate went up” could just be noise from a model version update or broader industry swings.

1. Why GEO Needs Experiment Design, Not “Launch It and See”

Traditional SEO has search engine results pages — ranking shifts can be tracked day by day and compared against competitors’ changes over the same period. GEO faces natural-language answers generated by large models: the same question asked today and tomorrow may get different answers; a different IP, a different account, a different time zone may also produce different answers. Without a fixed yardstick and a control design, teams fall into the most common loop: this month citations are up, the team celebrates; next month they’re down, and everyone starts wondering which article broke it. In between, there’s no conclusion at all.

This shares the same difficulty as Lead attribution: many touchpoints, long paths — without attribution design, you’re just going on feel. The difference is that GEO’s “user” is the model, whose behavior is more stable but less visible, which makes it even more necessary to design controlled experiments by hand and crack the black box open a little.

Common mistakes	Consequence	The right way
Check citation rate a week after changing content	Sample too small, results are random	Run the control baseline for a full 6 to 8 weeks
Change title, structure, and backlinks all at once	Can’t attribute to a specific action	Change only one variable at a time
Swap in a new set of questions each time you test	Sample drift, trends aren’t comparable	Fix the question set, reuse it across quarters
Only look at whether your own brand got mentioned	Miss overall shifts in the industry	Add a neutral control query
Ask once and call it done	Model sampling is inherently random	Run each query 3 times and take the mode

2. Step One: Lock Down the Hypothesis Before the Experiment Starts

A GEO hypothesis must be falsifiable. A valid hypothesis looks like this: “After adding three comparison data points with numbers to the FAQ module, the brand citation rate in the target question set rises from baseline X% to above Y%.” An invalid hypothesis looks like this: “Optimize the content a bit and hope AI cites us more.” The latter can’t be falsified, so it can’t be tested.

A hypothesis has to land on a specific variable. There are three types: page-level (title, H2 structure, opening answer, table density, Schema markup), site-level (internal link structure, crawl frequency, page authority flow), and external-level (fact sources cited by third-party sites, wiki/encyclopedia entries, press releases). You can’t mix all three in one experiment. For how to build the question set, see Monthly monitoring of a fixed question set and mining search queries on your own site for the method.

Hypothesis type	Example	Metric to measure	Test window
Page-level hypothesis	Lead with a numeric answer in the first paragraph: citation rate +10pp	Citation rate, first-citation position	4-6 weeks
Structure-level hypothesis	Add a comparison table: factual answers get cited more	Structured answer hit rate	6-8 weeks
External-level hypothesis	Add third-party fact sources: brand name mention rate rises	Brand mention rate, misstatement rate	8-12 weeks
Entity-level hypothesis	Complete the organization entity markup: AI correctly identifies the company’s full name	Entity recognition accuracy	6-8 weeks

Step 2: Variable control and comparison design

The core of experiment design is holding every other variable constant. A B2B standalone site doesn’t get enough traffic to run strict A/B bucketing (sending half your users to the old version and half to the new one), so GEO experiments rely more on time-series comparison: measure a 2-week baseline, ship the change, measure for 6 weeks, then look at the difference before and after.

Three confounders you must hold fixed. First, the question set stays the same, down to the wording. Second, the sampling environment stays the same: same region, same account state, same time of day. Third, keep a control set of queries: run a few neutral queries unrelated to your change alongside the rest. If those move too, it’s a model version update or industry noise, not your doing. For how to layer your content, see Fact, opinion, and claim: three layers of content.

Control	Baseline requirement	Experiment requirement	Common failure point
Question set	30-50 queries, covering all categories	Same set, no additions or removals	Adding new queries mid-test to pad the results
Sampling environment	Fixed IP region/language/time slot	Exactly the same	Running a mix from home and office networks
Control group	5-8 neutral queries	Monitor continuously	Forgetting to set up a control group
Model version	Record the current version number	Re-measure the baseline across versions	Comparing directly after a major model version upgrade
Page scope	Change only the target page	Freeze changes to all other pages	Casually editing the nav and site-wide styles along the way

Step 3: Sampling method and four core metrics

You can’t ask each query just once. Large models have a temperature parameter, so asking the same question three times in a row can surface different sources. The standard approach is to ask each query 3 times, count how often your domain appears in the answers, and take the mode as that query’s result; appearing in 2 of 3 runs counts as cited. Run one round at a fixed time each week so weekday-versus-weekend traffic differences don’t skew your read.

Rank the four core metrics by priority: citation rate (the share of target queries where the answer mentions your domain), first-citation position (where you rank among the sources cited in the answer — first carries the most weight), brand-name mention rate (whether the company or product name appears correctly in the answer), and negative misstatement rate (whether the answer contains factual errors about you). For how to build the monitoring dashboard, see AI Engine Citation Monitoring Dashboard.

Metrics	Baseline reference range	Meaningful improvement threshold	Tool / data collection method
Citation rate	5% to 15% for new sites	Net gain ≥10 percentage points	Manual or API sampling of each question in the set
First-citation position	3rd to 8th	Share in top 3 ≥30%	Order in which domains appear in the answer text
Brand-name mention rate	30% to 60%	Gain ≥15 percentage points	Regex match on full or abbreviated company name
Negative misstatement rate	Under 5% is healthy	Any rise triggers the correction process	Manual review + keyword alerts
Answer adoption rate	Depends on the industry	Rises for 3 consecutive weeks week-over-week	Compare fixed weekly rounds

5. Step four: attribution — rule out market-wide noise

After the experiment ends, look at the control queries first. If the control group also rose by 5 percentage points overall, that points to market-wide drift from a model version update or a shift in industry interest, and you need to subtract it from your true net gain. Net gain = change in the test group minus change in the control group. Only when net gain reaches the threshold do you count the action as effective.

Another common trap is that correlation is not causation. Say you changed a page, and at the same time a news story about something similar in your industry came out. AI answers cite you more often, but the news event may be driving that, not your page change. In that case, mark the experiment as “questionable” and repeat the same change next round to see whether it reproduces. One reproduction counts as verification; two count as a conclusion. For how to choose fact sources, see Off-site Fact Sources and AI Answers.

Attribution scenario	Signal	What to do
Test group rises, control group flat	Net gain hits the threshold and reproduces	Action works — lock it in and scale it
Test group rises, control group rises too	Market-wide drift in sync	Subtract, then judge again — may be ineffective
Test group rises, control group falls	Possible share capture	Effective — watch competitor positions closely
Test group falls, control group flat	The change has side effects	Roll back and pinpoint the damage
Everything swings wildly	Model version upgrade period	Pause judgment until the version stabilizes

6. Experiment cadence: how a one-person team schedules the work

Small teams should not run three experiments at once. A quarterly cadence works well: one main hypothesis per quarter, run over 6 to 8 weeks; while it runs, monitor the question set every week and leave the target pages alone. Reuse the same set of queries across quarters so you can spot year-over-year trends. For how to draw the line between content production and human AI work when several people collaborate, see AI Content and the Human Boundary.

Stage	Length	Action	Output
Baseline period	2 weeks	Run the question set and record the four metrics	Baseline Data Card
Change Window	1 week	Ship a single variable change	Change List
Observation Window	4-6 weeks	Run weekly, with a control group	Weekly Monitoring Sheet
Attribution Window	1 week	Subtract the market baseline, judge net lift	Experiment Conclusion Card
Review Window	1 week	If it works, lock it in; if not, archive it	Next Quarter Hypothesis List

FAQ

How long does a GEO experiment need to run before you can draw a conclusion?

6 to 8 weeks is the minimum bar. The first 2 weeks are your baseline, the middle 4 to 6 weeks are observation, and the last week is attribution. Anything under 4 weeks is only a reference, not grounds for deciding “this action worked,” because large models have week-level sampling variance and version updates. If your site has low traffic and a small citation base, stretch it to 10 weeks rather than drawing conclusions from 2 weeks of data.

Can I skip the control group and just make the change, then see if my numbers go up?

No. Without a control group, you can’t tell “my change worked” apart from “the whole industry or model version is rising.” The cheapest approach is to pull 5 to 8 neutral queries from your question set that have nothing to do with your change and run them as a control group every week. How much they move in sync is your noise, and you subtract it from your test group’s results. Those 5 to 8 queries need no extra content investment, just a few more runs.

Can one experiment test two changes at once?

Not recommended, unless the two changes are strongly related and you accept fuzzy attribution. GEO has a small sample size and high noise to begin with; if you change the title and add backlinks at the same time, even if results go up you won’t know which one did it, and you can’t repeat it next time. The right approach is serial: test page-level changes first, lock them in, then test external-level ones. If resources are really tight, at least keep the two changes on different pages and use page-to-page comparison to roughly separate them.

How do I choose the target question set, and how many questions is right?

30 to 50 is the sweet spot for a one-person team. Sources, in priority order: on-site search terms, high-frequency customer service questions, topics you already have content for and want cited, and questions from competitors’ FAQs. Split into four categories of roughly a quarter each: brand terms, product terms, general industry terms, and comparison/decision terms. Keep it fixed each quarter, only add up to 10 new ones when a new business line appears, and tag new versus old samples separately.

Should I sample automatically via API or ask questions manually?

Run 5 to 10 manually first to learn the format, then batch-run via API. Three things to watch: fix the region parameter and system prompt so you aren’t flagged as a bot; add a random delay to each query; run each one 3 times and take the mode. Running 50 queries a week purely by hand takes 2 to 3 hours; automation cuts it to 20 minutes. For log analysis methods, see AI Crawler Log Analysis.

What do I do if the experiment fails, should I roll back immediately?

First, decide whether it really failed. If the control group dropped at the same time, or the model just had a major version update, it doesn’t count as a failure. Only roll back once you’ve confirmed a net decline of more than 5 percentage points that hasn’t recovered for 2 straight weeks, and note which change direction caused the harm. Failed experiments are valuable too — they rule out a hypothesis. Writing the failure into your quarterly review library matters far more than forcing a change live that does nothing.

Final thoughts

The biggest difference between GEO and traditional SEO is that its “results” hide inside the model’s natural-language output, not a row of numbers sitting in front of you like a ranking. Precisely because you can’t see it, you need to turn experimentation into a process: fix your question set, control the variables, keep a baseline, run the full cycle, and subtract the overall trend. Without these, your team swings between “feels better” and “feels worse” every month, and a year later you have no conclusions you can build on.

Start with the smallest possible loop: this quarter, define just one hypothesis, build a 30-question set, run a 2-week baseline, change one variable, and observe for 6 weeks. Completing one full loop in a quarter is far more useful than launching ten “GEO optimization actions” at once. GEO’s compounding returns come from a repeatable experimental method, not from scattered attempts.

One line worth remembering: GEO optimization without a control group is like shooting hoops blindfolded — whether it goes in is pure luck, and the next game starts from zero again.