Monthly monitoring with a fixed question set: how to quantify AI citation rate

Bottom line first: The easiest thing to fake in GEO is performance you can’t disprove. To make it falsifiable, you have to lock down the method first: fix a set of questions whose members rarely change, fix the models and the phrasing, sample in the same week every month, then track five metrics. About 40 questions, 1–2 hours per sampling run, one-page report.

1. The easiest thing to fake in GEO is “performance you can’t disprove”

The most common claim in GEO services is “AI mentions you now.” It’s almost impossible to argue with, because ask a different question, use a different model, or try another day and the conclusion can flip. It proves neither that the work succeeded nor that it failed.

The problem is the sampling method: think up a question on the spot, ask once, glance at the answer. That gives you a sample size of 1, where randomness far outweighs signal. Drawing a conclusion from a single result is like flipping a coin once.

To make it falsifiable, you need to fix four things:

Element	If you don’t fix it	If you do fix it
Question set	Different questions every time, so nothing is comparable	The same questions month over month, so trends are readable
Models and entry points	Switching models means switching environments, so results aren’t comparable	Same model, same entry point, so any change comes from your content
Sampling window	Sampling in different weeks leaves you exposed to industry cycles	Same week every month, which filters out seasonal noise
Account and state	Login status and region change the answers	Same account state, which cuts down on environmental variables

The goal of this method isn’t to produce a precise number like “citation rate is 23%” — it’s to read the direction. When the month-over-month difference is within 5 percentage points, you can’t treat it as a conclusion; only three consecutive months moving the same direction count as a signal. This baseline has to be stated up front, or the report itself becomes just another way to fake it. It can only observe whether you appear in the answers, not how many people saw you — impressions and clicks belong to a different funnel and have to be measured separately. We developed this method across multiple client sites, and Crater‘s documentation includes a related index.

2. How to build the question set: three tiers, about 40 questions

The question set is the core asset of this method. It has to meet three conditions: the phrasing is what real buyers would use, it covers the full path from awareness to decision, and its members stay relatively stable.

We recommend building it in three tiers, with about 40 questions in total:

Tier	Question type	Videos	Example phrasing	What it mainly shows
Tier 1	Brand-term questions	10–12 questions	What does a company do, how good is a brand’s product	Whether the model knows the brand, whether the entity is clear
Tier 2	Category-level questions	12–15 questions	How to choose a type of equipment, what tolerance a process can hold	Whether you enter the category candidate pool
Tier 3	Purchasing decision questions	12–15 questions	How to screen Chinese suppliers, typical lead times for a product type	Whether you appear in answers at the purchasing stage

The three tiers do completely different jobs. Tier 1 tests awareness, Tier 2 tests category coverage, Tier 3 tests the purchasing stage.Testing only brand terms gives you inflated results—the model may know your name but never mention you in the questions buyers actually ask when deciding.

Where do the questions come from? Don’t make them up in a vacuum—pull from these four sources:

Source	How to pull them	Suggested share	Value
Questions sales reps actually get asked	Pull the exact wording from inquiries and chat logs	40%	The most realistic phrasing
On-site search terms	Deduplicate the search log and take the top 30	20%	Reveals intent gaps among existing visitors
AI autocomplete and related questions	Enter a category term and see what the model suggests	25%	Shows how people currently ask
Where competitors appear in answers	Note the phrasings and scenarios where competitors get mentioned	15%	Find the competitive gaps

Once the question set is locked, don’t change it casually. When you need to add questions, replace rather than append: add 2, drop 2, and note the replacement date. Otherwise, three months later you’ll be looking at a list you can’t compare over time.

3. Sampling discipline: five fixed items

The method holds up on discipline. Fix these five items for every sampling round; if any one changes, flag that period’s data separately.

Fixed item	Specific requirements	What to do if violated
Fixed models and entry points	Pick 2–3 models, record the exact version and entry type	List data from a changed model in its own column; don’t compare it directly with the previous period
Fixed phrasing	Use the question set verbatim—no rewording, no qualifiers	Reworded results don’t count for that period
Fixed time window	Monday morning of the same week each month	Flag data that deviates by more than 3 days as anomalous
Fixed account state	Same account, same network environment, same web-access setting	Rebuild the baseline when the state changes
Fixed retention	Save the original questions and full answer text, with screenshots	Data without original records is considered invalid

On phrasing, you have two options. Raw phrasing means entering the question as written—closer to real usage, but the answers are more random. Qualified phrasing adds “please list suppliers” after the question—more likely to trigger a brand mention, but the conclusions skew optimistic.We recommend raw phrasing as the main approach, with qualified phrasing only as a supplement when judging the category candidate pool. Record the two separately; don’t blend them into one metric. Taking the flattering numbers from qualified phrasing to a report is building yourself a fake baseline.

One more rule: ask each question only once. Don’t keep rephrasing until you get an answer you like. If you want to check whether a result is stable, retest it in the next cycle.

4. How the five metrics are defined

You have to lock down the definitions first, or every report will use a different calculation. These are the five we use.

Metrics	Definition	How to calculate	Observation period
Mention rate	Share of answers that include your brand name	Questions mentioning the brand ÷ total questions	Monthly
First-mention rate	Share of answers where your brand is the first entity named	First mentions ÷ total questions	Monthly
Source citation rate	Share of answers whose cited sources include your website	Questions citing your website ÷ questions mentioning the brand	Monthly
Answer accuracy rate	Share of answers whose description of you matches the facts	Questions with an accurate description ÷ questions mentioning the brand	Monthly
Head-to-head win rate	Share of answers where you’re described more favorably when a given competitor appears alongside you	Favorable answers ÷ questions where both appear	Monthly

Three definition details need to be called out separately.The denominator for mention rate is all questions, so it stays comparable month to month;the denominator for source citation rate is questions that mention the brand, otherwise it gets diluted by the many questions that don’t mention you;answer accuracy rate needs a human standard, so write a fact sheet up front listing hard facts like capacity, certifications, MOQ and lead times, and check each one against it.

Reference ranges for the five metrics (industrial categories as an example):

Metrics	First-quarter range	Range after six months	Description
Mention rate	5%–15%	15%–35%	Varies a lot with how competitive the category is
First-mention rate	0%–5%	5%–15%	Typically low in high-ticket categories
Source citation rate	10%–30%	30%–60%	Depends on your website content structure
Answer accuracy rate	40%–70%	70%–90%	Description errors are common early on
Head-to-head win rate	20%–40%	40%–60%	Small sample, so watch the trend, not the absolute number

5. Decision rules and thresholds, including “not enough data to decide”

Data alone isn’t enough; you also need decision rules. Without them, everyone reading the report draws their own conclusions. We use four tiers, and one of them is explicitly called “not enough data to decide.”

Per-question scoring:

Status	Condition	How to record it
Mentioned and cites your website	The answer mentions the brand and cites your website	3 points
Mentioned, no citation	The answer mentions the brand but doesn’t cite your website	2 points
Not mentioned	The brand name doesn’t appear in the answer	0 points
Incorrect answer	The brand is mentioned, but a hard fact is described incorrectly	Flag separately, exclude from scoring
Can’t be judged	The answer is generic or unrelated to the question	Mark as invalid, drop from this cycle

Overall conclusion:

Conclusion	Condition	Can it be reported externally?
Clear improvement	Mention rate up at least 5 percentage points from the previous cycle, and moving the same direction two cycles in a row	Yes
Possible improvement	Up by less than 5 percentage points, or up in only one cycle	No, write “to be watched”
No clear change	Change within plus or minus 3 percentage points	No, write “no change”
Not enough data to decide	Fewer than 30 valid samples, or the model or question wording changed this cycle	No, state the reason

“Not enough data to decide” is a required item in an honest report. There are usually three triggers: fewer than 30 valid samples in the current period, a model version update that shifts answer style across the board, or a breakdown in sampling discipline (asking reworded questions, switching accounts). In all three cases, forcing a conclusion does more harm than giving none — it pushes people to adjust content in the wrong direction.

There is one more decision principle that is easy to overlook:A drop in answer accuracy takes priority over mention rate. If the model mentions you a lot but describes you as a different type of supplier, or gets your capacity and certifications wrong, that mention is a liability. In that case the first priority is correcting the source of truth, not pushing mention rate higher.

6. Mapping from monitoring to action

The value of monitoring is that it points to specific actions. Below are the mapping rules we use, going straight from symptom to action.

Symptom	Verdict	First-priority action	Observation window
Low mention rate, normal accuracy	Not enough corpus density, the model has no material	Increase the density of factual content on your site; add specs, certifications, process details	8–12 weeks
Mentioned, but no cited source	Content can be read, but its structure cannot be extracted	Add conclusion paragraphs, tables, and FAQs so facts come in blocks	4-8 weeks
Low answer accuracy	The source of truth is wrong or outdated	Fix incorrect descriptions on your official site and public materials	4-8 weeks
Brand terms return answers, category terms do not	The entity is clear but category coverage is insufficient	Add category-level selection and comparison content	8–12 weeks
Category terms are mentioned, procurement terms are not	Missing content for the decision stage	Add procurement concerns: lead times, MOQ, certifications, after-sales	8–12 weeks
Competitors get mentioned, you do not	They have clear content for certain question phrasings	Build pages targeting those specific question phrasings	8–12 weeks
No change in any metric	Content updates have stalled, or there is a problem with the channel	First confirm whether content has been updated and whether crawlers are reading it	4 weeks

How to use this table: after each monthly report, pick only the top symptom and act on it. Do not change three things at once.Change one variable at a time, so next month you know which change made the difference. If you change content structure, update frequency, and factual descriptions all at once, you cannot attribute the results even if the data improves.

There are five main supporting content actions: write key facts as standalone paragraphs, add 5–8 FAQs to every page, add certification and compliance pages, unify brand wording and entity information, and add multilingual versions. Most take effect within 4–24 weeks, and the later ones are slower.

7. Report template, retention period, and when this does not apply

Keep the report short. A one-page table plus an archive is enough to support decisions.

Report section	Content	Length
Conclusion for this period	One-line conclusion, taken from the four-tier assessment	1 line
Five-metric table	Current vs. previous period values, with direction of change marked	1 table
List of anomalies	Questions with wrong or undecidable answers, listed one by one	3–8 items
This cycle’s action	Write just one, taken from the action map	1 line
Next cycle’s validation point	The one metric that confirms whether this cycle’s action worked	1 line

Retention requirements for source material:

What to retain	How long to keep it	Purpose
Original questions and full answer text	12 months	Review and trace changes in methodology
Answer screenshots	12 months	Respond to outside challenges
Monthly metrics table	Long term	See annual trends
Question set version log and fact-check table	Long term	Ensure comparability over time and accuracy of judgments

There’s a reason to keep data for 12 months: the impact of model version updates often shows up late, and it’s only six months later that you look back and notice a step change in the data. Without the original answers, you can’t tell whether that shift came from your content or from the model.

This method doesn’t apply in three situations. First, when the language of the question set doesn’t match the target market — testing a German market with English questions is meaningless. Second, when the citable content on your site is still very early — testing will just give you a row of zeros, so build content first. Third, when you treat monitoring as an acceptance tool — it can’t be used to prove how much a given deliverable “improved things,” and once it’s used that way, sampling discipline slips and the data stops being trustworthy.

FAQ

How many questions are enough?

Around 40 is the balance point — one sampling round takes 1–2 hours and can be read at different levels. Under 15, fluctuations can’t be explained and you can only see whether you’re mentioned at all; 20–30 suits a single category in a single market; above 60, precision gains are limited unless you’re covering multiple markets and languages. One rule to keep: add new questions by replacing old ones rather than appending, keep the total stable, and log when each replacement happened — otherwise you can’t compare across months.

How often should you test?

Once a month, on the Monday of the same week each time. Testing more often isn’t necessary, because model answers are inherently random and week-to-week differences are noise. Testing less often doesn’t work either — once a quarter, and you can’t attribute what happened in between. You can add extra tests at key moments: 4 weeks after a major content overhaul, 2 weeks after a model version update. Keep extra test data separate; don’t merge it into the regular series.

Which model should you test with?

Pick 2–3 models your target market actually uses, and stick with them — don’t switch every time. Base the choice on usage habits in the target market, not on which model gives you flattering answers. You must record the specific version and entry type, including whether web search is on and whether you’re logged in. After a model update, the answer style may shift across the board — in that case, rebuild the baseline and look at the periods before and after separately rather than subtracting one from the other.

Large model answers are different every time — how do you handle that?

Answers vary randomly, so never draw conclusions from a single run. There are three ways to handle this: make the question set large enough that 40 questions smooth out the noise from any one; only treat the same directional change across two or three consecutive periods as a signal, and ignore gaps under 5 percentage points; and keep the raw answer text so you can trace back when something looks off. If one question swings wildly three periods in a row, flag it as unstable and handle it separately.

Should we also track AI search entry points in client apps?

Yes, but as a separate supplementary series, not mixed in with the main one. Answers from client app entry points vary a lot depending on region, account status, and whether the device is online, so they’re less stable than web. We suggest keeping the main series in one fixed environment and testing client app entry points once a quarter on their own, to see whether you get mentioned in real user scenarios, and labeling those findings as reference only. Always keep the two datasets in separate tables.

Should we run this monitoring in-house or outsource it?

Either works, depending on whether you have consistent people and consistent time. In-house means you understand the business and can judge whether answers are accurate; the hard part is setting aside 1–2 hours every month for 12 straight months, and this is the first thing to get cut when people are busy. Outsourcing gives you steady discipline, but the risk is that the vendor doesn’t understand your business and treats wrong descriptions as correct. A middle path: outsource sampling and archiving, and have your team produce the fact-check table that determines accuracy.

Final thoughts

GEO easily turns into a vague line item because it has no real-time data like ads and no clear position like rankings. A fixed question set is essentially trading discipline for credibility —same group of people, same phrasing, same time window, turning something random into something comparable.

It can’t promise results down to the decimal point, and it can’t tell you “how many times better” you got. It can only tell you three things: whether this month is better, worse, or unchanged from last month; which questions mention you and which don’t; and whether the next step is to fix content, fix structure, or fix facts.

Only when it can honestly say “there isn’t enough data to decide” has this method actually proven itself. If you want to start by building the question set, you can pick real questions people ask from the topics in the Insights section and fill in the rest using the three-tier structure above. If you need to define the question set and metric definitions together, you can Contact Us.