- Owner
- Analyst
- Cadence
- Same day each month
- Time box
- Three to five hours
- Output
- Capture sheet and scoring sheet
Why the same test every month
Bay Area AEO's method is a monthly re-test: it builds a list of the real questions a business's customers ask AI and runs them across every major AI platform each month. The client sees how often they are mentioned, how they are described, which competitors appear and which sources the AI cites. Re-testing monthly is how progress becomes visible. A fixed panel run the same way each month turns a stream of individual answers into a trend the owner can read.
Assistants vary. The same question can produce a different list of names an hour later. The answer to that variation is a stable method, applied every month, with enough questions that a single odd answer does not move the result much.
Settings that stay fixed
| Setting | Rule |
|---|---|
| Panel | The frozen panel for this quarter, numbered, unchanged |
| Platforms | The same platforms as the audit: ChatGPT, Gemini, Perplexity, Google AI Overviews and Claude, plus any added at the quarter |
| Session | Fresh chat or private window per question; logged-in state recorded and kept the same |
| Web search | On where the platform offers it, and recorded |
| Location | Set to the question's city where the platform allows it |
| Day | The same day of the month, within a two-day window |
| Capture | Screenshot or shared link for every answer, filed by question number |
The scoring sheet
Every answer is scored on the same four measures. The definitions below are what make one month comparable with the next.
| Measure | How it is counted | This month |
|---|---|---|
| Mention rate | Answers that name the business, divided by all answers in the panel run | |
| Share of answer | Of all business names across all answers, the share that are this business | |
| Description accuracy | Answers naming the business whose description matches the fact sheet, divided by answers naming it | |
| Source citation count | Times the business's own pages or listings are cited across all answers | |
| Top competitors | The three businesses named most often, with counts | |
| Top cited sources | The five sources cited most often, with counts |
Procedure
- Open last month's sheet
Copy its settings line by line into this month's sheet before asking anything.
- Run the panel
Ask every question on every platform under the fixed settings, one platform at a time per question.
- Capture and fill
File each answer and fill its row: named or not, position, description, others named, sources cited.
- Score
Compute the six measures with the definitions above. Do not change a definition mid-quarter.
- Compare
Set this month's scores beside last month's and the baseline audit. Mark every change of two or more answers on any measure.
- Explain the changes
For each marked change, look at the work log for the period and the sources cited, and write one line on what most likely moved it.
Reading the changes
The timeline Bay Area AEO publishes tells the analyst what to expect. Listing and data fixes can start showing up within a few weeks, so description accuracy and mention rate on platforms that read live listings are the first measures to move. Content, reviews and third-party mentions build over two to four months, so the sources cited and the share of answer usually move later. When a measure moves, check which sources the answers cited this month; a new citation of a page or listing the team fixed is the clearest sign the work reached the answer.
Look at answers, not just numbers. The most persuasive line in any month is a quoted answer that now names the business with the right description, set beside the baseline answer for the same question. Choose two or three of those for the report every month.
When a platform changes
Assistants release new models and features often. When a platform changes in a way that affects answers, such as a new search feature, note the date on the scoring sheet. Keep running the panel the same way; the note explains a jump that has nothing to do with the business's own work, and the next quarter's baseline absorbs it.
Done when
Panel size and noise
With twenty questions on five platforms, a month produces a hundred answers, and a move of two or three answers is easy to see against that base. A smaller panel moves more from chance alone, which is why the prompt panel SOP sets a floor of ten questions and why the analyst marks changes of two or more answers, not one. If the owner wants a city or service tracked closely, add questions for it at the next quarterly freeze; more questions on the same topic steady the reading.
Filing the month
Name the folder with the date and file the capture sheet, the screenshots and the scoring sheet inside it. The report links to this folder, and next month's analyst opens it first.