Evaluating an SEO agency’s results means measuring whether the work they have done over a defined period has produced outcomes that justify the spend. The honest answer requires looking past vanity metrics – keyword rankings, traffic alone, the number of links built – and into the leading and lagging indicators that connect SEO activity to revenue or pipeline. It also requires distinguishing what the agency actually caused from what would have happened anyway.
Buyer-side evaluation is its own discipline. The agency’s reporting is, naturally, framed to present their work favourably. The buyer’s job is to read past that framing and ask: did organic visibility on commercially relevant queries improve, did qualified organic traffic grow, did organic-attributed pipeline or revenue grow, and was attribution honest about cohorts and lag? A good evaluation framework holds the agency accountable without expecting magic in unrealistic timeframes.
This article gives a buyer-side measurement framework: the KPIs that matter, the attribution and cohort approach that separates real lift from noise, the leading and lagging indicators that should appear in reports, what to ask for in monthly reviews, and the red flags in agency reporting that signal the work is not delivering.
Key Takeaways
- Agency results should be evaluated on commercial KPIs (qualified organic traffic, organic-attributed pipeline, revenue), not on rankings or traffic alone.
- Honest reporting includes baselines, cohort definitions, time-lag acknowledgments, and a clear narrative of what was done and what it caused.
- Red flags in agency reporting include rank-only dashboards, vanity-metric inflation, missing baselines, and inability to separate organic lift from paid or brand-search noise.
What the agency report should be telling you (and often is not)
A good monthly or quarterly report from an SEO agency should answer four questions: what did we do this period, what changed in the metrics that matter, what is our read on causation, and what is the plan for the next period. Many agency reports answer the first and the second only – a list of activities, a dashboard of metrics that have moved – and skip the causation read and the forward plan. That asymmetry is the first thing to flag.
Activities done. Specific deliverables: pages published, technical fixes shipped, links earned, schema deployed, internal-link work completed. This is the input layer; it is necessary but not sufficient.
Metric movement. Should include rankings on the agreed target-keyword cohort, organic sessions to target pages, conversion rate on those pages, organic-attributed leads or revenue. With baselines so movement is interpretable.
Causation read. The agency’s honest read on what their work caused versus what would have happened anyway. This is the section most reports skip. Good agencies are willing to say ‘this lift was likely seasonal, not our work,’ and ‘this drop was probably algorithm-related, here is our recovery hypothesis.’
Forward plan. The hypothesis-driven next-period plan: what is being tested, what is being scaled, what is being deprecated. A report that does not include a forward plan is closing the loop too late.
The KPIs that actually evaluate SEO results
SEO has many tracking metrics; only a subset are commercial KPIs that should drive agency evaluation. The right portfolio mixes leading indicators (early signals that the work is on track) with lagging indicators (eventual outcomes that justify the spend).
Leading indicators (months 1-4). Visibility on the commercial target-keyword cohort (impressions and average position in search console category tools, scoped to the keywords that matter). Indexing health on the target pages. Click-through rate on commercial queries. Crawl-budget efficiency on enterprise sites. AI-engine mention coverage on target prompts (newer, but increasingly material).
Mid-stage indicators (months 3-8). Organic sessions on target pages. Page-level conversion rate. Engagement metrics on landing pages (scroll depth, time-on-page, but with the understanding that these are easily gameable). New ranking arrivals on long-tail queries that signal topical authority is building.
Lagging indicators (months 6-12+). Organic-attributed leads or pipeline (with a defined attribution model). Organic-attributed revenue or won deals. Customer acquisition cost on the organic channel relative to paid. Retained-customer rate of organically-acquired customers (sometimes higher than paid, which is part of the long-term ROI story).
What is not a KPI on its own. Total organic traffic without cohort scoping (can be inflated by branded search or unrelated long-tail). Backlink count without quality scoring. Domain rating shifts (lagging and proxy-only). Average rank across all keywords (mixes commercial and informational, hides the metrics that matter).
Cohort analysis and attribution: separating lift from drift
The hardest measurement question in SEO evaluation is causation: did the agency’s work cause the lift, or would it have happened anyway? Cohort analysis is the operational answer.
Target-keyword cohort. Define a cohort of keywords the agency is being paid to influence (the commercial queries scoped in the engagement). Track average position, impressions, and clicks for the cohort over time. Compare against a control cohort (informational or branded queries the agency is not actively working). Lift in the target cohort that does not show up in the control cohort is more attributable to the agency’s work than to general market drift.
Target-page cohort. Define the pages the agency has worked on (newly published or actively optimised). Compare organic sessions and conversions on the target-page cohort against a control set of unworked pages. Same logic: lift on worked pages relative to unworked pages is more attributable.
Time-lag acknowledgment. SEO work has lag. Technical fixes can move metrics within weeks; new content typically takes three to six months to stabilise; backlink work takes longer. A good evaluation framework matches metric expectations to the realistic lag for each work type.
Algorithm-update normalisation. Major algorithm updates (Google core updates, AI-overview rollouts, spam updates) cause market-wide volatility that is not the agency’s fault or credit. Honest reports normalise around update windows or note them explicitly.
Branded vs non-branded split. A jump in organic traffic that comes entirely from branded search (people searching your brand name) is usually demand-generation work or PR, not the SEO agency’s work. Always split branded from non-branded in the reporting, and evaluate the agency primarily on non-branded movement.
What to ask for in monthly and quarterly reviews
The format of the review matters as much as the metrics in it. A good operating cadence has a monthly tactical review and a quarterly strategic review.
Monthly tactical review. Activities done. Cohort metrics with baselines. Wins and losses since last month. The two or three things that need decision or input from the buyer. The tactical plan for the next month. Should take forty-five to sixty minutes; longer is usually a sign the report is buried in detail without a clear narrative.
Quarterly strategic review. Cohort metrics with quarter-over-quarter comparison. Lagging indicators (organic-attributed pipeline, revenue). The hypothesis-test result on the bets that were placed last quarter. The strategic plan for the next quarter, including what is being deprecated. Forecasts for the next two quarters with stated assumptions.
Specific things to ask for. Search console data filtered to the target-keyword cohort, exported, with month-over-month and year-over-year comparison. The list of pages worked on this period and their before/after metrics. The list of links earned (with URLs, anchors, and the agency’s notes on relevance). The technical-fix list with deployment dates. AI-engine mention coverage on target prompts (if AI search is in scope). The agency’s honest read on what is and is not working.
Things to push back on. Reports that lead with average rank across all keywords. Dashboards that mix branded and non-branded traffic. Activity logs without metric outcomes. Vague claims about ‘authority building’ without specific link or content evidence. Refusal to share underlying search console or analytics data when asked.
Red flags in agency reporting
Specific signals in agency reporting suggest the work is not delivering or the reporting is not honest. None of these is fatal on its own; the pattern matters.
Rank-only dashboards. A report that leads with rank movements and never gets to organic sessions, conversions, or revenue is hiding the commercial picture. Rank is a leading indicator at best.
Average rank across all keywords. This metric mixes the long tail (where movement is easy and cheap) with commercial head terms (where movement is hard and valuable). Aggregate averages hide whether the agency is actually moving the queries that matter.
Missing baselines. Metric numbers without a baseline are uninterpretable. ‘Organic traffic up 30 percent’ is meaningless without knowing the starting point and the comparison period. Honest reports always show baselines.
Branded-search inflation. Organic traffic up 40 percent but branded search up 60 percent means non-branded probably went down. Reports that do not split branded from non-branded are obscuring this.
Vanity-metric inflation. Domain rating up, backlink count up, total impressions up – these can all rise without any commercial outcome. If the report leads with these and ducks revenue or pipeline, it is buying time.
No causation language. A report that lists activities and metric movements but never says ‘this caused that’ or ‘this did not work, here is our hypothesis why’ is not actually thinking about the work.
Refusal to share raw data. The buyer should always have access to the underlying search console and analytics data, not just the agency’s filtered dashboard. A refusal is a process problem.
Pattern of activity without movement. Six months of detailed activity logs and no movement on the commercial cohort is the clearest signal the work is not the right work.
Conclusion
Evaluating SEO agency results is a buyer-side discipline that does not happen on its own. The framework is: define commercial KPIs (organic-attributed pipeline and revenue, not rankings alone), set up cohort analysis so lift can be separated from drift, set realistic timeframes for leading and lagging indicators, demand honest reporting that includes causation reads and forward plans, and watch for the red flags (rank-only dashboards, missing baselines, branded-search inflation, refusal to share raw data, activity without movement). The agencies that deliver are usually willing to be measured this way; the ones that resist the framework are often the ones whose work would not survive scrutiny. Setting the evaluation contract early – what KPIs, what cadence, what data access, what timeframes – protects both sides and turns the engagement into a partnership rather than a black box.
Frequently Asked Questions
How long should I wait before evaluating an SEO agency’s results?
What is the most important KPI for evaluating SEO results?
How do I know if the agency caused the lift or it would have happened anyway?
Is rising domain rating or backlink count a real result?
What red flags suggest an SEO agency is not delivering?
Should I evaluate AI-search citation outcomes as part of agency results?
If you want a buyer-side review of an existing or proposed SEO engagement – KPI framework, cohort setup, reporting audit – we can scope it.
