By Dean McCoubrey Chief AI Strategist
Forgive the Kennedy-shaped phrasing, but ask not which AI tools the agency uses. Ask what those tools have changed about its thinking.
If the answer is only speed, volume, or cheaper production, you are not looking at AI capability. You are looking at a faster factory.
The short answer: Evaluating whether a marketing agency genuinely understands AI comes down to seven questions.
Ask whether AI changes the decisions they make, not just the speed.
Ask for a campaign where AI shifted strategic direction mid-flight.
Ask what they rejected from the AI output.
Ask about proprietary IP.
Ask how they govern brand outputs and handle data at scale.
Ask how they integrate with your systems.
Ask how they connect AI activity to revenue and build your capability over time.
This guide also covers how to choose the right partner model before you start shortlisting, and how to run the process without getting trapped in polished decks.
AI is now table stakes in marketing. According to Salesforce’s State of Marketing 2026, 87% of marketers now use generative AI in at least one workflow. That number sounds impressive. It also means that claiming to “use AI” tells you almost nothing about an agency’s actual capability.
The gap between AI adoption and AI capability is where most agencies are quietly bluffing. When almost every agency will tell you they use AI, asking whether they use it stops being useful. What you actually need to know is whether the AI they use has changed anything that matters: the quality of their thinking, the rigour of their governance, and the commercial results they can prove.
This guide is written for CMOs and marketing directors who are actively shortlisting partners and need something more useful than a demo of tools they have already seen.
AI theatre versus genuine AI capability
Some agencies will be fluent in the language of AI before they are mature in the practice of it. The tools arrived fast, the vocabulary spread faster, and the pitch decks followed immediately. What takes longer to build is an operating model where AI actually changes how decisions get made.The distinction worth naming: AI theatre is visible AI usage that leaves strategic judgment, governance, and measurement unchanged. The agency uses AI to produce content faster, to generate more options, to compress timelines. The work looks different. The thinking does not.
| AI Theatre | Genuine AI Capability | |
|---|---|---|
| Where AI sits | Production layer only | Across strategy, planning, delivery, and measurement |
| What changes | Speed and volume | Decisions, recommendations, and outcomes |
| Brief quality | Generic or templated; prompts fed without structured context | Client-specific, hypothesis-driven; the brief is the thinking |
| Iteration logic | Volume and speed; more options, faster | Selective and editorially controlled; rejection is part of the process |
| How it shows up in a pitch | Tool demonstrations, workflow diagrams | Case examples with changed direction and proven results |
| Governance | Absent or informal | Documented, with clear brand safety and approval frameworks |
| Measurement | Efficiency metrics (time saved, content volume) | Commercial metrics (revenue lift, pipeline, CAC, LTV) |
| Who owns it | One AI specialist or department | Embedded across the team |
The Real Story Group’s framework for evaluating AI capability draws a useful distinction between three types of AI: generative AI (writing, editing, creative production), insights AI (audience analysis, attribution, predictive modelling), and decisioning AI (next best action, dynamic optimisation, journey testing). Most agencies have invested in the first. Fewer have built the second. The third is where genuine competitive advantage lives, and it is the hardest to fake in a pitch.
Why the Standard Partner Models Leave Buyers Exposed
Most buying committees frame the market in three categories: traditional agency, AI-native vendor, and consultancy. Each has a genuine strength and a structural gap. Understanding where each breaks down is more useful than picking a category.
According to Forrester’s 2026 Buyers’ Journey Survey, 94% of B2B buyers used AI in their most recent purchase process. Vendors know this. The pitch has adapted accordingly. What has not adapted is the way most buyers evaluate what they are actually buying.
| Model | Strength | Limitation | What it means for buyers |
|---|---|---|---|
| Traditional agency | Strong delivery and creative execution | AI often bolted on, not systemic; operating model rarely redesigned | You get familiar outputs, not better decisions |
| AI-first vendor | Speed, automation, and production scale | Often lacks strategic judgement, organisational empathy, and commercial context | You get more output, not smarter strategy |
| Consultancy | Strategic clarity and operating model design | Rarely executes; capability often stops at the recommendation | You get a framework, not a growth engine |
| Strategic growth partner | Better decisions, connected execution, and embedded capability | Fewer in the market; harder to evaluate on standard scorecards | You get advantage that compounds, if you know what to look for |
AI has democratised execution. It has not democratised judgement. The partners worth choosing understand that distinction.
Which model fits your situation?
Before you start shortlisting, be clear about what your business actually needs. For a deeper look at how AI-first and AI-augmented operating models differ in practice, that guide is worth reading alongside this one.
- High data volume, performance media, programmatic — AI-first structure, with clean data and a clear measurement framework already in place
- Complex brand, B2B relationships, regulated sector — AI-augmented, human-led model with strong governance
- Early-stage AI maturity internally — AI-augmented partner who builds your capability alongside delivering results
- Strategic transformation, not just execution — a growth partner who connects strategy to commercial outcomes, not a vendor who executes briefs faster
The right model depends on your situation. Getting this wrong is not a minor procurement error. It is a competitive disadvantage with a compounding cost.
Seven Questions to Ask Any Agency in a Pitch
These questions are not designed to catch agencies out. They are designed to surface the difference between an agency that has thought carefully about how AI changes their work, and one that has learned to talk about it fluently. Both will give you an answer. The quality of the answer is the data.
1. How does AI change the decisions you make, not just the speed at which you make them?
This is the foundational question. It separates capability from convenience.
One concrete place to look before the pitch even begins: ask to see the brief that preceded a recent deliverable. Agencies that genuinely integrate AI start every workflow with structured thinking, not a prompt. A real AI-integrated brief documents the commercial objective, the audience context, the constraints, and the hypotheses being tested. Without it, the output is generic by design.
- Strong answer: The agency describes a specific moment where AI-generated insight led to a different strategic recommendation, a changed channel mix, or a revised creative direction. They can name the decision and explain the outcome. They can show you the brief that preceded the work.
- Red flag: The answer focuses on workflow efficiency, turnaround times, or content volume. Speed is not strategy.
2. Can you show me a campaign where AI-driven insight changed the creative or media direction mid-flight?
This tests whether AI is embedded in active decision-making or only used in pre-production.
- Strong answer: A concrete example with a before and after. What did the AI surface? What changed? What was the commercial result?
- Red flag: Vague references to “ongoing optimisation” with no specific example. If they cannot name the campaign, they probably cannot name the outcome.
3. What did you reject from the AI output, and why?
Any agency can run a prompt. The skill is in evaluating what comes back. Agencies that understand AI can walk through specific examples of output they discarded, rewrote, or challenged. That editorial judgement is the actual product.
- Strong answer: “The model gave us three headline directions. We rejected two because they did not reflect the brand’s commercial positioning. Here is the version we rewrote and the reasoning behind it.”
- Red flag: Enthusiasm about speed and volume with no mention of quality control or rejection logic. If an agency cannot describe what they decided not to use, they may not be making those decisions at all.
4. What proprietary methodology or IP sits behind your AI capability?
Licensed tools are accessible to every agency in the market. They create speed, but they rarely create differentiation. The stronger buying signal is whether an agency has translated its experience into proprietary thinking: frameworks, diagnostics, prompt systems, scoring models, or repeatable processes that are genuinely theirs.
| Licensed tools | Proprietary IP |
|---|---|
| Available to all agencies | Built from the agency’s own experience |
| Creates speed | Creates consistency and originality |
| Replicable by competitors | Defensible and compounding |
| Reflects vendor capability | Reflects agency judgement |
| Depreciates as tools evolve | Strengthens over time |
- Strong answer: Documented methodology that shapes how the work gets done. The agency’s IP should be visible in how they think about your brief, not just in what tools they mention.
- Red flag: A slide deck listing third-party platforms with no explanation of how those platforms are directed, combined, or quality-controlled.
5. How do you maintain brand quality and governance, including data, privacy, and responsible AI?
UK Government AI adoption research identifies brand safety, trust, and measurement as the central concerns in AI adoption. A credible agency should have thought about this carefully — and not just the brand safety dimension.
Data and AI laws have grown by 400% since 2016, with EU AI Act enforcement now underway and the FTC taking a strong stance against misleading AI claims. Enterprise buyers are increasingly expecting agencies to explain data provenance, obtain explicit consent for AI training use, and provide clear documentation of how data moves through their systems.
- Strong answer: A documented governance framework covering review layers, approval processes, risk ownership, data provenance, consent, storage, model usage policies, retention periods, and a named person accountable for AI governance within the agency.
- Red flag: Governance treated as an afterthought, or described as “we always have a human check it” without any structure behind that claim. No named accountability. Vague answers about which AI models are used and how client data is handled.
The IAB’s State of Data 2025 found that around 70% of marketers have experienced AI-related incidents in live campaigns, yet only approximately one-third have formal AI governance tools in place. A partner who cannot answer this question in plain language is not ready for procurement scrutiny, let alone board-level accountability.
6. How do you integrate with our data environment, systems, and governance constraints?
This is where most AI partnerships fail in practice. A partner who cannot map cleanly to your data environment, existing workflows, and governance requirements will create friction that erodes the commercial case before it has a chance to prove itself.
The INFUSE Voice of the Buyer 2026 report, based on 310 enterprise buyers, found that proven integration was the top gating factor in AI vendor selection at 47%. More than quantified ROI. More than plain-language clarity. Buyers have been burned by partnerships that looked good in the pitch and broke down in the data layer.
- Strong answer: The agency can describe specific integrations they have managed, what broke, and how they handled it. They ask you about your stack before they answer this question.
- Red flag: Vague answers about APIs and “seamless integration.” If they have not asked about your data environment, they are not thinking about it.
7. How do you connect AI activity to commercial outcomes, and how do you build our capability over time?
This is the question that most directly separates genuine AI capability from efficiency theatre. It also combines two sides of the same accountability question: does working with this partner make you more commercially effective, and does it make your organisation more capable, or just more dependent?
Writer’s 2026 enterprise AI report found that while 59% of companies invest over £1 million annually in AI, only 29% report significant returns. That gap lives in the space between outputs and outcomes.
- Strong answer: The agency can describe how AI-assisted activity connects to pipeline, revenue, customer acquisition cost, or retention. They measure commercial impact, not just activity. They also document their process, share their reasoning, and treat knowledge transfer as part of the engagement.
- Red flag: All measurement is expressed in efficiency terms: time saved, content produced, impressions generated. No link to revenue. No answer to what your team looks like after 12 months with them.
The question worth adding at the end of any pitch:
“Will working with you make our team more capable over time, or more dependent on you?”
Strong AI partners improve your organisation’s judgement and AI fluency alongside delivering commercial results. Weaker ones keep the process opaque. That is the difference between a vendor and a growth partner.
A note on answers that feel rehearsed: What you are listening for is not spontaneity. It is specificity. A rehearsed answer with a real example behind it is still a good answer. A fluent answer with no example behind it is not.
What Genuine AI Integration Looks Like in Practice
Beyond what an agency says in a pitch, there are observable signals in how they work. Genuine AI integration tends to show up across the full workflow, not concentrated in a single team or a single stage of delivery.
Observable signals worth looking for
- AI in insight generation, not just production. The agency uses AI to surface audience signals, identify anomalies, and model attribution, not only to generate copy or creative variants.
- First-party data at the centre. AI capability without data infrastructure is limited. An agency that has genuinely embedded AI will have a clear view of how client first-party data feeds into their models and tools.
- Human editorial control over AI outputs. The most credible agencies describe a deliberate tension between AI-generated material and human judgment. They have not automated away the thinking; they have changed where the thinking happens.
- Measurement that includes AI-native channels. As AI-powered search and answer engines reshape how buyers discover brands, the agencies ahead of this shift are already measuring prompt visibility and answer engine inclusion alongside traditional metrics. If an agency has no view on this, they are measuring the last decade, not the next one.
- Cross-team adoption. AI capability that lives in one person or one department is fragile. When it is embedded, it shows up in how strategists brief, how planners model, and how account teams report.
The real test is coherence. An agency with genuine AI capability can describe how AI connects strategy to execution to measurement as a single system. An agency performing AI capability tends to describe each application separately, because there is no system behind them.
The Red Flags That Reveal AI Theatre
None of these signals is definitive in isolation. Several together are worth taking seriously.
- The answer to “how do you use AI?” is a list of tools. Naming platforms is not evidence of capability. Ask what those tools changed about the work. If the answer circles back to tools, you have your answer.
- They cannot name a decision AI changed. Fluent answers with no example behind them are a reliable signal. If AI has genuinely shifted how an agency thinks, they should be able to describe a specific instance without pausing to construct one.
- One person owns the AI capability. A single specialist is a dependency, not an operating model. When AI is genuinely embedded, it shows up in how strategists brief, how planners model, and how account teams report. Not in one person’s job title.
- The pitch is a demonstration, not a dialogue. Agencies with genuine capability ask questions. They want to understand your data, your governance requirements, your measurement framework. Agencies performing capability tend to show you things and wait for applause.
- They measure AI success by efficiency alone. Time saved and content volume are real gains. But if the agency cannot connect AI activity to pipeline, revenue, or commercial lift, they are optimising the wrong thing. Spencer Stuart’s CMO research identifies ROI and commercial proof as the central anxieties driving senior marketing decisions in 2026. An agency that cannot speak to those is not yet speaking the right language.
- Governance is vague or overly technical. A partner who cannot explain their governance model in plain language is not ready for procurement scrutiny, let alone board-level accountability.
How to Run the Shortlist Without Getting Trapped in Decks
The standard agency selection process, long RFP followed by multiple pitch presentations and chemistry scores, is not well-suited to evaluating AI-enabled partners. It rewards polished presentation over applied thinking. Here is a more useful approach.
- Shortlist three partners maximum. More than that and the evaluation becomes a comparison of decks rather than a genuine assessment of capability. Use the seven questions above to filter before you invite anyone to pitch.
- Replace the chemistry pitch with a working session. Give each shortlisted partner the same live commercial problem and ask them to work through it with you. How they think is more revealing than what they present.
- Define a pilot with the right success metrics. A 30 to 60-day pilot is a sensible starting point, but only if the pilot measures decision quality and commercial learning, not output velocity. If you measure the pilot on volume, you will select for volume.
- Insist on the delivery team being in the room from day one. The gap between pitch team and delivery team is where most agency relationships start to erode. Require it upfront, not as a negotiating point later.
Key takeaway: You are not buying AI capability. You are buying better commercial judgement. The process should be designed to test that, not to test who can produce the most impressive slide deck in three weeks.
The INFUSE Voice of the Buyer 2026 report found the market has moved decisively from hype to proof. Proven integration, quantified ROI, and governance clarity are now the top gating factors. Structure your shortlist process around those criteria, not around which agency has the most impressive tool stack.
A Practical AI Agency Evaluation Scorecard
This scorecard is designed to be used during or immediately after a pitch process. Score each dimension 1 to 3. A score below 20 out of 30 warrants deeper scrutiny before any shortlist decision is made.
| Dimension | 1 – Weak | 2 – Developing | 3 – Strong |
|---|---|---|---|
| Decision impact | AI used only in production | AI informs some recommendations | AI demonstrably changes strategic decisions |
| Case proof | No specific examples | Examples exist but outcomes unclear | Named campaigns with measurable commercial results |
| Governance | No framework described | Informal review process | Documented governance with clear accountability |
| Stack clarity | Tool list with no rationale | Some explanation of tool choices | Principled, agnostic-where-appropriate stack with clear logic |
| Data integration | No mention of first-party data | Some data integration described | First-party data central to AI capability |
| Measurement quality | Efficiency metrics only | Mix of efficiency and commercial | Commercial outcomes: revenue, CAC, LTV, pipeline |
| Cross-team adoption | AI sits with one specialist | Some team-wide use described | AI embedded across strategy, planning, and delivery |
| Strategic fluency | Tool-led answers | Process-led answers | Outcome-led answers with commercial framing |
| AI-native search | No awareness of answer engines | Awareness but no measurement | Active measurement of AI-native discovery and visibility |
| Commercial accountability | No link to revenue | Indirect commercial references | Direct connection between AI activity and business outcomes |
Score interpretation:
- 25-30: Strong capability. AI appears genuinely embedded in operating model and commercial thinking.
- 20-24: Developing capability. Probe the gaps before committing, particularly on governance and commercial proof.
- Below 20: Proceed with caution. The agency may be fluent in AI language without the operating model to back it up.
The scorecard is a decision-support tool, not a replacement for judgment.
For use during the pitch conversation itself (before the scorecard is applied):
- Show me the brief that preceded this work.
Considered answer: documented, client-specific, hypothesis-driven.
Worth noting: a generic template or no brief at all. - What did you reject from the AI output, and why?
Considered answer: specific examples with clear editorial reasoning.
Worth noting: “we review everything” with no specifics. - How do you validate AI outputs commercially?
Considered answer: named metrics and documented feedback loops.
Worth noting: volume and speed claims with no outcome data. - What would you not use AI for in my project?
Considered answer: an honest, specific list of decisions that require human judgement.
Worth noting: “AI can do it all” or vague reassurance. - Who is accountable for AI-assisted work?
Considered answer: a named person with a clear sign-off process.
Worth noting: “the team reviews it.” - Can you show me your process model?
Considered answer: a diagram with decision points and validation stages.
Worth noting: a list of tools. - Will working with you make our team more capable over time, or more dependent on you?
Considered answer: knowledge transfer built into the engagement.
Worth noting: process kept opaque; dependency by design.
A strong score in a pitch does not guarantee strong delivery. But a weak score in a pitch is usually a reliable signal.
For a broader view of how AI-first agencies are positioning themselves in the UK market, the best AI-first marketing agencies in the UK for 2026 guide provides useful context on how genuine capability is being built and described.
The Honest Question to Ask Yourself
The right question at the end of any agency pitch is not “do they understand AI?” Almost everyone will pass that test in 2026. The question is whether AI makes their thinking better, or just their production faster.
The former is rare. The former is what changes commercial outcomes.
An agency that uses AI to compress timelines and increase output volume is offering you efficiency. That has value. But an agency that uses AI to improve the quality of its decisions, to surface insights that change strategic direction, and to connect marketing activity to commercial results is offering you something different. The distinction is worth paying attention to.
Most companies ask what AI can do. The organisations that pull ahead ask a harder question: what should AI help us become? The answer is not a better content machine. It is a sharper, faster, more confident decision-making organisation. AI makes production abundant. Judgement remains scarce. Choose the partner who brings both.
For a broader view of how AI-first and AI-augmented agencies are positioning themselves in the UK market, the best AI-first marketing agencies in the UK for 2026 guide provides useful context on how genuine capability is being built and described.
Frequently asked questions
What questions should I ask an agency about AI?
Ask five questions: how AI changes the decisions they make (not just the speed); whether they can show a campaign where AI shifted creative or media direction mid-flight; how they maintain brand governance when AI generates outputs at scale; what their AI stack looks like and where they are tool-agnostic; and how they connect AI-assisted activity to commercial outcomes such as pipeline, revenue, or customer acquisition cost.
How do I spot AI theatre in a marketing agency?
AI theatre is visible AI usage that leaves strategic judgment, governance, and measurement unchanged. The clearest signals: the agency leads with tool names rather than outcomes; their AI capability sits with one specialist rather than being embedded across the team; there is no documented governance framework for AI-generated brand outputs; and all measurement is expressed in efficiency terms rather than commercial results.
What does a genuinely AI-first agency look like?
A genuinely AI-first agency uses AI across the full workflow, from insight generation and planning through to delivery, optimisation, and measurement. It can show specific examples where AI changed a strategic recommendation, not just sped up production. It has a documented governance framework. It measures commercial outcomes, not just efficiency gains. And it has a view on AI-native search and answer engine visibility, not only traditional channel metrics.
How do I evaluate AI capability without being misled by a good pitch?
Use the five-question framework and the 10-dimension scorecard in this article. A score below 20 out of 30 warrants deeper scrutiny. Pay particular attention to specificity: an agency with genuine capability will give you named examples with outcomes. An agency performing capability will give you fluent answers with no example behind them.
Why does AI governance matter when evaluating a marketing agency?
Because AI-generated outputs at scale carry real brand risk. If an agency cannot describe who reviews AI-generated material, how brand standards are maintained, and who owns accountability when something goes wrong, that is not a minor gap. It is a governance failure waiting to happen. The UK Government’s AI adoption research identifies brand safety and trust as central concerns in AI adoption for exactly this reason.