How to evaluate an AI support agent before and after every prompt change

Teams ship an AI agent into support, watch it work well in a demo, and then discover in week three that it confidently answered a billing question with a policy that changed last quarter. A benchmark would not have caught this. A golden set built from your own tickets would have, a gap the ai support onboarding checklist should include as a launch step, not an afterthought.

This guide is for whoever owns quality on the support agent: an engineer wiring up evaluation, a support lead reviewing agent output, or a founder deciding whether the next prompt change is safe to ship. It assumes the agent already answers from your content, covered in RAG for customer support, and focuses on how you know whether it is any good.

Why public benchmarks do not tell you what you need to know

Public LLM benchmarks measure general capability: reasoning, coding, broad knowledge. They say nothing about whether your agent knows your refund window, correctly refuses to promise a feature you do not have, or hands off cleanly when a customer is angry about a billing error.

A model that scores well on a general benchmark can still fail your support use case entirely, because the failure modes that matter in support are narrow and specific to your business, not general reasoning gaps. Conversely, a smaller or older model grounded well in your content can outperform a larger ungrounded one on your actual tickets.

Twig's research on common complaints about AI customer support tools lists hallucinated answers, missing escalation paths, robotic tone, and poor integration as the top user complaints, none of which a general capability benchmark measures. These are evaluation targets specific to the support domain, and they only show up when you test against support conversations.

The honest conclusion is that a benchmark score is a weak proxy at best. The only evaluation that predicts real performance is one built from the same kind of tickets your agent will actually face, scored against the answer a competent human on your team would have given.

What a golden set is and why 200 tickets is the right size

Communicate.so

A golden set is a curated collection of real, resolved support tickets paired with the answer your team actually gave. It is not synthetic, and it is not written from imagination. Every entry comes from a conversation that really happened, with a customer question and a human-verified correct response.

Two hundred tickets is a workable size for most support operations, large enough to cover the recurring question categories and the genuine edge cases, small enough that a person can review the full set in a day or two when building it and again after any major change.

Pull the set from your ticket history with a deliberate sampling strategy rather than a random scrape. Include the most frequent question types in proportion to their real volume, then deliberately over-sample edge cases and past mistakes, because a golden set that mirrors only the easy majority will not catch what actually breaks an agent, an approach consistent with how HappySupport frames deflection measurement as a maturity curve rather than a single snapshot.

Each entry needs four fields at minimum: the customer's original question with full context, the correct answer as your team gave it, the source document or policy that answer relies on, and a difficulty tag such as common, edge case, or policy-sensitive. The source field matters because it lets you verify the golden answer is still accurate when your policies change.

Refresh the golden set on a fixed cadence, not only when something breaks. A quarterly review that checks each entry's source document against current policy catches drift before a customer does, and ties directly into how you should already be maintaining the content your agent trains on, covered in train AI on your help center.

Scoring: what a rubric needs to avoid reviewer disagreement

A vague rubric produces inconsistent scores, and inconsistent scores make regression testing worthless, because you cannot tell whether a score dropped due to a real regression or due to a different reviewer's mood. The fix is a rubric with explicit, binary-leaning criteria instead of a single 1 to 10 quality slider.

Score each golden set answer against four dimensions separately: factual accuracy against the source document, whether the agent stayed within its actual capabilities and did not promise something untrue, tone and clarity appropriate to the situation, and whether an escalation happened when one should have. Each dimension gets a pass, partial, or fail, not a single blended number.

Factual accuracy and escalation correctness can be partially automated, because you can programmatically check whether the model's cited source matches the golden set's source, and whether an escalation flag fired when the golden answer says it should. Capability honesty and tone are harder to automate reliably and benefit from a second model acting as a grader plus a periodic human spot check, an approach consistent with Anthropic's documentation on evaluation tooling for language model applications.

Run every score with at least two independent reviewers on a sample of the set, and measure their agreement rate. If two reasonable people score the same answer differently more than roughly one time in ten, the rubric is still too vague and needs sharper pass conditions before you trust it for regression decisions.

Automated scoring versus human review: where each earns its keep

Communicate.so

Automated scoring is fast and cheap, which makes it the right tool for every prompt change, but it misses nuance a human catches immediately. Use it as the first gate on every change, not the only gate.

A second LLM acting as a grader, given the golden answer and the model's actual answer, can flag likely factual mismatches and missing citations reliably enough to catch the majority of regressions before a human ever looks. This is cheap enough to run on every single prompt or model change without friction.

Human review earns its keep on tone, capability honesty, and the genuinely ambiguous cases where an automated grader's judgment is unreliable. Run a full human pass over the entire golden set before a major change, such as a new base model or a significant prompt rewrite, and a smaller spot check of twenty to thirty entries for minor changes.

The two methods are not redundant. Automated scoring catches drift fast enough to block a bad deploy in minutes. Human review catches the subtler failures, like an answer that is factually correct but promises something the product does not actually do, that automated graders routinely miss.

Log every scoring run with the model version, the prompt version, and the full set of per-entry scores, not just an aggregate. When a regression appears, you need to see which specific entries dropped and why, not just that the average score moved, a level of traceability that matters just as much for actions as it does for answers, covered in ai agent actions and APIs.

Regression testing a prompt change before it ships

Treat every prompt change, every model swap, and every retrieval configuration change as a candidate for regression, the same discipline you would apply to a code change in any other part of the system. Run the full golden set against the new configuration before it reaches a single customer.

Compare the new scores entry by entry against the last known-good baseline, not just the aggregate average. An aggregate can hold steady while a handful of policy-sensitive entries quietly fail, and those are exactly the entries where a wrong answer costs the most.

Set a hard gate: no entry that previously passed factual accuracy or correct escalation may fail after the change, without an explicit, reviewed reason. This mirrors how Google describes gated deployment checks for ML pipelines, where a regression on a critical metric blocks the release rather than triggering a note for later.

Keep the last several baseline runs archived, not just the most recent one. When a regression appears two changes later, you need to bisect between the last good state and the current one, and that requires more than a single snapshot of scores.

Communicate.so

Building the golden set from real tickets: a worked procedure

Start by pulling the last three to six months of resolved tickets from your support system, filtered to conversations with a clear resolution and no unresolved dispute. This avoids polluting the set with cases where the human answer itself was uncertain.

Bucket the pulled tickets by topic using whatever tagging your support tool already has, then sample proportionally to real volume for the common categories. A category that makes up thirty percent of your real tickets should make up roughly thirty percent of the golden set, not an equal split across every category regardless of frequency.

Deliberately add a smaller, dedicated bucket of edge cases: past mistakes the agent or a human made, ambiguous policy questions, and any incident where a customer escalated because the first answer was wrong. These entries are worth more than their share of volume because they are where a regression is most likely to hide, a pattern documented incidents like the Feb 2026 hallucinated downgrade policy incident at a cloud storage company make concrete, where a chatbot cited a policy that never existed.

Have a second person independent of the agent's build verify each golden answer against the actual current policy before it enters the set. An answer that was correct when the ticket was resolved six months ago may no longer be correct if your policy changed since, and an unverified golden set will quietly teach you to trust wrong answers.

Common pitfalls that quietly break an evaluation program

Communicate.so

The most common failure is a golden set that quietly goes stale. A team builds a careful set of 200 tickets, ships it, and never revisits the source policies behind each answer. Six months later a third of the entries still show the old refund window as correct, and every regression check against them is testing the agent against a policy that no longer exists.

The fix is not complicated but it is easy to skip under deadline pressure: tie each golden set entry to its source document with a link or an ID, and run a lightweight audit whenever that source document changes. A policy update should trigger a review of every golden set entry that cites it, not wait for the next scheduled quarterly pass.

A second common pitfall is scoring drift between reviewers over time, even with a good rubric. A reviewer who has scored two hundred entries in one afternoon tends to loosen their standards by the end of the session, a pattern well documented in human annotation research summarized by NIST's AI risk management framework discussion of measurement consistency. Break scoring sessions into smaller batches across multiple days and re-score a handful of early entries at the end to check for drift.

A third pitfall is over-fitting the prompt to the golden set itself. If every prompt iteration is tuned until it scores perfectly against the same 200 entries, the prompt can end up memorizing quirks of that specific set rather than generalizing to new questions. Hold out a smaller separate set of tickets, refreshed independently, purely for a final sanity check before shipping a change that scored well against the main golden set.

A fourth pitfall is measuring only accuracy and ignoring escalation behavior, which is where real damage tends to concentrate. An agent that answers 95% of questions correctly but fails to escalate the remaining 5% when it should is worse in practice than one that escalates slightly more often, because a wrong answer given with confidence does more harm than a clean handoff to a person, a distinction covered in AI to human handoff.

The last pitfall worth naming is treating evaluation as a one-time project instead of a recurring practice tied to every change. A golden set built once at launch and never run again is a snapshot of quality on day one, not a guarantee about the agent running in production a year later after a dozen prompt tweaks and a model migration.

A sixth pitfall, subtler than the rest, is letting the person who wrote the prompt also grade its output against the golden set. Self-grading introduces a quiet bias, because the author tends to read their own intent into an ambiguous answer rather than judging it as a customer would. Assign scoring to someone who did not write the change under review, even if that means a support lead grades an engineer's prompt edit or a second engineer reviews the first one's work.

Watch also for a golden set that grows lopsided toward whichever failure mode caused the most recent incident. Teams naturally add entries for the exact scenario that just went wrong, which is useful, but over several incidents the set can drift away from proportional coverage of your real ticket volume. Periodically re-check the sampling balance against current ticket data, not just against the list of past fires you have fought.

Where communicate fits, honestly

Communicate exposes per-conversation transcripts and outcomes through analytics, which is the raw material for building and refreshing a golden set: real questions, real answers, and whether the conversation resolved or escalated to a human.

The platform does not ship a built-in automated regression pipeline against a golden set today, that evaluation layer is something you build around the exported conversation data using the procedure in this guide, run against your own scoring rubric and thresholds.

The honest limits: there is no native A/B testing between prompt or model versions inside the product, and the single-model setup on gpt-4o-mini through OpenRouter means model comparison is not the axis you tune. Entry is a one-time $1 activation with 100 test credits, detailed on the pricing page. Questions go to [email protected].

Key takeaways

• A public LLM benchmark predicts almost nothing about support performance, because the failure modes that matter are specific to your policies, product, and edge cases.
• Build a golden set of roughly 200 real, resolved tickets, sampled proportionally to real volume with a deliberate over-sample of edge cases and past mistakes.
• Score against explicit, binary-leaning criteria across four dimensions: factual accuracy, capability honesty, tone, and correct escalation, not a single blended quality number.
• Use fast automated scoring as the first gate on every change, and human review as the deeper check on tone, capability honesty, and ambiguous cases.
• Regression-test every prompt, model, or retrieval change against the full golden set before it ships, comparing entry by entry against the last known-good baseline.

Ready to build the feedback loop that catches a regression before a customer does? Start with a one-dollar account activation that includes 100 test credits, pull your first hundred resolved tickets, and build the first version of your golden set this week. The ai agent guardrails guide covers the refusal thresholds a golden set should specifically test.

Frequently asked questions

What is a golden set in AI support agent evaluation?

A golden set is a curated collection of real, resolved support tickets, each paired with the answer your team actually gave and verified as correct against a specific source document. It differs from a synthetic test set because every entry reflects a conversation that genuinely happened, which is why it predicts real performance far better than a public benchmark, as covered in the golden set section above and in RAG for customer support.

Why is 200 tickets a reasonable golden set size?

Two hundred is large enough to cover the recurring question categories in rough proportion to their real volume, plus a meaningful sample of edge cases, while staying small enough that a person can review the entire set in a day or two. Smaller teams can start with fewer and grow the set as new failure modes surface.

Why do public LLM benchmarks fail to predict support performance?

Because they measure general reasoning and knowledge, not whether an agent knows your specific refund window, correctly refuses to promise a feature you do not have, or escalates cleanly when a customer is upset. Twig's research on common AI support complaints lists hallucination and missing escalation paths as top issues, and neither shows up in a general capability benchmark.

What fields should each golden set entry include?

At minimum: the customer's original question with full context, the correct answer as actually given, the source document or policy the answer relies on, and a difficulty tag such as common, edge case, or policy-sensitive. The source field is what lets you verify an entry is still accurate as policies change.

How often should a golden set be refreshed?

On a fixed cadence, quarterly at minimum, checking each entry's cited source against current policy, not only after something breaks. This mirrors the ongoing maintenance a knowledge base itself needs, covered in train AI on your help center, because a stale golden set will silently validate wrong answers.

What are the four scoring dimensions for an agent's response?

Factual accuracy against the cited source, capability honesty meaning no promise of a feature or policy that does not exist, tone and clarity appropriate to the situation, and correct escalation when confidence should have been low. Each dimension should be scored separately as pass, partial, or fail, not blended into one number.

Why avoid a single 1 to 10 quality score?

A single blended score hides which specific dimension failed and makes reviewer disagreement hard to diagnose. Explicit, near-binary criteria across separate dimensions let two reviewers converge more reliably and let you see exactly what broke when a regression appears.

How do you know if a scoring rubric is too vague?

Measure agreement between two independent reviewers scoring the same sample of answers. If they disagree on more than roughly one in ten entries, the rubric's pass conditions are not specific enough to trust for regression decisions, and it needs sharper, more concrete criteria.

Can automated scoring fully replace human review?

No. Automated scoring, often a second model acting as a grader, catches likely factual mismatches and missing citations fast enough to run on every change, an approach consistent with Anthropic's evaluation tooling documentation. It misses subtler issues like an answer that is factually correct but promises a capability the product does not have, which still needs human judgment.

How often should human review run versus automated scoring?

Run automated scoring on every prompt or model change, since it is cheap and fast enough to block a bad deploy in minutes. Run a full human pass over the entire golden set before major changes like a new base model, and a smaller spot check of twenty to thirty entries for minor prompt edits.

What counts as a regression when comparing evaluation runs?

Any previously passing golden set entry, particularly one scored on factual accuracy or correct escalation, that fails after a change, without an explicit and reviewed reason for the change in outcome. Comparing only the aggregate score across a run can hide individual entries that quietly failed.

Should you compare aggregate scores or entry-by-entry scores?

Entry by entry, always, in addition to the aggregate. An average score can stay flat while a handful of policy-sensitive entries silently fail, and those specific entries are usually the ones where a wrong answer costs the most in a real conversation.

How do you build a golden set without inventing test cases?

Pull real, resolved tickets from the last three to six months of your support history, filtered to conversations with a clear resolution and no ongoing dispute. Bucket by topic and sample proportionally to real ticket volume, then deliberately over-sample edge cases and past incidents, such as the Feb 2026 hallucinated downgrade policy incident where a chatbot cited a policy that never existed.

Why over-sample edge cases in a golden set instead of sampling purely by volume?

Because edge cases are disproportionately where regressions hide. A prompt change is far more likely to break a rare, policy-sensitive answer than a common, well-covered one, so a golden set that mirrors only the easy majority will miss exactly the failures that matter most.

How do you verify a golden answer is still correct before adding it to the set?

Have someone independent of the agent's build check the answer against the current, live version of the source policy or document, not the version that was true when the ticket was originally resolved. Policies change, and an unverified golden set will quietly validate an answer that is now wrong.

What should trigger a full golden set re-run versus a spot check?

A full run against every entry belongs before any major change: a new base model, a significant prompt rewrite, or a retrieval configuration change. A smaller spot check of twenty to thirty entries is usually sufficient for a minor wording tweak that touches a narrow part of the prompt.

How does golden set evaluation relate to action safety?

The same regression discipline applies to actions as to answers: any change to how the agent proposes or scopes an action call should be tested against a golden set of past action decisions, logged with the same traceability described in ai agent actions and APIs, so a regression in action-triggering behavior is caught before it reaches a real customer.

What is the biggest mistake teams make when evaluating a support agent?

Trusting a general benchmark or a small handful of manually spot-checked conversations instead of building a systematic, regularly refreshed golden set from real tickets. A benchmark score can look good while the agent is quietly wrong on the exact questions your customers actually ask.

How do you prevent reviewer scoring drift during a long evaluation session?

Break scoring into smaller batches spread across multiple days rather than one long session, and re-score a handful of early entries at the end to check whether standards loosened. Reviewers who score two hundred entries in one sitting tend to grow more lenient toward the end, which quietly distorts the results.

Should a golden set include tickets where the human answer itself was uncertain?

No. Filter the source tickets to conversations with a clear resolution and no ongoing dispute before building the set. A golden set entry built from an uncertain human answer teaches the evaluation process to trust an answer that was never actually verified as correct in the first place.