Evaluating an AI assistant means measuring, with real questions and criteria defined in advance, whether it answers correctly, relies on the right sources, declines when it should, and does so at acceptable cost and speed. You do it before launch, on every prompt or model change, and continuously in production. Without evaluation, your assistant's quality is an opinion; with it, quality becomes a number you can improve.
Why "it looked good in the demo" fails
Almost every AI assistant impresses in its first demo. The problem is that demos are built from questions the team already knows will work. Production brings different ones:
- The long tail: customers who misspell, cram three topics into one message or ask for things the assistant should never answer.
- Non-determinism: the same model can answer the same question differently. One passing test does not guarantee the next.
- Invisible changes: a "small" prompt tweak fixes one case and breaks ten others. Your provider updates or retires a model. New documents land in the index.
- Confident errors: language models rarely volunteer "I'm not sure." A well-written wrong answer goes unnoticed if nobody measures it.
The fix is not more manual spot-checking; it is an evaluation system that runs the same way every time.
The golden set: real questions with expected answers
A golden set is a collection of real questions with the correct answer, the source that supports it and the grading criteria, used as a fixed benchmark for the assistant. It is the most valuable asset in the project, and it should be built by someone who knows the business, not just the engineers.
- Collect real questions: support chats, emails, tickets, intranet search logs. Strip personal data before using them.
- Start with 50 to 100 questions and grow to several hundred over time. More questions with automated grading beat a few hand-graded ones.
- Cover categories: common questions, hard cases, ambiguous questions, out-of-scope requests (the assistant should hand off or decline) and manipulation attempts.
- For each question record: the expected answer or facts it must contain, the source document, what it must never say and whether it should escalate to a human.
- Version the set: when a policy changes, update the expected answer; when production surfaces a failure, add it as a new case.
Which metrics to track
No single metric captures quality. These are the ones we recommend for a business assistant:
- Correctness: is the answer right and complete against the expected one?
- Groundedness: is every claim supported by the retrieved documents, or did the model add its own? Critical for enterprise RAG systems.
- Citation accuracy: does the cited source actually say what the answer claims?
- Refusal behavior: does it decline when evidence is missing or the request is out of scope? And the reverse: does it avoid refusing legitimate questions out of excess caution?
- Safety: no leaking other customers' data, no following injected instructions, no invented prices or commitments.
- Latency: time to first token and total time, at p50 and p95. On chat and voice channels, delays are noticed immediately.
- Cost per conversation: input and output tokens, retrieval and tool calls, summed over the whole conversation rather than per message.
- Resolution rate: the share of conversations resolved without a human, confirmed by the user or by the absence of a follow-up complaint.
- Escalation rate: how many conversations go to a person, and whether those handoffs were correct. Low escalation is not automatically good: it can mean the assistant answers what it should not.
How to grade: rules, humans and LLM-as-judge
There are three ways to grade, and you should combine them:
- Code-based checks: is the JSON valid? Is there a citation? Is the policy number correct? Does the answer quote a price that is not in the catalog? Cheap, fast and deterministic.
- Human review: the quality reference, but slow and expensive. Use it to calibrate and to review samples, not on every run.
- LLM-as-judge: a model grades responses against a rubric. A 2023 study (Zheng et al.) found that strong judges like GPT-4 agree with human preferences more than 80% of the time, about the same as humans agree with each other, but it also documented biases: favoring the first answer shown, favoring longer answers and favoring outputs from their own model family.
To make an automated judge trustworthy:
- Use concrete, binary criteria ("does it mention the 30-day window? yes or no") instead of "rate from 1 to 10".
- Use a different model from the one that generates the answers, as Anthropic's evaluation documentation recommends.
- Calibrate against humans: have a domain expert grade 50 to 100 responses, compare with the judge and refine the rubric until agreement is high. Recalibrate whenever you change the judge model.
- Manually review cases where the judge and the code checks disagree.
Regression testing on every change
Every change to the prompt, model, retrieval settings or document set should run against the golden set before reaching production, just as code runs through automated tests.
- Baseline: store results for the current version so you can compare.
- Compare by category: a stable average can hide that common questions improved while safety cases regressed.
- Blocking thresholds: for example, zero regressions on safety cases and no more than a one-point drop in overall correctness.
- Pin model versions rather than using aliases that change on their own, and re-evaluate before migrating when a provider announces a deprecation.
- Automate: run evaluations in your CI/CD pipeline and publish a report with each build.
Production monitoring and feedback loops
The golden set measures what you expected; production shows you what you did not. Good monitoring includes:
- Explicit feedback: thumbs up or down, with an optional reason.
- Sample review: every week, a business owner reviews a random sample of conversations plus every thumbs-down and escalation.
- Online automated grading: the judge scores a percentage of real conversations to catch drops in groundedness or citation quality.
- Operational alerts: spikes in latency, cost per conversation, tool errors or unanswered questions.
- Close the loop: every real failure becomes a golden set case so it cannot silently recur.
Mind privacy: conversation logs contain personal data. Define who can read them, how long they are kept, and de-identify what you reuse for evaluation, in line with state privacy laws, HIPAA where applicable and your SOC 2 controls. The Measure function of the NIST AI Risk Management Framework is a useful structure for documenting all of this for auditors and enterprise customers.
Red teaming: attack your own assistant
Red teaming means deliberately trying to make the assistant fail before a malicious user does. At a minimum, test:
- Instructions injected in the message or in documents and web pages the assistant reads. See our prompt injection guide.
- Attempts to extract the system prompt, other customers' data or documents outside the user's permissions.
- Requests for discounts, refunds or commitments the assistant is not authorized to make.
- Dangerous actions if the assistant uses tools: canceling orders, sending emails, editing records.
The OWASP Top 10 for LLMs makes a good checklist. Successful attacks also go into the golden set.
A simple scorecard to start with
| Metric | How it is measured | Example target | Frequency |
|---|---|---|---|
| Correctness | Golden set with calibrated judge | Set with the business based on use-case risk | Every change |
| Groundedness and citations | Judge plus check that the cited passage contains the claim | No unsupported claims on critical cases | Every change and weekly sample |
| Correct refusals | Out-of-scope questions in the set | Hands off or declines on all of them | Every change |
| Safety | Red-team cases | Zero regressions | Every change and quarterly |
| p95 latency | Telemetry | Based on channel (chat, voice, email) | Continuous |
| Cost per conversation | Token and tool telemetry | Within the business case budget | Weekly |
| Resolution and escalation | Logs plus sample review | Resolution trending up without wrong handoffs | Monthly |
Exact targets depend on the use case: an assistant that answers store hours tolerates more error than one explaining loan terms or benefits eligibility. If you are building a support bot, our AI chatbot on WhatsApp Business guide covers channel-specific details. Our AI development services include this evaluation setup from the first prototype.
Frequently asked questions
How many test questions do I need to evaluate an AI chatbot?
To start, 50 to 100 well-chosen real questions are enough to catch major problems and compare versions. Over time, grow to several hundred covering common questions, hard cases, out-of-scope requests and manipulation attempts. Representativeness matters more than volume: the questions should come from real customer or employee conversations, not be invented by the team that built the assistant.
What is LLM-as-a-judge, and is it reliable?
It means using one language model to grade another's answers against a rubric. It is reliable once calibrated: a domain expert grades a sample, you compare those grades with the judge and refine the rubric until agreement is high. Use concrete criteria, a different model from the one being tested and periodic re-checks, because judges have known biases such as favoring longer answers.
How often should an AI assistant be re-evaluated?
On every change to the prompt, model, retrieval configuration or documents, before it ships. Continuously in production, through weekly samples of real conversations and alerts on cost and latency. And whenever your provider releases a new version or deprecates a model, even if you changed nothing yourself, because the behavior of your assistant can shift underneath you.
Which metrics matter most for a customer service AI assistant?
For the business: resolution rate without human help, correct escalation rate and cost per conversation. For quality: correctness against expected answers, groundedness in current documents and declining when information is missing. As a hard requirement: zero safety regressions, meaning no exposure of other customers' data and no unauthorized promises about pricing, refunds or policy.
What is AI red teaming?
AI red teaming is deliberately attacking your own assistant to find failures before a malicious user does: prompt injection, system prompt extraction, access to data outside the user's permissions or unauthorized actions through tools. It happens before launch and periodically afterward, and every successful attack becomes an automated test so the same weakness cannot quietly return.
Conclusion
A trustworthy AI assistant does not come from picking the best model; it comes from a process: a golden set of real questions, metrics that reflect the business, an automated judge calibrated against humans, regression tests on every change, production monitoring and red teaming. That process turns quality into something you can manage and improve week after week.
If you want to launch an assistant with this level of control, or measure the one you already have, explore our AI development services or book a free consultation with our nearshore team in UTC-5.