How to Evaluate a RAG Release: 5 Production Gates
A RAG release should not ship because one aggregate score improved. It should ship when retrieval, citation support, safe refusal, tail latency and cost each stay inside a boundary the product team approved before the run. One strong metric cannot cancel a failure in another gate.
That turns evaluation into a release decision instead of a dashboard. Every gate needs a versioned dataset, a named owner, an explicit hold condition and a rollback trigger. The artifact below is a release card that can travel with a candidate from staging to production. These five gates cover RAG behavior and economics. They do not replace access-control tests, privacy review, prompt-injection testing or other security checks. Treat those as separate non-compensating prerequisites, especially when the corpus contains restricted data or the application can take actions.
The five-gate RAG release card
| Gate | Required evidence | Pass or hold rule | Primary owner | Rollback trigger |
|---|---|---|---|---|
| 1. Retrieval | Fixed queries, expected sources, retrieved IDs and ranks | Pass only when every critical slice meets its approved threshold and the candidate does not hide a regression behind the aggregate | Search or ML engineer | Required evidence disappears or falls below the usable rank boundary |
| 2. Citation support | Claim-to-source spans, source version, authority and freshness checks | Hold when a material claim lacks support, points to the wrong passage or relies on a source outside its freshness budget | RAG application and content owners | Unsupported or stale claims rise above the approved incident boundary |
| 3. Safe refusal | Paired answerable and unanswerable cases, plus conflicting and forbidden-source cases | Pass only when false answers and false refusals stay inside separate approved limits | Product and AI engineer | The system answers a critical no-evidence case with unsupported content |
| 4. Tail latency | End-to-end and stage timings by query class, load level and cache state | Hold when the relevant percentile breaches its SLO, even if the average improves | SRE or platform engineer | Tail latency or timeout rate breaches the rollback condition |
| 5. Cost per successful answer | Provider usage, retrieval, reranking, validation, retry and fallback cost per case | Pass when successful answers stay inside the per-class budget without quality or refusal regressions | Product and platform owner | Cost per successful answer breaches budget or climbs with falling success rate |
For a wider view of the corpus, retrieval, evaluation and operating boundaries around these checks, the Pharos Production page on production RAG system layers maps the surrounding delivery scope. The scorecard here has a narrower job: decide whether one named candidate can take traffic.
Build a release dataset that can fail
Start with the product contract, not a metric catalog. Define which query classes the system must answer, which sources it may use, when it must refuse and which user-visible delay is acceptable. A support assistant and a research tool can use the same retrieval stack while needing different evidence rules.
The dataset should contain real production-shaped queries or expert-written equivalents. Add reviewed synthetic cases only to extend coverage, not to stand in for ground truth. Keep at least these query classes separate:
| Query class | Expected behavior |
|---|---|
| Ordinary answerable lookup | Retrieve the required source and answer from it |
| Hard answerable query | Combine the necessary passages without dropping a qualifier |
| No approved answer | Refuse or state the evidence gap |
| Conflicting or stale sources | Apply the source policy, qualify the answer or refuse |
| Restricted source | Exclude unauthorized evidence before it reaches the model |
Freeze the corpus snapshot, chunking and embedding configuration, retriever and reranker versions, prompt, model, evaluator and dataset revision for each run. If those inputs change together, a score delta cannot identify the cause. Compare the candidate with the last accepted baseline on the same snapshot whenever the release question allows it.
Do not copy a threshold from a vendor tutorial. Set each boundary from labeled cases, the current baseline, measurement variability and the consequence of a false pass. A high-risk query class may need a hard invariant while a low-risk discovery query can use a review band.
Retrieval gate 1: Surface the required evidence
Score retrieval before looking at the generated answer. Record the expected document or passage for each labeled case, then preserve the IDs and ranks returned by the candidate. Required-document hit rate, recall at k, mean reciprocal rank or nDCG can be useful, but the metric must match the product question. A release card should report results by query class as well as in aggregate. If common lookups improve while multi-passage queries lose their required second source, the aggregate can still look healthy. The slice failed.
The pass rule has two parts. Every critical slice must clear its predeclared boundary, and the candidate must not regress against the accepted baseline beyond the approved tolerance. When retrieval fails, inspect ingestion, authorization filters, query rewriting, ranking and context truncation before changing the generation prompt.
Citation support gate 2: Check current claims
A citation is not valid merely because the URL opens. For every material claim, verify that the cited passage entails the claim, the citation resolves to the intended document version, the source is allowed for that query class and its age stays inside the declared freshness budget. Claim-level checks expose failures that answer-level groundedness can hide. A response may be mostly supported while one date, limit or exception comes from an obsolete source. Store the claim-to-span mapping so a reviewer can inspect the exact evidence rather than rerun the whole conversation.
Hold the release when a critical claim has no supporting passage, when the link resolves to different content or when a newer authoritative source conflicts with the cited version. Do not average these cases into a general citation score if the product contract treats them as blockers.
Safe refusal gate 3 works in both directions
A RAG system can fail by answering without evidence or by refusing when adequate evidence exists. Measure those errors separately. An overall refusal rate cannot distinguish a cautious system from a broken retriever. Create paired cases with similar wording but different evidence states. One case has a current, authorized source and should be answered. Its pair removes that source, replaces it with conflicting material or makes it inaccessible to the test identity. The expected behavior should say answer, qualify, ask for clarification or refuse.
The release rule should weight consequences. A false refusal may frustrate a user, but a confident false answer in a policy, financial, clinical or operational workflow can carry a different cost. Keep a manual review band for borderline cases and calibrate any model-based judge against expert labels before using it as a gate.
Tail latency gate 4 must fit the product SLO
Measure the whole request and every stage that can own a delay: query processing, permission filters, sparse and vector retrieval, reranking, context assembly, generation, citation validation, retries and streaming. Report the percentile named in the service-level objective, not only a mean that hides slow requests. Run the same query slices under the load and cache states the release will encounter. Separate cold and warm paths. A candidate that improves retrieval quality by adding a reranker may still fail if time to first token or end-to-end tail latency crosses the user-visible boundary.
The gate passes when each required query class remains inside its SLO and the measurement is comparable with the baseline. If infrastructure, region or load generator changed, record that break instead of presenting the result as a clean comparison.
Cost gate 5: Tie spend to successful answers
Cost per request rewards cheap failures. Track cost per successful answer as the primary economic signal, then keep per-request cost as a diagnostic. Include embedding or search calls, reranking, model input and output, grounding checks, retries, fallbacks and any evaluator that runs synchronously in the user path. Segment the result by query class. Multi-source research may have a different budget from a short support lookup. Also report the success denominator, because a lower average cost means little if the candidate refuses more answerable questions or skips citation validation.
Set the gate against an approved budget and compare it with the accepted baseline on identical cases. A cost increase may be acceptable when it resolves a measured quality failure, but that trade needs an explicit decision rather than a hidden weighted score.
Route each failure to the first useful owner
| Failure pattern | First evidence to inspect | Likely first owner | Do not start with |
|---|---|---|---|
| Required source never appears | Ingestion record, ACL decision, query transform and candidate ranks | Data, search or ML engineer | A larger generation model |
| Source is retrieved but omitted from context | Context selection, deduplication, ordering and token budget | RAG application engineer | Re-embedding the corpus |
| Context is correct but a claim is unsupported | Prompt version, answer trace and claim-to-span result | AI application engineer | Raising top k blindly |
| Answer appears on a no-evidence case | Refusal policy, paired case and judge label | Product and AI engineer | Lowering every retrieval threshold |
| Tail latency regresses | Stage spans, cache state, retry path and load profile | SRE or platform engineer | Averaging the slow slice away |
| Cost rises while success falls | Per-stage usage, retry count, fallback path and success label | Product and platform owner | Declaring a better composite score |
This routing order is a diagnostic aid, not an ownership chart for every organization. The point is to move from a failed gate to the earliest stage that can explain it.
Write the go, hold or rollback record
The release record should make the result reproducible without relying on a dashboard that will change. Store the candidate and baseline IDs, corpus and dataset revisions, pipeline component versions, evaluator versions, gate results by slice, approved thresholds, owners and the final decision.
Record exceptions as waivers with a reason, approver and expiration date. A warning with no owner or expiry is a delayed release decision. If production monitoring later crosses a rollback boundary, the same record should identify the last accepted configuration and the person authorized to restore it.
Offline evaluation also needs a production continuation. Sample traces under the applicable privacy and retention policy, watch the same quality, refusal, latency and cost signals, then promote confirmed incidents and new query classes into the next dataset revision.
The smallest complete record ends with release_id, baseline_id, dataset_version, corpus_version, five gate results, one decision, one owner per failure and one rollback target. If any of those fields is missing, the release is not yet reproducible.
Comments
Post a Comment