Section

data-ai

Kind

Editorial

Reading time

4 min

Updated

8 September 2026

Evaluating a Model Change Without Fooling Yourself: Metrics, Rubrics and Observability for AI systems

Can your team tell whether the new prompt or model is really better? [Evaluating model changes is notoriously hard, because the evaluation requires a apples-to-apples test with the same expensive latency, quality, and cost metrics. That’s why the best practice is to compare model versions in a fixed golden task set, under the same exact rubric and resource constraints, and record the effect on latency and costs as well as the familiar metrics of accuracy or success. Evaluation of a model or prompt change should also include human review when the output gambles with important or subjective portions of business value, and confirm everything on a shadowed live traffic sample before stepping up to an A/B test.]

Model changes are To evaluate a change, you must first compare the changed system to a fixed baseline using the same task set, scoring rubric, and conditions and this infers a bias-free, apples-to-apples comparison. A baseline is typically [a held-out set: a golden collection of tasks that have been curated for evaluation but not adjusted for model quality., as well as a 1–10% sample of the target traffic that must be reviewed by humans for quality against a structured rubric.] The Stanford guide recommends [using a rubric-based human review], and tracking the cost and latency of generating the answers in the same evaluation.

Comparing a change to a fixed baseline avoids falling into the trap of gameable metrics: detailed but non-generalizable behavior that boosts the score on the training set but not on novel input, as when a change improves one example without being [better on a held-out task set..] With a held-out evaluation set, models consistently raise the bar on the full range of tasks, rather than moving their score up in a training-targeted direction. Even better, the held-out set can be consistently rated by multiple participants with human consistency checks and a rigorous rubric. It is the way to [withstand self-persuasion and the temptation to effect a well-crafted answer that misleads the annotator into a judgment the model can’t reproduce. ]

For the most difficult evaluations—those where [the criterion is subjective or important—for example, when it involves reasoning, tone, coherence, or factual accuracy—the guide recommends using rubrics that are [transparent and stringent. This is because any opaque criterion for subjective evaluation is a playground for self-persuasion.]] A good rubric thinks through the edges and the spectrum of how people in the target role could react, and [offers clear patterns and examples on regular examples and edge cases in the held-out set, to reduce ambiguity and difference between reviewers. At the same time, a checklist can logged for a subset of [all cases, as well as [a one- or two-line overall satisfaction score]](https://www.codse.com/blog/ai-agent-evaluation-production-observability-cost-attribution).]

A fixed rubric and a golden set are not the only protection against self-persuasion; they are also the first step to evaluating a change that avoids ignoring latency and cost. Because if the change comes with [a cost or latency advantage, this matters just as much as the answer quality in production. Indeed, many model changes are designed to reduce latency, or to trade off latency for less expensive units. [The cost and latency of producing a held-out result matters most when trying to determine the net change to the operation.']

Even with a fixed held-out task, a baseline rubric, and the same or better latency-cost tradeoff, a single example of a better answer or a win in online tests does not prove that the new configuration is better (https://preporato.com/blog/ccar-p-evaluation-strategy-eval-datasets-metrics-guide). To [move from a win on the golden set to production readiness, the winning model of prompt must be exposed to real traffic and then incrementally reviewed with a mix of human and automatic quality assessment until it converts as well as or better than the existing alternative] To [shadow-test, the model and prompt are run on real user input, usually in a staging environment, with the output actually sent to an endpoint like a database or logging pipeline, and then analyzed against the existing baseline on the full range of metrics (quality, latency, cost, safety, and security) (https://www.codse.com/blog/ai-agent-evaluation-production-observability-cost-attribution).] If the shadow-test matches or exceeds the existing quality on the golden task set, ambitiously timed response targets, and a holistic cost specification (total tokens, successful requests, failure-finding), then [it is ready for A/B testing on live traffic.] (https://preporato.com/blog/ccar-p-evaluation-strategy-eval-datasets-metrics-guide) There are parts of this procedure—especially the metrics definition and the shadow-test metrics tracking in production observability—where human review is essential, but these parts are no longer viewed as a subjective judgment where the AI’s answer could game the evaluator, but are an, almost mechanical application of the rules.

[The end result of this quality, latency, and cost laser focus is not to maximize any one of them in a complex trajectory-down-the-hill of training data, but to find a well-rounded combination using a stable evaluation set and scoring criteria. This lets teams introduce new models, tools, and functions, with full confidence that the addition is a productive move for the ultimate stakeholders.]