A passing eval doesn't watch production for you
By the end of the last lesson, you had traces and spans running through every layer of the pipeline — every retrieval call, every generation step, every tool invocation, captured and searchable. That's raw signal. It tells you what happened on any single request you go looking for. It doesn't, by itself, tell you what to do about it.
This lesson is about turning that signal into decisions. Two decisions, specifically: what to actually watch on an ongoing basis so degradation surfaces before a user has to file a complaint about it, and how to ship a change to a model or prompt without finding out it was a bad idea only after it's serving one hundred percent of your traffic.
Those two problems are related, and they compound if you get either one wrong. Weak monitoring means you don't notice a bad rollout until it's already caused damage. A reckless rollout process means that even great monitoring is discovering problems after the fact instead of before. We'll take them one at a time, starting with what to watch.
Cost, Latency & Drift
Here's an uncomfortable fact about running an LLM feature in production: a request that costs two-tenths of a cent is completely invisible at a thousand users a day, and a six-figure line item at a hundred thousand. Nobody notices the cost problem while it's small. They notice it on the monthly cloud bill, months after the decision that caused it shipped. The fix isn't complicated — track per-request token cost in the same dashboard as latency and error rate — but it has to be there from day one, because by the time finance asks about it, the answer is supposed to already exist.
Latency needs the same treatment, and it needs to be measured at the right grain. Time-to-first-token is what a user actually experiences as responsiveness; time-per-output-token is what determines whether a streaming response feels smooth or stutters word by word — cross roughly 100 milliseconds per token and users start to notice the gaps. And track percentiles, not averages. A p50 that looks great can hide a p99 that's making ten percent of your users wait eight seconds, and averages will never tell you that story.
The failure mode that neither cost nor latency dashboards catch is drift — the shape of responses quietly changing over time. Output distribution drift shows up as response length, sentiment, or refusal rate shifting without any code change on your side, often because a model provider updated something behind a version string you didn't touch. In a RAG system specifically, embedding drift is its own variant: the embedding model or the index changes, semantic similarity shifts, and retrieval starts returning subtly worse documents — while response length stays normal and every request still returns HTTP 200. Aggregated across enough traces, this telemetry is what surfaces cost and latency creep and drift long before anyone would spot it by eyeballing individual requests.
The reason drift deserves its own category, and not just a line on the same dashboard, is that it's exactly the failure standard monitoring is structurally blind to. Nothing crashes. Nothing looks wrong from the outside. A normal-looking latency graph and a 200 status code coexist perfectly well with the model quietly getting worse underneath them.
The catch: None of these dashboards catch a quality regression on their own — a model can get measurably worse at answering questions while cost, latency, and response length all stay flat. The fix is to run the same judge and rubric scoring you built for CI against a sampled slice of live production traffic, on an ongoing basis rather than only at deploy time. It's cheaper than scoring everything, it catches degradation that happens between deploys instead of only at them, and it's what actually keeps you from finding out three weeks later that something broke the day after your last release.
Shadow & Canary Rollout
Seeing a problem and safely fixing it are two different skills, and the gap between them is where most production incidents actually live. Rolling a new model or a new prompt straight to all of your traffic means the first signal you get that something regressed is real user impact — a support ticket, a spike in complaints, someone screenshotting a bad answer. That's a completely avoidable way to find out.
The safer sequence starts with shadow traffic: duplicate live production requests to the candidate model or prompt, log what it produces, and compare that output against what your current production system actually returned — all without ever showing the candidate's answer to a real user. The risk is zero, because nobody downstream ever sees it. The tradeoff is that you only get comparative output quality, not real user reaction, since nobody actually reacted to anything.
Once shadow traffic looks clean, canary is the next step: route a small real percentage of traffic to the new version and watch it closely — latency percentiles, cost per request, refusal and error rate, and the output-length distribution, plus any explicit user feedback signals you have access to. This is where you find out how the candidate behaves under real usage patterns and real user reactions, at a blast radius small enough that a bad surprise costs you very little. Only after canary holds up do you ramp toward full traffic.
The tradeoff: Skipping shadow because the new version "seems fine" is the single most common way a canary ends up catching something that should have been caught earlier, for free, with zero users involved. Shadow traffic costs you nothing but compute and a little pipeline plumbing — there's rarely a good reason to skip straight to canary.
Feedback Loops
Every production system is already collecting a signal most teams don't fully use: what users do after they get an answer. A thumbs up or thumbs down, an explicit correction, a user retrying the same question phrased a different way, an escalation to a human — none of that requires you to build anything new. It's free, continuous, and it's happening whether you capture it or not.
The highest-value thing to do with that signal isn't to stare at a dashboard of satisfaction percentages. It's to route the negative signal back into the golden dataset you built in Lesson 4. A real production failure that a real user flagged is a better regression-test case than another synthetic example your team wrote in isolation, because it's proof — not a guess — that this exact input breaks something for someone.
That's what closes the loop between the earlier lessons and this one. Production observability doesn't just watch the system from the outside; it manufactures the next generation of the eval suite that gates the next change. A golden dataset that never gets refreshed with real failures slowly drifts out of sync with what your users are actually doing, and it starts giving you false confidence — passing evals on cases nobody encounters anymore, missing the ones that actually matter now.
The gotcha: Feedback signal is noisy and biased by nature — users who are frustrated enough to leave a thumbs-down are a self-selected sample, not a representative one. Treat flagged failures as candidates for the golden dataset that need triage, not as an automatic, unfiltered feed. The value is in the real failure cases they surface, not in the raw satisfaction rate.
Evaluation before, observability after, feedback in between
Zoom out and the last several lessons form one continuous system, not a checklist. Evaluation happens before deployment — the golden datasets, the judge scoring, the CI quality gates that decide whether a change is even allowed to ship. Observability happens after deployment — the traces, the cost and latency dashboards, the drift detection that tells you whether what shipped is still behaving the way it did when it passed CI. And feedback loops are the connective tissue: they take what observability finds in production and feed it back into the eval suite that gates the next change, so the system that checks your work keeps getting sharper instead of going stale.
Shadow and canary rollout are what make the "after deployment" phase survivable — they're the difference between finding a regression in a controlled, low-blast-radius test and finding it in a user's screenshot. None of this replaces good evals. It's what happens in the gap between an eval suite giving you a green checkmark and a system actually being trustworthy at scale, over time, as the world underneath it keeps shifting.
In the next lesson, we'll see what happens when this entire picture gets harder all at once — because everything we've covered so far assumed the system was answering one question at a time. Lesson 8 is agent-specific evaluation, where the system isn't just answering, it's taking a sequence of actions, and every one of those extra steps and tool calls is another place things can go wrong.