Scoring the answer was never the whole job
Everything up to this point in the course has scored one request and one response. A user asks something, the system answers, and evaluation — deterministic checks, a calibrated judge, a human rater — looks at that single pair and renders a verdict. An agent breaks that assumption completely, because an agent doesn't produce an answer, it produces a sequence of actions that eventually arrives at one.
That sequence is where things actually go wrong. An agent can call a destructive tool it never needed to call, or read from a data source it had no business touching, or take four inefficient detours before landing on the right answer — and if you only score what it says at the end, none of that shows up. The final answer can be completely correct while the path that produced it did something you'd never have approved if you'd been watching.
That's the gap this lesson closes: scoring only the final answer misses most of what can go wrong in an agentic system. We'll cover the three layers mature agent evaluation actually scores, why tool-call correctness gets pulled out of judge-based scoring entirely, and the benchmarks people reach for when they need a standardized comparison point.
The trajectory is where the risk lives
Here's a scenario that final-answer evaluation is structurally incapable of catching: an agent is asked to update a customer's shipping address, and it gets the right address into the right field, so the final answer looks perfect. But to get there, it first queried the entire customer table instead of the one record it needed, and it called a delete-and-recreate tool instead of an update tool because that path happened to also work. Nothing about the outcome tells you that happened. You'd have to have been watching the whole time.
That's why mature agent evaluation scores three separate layers instead of one. The final-answer layer is what you already know — score only the last message, exactly the way you'd evaluate a single-turn system, ignoring everything that came before it. The trajectory layer scores the full sequence of intermediate steps: which tools got chosen, whether the reasoning behind each choice holds up, whether the arguments passed to each call were correct, and whether the order those steps happened in made sense. The per-turn layer applies when the agent isn't just answering one request but running an ongoing multi-turn session, and it scores each turn's contribution to that session rather than treating the whole conversation as one lump verdict.
Put together, those three layers catch the failure mode that matters most in systems with real tool access: a correct final answer reached through a wrong or unsafe trajectory. The destructive tool call that wasn't needed, the context that got read when it shouldn't have been touched — both of those look completely fine if you only check what the agent said at the end. They only show up when something is actually watching the steps in between.
Once you accept that the trajectory needs its own scoring pass, a design question follows immediately: does every step in that trajectory need the same kind of scoring? It doesn't — and the sharpest illustration of that is the part of the trajectory that shouldn't be judge-scored at all.
The catch: Trajectory scoring is more expensive to build than final-answer scoring — it requires capturing and structuring every intermediate step, not just the last message, which means your evaluation harness needs full tracing wired in before it can score anything. Teams that skip straight to "does the agent give the right answer" often don't discover they need trajectory eval until a destructive tool call already happened in production.
Not everything in the trajectory needs a judge
It would be easy to assume that scoring a trajectory means running every step through an LLM judge, the same way you'd score an open-ended answer. That assumption is wrong for a specific, important piece of the trajectory: which tool the agent picked and what arguments it passed to it.
Tool-call correctness is checked deterministically, not with a judge. At each step, you compare the agent's selected tool name and its arguments against the expected call for that step, and tool selection precision falls out as a simple ratio — correct tool calls divided by total tool calls. There's no ambiguity to adjudicate here: either the agent called update_address with the right customer ID, or it didn't. Either the required parameter was present and correctly typed, or it wasn't.
That's exactly why a judge is the wrong tool for this layer. Tool names, required parameters, and expected structured outputs are exact-match problems — the kind of thing a few lines of comparison code check in milliseconds with zero ambiguity. Spending judge tokens on something checkable in code buys you nothing: you pay the latency and cost of an LLM call, and you introduce judge-bias risk, for a verdict that deterministic code would have given you for free and with total confidence. The right instinct is the opposite of throwing more model at the problem — reserve judge scoring for the parts of the trajectory that are genuinely subjective, like whether the overall plan the agent chose was a reasonable one given the request.
So the trajectory splits cleanly in two: the mechanical layer — did it call the right tool with the right arguments — gets checked in code, and the judgment layer — was this a sensible plan — gets checked by a judge. Knowing where that boundary sits is most of what separates a well-designed agent eval harness from an expensive, noisy one. But even a well-designed harness built entirely in-house eventually needs a way to compare against something outside itself.
The tradeoff: It's tempting to route everything through the judge because it's less code to write upfront — one general-purpose scoring call instead of a purpose-built comparator for every tool signature. That shortcut costs you exactly where it matters most: the deterministic layer is supposed to be your cheapest, fastest, most trustworthy signal, and judge-based tool checking throws away all three properties for no gain in accuracy.
A leaderboard tells you less than your own logs
Once you've built trajectory scoring and split out the deterministic tool-call layer, a reasonable next question is how your agent stacks up against agents other people have built. That's what the standardized benchmarks are for, and three show up repeatedly in agent evaluation: AgentBench, WebArena, and SWE-bench.
Each targets a different slice of agent capability. AgentBench spans a range of interactive environments to give a general read on agent reasoning and tool use. WebArena scores agents navigating and completing tasks on realistic websites — the kind of multi-step, tool-mediated work a browsing agent actually has to do. SWE-bench evaluates whether an agent can resolve real GitHub issues by producing a working code patch, which makes it the closest thing the field has to a standardized test of coding-agent competence. All three give you a comparison point that's reproducible across labs and papers, which is genuinely useful when you're deciding between base models or evaluating whether a new agent architecture is worth adopting.
What none of them give you is a verdict on your system. A benchmark score tells you how an agent performs on tasks someone else designed, against a rubric someone else wrote, using tools that almost certainly aren't the ones your agent actually calls. Your production agent might use a proprietary internal API, operate inside a domain with its own vocabulary and failure modes, and be judged by users who care about things no public benchmark encodes. None of that shows up in an AgentBench leaderboard position.
That's the same lesson this course has returned to from the very first lesson on reference-based metrics: a general-purpose benchmark is a useful sanity check and a poor substitute for your own golden dataset. Standardized benchmarks earn their place as a cross-system comparison point, not as a replacement for the application-specific trajectory eval built from your own tool set and your own failure modes — which is the thing that actually tells you whether your agent is safe to ship.
The gotcha: A strong SWE-bench or WebArena score is easy to over-read as "this agent is production-ready." It tells you the underlying model and scaffolding are competitive on a fixed, public task distribution — it says nothing about how the same agent behaves on your tools, your data, and the edge cases your users actually hit. Treat benchmark performance as a floor, not a substitute for eval against your own trajectory data.
Evaluation before, observability after, feedback in between
Every practice in this course sits on one side or the other of a single moment: deployment. Before it, evaluation — deterministic checks, calibrated LLM-as-judge, human-validated rubrics, golden datasets, CI gates — answers "is this change safe to ship." After it, observability — traces, the OTel GenAI vocabulary that makes them portable across vendors, drift detection, cost and latency tracking — answers "is it still safe, three weeks later, against traffic nobody tested for." Safe rollout patterns are the bridge between the two: they let you find out an answer is wrong from a canary's dashboard instead of from a user.
The discipline that separates demos from products isn't a single tool, and it isn't a task you finish. It's a loop: evaluate before you ship, observe after you ship, and route what production teaches you back into the next evaluation. A system with a great CI score and no production observability will eventually ship a silent regression it never notices; a system with rich observability and no eval gate will ship regressions on purpose, one merge at a time, and only find out from the graphs afterward. Neither half works without the other.
Agent orchestration is where this loop gets tested hardest. More steps means more tool calls, more tool calls means more ways for a trajectory to go wrong that a final-answer score can't see, and every practice this course covered — reference metrics, calibrated judges, golden datasets, CI gates, tracing, drift detection, safe rollout, and now trajectory-aware agent eval — is what it takes to keep that loop closed once the system doing the acting isn't just answering questions anymore, but doing things.