Passing the gate isn't the finish line
Passing the CI quality gate from the last lesson feels like a resolution. The prompt change is versioned, the golden dataset ran clean, the score cleared the threshold, and the pull request merges. But all that gate actually proved is that the system was good on one day, against one fixed dataset, under one set of conditions you thought to write a test for.
It says nothing about whether the system is still good three weeks from now, against the traffic your actual users send — traffic nobody wrote a test case for, because nobody could have anticipated it. A user pastes in a 40,000-token document. A tool call returns malformed JSON for the first time. A model provider quietly updates the weights behind an API you don't control. None of that shows up in a pre-merge eval run.
That's the job that starts once the gate closes behind you: observability. Not "did it pass," but "what actually happened on this specific live request," aggregated over time so you can notice when the trend starts moving the wrong way before your users notice for you. That starts with a single idea — the trace — and we'll build outward from there.
The unit of observability is one request
When an eval score drops from 92% to 85%, the number alone tells you almost nothing about what to fix. Somewhere inside that 7-point gap are real requests that went wrong for real reasons, and the aggregate score has already thrown away exactly the information you need to find them.
That's what a trace gives you back. A trace is one end-to-end request, from the moment it enters your system to the moment a response goes out. Inside it, every LLM call, every retrieval step, every tool invocation is a span — a timed, attributed unit of work with its own inputs, outputs, and duration. For a single API call, a trace might be one span. For a multi-step agent, it's a tree: a top-level agent span with child spans branching off for every reasoning step, every tool call, every handoff to a sub-agent.
The tree structure is what makes traces useful for debugging rather than just logging. When something goes wrong deep in a multi-hop agent run, you don't want a flat wall of log lines you have to reconstruct into a sequence by hand — you want to open the trace, see the shape of the tree, and click straight down to the one child span where retrieval returned garbage or a tool call used the wrong argument. That's the specific mechanism that explains why the aggregate score is 85% instead of 92%: not a global property of the system, but a concrete, reproducible failure in one branch of one request.
The catch: A trace is only as useful as what you chose to capture as span attributes when you instrumented the code. If you didn't record the retrieved documents, the tool arguments, or the intermediate reasoning step, the trace shows you that something happened in that span — not what. Instrumentation decisions made on day one determine what's debuggable on day ninety.
A shared vocabulary for spans
Traces and spans aren't a new idea — distributed tracing has existed for web services for years. The problem specific to LLM and agent systems was that nobody agreed on what a span for a model call should actually contain. A span from a LangChain agent recorded different attributes, under different names, than a span from a raw API call to the same model. Every observability vendor that wanted to support both had to write bespoke translation logic per framework, and that translation logic broke every time a framework changed its internals.
Since 2024, the OpenTelemetry GenAI Special Interest Group has been standardizing that vocabulary. The GenAI semantic conventions define consistent attribute names for the things every LLM call has in common: which model actually handled the request, how many input and output tokens it consumed, what tools it called and with what arguments, and — opt-in, because it's often sensitive — the full prompt and completion content.
The payoff is that instrumentation stops being per-vendor work. You instrument your code once, against the OTel GenAI conventions, and any conventions-compatible backend understands the resulting spans without translation — a span from a LangChain agent and a span from a raw API call look the same shape to the backend, because they're using the same names for the same concepts. That's what turns "add observability" from an N-vendor integration problem into a one-time investment.
Which backend you point that standardized telemetry at is a separate decision — and it's one where the honest answer today is "it depends," because the platforms built to receive this data are optimized for different things.
The catch: Adoption is real but partial. Some vendors are fully compatible with the GenAI conventions out of the box; some approximate the schema and translate under the hood; some still want you on a proprietary SDK if you want their full feature depth. Standardization reduces the instrumentation burden — it doesn't yet guarantee you can point the same OTel exporter at any backend and get identical fidelity. Check current vendor docs before you assume portability.
Where the traces actually go
Once you're emitting standardized spans, you need somewhere to send them — and the honest answer is that no single platform is the obvious default, because they're each optimized for a different point on the tradeoff between integration depth and integration cost.
If you're already committed to LangChain or LangGraph, LangSmith gives you the deepest integration and functions almost like an IDE for agent execution, at the cost of coupling you to that framework. Langfuse takes the opposite bet: framework-agnostic via OTel, open source, and it gives you full feature parity whether you self-host it or use the hosted version — a natural fit if vendor lock-in is a concern. Arize Phoenix and its hosted sibling Arize AX come from an ML-observability heritage and go deepest on eval primitives and drift and embedding analysis specifically, which matters a lot for the concept we're building toward next. Helicone sits at the cheap end of the integration-cost axis: it's a drop-in proxy, so you change one base URL and start getting traces with zero SDK changes — you trade away framework-level structure for that simplicity. And if you're already standardized on Datadog or Weights & Biases for other observability or experiment tracking, their native LLM offerings — Datadog LLM Observability and W&B Weave — are usually the pragmatic choice over introducing a new vendor.
The general shape of the tradeoff: a proxy-based tool like Helicone only sees API-call-level detail because that's all that passes through the proxy, while a framework-native tool like LangSmith sees the full agent-execution structure because it's instrumented from inside the framework. Neither is strictly better — the right one depends on whether you need that deep structural visibility or just want the fastest possible path to "we have traces."
The tradeoff: Picking a platform is easy to defer and expensive to redo. If you instrument against generic OTel GenAI conventions rather than a vendor SDK, switching backends later is mostly a config change. If you build directly against a proprietary SDK for its deeper feature set, migrating means re-instrumenting. Decide up front how much you're willing to lock in for how much extra depth.
Telemetry isn't a story until you aggregate it
A single trace answers a single question: what happened on this one request. That's essential for debugging the specific failure that dragged your eval score down. But one trace, on its own, doesn't tell you whether your system is healthy — it tells you about one moment in time.
The real value shows up when you stop looking at individual traces and start aggregating them: watching cost and latency trend upward across thousands of requests, or watching the shape of responses quietly drift because a model provider changed something upstream, or because your retrieval index started returning subtly worse documents while every conventional health check — response length, HTTP 200 rate — kept reporting green. Spans and standardized conventions are what make that aggregation possible in the first place; without them you'd just have thousands of incompatible log lines.
In the next lesson, we'll take that raw trace telemetry and turn it into the actual signals that matter in production — cost, latency, and drift — and look at how to roll out changes safely once you can finally see what's happening while it happens.