AI Agent Observability: Tracing, Metrics, and Evals Explained

What is AI agent observability, and how is it different from APM?

AI agent observability is the practice of capturing, tracing, and evaluating everything an AI agent does while handling one request — not just whether it returned a 200 in 800ms. Classic application performance monitoring (APM) treats a request as one span with a duration and an error flag, which works when the server does one deterministic thing. An agent is different: a single user request can fan out into a planning step, several nested model calls, multiple tool calls, a retrieval step, and a handoff to another sub-agent — and any one of those can be the reason the response was slow or wrong.

Elastic's engineering team and Honeycomb both make the same point from different angles: you need the whole span tree — the parent-child relationship between invoke_agent, chat, and execute_tool spans — to debug an agent, because the aggregate request duration hides which exact step caused the problem.

Why agent observability matters now

Once an agent has more than one tool, or hands work off between sub-agents, a plain error log stops being enough. A request that "worked" in the sense of returning an answer can still have silently called the wrong tool, burned 10x the tokens it needed to, or lost context across a multi-turn session — none of which shows up as an HTTP error. Without a trace of every step, you're debugging by guessing which of the five things that happened inside the request actually went wrong.

This is also why the industry converged on a shared standard rather than every vendor shipping a proprietary tracing format: the OpenTelemetry GenAI Semantic Conventions mean a trace produced by one SDK can be read by Langfuse, Phoenix, MLflow, or a generic OTel backend like Honeycomb or Elastic, instead of locking you into one vendor's trace format.

How agent tracing actually works (OpenTelemetry GenAI semconv)

The OpenTelemetry GenAI semantic conventions define standard span names for the operations an agent performs — invoke_agent, chat, execute_tool, and plan among them — plus a common set of attributes carried on every span:

A simplified trace for "research a topic, then write it up" looks like a tree: a root invoke_agent span for the researcher agent, with child spans for a chat call to the model and an execute_tool call to a search tool, then a sibling invoke_agent span for a writer agent it handed off to, which itself nests its own chat and execute_tool spans — exactly what the hero diagram above shows.

with tracer.start_as_current_span(
    "invoke_agent", attributes={"gen_ai.agent.name": "researcher"}
) as agent_span:
    with tracer.start_as_current_span(
        "chat", attributes={"gen_ai.request.model": "claude-opus-4"}
    ):
        response = model.chat(messages)

    with tracer.start_as_current_span(
        "execute_tool", attributes={"gen_ai.tool.name": "web_search"}
    ):
        results = web_search(query)

One caveat worth flagging: TrueFoundry's writeup notes the spec is still marked "Development" status as of mid/late 2026 — not yet frozen or stable — so attribute names can still shift between releases. Pin a version and expect to update instrumentation as the spec matures, the same way you'd treat any pre-1.0 standard. MLflow's docs on the convention are a good place to track what's actually stabilized.

Langfuse vs LangSmith vs Arize Phoenix vs MLflow

All four read and/or emit OpenTelemetry-shaped traces, but they differ on license, self-hosting difficulty, and what they're actually strongest at:

Per MLflow's own comparison and Arize's evaluation-platform breakdown, the practical default is: Langfuse if you want an OTel-native, self-hostable option with a dedicated multi-agent "Agent Graph" view; LangSmith only if you're already committed to LangChain/LangGraph and fine with a closed-source, cloud-first tool; Arize Phoenix when RAG and retrieval-quality metrics (faithfulness, relevance) matter as much as tracing; and MLflow if you already run it for experiment tracking and want tracing in the same place — it's exported native GenAI semconv support since MLflow 3.11. A fifth option, Braintrust, is worth a mention for eval-driven workflows but has no self-hosting path, per ZenML's Langfuse vs Phoenix comparison.

If you're weighing Langfuse against Phoenix specifically, Arize's own FAQ on the two is a reasonably balanced starting point even though it's written by one of the vendors.

Trajectory evals vs output-only evals

Tracing tells you what happened; evaluation tells you whether it was good. The mistake teams make is scoring only the final output — does the answer look right? — because that check misses an entire class of agent failures. Per Arize's research on agent evaluation, trajectory evaluation — scoring the sequence of tool calls and decisions the agent took, not just the end state — catches failures an output-only check can't:

In practice this means scoring traces, not just outputs: given the full span tree for a request, check whether the tool call sequence matches what a correct run should look like, in addition to whether the final text response reads fine. If you're building retrieval into the agent, the same principle applies — faithfulness and context-precision metrics score the retrieval step, not just whatever the model wrote afterward.

How do you observe a multi-agent system specifically?

Once you're running multiple agents coordinating on one task, one rule matters more than any tool choice: Honeycomb's instrumentation guide specifies that each agent should carry its own gen_ai.agent.name attribute, and the calling agent — not the callee — is responsible for emitting the invoke_agent span for that handoff.

Get that backwards and your tracing UI shows two unconnected traces instead of one agent handing off to another — which defeats the purpose of tracing a multi-agent run in the first place. This is also where a tool's dedicated "agent graph" view (Langfuse has one) earns its keep: it renders the handoff chain visually instead of making you reconstruct it from a flat span list.

How to get started with agent observability

This also matters for the operational side of agents more broadly — see our MLOps best practices for how tracing fits into the rest of the production lifecycle, and our piece on AI agent memory if context loss across sessions (one of the trajectory-eval failure modes above) is what's biting you. If your agent runs untrusted code or tools, pair observability with proper sandboxing — tracing tells you what happened, sandboxing limits the damage when it's something you didn't want.

Frequently Asked Questions

References

• • OpenTelemetry — GenAI observability blog post
• • Elastic Observability Labs — AI agent observability with OpenTelemetry
• • TrueFoundry — OpenTelemetry GenAI semantic conventions
• • Honeycomb — instrumenting AI agents: the agent timeline, an OpenTelemetry guide
• • MLflow docs — GenAI semantic conventions
• • MLflow — top 5 agent observability tools
• • Arize — LLM and agent evaluation platforms
• • Python Databench — LLM observability: Langfuse, LangSmith, Arize Phoenix (2026)
• • ZenML — Langfuse vs Phoenix
• • Arize Phoenix FAQ — Phoenix vs Langfuse key differences