Intermediate

LLM Tracing & Observability

When an LLM application produces a wrong or unexpected output, you need to know exactly what prompt was sent, what the model returned, how long it took, and what tools were called in what order. Without tracing, debugging production LLM failures is guesswork. Tracing captures this context as structured, queryable data.

What to Capture

A complete LLM trace should include:

Per-request data

• Full system prompt (with version/hash)
• User message content
• Model response content
• Model name and version
• Input and output token counts
• Latency: TTFT, total duration
• Cost (tokens × price per token)

Per-session data

• Session/conversation ID
• User ID (anonymised)
• All turns in the conversation
• RAG retrieval results and scores
• Tool call names, inputs, outputs
• Any evaluation scores (LLM-as-Judge)

OpenTelemetry for LLMs

OpenTelemetry (OTel) is the open standard for distributed tracing. The OpenTelemetry Semantic Conventions for LLMs (GenAI conventions) define standard attribute names for LLM-specific data, enabling interoperability between instrumentation libraries and observability backends.

# Key OTel GenAI attributes
gen_ai.system           # "openai", "anthropic", "google"
gen_ai.request.model    # "gpt-4o", "claude-sonnet-4-6"
gen_ai.request.max_tokens
gen_ai.request.temperature
gen_ai.response.finish_reason
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens

Tracing Tools

LangSmith

The dominant tool for LangChain/LangGraph applications. Automatically traces all chain and agent runs. Provides a UI for inspecting each trace, filtering by latency or error, and running evaluations. Free tier available; paid for volume.

Best for: teams already using LangChain/LangGraph

Helicone

Proxy-based: route your OpenAI/Anthropic calls through Helicone's endpoint, zero code change required. Captures all request/response pairs with cost and latency. Also supports custom properties for filtering (user ID, feature name).

Best for: quick setup without code changes; framework-agnostic

Arize Phoenix

Open-source, self-hosted. OTel-native. Supports LLM traces, RAG retrieval evaluation, embedding drift detection, and online evals. Strong for teams that need on-premises observability.

Best for: enterprises with data residency requirements; open-source preference

Anthropic Console

The Workbench and API logs in the Anthropic Console provide basic tracing for Claude requests - useful for early debugging but lacks multi-step agent visibility and custom filtering.

Best for: simple Claude-only applications; no extra infrastructure

Sampling Strategy

Logging every token of every request at scale is expensive. Use sampling:

Always log errors - 100% of failed requests should be captured
Always log slow requests - P99 latency outliers are often bugs
Sample successful requests - 5–20% is usually enough for trend monitoring
Log all requests for new features - full visibility until the feature is stable

Tracing Checklist

Capture prompt version/hash alongside every logged request
Log token counts and cost on every request for dashboards
Add a conversation ID so multi-turn sessions can be reconstructed
Include user-facing error messages if the model produced an invalid response
Set up alerts on P95 latency and error rate before going to production

Tracing LLM Calls in Production