Intermediate

LLM Logging Best Practices

Logging for LLM applications is distinct from traditional application logging. A HTTP request log tells you what URL was hit; an LLM log needs to capture what the model was told, what it said, how much it cost, and whether the output was good - all in a structured, queryable format. Getting this right from the start saves enormous debugging time later.

What to Log

Log every LLM call as a structured record with at minimum:

{
  "timestamp": "2026-06-26T12:34:56Z",
  "request_id": "req_abc123",
  "session_id": "sess_xyz789",
  "user_id": "user_hash_789",   // hashed, not raw PII
  "feature": "support-classifier",
  "model": "claude-sonnet-4-6",
  "prompt_version": "v3",
  "input_tokens": 542,
  "output_tokens": 38,
  "latency_ms": 1240,
  "cost_usd": 0.00089,
  "finish_reason": "end_turn",
  "error": null,
  "eval_score": null            // populated later by evaluation pipeline
}

Do not log full prompt/response text in the same system as other application logs - it is large, PII-sensitive, and needs different retention rules.

Prompt and Response Logging

Full prompt and response text should be stored separately from metadata, with stricter access controls and shorter retention periods. Options:

Dedicated LLM observability tool

LangSmith, Helicone, Arize - store prompt/response with the request metadata in the same queryable interface. Built-in PII redaction in some tools.

Object storage + metadata DB

Store prompt/response blobs in S3/GCS (with encryption at rest); store metadata in Postgres or ClickHouse. Join on request_id for analysis. Full control but more infrastructure.

PII Handling

Critical: Treat all LLM inputs as potentially PII

• User messages often contain names, emails, addresses, medical info
• LLM responses may echo PII from the input
• Store in a separate system with stricter access controls
• Hash or anonymise user identifiers in metadata logs
• Implement deletion capability to comply with GDPR right-to-erasure requests
• Check whether your LLM provider uses API inputs for model training - opt out if not desired

Retention Policy

Data type	Typical retention	Rationale
Request metadata (no text)	12–24 months	Cost tracking, trend analysis, capacity planning
Full prompt/response text	30–90 days	Debugging window; PII minimisation
Eval dataset entries	Indefinite	Curated; used for regression testing - do not auto-delete

Building a Log Pipeline

Emit structured JSON logs from your application for every LLM call
Ship metadata to your existing observability stack (Datadog, Grafana, CloudWatch)
Ship prompt/response text to a separate, access-controlled store
Build a dashboard for token cost and latency from day one - not after the bill arrives
Add an async evaluation step that scores logged responses and writes scores back to the metadata record

Logging Prompts & Responses Safely