跳到主要内容

First-Party OpenTelemetry Tracing: Two Paths, One Trace

高级

For most of 2026, tracing a Claude API call meant writing the span yourself: open a span, call messages.create, copy usage onto attributes, close the span. Agent observability is the page about doing that well — and it is still the page about what the names mean.

This page is about the part that changed. The API and the SDK now emit those spans themselves, in beta, and they emit considerably more than you were writing by hand: a span per model request inside one API request, a span per server tool the API ran for you, a span per compaction, a span per Managed Agents turn. Things that happen on Anthropic's side of the wire, which hand-instrumentation structurally cannot see.

There are two independent paths to turn this on, and the most expensive thing to misunderstand is that they overlap. Enable both without reading one paragraph of the docs and every Messages request shows up twice, with the same span name, and your token dashboards double.

What you'll learn
  • Choose between SDK tracing, API tracing, or both — and know exactly what each one can and cannot see
  • Set up each path: a tracer provider in your code, a destination in the Console or via the Admin API
  • Sum tokens correctly across overlapping spans — the one rule that stops double counting
  • Pick a content mode deliberately, and know what overrides your choice (HIPAA/FERPA/K-12, zero data retention)
  • Join API spans into your application's traces with traceparent — and avoid the sampled flag that deletes them
  • Survive the delivery rules: at-least-once, best-effort, and automatic disabling after 24 hours of failures

The two paths​

They are not two ways to do the same thing. They see different halves of the run.

What you getSDK tracingAPI tracing
Messages API and Sessions API requestsyesyes
Token usageyesyes
Each model request inside one API request—yes
Managed Agents turns—yes
Server tools and MCP connector tools, run by the API—yes
Your own client tools, run by the tool runneryes—
Tools in a self-hosted sandboxyes—
Other requests, such as file uploadsyes—
How you set it upcodeConsole or Admin API
Who can set it upany developeran admin
Which clients it coversonly clients using the SDKany client, including curl

The split follows one line: the SDK sees what your process does, the API sees what Anthropic's side does. A server tool like web search runs inside the API request — your process never observes it, so only API tracing produces that span. Your own get_weather function runs in your process — only SDK tracing produces that one.

Which is why "both" is the real answer for most production setups, and why the overlap rule below is not an edge case.

Read that shape carefully, because it contains the trap. anthropic.messages.create appears four times for a two-request conversation: twice from the SDK (kind CLIENT) and twice from the API (kind SERVER), with each API span parented to its SDK span. They are not duplicates in the sense of a redelivery — they are two legitimate observations of the same request, from both ends.

Set up SDK tracing​

SDK spans need no permissions and no Console access. The SDK uses whatever tracer provider your application already registered, so there is no Anthropic-specific exporter to wire up.

Guided walkthrough1 of 4
  1. Spans ship in the current Python, TypeScript, C#, Go, Java, PHP and Ruby SDKs. On Java the minimum is `anthropic-java` 2.73.0. Pin the version — the beta note says a new SDK release may rename a span, an attribute, an option or an environment variable.

Minimum viable SDK tracing — console exporter first (Python)

from anthropic import Anthropic
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

# Register BEFORE the first traced call. Swap ConsoleSpanExporter for an
# OTLP exporter once you can see spans in the terminal.
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)

# Record prompts and responses, and label this conversation.
client = Anthropic(open_telemetry={"traces": {"content_mode": "content"}})

tracer = trace.get_tracer("my-service")
with tracer.start_as_current_span("answer_question"):
  message = client.with_options(
      open_telemetry={**client.open_telemetry, "conversation_id": "conv_42"}
  ).messages.create(
      model="claude-opus-5-5",
      max_tokens=1024,
      messages=[{"role": "user", "content": "Hello, Claude"}],
  )

The SDK options worth knowing​

Option (Python spelling)Environment variableDefault
traces.content_modeANTHROPIC_OPEN_TELEMETRY_TRACES_CONTENT_MODEmetadata_only
traces.max_content_bytesANTHROPIC_OPEN_TELEMETRY_TRACES_MAX_CONTENT_BYTES0.75 MB
conversation_id—none
tracer_provider—the registered global provider
propagationANTHROPIC_OPEN_TELEMETRY_PROPAGATIONon
open_telemetry (off switch)ANTHROPIC_OPEN_TELEMETRYon

Two behaviors here will cost you an hour each if you meet them cold:

  • Passing open_telemetry to the client overrides the environment variables as a whole, nested keys included. Any key you leave out of the object silently reverts to its table default even if its variable is set. Build the object from the existing one rather than from scratch.
  • On TypeScript with @opentelemetry/instrumentation-undici — which @opentelemetry/auto-instrumentations-node pulls in — set propagation: false. A bug in that package adds a second traceparent header to each request, which makes the trace context invalid and breaks the join silently.

Set up API tracing​

API tracing needs no code change at all. You register a destination — your observability platform's OTLP endpoint plus credentials — and the API posts spans to it.

What you need before you start:

  • The admin role in the Console organization. The developer role can view destinations but not create or change them.
  • An OTLP/HTTP endpoint accepting protobuf, over HTTPS, on port 443. This is the requirement that bites self-hosters: a standard OpenTelemetry Collector listens for OTLP/HTTP on port 4318, so you need a load balancer, reverse proxy, or a reconfigured receiver in front of it. There is no gRPC option and no non-443 port.
  • Credentials for that endpoint, which go in request headers. The Console will not show the values again after you save.

Create a telemetry destination with the Admin API (curl)

curl --fail-with-body -sS "https://api.anthropic.com/v1/organizations/telemetry_destinations" \
-H "x-api-key: $ANTHROPIC_ADMIN_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "anthropic-beta: telemetry-destinations-2026-08-11" \
-H "content-type: application/json" \
-d '{
  "endpoint_url": "https://gateway.example.com/otlp",
  "headers": {"authorization": "Bearer YOUR_OBSERVABILITY_API_KEY"},
  "traces": {"content_mode": "metadata_only"},
  "workspaces": {"type": "all"}
}'

Note the endpoint URL shape: give the base OTLP URL and let the API append /v1/traces itself. A path prefix is allowed. If your platform receives traces somewhere else entirely, set the optional traces endpoint URL — but it must share the same host.

Three settings deserve a deliberate decision rather than a default:

  • Workspaces — all, selected, or all except. all includes workspaces created later, which is usually what you want and occasionally exactly what you don't. A destination only receives spans from the workspace where the request or session actually ran.
  • Content mode — covered below.
  • Maximum content size — only applies in full-content mode. Choosing a listed partner in the Console pre-fills a value that platform accepts; with Custom you are on your own.

Then verify, in this order: send a test span from the destination's menu and confirm an anthropic.test span arrives; send a real request in the chosen workspace; check the Delivery column reads Healthy; filter your platform on service.name = claude-platform. Allow about a minute at each step — both destination changes and the health column lag by roughly that much.

Content modes, and what overrides your choice​

Every destination and every SDK client has one of two content modes.

  • Metadata only (metadata_only) — the default. IDs, timing, status, token counts, and text you chose yourself: agent names, tool names and descriptions, MCP server names.
  • Full content (content) — everything above plus model prompts and responses. In the span reference these attributes carry a Content badge: gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.call.arguments, anthropic.tool_result.content.

Full content is what makes traces usable for the thing teams actually want — reading a failed conversation, or building evals from real traffic. It is also what moves your prompts and your users' inputs into a second system, under that vendor's retention and access controls rather than yours. The same calculus as what your agent uploads: enabling it grants your observability provider roughly the clearance your application data has.

Two overrides are worth committing to memory, because both produce confusing support tickets:

  • A HIPAA, FERPA, or K-12 data-handling restriction with Anthropic forces destinations to metadata only, whatever the destination says. SDK spans are not affected — your own process can still emit content. So the symptom is "full content is on, the Console agrees, and the prompts only appear on the CLIENT spans."
  • Zero data retention is compatible with full content. For Messages API requests the API holds exported content only in memory until delivery. Managed Agents is not ZDR-eligible in the first place, and its retention is unchanged by using OpenTelemetry.

Content limits and truncation​

Each span holds up to 0.75 MB of content by default — chosen to fit a 200k-token context window of English text while staying under the 1 MB request limit common to observability platforms and proxies. A full 1M-token window needs roughly 4 MB.

When content exceeds the limit, the span is shortened and anthropic.content.truncated is set to true. The drop order is specified, and it matters for anyone building evals from traces:

  • Request spans drop the system prompt first, then the earliest messages.
  • Tool spans drop blocks from the end of the result.
  • Images and documents appear only as their type — never as bytes.

So a naive "replay every trace as an eval case" pipeline will quietly feed you system-prompt-less conversations unless you filter on anthropic.content.truncated. Make that filter the first line of the pipeline, not a later discovery.

The one rule: count each token once​

This is the section to read twice. Token usage attributes appear on several spans per request, by design, because each span reports its own step. Summing everything you receive overcounts; summing the wrong level undercounts.

Guided walkthrough1 of 5
  1. If a Messages request has both an SDK span and an API span, sum the API spans. Both are named `anthropic.messages.create`, so you cannot filter by name — tell them apart by span kind: the SDK span is CLIENT, its child API span is SERVER. Filtering on `service.name` = `claude-platform` works too.

One more thing that is not a counting rule but will save you from a bad number: gen_ai.usage.input_tokens already includes cache reads and cache writes. Adding gen_ai.usage.cache_read.input_tokens to it double-counts the cheapest tokens you have. See prompt caching for what those tokens cost relative to each other.

Traces are not a billing record

Anthropic states plainly that exported spans are not guaranteed to be a complete record of usage. Delivery is best effort, so a span can be lost; and a caller can suppress a request's spans entirely by turning off the sampled flag in its traceparent. For per-key accounting — spotting a leaked key, reconciling an invoice — use the Usage API, which counts requests that produced no spans at all. Traces are for debugging and for cost attribution; they are not the ledger.

Joining API spans into your own traces​

By default, API spans live in their own traces — technically correct and nearly useless, because you cannot tell which feature or user caused the call.

The API reads the W3C traceparent header from each request. With the SDK, you get this for free as long as you called the SDK inside your own span and registered the W3C trace context propagator. (If your application standardized on B3 or another propagator, add W3C alongside it — the SDK writes headers with your propagator.) With any other client, inject the active span's context into the headers yourself before posting.

Then there is the flag that surprises everyone:

The sampled flag is an on/off switch for spans, not a hint

If the sampled flag in the incoming traceparent is off, the API emits no span for that request and none of the spans under it. Head sampling in your application therefore decides whether Anthropic-side telemetry exists at all. A 1%-sampling service that enables API tracing gets 1% of model-request and server-tool spans — and will conclude the feature is broken. Either sample at the tail in your collector, or force the sampled flag on for the paths you need traced.

The propagation goes one hop further than most people expect. When the Messages API calls your MCP server through the MCP connector, it passes a W3C traceparent in params._meta, so a server that reads it can attach its own spans under the API's. In a tools/call request that parent is the anthropic.mcp_tool_use span; in other messages it is anthropic.messages.create. An untraced request sends no traceparent at all. On the client side the picture is patchier: the Python mcp package from version 2.0 and the C# ModelContextProtocol package forward trace context in _meta; the TypeScript, Go, Java, Ruby, and PHP MCP SDKs do not do it on their own.

Delivery rules, and how a destination dies​

API-side export is a push from Anthropic's infrastructure to your endpoint, which means it has its own failure semantics — closer to a webhook than to an OTLP exporter in your process.

PropertyBehavior
DuplicationAt least once. Dedupe by trace ID + span ID.
LossBest effort. A lost span is never resent.
CompressionEach delivery's first request is gzipped with Content-Encoding: gzip. On a 400 or 415, the same spans are resent immediately, uncompressed.
RetriesLimited retries after a timeout, a connection/TLS/DNS failure, or a 408, 429, or 5xx. Other failures are not retried.
TimeoutsTLS handshake within 5 s, response headers within 5 s, 10 s total per delivery — including the uncompressed resend.
User agentUser-Agent: Claude-Platform. Useful for an allowlist rule.

And three ways the API will disable a destination outright, stopping export until a human re-enables it:

  1. The endpoint returns a 3xx redirect. Redirects are not followed — point the destination at the final URL.
  2. The endpoint's hostname resolves to a non-public IP address. There is no tunnel to your VPC; expose a public collector or use a managed platform.
  3. Every delivery fails for 24 hours.

That third one is the operational hazard. A rotated observability API key that nobody notices becomes a destination that is Failing for a day and then Disabled — and re-enabling is a manual click in the Console (with the workspace filter set to All), not an automatic recovery. If you alert on one thing from this page, alert on the Delivery state.

Limits per organization​

LimitValue
Destinations, including disabled ones10
Workspaces selected for one destination1,000
Request headers per destination20
Header name length128 bytes
Header value length4,096 bytes
All header names and values together16,384 bytes
Endpoint URL length2,048 characters

Reading a Managed Agents session​

Sessions break the one-trace assumption, and the fix is a grouping key rather than a trace ID.

A session's spans can land in several traces — a later request sent from a new root trace starts a new one — and with multiagent orchestration a session has several threads. So:

  • One thread, as a transcript — group by gen_ai.conversation.id (the session thread ID), sort by start time.
  • A whole session, all threads — group by anthropic.session.id, across traces.

The span shapes are worth internalizing because they do not match the generic GenAI conventions you may already have dashboards for. anthropic.session.turn carries gen_ai.operation.name = invoke_agent and spans one thread's running state; a turn that a request started is a child of that request's span, while a turn that starts because another thread replied has no parent at all. In full-content mode the turn span holds the thread's content: gen_ai.system_instructions on a thread's first turn only, gen_ai.input.messages, gen_ai.output.messages. Advisor threads are deliberately thinner — gen_ai.agent.name is anthropic.advisor, and the instruction and input attributes are absent.

Four cases produce no API tool span at all, and each has caused someone to think their tracing was broken: a tool your application runs (the SDK tool runner emits that span); a tool in a self-hosted sandbox (the SDK environment worker emits it, grouped under anthropic.environment.work); a multiagent or advisor tool; and a tool use denied under the auto permission policy or by a user.tool_confirmation event. Denied calls leave no tool span — if you are auditing what an agent tried to do, read the session events, not the trace.

Common mistakes​

  • Summing every anthropic.messages.create span you receive. Two legitimate observations per request, same name. Filter by span kind or service.name first. This is the mistake that produces a plausible, wrong, exactly-doubled cost number.
  • Expecting spans at your usual sampling rate. A sampled-off traceparent suppresses the API spans completely. Head sampling in your service is head sampling of Anthropic's telemetry.
  • Pointing a destination at a bare Collector. OTLP/HTTP on 4318 is not an HTTPS endpoint on 443. Put a proxy in front, or the destination never delivers.
  • Treating full content as a logging flag. It relocates prompts and user input into your observability vendor. Decide it with whoever owns data handling, and know that a HIPAA/FERPA/K-12 restriction silently overrides it for API spans only.
  • Building evals from traces without checking anthropic.content.truncated. Truncation drops the system prompt first. Your eval corpus will be missing exactly the part that determined behavior.
  • Letting a Failing destination sit. 24 hours of failures disables it, and re-enabling is manual. Alert on the Delivery state, not on span volume — span volume looks fine right up until it is zero.
  • Hard-coding these attribute names into forty panels. The spans follow the GenAI conventions at a specific commit (e57c543b, September 24, 2026) of a repository with no tagged release, and the beta note reserves the right to rename. The mapping-layer habit from agent observability applies here with extra force.

Where the spans go​

Any platform that accepts OTLP works. Anthropic documents integration guides with Arize AI, Braintrust, Databricks, Datadog, Grafana Cloud, Honeycomb, Langfuse, LangSmith, New Relic, PostHog, Sentry, and Splunk; choosing a listed provider in the Console asks for that platform's own credentials and pre-fills a content size it accepts, instead of asking for a raw endpoint and headers.

For Claude Code, the Agent SDK, and Cowork, this is a different system with different switches — CLAUDE_CODE_ENABLE_TELEMETRY and friends, covered in agent observability. Enabling destinations here does nothing for them, and vice versa.

Check yourself

0/3
  1. You enable both SDK tracing and API tracing, then sum gen_ai.usage.input_tokens over every anthropic.messages.create span. What happens?
  2. Your service head-samples 1% of traces and you just added a telemetry destination. Why are almost no model-request spans arriving?
  3. A destination shows Failing for a day because the observability API key was rotated. What happens next?
按 Enter 或空格键翻转卡片。使用左右方向键在卡片之间切换。已显示术语。
1 / 14
Key takeaways
  • Two paths, two halves of the run: the SDK sees your process, the API sees Anthropic's side. Most production setups want both.
  • Both paths emit anthropic.messages.create for the same request. Pick one source by span kind before you sum anything.
  • Within API spans, sum one level — messages.create or model_request, never both — and add compaction from exactly one place.
  • Spans are for debugging and attribution, not billing: delivery is best effort and a sampled-off traceparent erases a request entirely. Use the Usage API for the ledger.
  • The sampled flag is an on/off switch, not a hint. A 1%-sampling service gets 1% of Anthropic-side spans.
  • Full content relocates your prompts into your observability vendor. A HIPAA/FERPA/K-12 restriction overrides it for API spans only — a very confusing symptom if you do not know it.
  • Check anthropic.content.truncated before building evals from traces: truncation drops the system prompt first.
  • Alert on a destination's Delivery state. 24 hours of failures disables it, and recovery is a manual click.

Next​

  • Agent observability — what the gen_ai.* names mean, why none of them are stable, and the Claude Code telemetry switches this page does not cover
  • Cache diagnostics — once the spans show a cache read you did not expect, the per-request fields that explain why
  • Managed agents — the sessions, turns, and threads whose spans this page teaches you to group

Sources & further reading​