TraptureIQ Guides Observability
Your agent didn’t crash. It quietly did the wrong thing.
Traditional monitoring answers “is it up”. An agent is always up. It answers confidently, calls the wrong tool, hands off to the wrong specialist, and returns something plausible, with a 200 status and no error anywhere. Debugging that means looking at a different set of layers than the ones your APM already watches.
The failure mode that costs the most time is not the exception. It is the run where every span is green, the latency is normal, the status is 200, and the answer was wrong. Nothing in a conventional monitoring stack has an opinion about that, because nothing in it is looking at the layer where the mistake was made.
Four layers, four different questions
An ADK trace is not flat. It is a nest of spans, and each level can only answer certain questions. Ask the wrong level and you get a confident, useless answer.
- invocation: the whole turn. Useful for latency and success rate, and nothing else: this span carries zero attributes of its own. Every trace-level figure has to be rolled up from its children.
- agent_run: which agent handled the turn and who it handed to. This is where routing bugs live, and they are invisible one layer up.
- call_llm: model, tokens, cost. Also where a retry hides: two calls look identical whether the agent asked twice on purpose or the first attempt failed.
- execute_tool: did the tool work, and how slow was it. Usually the real critical path in an agent that feels sluggish.
The error count almost everyone gets wrong
This one is worth stating plainly because it produces confident, wrong dashboards. When a tool raises, the exception propagates upward through spans that are themselves perfectly healthy. The tool span is marked errored, and so are its ancestors.
One broken tool, three error statuses
Count spans with an error status and you report three problems. Try to be clever and exclude "parents of an error" and you still get it wrong, because the exception passes through healthy call_llm and generate_content spans on the way up. Our first implementation reported two error groups for one broken tool.
Correct attribution requires walking the full descendant tree to find where the failure originated, then counting that once. The difference between "three intermittent errors" and "one broken tool" is the difference between a week of investigation and an afternoon.
Why the user is missing from your traces
A question every team asks in week two is "which users are hitting the errors", and the answer comes back empty, permanently.
The cause is structural. From ADK 2.1.0, user.id appears only on log records, never on spans, and only when message content capture is switched on. With ADK's defaults it is sent nowhere at all, so no query can produce a per-user view. You need the logs endpoint, the content-capture setting, and a backfill that works in both directions, because traces and logs flush on independent schedules: the log carrying the user id may land after the trace it belongs to.
# the last two lines are the ones people skip, and they are why # "cost by user" and "errors by user" come back empty on ADK 2.1.0+ export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://api.traptureiq.com/v1/traces" export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=tiq_..." export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="https://api.traptureiq.com/v1/logs" export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
The trade-off is real: ADK has no setting that sends the user without the message text, so that last line also ships prompts and responses in the logs. On ADK 2.0 and older the user id was on the spans and none of this applies.
The journey is the unit that matters
A single trace tells you what happened in one turn. It cannot tell you that the user asked the same question three different ways, got a defensible answer each time, and left without what they came for. Every individual turn passes review. The session fails.
That is why the session-level journey (every turn in order, with hand-offs, tool calls and errors on one timeline) is where agent debugging actually happens. It is also the view that makes hand-off bugs obvious: a specialist agent that keeps returning control to the router without doing anything looks fine per-trace and absurd in a journey.
What ADK does not tell you, and what can be derived
Being precise about the boundary matters more than a longer feature list. Two examples worth knowing before you plan around them:
- Memory and artifacts emit no telemetry at all. ADK produces zero spans and zero events for either. Anything you see reported about memory reads or artifact writes is derived from adjacent signals: legitimate, useful, and not the same thing as a measurement. Treat any tool that claims direct memory telemetry with suspicion.
- Local versus deployed cannot be detected. It has to be declared with
OTEL_RESOURCE_ATTRIBUTES. Otherwise your laptop runs and your production traffic are the same population.
These are not readings of the documentation. We captured raw OTLP protobuf from a live ADK agent (plain turn, tool call, deliberately failing tool, sub-agent transfer) and decoded it. Three findings from that capture each broke a first-cut implementation that had looked correct in review.
Where the trace stops and the graph starts
Traces answer "what happened in this run". They do not answer "how is this system put together, and is it running the way it was designed", which is a question about structure rather than events. That is a different derivation, covered in agent topology, and it is where a ParallelAgent that quietly runs serially finally becomes visible.
Questions people actually ask
How do I trace a Google ADK agent?
ADK emits OpenTelemetry spans natively, so setting OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS sends full traces to any OTLP collector with no code change. Add OTEL_EXPORTER_OTLP_LOGS_ENDPOINT for the log stream, which carries the user id that spans do not.
Why does one broken tool show up as several errors?
A tool exception propagates upward through healthy parent spans, so a single failure marks the tool span and its ancestors as errored. Counting error statuses directly reports two or three problems where there is one. Correct attribution needs a full descendant walk to find the deepest originating span.
Why does my ADK observability show zero users?
From ADK 2.1.0 the user id appears only on log records, never on spans, and only when message content capture is on. With default settings per-user views are permanently empty. Configure the logs endpoint, set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true (which also sends prompt and response text), and backfill across both signals, since traces and logs flush independently.
Does adk run produce telemetry?
No. adk run does not initialise OpenTelemetry, so it exports nothing regardless of environment variables. Use adk web or adk api_server when you want telemetry from a local session.
What is a user journey in agent observability?
A session-level view of every turn a user took, including agent hand-offs, tool calls and errors, in order. It is the level at which you can see that an agent answered each turn correctly and still failed the user overall.