Troubleshooting
Find your symptom, then the cause and the fix. Almost every case is a variable that never reached the agent.
Find your symptom
| What you see | Most likely cause | Fix |
|---|---|---|
| Nothing at all, agent running locally | Exporter package missing, endpoint without /v1/traces, malformed key header, variables in .env instead of exported, or adk run | Nothing arrives from a local agent |
| Nothing on the very first run of a new account | The first export sets up your workspace and can miss the exporter's window | Send a second message |
| Works locally, nothing once deployed | The variables never reached the deployed runtime | Nothing arrives from a deployed agent |
| Agent Engine, nothing arrives | ADK older than 2.11 | Agent Engine on ADK older than 2.11 |
| Every model call fails with a 404 after a Cloud Run deploy | The deploy set GOOGLE_CLOUD_LOCATION to your region; the model only exists on global | Model calls 404 after deploying |
| Calls arrive but cost is zero | The model is not in your price catalogue | Costs show as zero |
| Prompts and responses showed locally, missing once deployed | adk deploy turns span content capture off | Prompt and response text is missing |
| Logs rows read “content hidden” | Log content capture is off (ADK's default) | Prompt and response text is missing |
| By user shows one unnamed row; Unique users is zero | No logs endpoint, or log content capture off | No per-user breakdown |
| One agent shows up as two | The agent name changed between runs | One agent appears twice |
| An eval run fails on every case | Private or local URL, or an agent that requires Google sign-in | Eval runs fail |
| An HTTP error code | Key, subscription, membership or rate limit | Error codes |
Nothing arrives from a local agent
Work through these in order. The first is the most common by far.
1. The OTLP exporter package is missing
google-adk does not bundle opentelemetry-exporter-otlp-proto-http, and without it the export fails silently. Your agent works perfectly and sends nothing. Install it in the same Python environment the agent runs in:
pip install opentelemetry-exporter-otlp-proto-http
2. The endpoint URL is missing its path
It must end in /v1/traces. Pointing at the host alone silently sends nowhere useful.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://api.traptureiq.com/v1/traces"
# no path: the exporter posts to the host root OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://api.traptureiq.com"
3. The API key header is malformed
OTEL_EXPORTER_OTLP_HEADERS takes key=value pairs, comma separated, no Bearer, no colon, no quotes around the value.
OTEL_EXPORTER_OTLP_HEADERS="X-API-Key=tiq_your_key_here"
# colon instead of = OTEL_EXPORTER_OTLP_HEADERS="X-API-Key: tiq_your_key_here" # Bearer scheme: TraptureIQ reads X-API-Key, not Authorization OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer tiq_..."
4. The variables are in a .env file instead of exported
adk web reads the OpenTelemetry variables once, at startup, from the terminal. The agent's .env is loaded later, on its first request, which is too late: nothing is exported. Run export in the terminal you start adk web from, and again in any new terminal. Check with:
env | grep OTEL_
5. The agent was started with adk run
adk run never initialises OpenTelemetry, in any ADK version. The agent answers normally and nothing is exported. Use adk web or adk api_server.
6. It has not been long enough
Spans leave on the exporter's batch schedule, not the moment the agent replies. Allow 10–30 seconds. The very first run on a new account also sets up your workspace and can miss that window; send a second message.
Nothing arrives from a deployed agent
The deploy succeeded and the agent answers, so the variables almost certainly never reached the runtime. Each target takes them differently; the full commands are on Connect your agent.
| Target | What goes wrong | Fix |
|---|---|---|
| Cloud Run | The variables are in the agent's .env. adk deploy cloud_run does not turn it into environment variables. | --env KEY=VALUE on ADK 2.9 and newer; -- --set-env-vars after the agent folder on 2.8 and older |
| Cloud Run, ADK 2.9+ | The deploy fails when you use the old -- --set-env-vars form | Use --env. ADK already passes --update-env-vars, and gcloud accepts only one of the two. |
| Cloud Run | A value containing a comma (two headers, several resource attributes) arrives split | Keep it to one --env entry, or set it afterwards with gcloud run services update --update-env-vars |
| GKE | The generated Deployment has no env: block, and adk deploy gke has no env flag | kubectl set env deployment/<service> ... after deploying |
| Agent Engine | ADK older than 2.11 | Upgrade and redeploy |
On Cloud Run, check what the running service actually has:
gcloud run services describe my-agent --region=$LOCATION_ID \ --format='yaml(spec.template.spec.containers[0].env)'
Model calls 404 after deploying
adk deploy sets GOOGLE_CLOUD_LOCATION to your --region. Models served only on the global endpoint (for example gemini-3.5-flash) then fail every call with a 404. Set it back: --env GOOGLE_CLOUD_LOCATION=global on Cloud Run (2.9+), GOOGLE_CLOUD_LOCATION=global in the agent's .env on Agent Engine.
Costs show as zero
The model is not in your price catalogue. Open Observability → Pricing and check the exact model name from the run in Traces. A new model release usually needs adding once. The call is stored with priced=false rather than a guessed cost, and later calls price correctly once the model is added.
Prompt and response text is missing
Token counts and cost never need this. Prompt and response bodies are two separate opt-ins, one for traces and one for logs:
| Where the text is missing | Set |
|---|---|
| Observability → Traces, after deploying | ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=true. Locally it defaults on, but every adk deploy writes false when it is not set. |
| Observability → Logs (“content hidden”) | OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true, plus OTEL_EXPORTER_OTLP_LOGS_ENDPOINT |
SPAN_AND_EVENT. On ADK 2.1 and 2.2 that value is not recognised for these log records, so the text stays hidden and the user id is dropped as well. true works on every ADK version.Both store your users' prompts and the model's replies in TraptureIQ. Leave them off where conversations carry personal or regulated data.
No per-user breakdown
From ADK 2.1.0, the user id travels only on log records, and only when log content capture is on. So per-user cost needs all three: a user id set on the ADK session, the logs endpoint, and content capture.
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="https://api.traptureiq.com/v1/logs" export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
ADK has no setting that sends the user without the message text. On ADK 2.0 and older the user id is on the spans and needs neither.
One agent appears twice
An agent is identified by its name, and a new name is a new agent. A typo, or a different OTEL_SERVICE_NAME between local and deployed runs, splits one agent in two. Keep the name stable and use OTEL_RESOURCE_ATTRIBUTES=deployment.environment=... to tell local from production instead.
Eval runs fail
- “resolves to a private, loopback or link-local address”. Eval calls your agent from TraptureIQ's servers, so
localhostand private addresses cannot work. Use a publicly reachable URL. - 403 from a Cloud Run agent. An ADK target sends no Google identity token, so a service deployed without public access refuses it. See Eval for what a target can be.
- “ADK session create failed”. Check the App name: it must be the agent's folder name, the one
adk weblists, not the service name.
Error codes
| What you see | What it means |
|---|---|
401 | Key is wrong, revoked, or the header never arrived. |
402 | Subscription lapsed. Sign in and reactivate under Account → Billing. Billing stays reachable. |
403 | The account is disabled, or you asked for a tenant you are not a member of. |
429 | Rate limited. Back off and retry. |
X-Request-ID header, and server-side errors are recorded against it. Sending us that id turns “it broke yesterday” into an exact record.Agent Engine on ADK older than 2.11
If your agent runs on Agent Engine, everything is configured correctly, and still nothing arrives. Check your ADK version:
adk --version
Agent Engine is supported on ADK 2.11.0 and newer, the version it is verified on end to end. The fix is to upgrade: , then redeploy. Nothing else is needed.
On 2.2 to 2.10 the deploy already reads the OTLP variables, so an old version is not by itself the reason nothing arrives; those versions are simply not verified or supported on Agent Engine. Upgrade first, then work through Nothing arrives from a deployed agent if it still fails.
On 2.1.x and older the problem is certain: adk deploy agent_engine produced a different kind of app that never read the OTLP variables, so they were set on the engine and silently ignored. Only if you are pinned to one of those versions and cannot upgrade, install the exporter from the agent package's existing __init__.py. agent.py is still untouched, and the call must come before from . import agent:
import os def _install_traptureiq_exporter() -> None: endpoint = os.environ.get("TIQ_TRACES_ENDPOINT") api_key = os.environ.get("TIQ_API_KEY") if not (endpoint and api_key): return try: from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor provider = trace.get_tracer_provider() if not isinstance(provider, TracerProvider): provider = TracerProvider( resource=Resource.create({"service.name": os.environ.get("TIQ_AGENT_NAME", "my-agent")}) ) trace.set_tracer_provider(provider) provider.add_span_processor( BatchSpanProcessor( OTLPSpanExporter(endpoint=endpoint, headers={"x-api-key": api_key}) ) ) except Exception as exc: # never break the agent because telemetry failed print("[traptureiq] exporter setup failed:", exc) _install_traptureiq_exporter() from . import agent # noqa: E402,F401 <- must stay last
# my_agent/.env: deliberately NOT the OTEL_ names, which this ADK ignores. # Using distinct names also means that if you later upgrade past 2.2.0, ADK and # this file cannot both install an exporter and double-count every call. GOOGLE_GENAI_USE_VERTEXAI=1 TIQ_TRACES_ENDPOINT=https://api.traptureiq.com/v1/traces TIQ_API_KEY=tiq_your_key_here TIQ_AGENT_NAME=my-agent
Still stuck
Check https://api.traptureiq.com/v1/health, then use the Contact us button in the corner of any console page, which captures the page you were on, or raise a ticket under Account → Support. You can also email traptureiq@techtrapture.com.