Documentation

Troubleshooting

Find your symptom, then the cause and the fix. Almost every case is a variable that never reached the agent.

Find your symptom

What you seeMost likely causeFix
Nothing at all, agent running locallyExporter package missing, endpoint without /v1/traces, malformed key header, variables in .env instead of exported, or adk runNothing arrives from a local agent
Nothing on the very first run of a new accountThe first export sets up your workspace and can miss the exporter's windowSend a second message
Works locally, nothing once deployedThe variables never reached the deployed runtimeNothing arrives from a deployed agent
Agent Engine, nothing arrivesADK older than 2.11Agent Engine on ADK older than 2.11
Every model call fails with a 404 after a Cloud Run deployThe deploy set GOOGLE_CLOUD_LOCATION to your region; the model only exists on globalModel calls 404 after deploying
Calls arrive but cost is zeroThe model is not in your price catalogueCosts show as zero
Prompts and responses showed locally, missing once deployedadk deploy turns span content capture offPrompt and response text is missing
Logs rows read “content hidden”Log content capture is off (ADK's default)Prompt and response text is missing
By user shows one unnamed row; Unique users is zeroNo logs endpoint, or log content capture offNo per-user breakdown
One agent shows up as twoThe agent name changed between runsOne agent appears twice
An eval run fails on every casePrivate or local URL, or an agent that requires Google sign-inEval runs fail
An HTTP error codeKey, subscription, membership or rate limitError codes

Nothing arrives from a local agent

Work through these in order. The first is the most common by far.

1. The OTLP exporter package is missing

google-adk does not bundle opentelemetry-exporter-otlp-proto-http, and without it the export fails silently. Your agent works perfectly and sends nothing. Install it in the same Python environment the agent runs in:

Terminal
pip install opentelemetry-exporter-otlp-proto-http

2. The endpoint URL is missing its path

It must end in /v1/traces. Pointing at the host alone silently sends nowhere useful.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://api.traptureiq.com/v1/traces"

3. The API key header is malformed

OTEL_EXPORTER_OTLP_HEADERS takes key=value pairs, comma separated, no Bearer, no colon, no quotes around the value.

OTEL_EXPORTER_OTLP_HEADERS="X-API-Key=tiq_your_key_here"

4. The variables are in a .env file instead of exported

adk web reads the OpenTelemetry variables once, at startup, from the terminal. The agent's .env is loaded later, on its first request, which is too late: nothing is exported. Run export in the terminal you start adk web from, and again in any new terminal. Check with:

Terminal
env | grep OTEL_

5. The agent was started with adk run

adk run never initialises OpenTelemetry, in any ADK version. The agent answers normally and nothing is exported. Use adk web or adk api_server.

6. It has not been long enough

Spans leave on the exporter's batch schedule, not the moment the agent replies. Allow 10–30 seconds. The very first run on a new account also sets up your workspace and can miss that window; send a second message.

Nothing arrives from a deployed agent

The deploy succeeded and the agent answers, so the variables almost certainly never reached the runtime. Each target takes them differently; the full commands are on Connect your agent.

TargetWhat goes wrongFix
Cloud RunThe variables are in the agent's .env. adk deploy cloud_run does not turn it into environment variables.--env KEY=VALUE on ADK 2.9 and newer; -- --set-env-vars after the agent folder on 2.8 and older
Cloud Run, ADK 2.9+The deploy fails when you use the old -- --set-env-vars formUse --env. ADK already passes --update-env-vars, and gcloud accepts only one of the two.
Cloud RunA value containing a comma (two headers, several resource attributes) arrives splitKeep it to one --env entry, or set it afterwards with gcloud run services update --update-env-vars
GKEThe generated Deployment has no env: block, and adk deploy gke has no env flagkubectl set env deployment/<service> ... after deploying
Agent EngineADK older than 2.11Upgrade and redeploy

On Cloud Run, check what the running service actually has:

Terminal
gcloud run services describe my-agent --region=$LOCATION_ID \
  --format='yaml(spec.template.spec.containers[0].env)'

Model calls 404 after deploying

adk deploy sets GOOGLE_CLOUD_LOCATION to your --region. Models served only on the global endpoint (for example gemini-3.5-flash) then fail every call with a 404. Set it back: --env GOOGLE_CLOUD_LOCATION=global on Cloud Run (2.9+), GOOGLE_CLOUD_LOCATION=global in the agent's .env on Agent Engine.

Costs show as zero

The model is not in your price catalogue. Open Observability → Pricing and check the exact model name from the run in Traces. A new model release usually needs adding once. The call is stored with priced=false rather than a guessed cost, and later calls price correctly once the model is added.

Prompt and response text is missing

Token counts and cost never need this. Prompt and response bodies are two separate opt-ins, one for traces and one for logs:

Where the text is missingSet
Observability → Traces, after deployingADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=true. Locally it defaults on, but every adk deploy writes false when it is not set.
Observability → Logs (“content hidden”)OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true, plus OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
Use true, not SPAN_AND_EVENTSome OpenTelemetry guides suggest SPAN_AND_EVENT. On ADK 2.1 and 2.2 that value is not recognised for these log records, so the text stays hidden and the user id is dropped as well. true works on every ADK version.

Both store your users' prompts and the model's replies in TraptureIQ. Leave them off where conversations carry personal or regulated data.

No per-user breakdown

From ADK 2.1.0, the user id travels only on log records, and only when log content capture is on. So per-user cost needs all three: a user id set on the ADK session, the logs endpoint, and content capture.

Terminal
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="https://api.traptureiq.com/v1/logs"
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

ADK has no setting that sends the user without the message text. On ADK 2.0 and older the user id is on the spans and needs neither.

One agent appears twice

An agent is identified by its name, and a new name is a new agent. A typo, or a different OTEL_SERVICE_NAME between local and deployed runs, splits one agent in two. Keep the name stable and use OTEL_RESOURCE_ATTRIBUTES=deployment.environment=... to tell local from production instead.

Eval runs fail

  • “resolves to a private, loopback or link-local address”. Eval calls your agent from TraptureIQ's servers, so localhost and private addresses cannot work. Use a publicly reachable URL.
  • 403 from a Cloud Run agent. An ADK target sends no Google identity token, so a service deployed without public access refuses it. See Eval for what a target can be.
  • “ADK session create failed”. Check the App name: it must be the agent's folder name, the one adk web lists, not the service name.

Error codes

What you seeWhat it means
401Key is wrong, revoked, or the header never arrived.
402Subscription lapsed. Sign in and reactivate under Account → Billing. Billing stays reachable.
403The account is disabled, or you asked for a tenant you are not a member of.
429Rate limited. Back off and retry.
Quote the request idEvery response carries an X-Request-ID header, and server-side errors are recorded against it. Sending us that id turns “it broke yesterday” into an exact record.

Agent Engine on ADK older than 2.11

If your agent runs on Agent Engine, everything is configured correctly, and still nothing arrives. Check your ADK version:

Terminal
adk --version

Agent Engine is supported on ADK 2.11.0 and newer, the version it is verified on end to end. The fix is to upgrade: , then redeploy. Nothing else is needed.

On 2.2 to 2.10 the deploy already reads the OTLP variables, so an old version is not by itself the reason nothing arrives; those versions are simply not verified or supported on Agent Engine. Upgrade first, then work through Nothing arrives from a deployed agent if it still fails.

On 2.1.x and older the problem is certain: adk deploy agent_engine produced a different kind of app that never read the OTLP variables, so they were set on the engine and silently ignored. Only if you are pinned to one of those versions and cannot upgrade, install the exporter from the agent package's existing __init__.py. agent.py is still untouched, and the call must come before from . import agent:

import os


def _install_traptureiq_exporter() -> None:
    endpoint = os.environ.get("TIQ_TRACES_ENDPOINT")
    api_key = os.environ.get("TIQ_API_KEY")
    if not (endpoint and api_key):
        return
    try:
        from opentelemetry import trace
        from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
        from opentelemetry.sdk.resources import Resource
        from opentelemetry.sdk.trace import TracerProvider
        from opentelemetry.sdk.trace.export import BatchSpanProcessor

        provider = trace.get_tracer_provider()
        if not isinstance(provider, TracerProvider):
            provider = TracerProvider(
                resource=Resource.create({"service.name": os.environ.get("TIQ_AGENT_NAME", "my-agent")})
            )
            trace.set_tracer_provider(provider)

        provider.add_span_processor(
            BatchSpanProcessor(
                OTLPSpanExporter(endpoint=endpoint, headers={"x-api-key": api_key})
            )
        )
    except Exception as exc:   # never break the agent because telemetry failed
        print("[traptureiq] exporter setup failed:", exc)


_install_traptureiq_exporter()

from . import agent  # noqa: E402,F401   <- must stay last
Only on 2.1.x and older. Remove it once you upgradeOn ADK 2.2.0 and newer this snippet is not just unnecessary, it is risky. ADK will not replace a TracerProvider that something else already installed, so if this runs first, ADK's own telemetry setup becomes a no-op and your Cloud Trace export stops, with no error anywhere. Upgrade and delete it.

Still stuck

Check https://api.traptureiq.com/v1/health, then use the Contact us button in the corner of any console page, which captures the page you were on, or raise a ticket under Account → Support. You can also email traptureiq@techtrapture.com.