Eval
Is the agent any good, and does it hold up under load?
Eval sets
Eval → Eval sets
A list of cases: an input, and what a good answer looks like. Run the set against a live agent endpoint and each case is scored by an LLM-as-judge on the metrics you pick.
Every new workspace is seeded with three security red-team sets: jailbreak and prompt injection, data-loss prevention, and responsible AI.
Open a seeded set or create your own and add cases, then Run eval against your agent's endpoint URL. The seeded sets are ordinary eval sets. Read the cases, edit them, delete the ones that do not apply to you.
Full walkthrough below.
Prompt changes are untestable by intuition. A set you can re-run turns “this feels better” into a number, and catches the regression where fixing one behaviour quietly breaks another.
Load tests
Eval → Load testsA traffic generator against an agent endpoint, reporting latency percentiles, throughput and error rate.
Give it the endpoint, choose concurrency and duration, and start. Watch p50 and p95 rather than the average. The average hides exactly the tail that users notice.
Agents degrade under concurrency in ways single requests never reveal: rate limits, tool timeouts, connection pools. It is also the cheapest way to find out what sustained traffic would actually cost. Run one, then read Cost for the same window.
Running your first eval
Open an eval set
Eval → Eval sets, then pick one of the seeded security sets or create your own and add cases.
Click Run eval
Choose a target (below) and the metrics to score. For a Cloud Run agent you can pick the service from your Google Cloud projects instead of pasting its URL.
Read the run
Each case gets a score and the judge's reasoning, so a low score comes with a stated reason rather than just a number you have to trust.
What an eval can call
Eval sets and load tests take the same two kinds of target:
| Target | Fields | What TraptureIQ sends |
|---|---|---|
| ADK agent | Base URL of an adk api_server (or adk web, or a Cloud Run service deployed with adk deploy cloud_run), App name, optional API key | Per case: create a session, POST /run_sse, delete the session. The API key, if set, goes in an X-API-Key header. |
| Webhook | URL, optional headers as a JSON object | POST {"question": "..."}, expecting {"answer": "...", "tool_calls": [...]} back. Works with any framework. |
- The URL must be publicly reachable. The calls come from TraptureIQ's servers, so
localhostand private addresses are refused. - An ADK target sends no Google identity token. A Cloud Run service deployed without public access answers 403. Put the agent behind a webhook you control, or evaluate a copy that allows unauthenticated calls.
- App name is the agent's folder name, the one
adk weblists, not the Cloud Run service name. - Agent Engine is not a direct target yet. Its API is not the ADK REST API; call it from a webhook.
Next: Tools.