TraptureIQ Guides Evaluation
Most agent evals measure something nobody would act on
A score of 0.83 on a dashboard changes nothing. The only eval that alters behaviour is the one wired to a decision: ship or don’t. Getting there means being honest about what an LLM judge can and cannot tell you, and refusing to let a broken judge quietly pass a release.
Agent evaluation has an adoption problem that looks like a tooling problem. Teams build an eval suite, run it weekly, watch the number, and ship anyway, because the number was never connected to anything. The suite becomes a report nobody reads, which is worse than no suite at all: it produces the feeling of rigour without the effect.
The only shape that changes behaviour
Everything else in this guide is downstream of that diagram. If the last step of your eval pipeline is a chart, the pipeline is optional. If the last step is exit 1, it is a control.
# in CI: the threshold is the decision, written down tiq-eval run <eval-set-id> --wait --fail-under 0.8 --require-pass
Write cases that encode a decision
Most eval sets are built by collecting prompts somebody found interesting. The useful version starts from the other end: what would make you refuse to ship? Each case should be a sentence you would defend in a release review.
- Regression cases: every bug you have already fixed. This is the highest-value category and the one teams skip. A bug that recurs twice was never fixed; it was patched.
- Routing cases: inputs where the right specialist matters more than the wording of the answer. Multi-agent systems regress here first, and accuracy scoring barely notices.
- Tool-call cases: the tool should be called, with these arguments. Deterministic, cheap and worth more than a rubric.
- Refusal cases: things the agent must decline. Without these, "helpfulness" optimisation quietly erodes safety behaviour.
Run against a real endpoint
A mocked model tests your prompt. It does not test tool wiring, hand-offs, timeouts, or the retry that doubles your bill. Those are where agent regressions actually live, so the eval has to drive a real endpoint, and incidentally a real endpoint means an eval run is itself a series of billed calls. Budget for it, and measure it: see ADK agent cost.
Using an LLM judge without lying to yourself
A judge model is the only practical way to score open-ended output at volume. It is also the component most likely to make your suite quietly meaningless, in three specific ways.
Judge unavailability must be loud. If the judge model is unreachable and the runner falls back to a default score, a broken pipeline produces a green build: the worst possible failure mode for a gate. Unavailable should be reported as unavailable, and the run should say so rather than degrade silently.
Unimplemented metrics should say unimplemented. There is real pressure to approximate a hard metric (hallucination detection is the usual candidate) and ship it as if it worked. An approximation presented as a measurement is worse than a gap, because people plan against it. We ship two metrics explicitly marked unimplemented rather than approximated, for exactly this reason.
Deterministic beats judged where possible. If the case has a ground truth (a tool was called, a field was extracted, a refusal happened), check it directly. Reserve the judge for questions that genuinely need judgement.
Security belongs in the eval suite, not a separate quarter
An agent with tools is an agent that can be talked into using them. Prompt injection is not a theoretical risk for systems that read email, browse pages or accept documents. It is the primary attack surface, and it regresses like any other behaviour.
- Jailbreak and prompt injection: instructions embedded in the data the agent reads, not just in what the user types.
- Data-loss prevention: does private context leak into a response when the input invites it?
- Responsible AI: behaviour under hostile or manipulative input.
These are worth seeding as templates so they exist on day one rather than after an incident. They are also the cases most likely to break when somebody "improves" a system prompt.
Load testing is an eval too
Correct-but-unusable is a failure. Concurrency changes agent behaviour in ways single runs never reveal: tool timeouts, rate limits, and the parallel block that turns out to be serial under contention. See agent topology.
One caution about percentile arithmetic, since load-test numbers get quoted in contracts: a straightforward percentile implementation is easy to get subtly wrong. We found an off-by-one that returned the maximum as the p99 of 100 samples, in two independent implementations. If you compute your own percentiles, test them against a known distribution before anyone plans capacity on them.
A suite you would actually keep
- Start with ten cases drawn from real failures, not imagined ones.
- Make the pass threshold explicit and put it in CI on day one, even at 0.6.
- Add every production bug as a case the day it is fixed.
- Keep security templates in the same suite so they run on every change.
- Re-run at model upgrades. That is the moment behaviour silently moves.
Questions people actually ask
How do I evaluate a Google ADK agent?
Define an eval set of cases with expected behaviour, run it against a real agent endpoint rather than a mock, score each case with metrics appropriate to the case type, and fail the build below a threshold. Running against a real endpoint matters because most agent regressions live in tool calls and hand-offs, which a mocked model never exercises.
Is LLM-as-a-judge reliable for agent evaluation?
It is reliable for comparative and rubric-shaped questions and unreliable for questions with a precise ground truth, where a deterministic check is better. The critical safeguard is that judge unavailability must be reported explicitly. A judge that silently returns a default score turns a broken pipeline into a green build.
What should an agent eval suite cover beyond accuracy?
At minimum: prompt injection and jailbreak resistance, data-loss prevention on responses that touch private context, responsible-AI behaviour on hostile input, tool-call correctness, and performance under concurrency. Accuracy alone misses every security regression.
How do I stop a bad agent change from reaching production?
Run the eval set in CI and gate on the exit code, with an explicit threshold and a required pass rate. A score written to a dashboard is advisory; a non-zero exit code is the only output that reliably blocks a deploy.