Agent Eval Suite From Traces

An evaluation-engineering team for production LLM agents. Traces are mined with several detection signals - explicit negative feedback, human takeover, turn-limit terminations, schema violations, tool errors followed by an answer anyway, and users rephrasing the same request - and clustered by mechanism rather than by wording, with frequency per thousand sessions and severity kept as separate numbers. The taxonomer gives each mode a definition and an explicit boundary against neighbouring modes, a justified per-mode coverage target, and strata by trigger shape and conversation depth, since most suites are entirely single-turn while most production failures begin after turn four. Case authoring is deliberately blind to grader implementation: the author produces a minimized, de-identified, fixture-pinned case plus expected behaviour in plain language and one passing and one failing example, and the grader designer builds checks against that statement without editing it, recording what each check cannot detect. Calibration compares every grader to a stratified human-labelled sample on balanced accuracy and failure-class recall rather than raw accuracy, rejects unstable judges, and returns cases to the author when human annotators barely agree with each other. The drift sentinel retires cases only on a cited product decision and refuses every other removal, and the regression gatekeeper audits the case-set digest for removals, weakened assertions and raised thresholds before reading any score, running repeats to separate regression from nondeterminism.