A benchmark scores a fixed dataset. The tools on this page score what the agent does while it runs: evaluate recorded traces, collect metrics from hooks, and turn errors into regression cases.

Score a trace

TraceEvalBridge runs evaluators over an AgentTrace and writes each score back onto the trace.

static TraceEvalBridge bridge(LLMJudge judge) {
    return TraceEvalBridge.builder()
        .evaluator(new AnswerRelevancyEvaluator(judge))
        .evaluator(new PIILeakageEvaluator())
        .failureThreshold(0.6)
        .onFailure(failed -> System.out.println(failed.traceId() + " "
            + failed.evaluatorName() + " scored " + failed.score()))
        .build();
}

static List<EvaluationResult> evaluateTurn(TraceEvalBridge bridge, String agentId,
                                           String question, String answer, Instant started) {
    AgentTrace trace = AgentTrace.builder(agentId).build();
    trace.addObservation(trace.startGeneration("chat")
        .startTime(started)
        .input(question)
        .output(answer)
        .endTime(Instant.now())
        .build());
    return bridge.process(trace);                         // also adds an "eval.<name>" score to the trace
}

process(trace) builds the evaluation input from the trace's root observations:

Input	Taken from
`userInput`, `agentResponse`	`input` and `output` of the last root `GENERATION`
`latencyMs`	That generation's start and end time
`actualToolSequence`	The names of the root `SPAN` observations
`metadata`	`trace_id`, `agent_id`, `session_id` and the trace's own metadata

A generation nested under a span is not read, and neither are prompt and completion; set input and output. A trace with no root generation input or output is skipped and returns an empty list.

Each result is added to the trace as a Score named eval.<evaluator name> with source HEURISTIC. onFailure is called for every score below failureThreshold, which defaults to 0.5, with the trace id, agent id, evaluator name, score, details and the evaluation input. An evaluator that throws is logged and left out of the results.

The trace's metadata reaches the evaluators, so a RAG evaluator can read retrieved_documents if you put it on the trace.

Collect metrics with eval hooks

An EvalHook receives events from the agent and records metrics into an EvalContext. Hooks need tnsai-quality on the classpath; without it, addEvalHook accepts the hook but nothing calls it.

public static final class LatencyHook implements EvalHook {

    @Override
    public void onAfterChat(EvalContext ctx, String response, long latencyMs) {
        ctx.recordMetric("chat_latency_ms", latencyMs);
    }

    @Override
    public void onAfterToolCall(EvalContext ctx, String toolName, Object result,
                                boolean success, long latencyMs) {
        ctx.incrementCounter(success ? "tool_ok" : "tool_failed");
    }
}

static EvalContext.MetricStats measure(Agent agent, List<String> prompts) {
    agent.addEvalHook(new LatencyHook());
    EvalContext session = agent.clearEvalContext();       // later calls record into this session
    prompts.forEach(prompt -> agent.chat(prompt, false));
    System.out.println("Failed tool calls: " + session.getCounter("tool_failed"));
    return session.getStats("chat_latency_ms");           // count, min, max, average, p50, p90, p99
}

Method	Called
`onAgentStart`	When a new eval context starts: on the first `chat` after a hook is added, or on `clearEvalContext()`
`onBeforeChat`, `onAfterChat`	Around each `chat` call, with the latency
`onError`	When `chat` throws, with phase `"chat"`
`onBeforeToolCall`, `onAfterToolCall`	Around each tool call the model makes, with its success and latency
`onGoalCompleted`	When you call `agent.recordGoalCompletion(goalId, success, details)`
`onAgentStop`	When the agent stops; the context is then marked complete

EvalHook also declares memory, communication and plan methods, but the agent never calls them.

A hook that throws is logged and does not affect the agent or the other hooks. agent.setEvalEnabled(false) pauses every hook.

EvalContext holds one session:

recordMetric(name, value) adds a value, and recordMetric(name, value, tags) adds it under the name with the tags appended. getStats(name) returns count, min, max, average, p50, p90 and p99.
incrementCounter(name) and getCounter(name) count events.
addListener receives each metric as it is recorded.
toMap() returns the session, its metrics and its counters for export.

agent.getEvalContext() returns the current session. agent.clearEvalContext() starts a new one and returns it.

Turn failures into regression cases

AutoHarnessLoop takes the errors you record, groups them, writes a test case for each group and runs those cases as a gate. The hook below records every chat error with the message that caused it:

public static final class FailureHook implements EvalHook {

    private final DefaultFailureMiner miner;
    private volatile String lastMessage = "";

    public FailureHook(DefaultFailureMiner miner) {
        this.miner = miner;
    }

    @Override
    public void onBeforeChat(EvalContext ctx, String message) {
        lastMessage = message;
    }

    @Override
    public void onError(EvalContext ctx, Throwable error, String phase) {
        miner.record(ctx.getAgentId(), phase, error.getClass().getName(),
            String.valueOf(error.getMessage()), Map.of("input", lastMessage));
    }
}

static boolean regress(DefaultFailureMiner miner, BenchmarkRunner runner) {
    AutoHarnessLoop loop = AutoHarnessLoop.builder()
        .miner(miner)
        .clusterer(new SimpleFailureClusterer())
        .generator(new DefaultEvalCaseGenerator())
        .runner(runner)                                   // runs the generated cases against your agent
        .passRateThreshold(0.9)
        .build();

    AutoHarnessLoop.LoopResult result = loop.run();       // clears the miner after the gate runs
    System.out.println(result.summary());
    return result.gateResult() == null || result.gatePassed();  // null: no failures recorded
}

run() goes through these steps:

Takes the recorded failures from the miner. With none, it returns at once: gateResult() is null and gatePassed() is false, so check for null as the example does.
Groups them. SimpleFailureClusterer groups by error type and phase.
Writes up to three cases per group with DefaultEvalCaseGenerator. Each case's input is the recorded input context value, or Reproduce failure: <message> when there is none. Its tags are autoharness, regression and the phase.
Runs the cases with your BenchmarkRunner and checks them against passRateThreshold (default 0.8) and scoreThreshold (default 0.7).
Clears the miner and returns a LoopResult with the failure, group and case counts, the gate result and a summary.

The generated cases have no expected answer, so choose evaluators for the runner that check whether the agent now answers without an error, not how good the answer is. runUntilPass() repeats run() up to maxIterations (default 5) and stops when the gate passes. The miner is empty after the first run, so unless new failures are recorded in between, the later runs find nothing and return without a gate result.

Evaluate traces and live runs

Score a trace

Collect metrics with eval hooks

Turn failures into regression cases

Related