# Evaluate traces and live runs

A benchmark scores a fixed dataset. The tools on this page score what the agent does while it runs: evaluate recorded traces, collect metrics from hooks, and turn errors into regression cases.

## Score a trace

`TraceEvalBridge` runs evaluators over an [`AgentTrace`](../observability/monitor.md#structured-traces) and writes each score back onto the trace.

<!-- java-contract: src/main/java/com/example/tnsai/docs/EvaluationTraceBridgeExample.java -->
```java
static TraceEvalBridge bridge(LLMJudge judge) {
    return TraceEvalBridge.builder()
        .evaluator(new AnswerRelevancyEvaluator(judge))
        .evaluator(new PIILeakageEvaluator())
        .failureThreshold(0.6)
        .onFailure(failed -> System.out.println(failed.traceId() + " "
            + failed.evaluatorName() + " scored " + failed.score()))
        .build();
}

static List<EvaluationResult> evaluateTurn(TraceEvalBridge bridge, String agentId,
                                           String question, String answer, Instant started) {
    AgentTrace trace = AgentTrace.builder(agentId).build();
    trace.addObservation(trace.startGeneration("chat")
        .startTime(started)
        .input(question)
        .output(answer)
        .endTime(Instant.now())
        .build());
    return bridge.process(trace);                         // also adds an "eval.<name>" score to the trace
}
```

`process(trace)` builds the evaluation input from the trace's root observations:

| Input | Taken from |
|-------|------------|
| `userInput`, `agentResponse` | `input` and `output` of the last root `GENERATION` |
| `latencyMs` | That generation's start and end time |
| `actualToolSequence` | The names of the root `SPAN` observations |
| `metadata` | `trace_id`, `agent_id`, `session_id` and the trace's own metadata |

A generation nested under a span is not read, and neither are `prompt` and `completion`; set `input` and `output`. A trace with no root generation input or output is skipped and returns an empty list.

Each result is added to the trace as a `Score` named `eval.<evaluator name>` with source `HEURISTIC`. `onFailure` is called for every score below `failureThreshold`, which defaults to 0.5, with the trace id, agent id, evaluator name, score, details and the evaluation input. An evaluator that throws is logged and left out of the results.

The trace's metadata reaches the evaluators, so a RAG evaluator can read `retrieved_documents` if you put it on the trace.

## Collect metrics with eval hooks

An `EvalHook` receives events from the agent and records metrics into an `EvalContext`. Hooks need `tnsai-quality` on the classpath; without it, `addEvalHook` accepts the hook but nothing calls it.

<!-- java-contract: src/main/java/com/example/tnsai/docs/EvaluationHooksExample.java -->
```java
public static final class LatencyHook implements EvalHook {

    @Override
    public void onAfterChat(EvalContext ctx, String response, long latencyMs) {
        ctx.recordMetric("chat_latency_ms", latencyMs);
    }

    @Override
    public void onAfterToolCall(EvalContext ctx, String toolName, Object result,
                                boolean success, long latencyMs) {
        ctx.incrementCounter(success ? "tool_ok" : "tool_failed");
    }
}

static EvalContext.MetricStats measure(Agent agent, List<String> prompts) {
    agent.addEvalHook(new LatencyHook());
    EvalContext session = agent.clearEvalContext();       // later calls record into this session
    prompts.forEach(prompt -> agent.chat(prompt, false));
    System.out.println("Failed tool calls: " + session.getCounter("tool_failed"));
    return session.getStats("chat_latency_ms");           // count, min, max, average, p50, p90, p99
}
```

| Method | Called |
|--------|--------|
| `onAgentStart` | When a new eval context starts: on the first `chat` after a hook is added, or on `clearEvalContext()` |
| `onBeforeChat`, `onAfterChat` | Around each `chat` call, with the latency |
| `onError` | When `chat` throws, with phase `"chat"` |
| `onBeforeToolCall`, `onAfterToolCall` | Around each tool call the model makes, with its success and latency |
| `onGoalCompleted` | When you call `agent.recordGoalCompletion(goalId, success, details)` |
| `onAgentStop` | When the agent stops; the context is then marked complete |

`EvalHook` also declares memory, communication and plan methods, but the agent never calls them.

A hook that throws is logged and does not affect the agent or the other hooks. `agent.setEvalEnabled(false)` pauses every hook.

`EvalContext` holds one session:

- `recordMetric(name, value)` adds a value, and `recordMetric(name, value, tags)` adds it under the name with the tags appended. `getStats(name)` returns count, min, max, average, p50, p90 and p99.
- `incrementCounter(name)` and `getCounter(name)` count events.
- `addListener` receives each metric as it is recorded.
- `toMap()` returns the session, its metrics and its counters for export.

`agent.getEvalContext()` returns the current session. `agent.clearEvalContext()` starts a new one and returns it.

## Turn failures into regression cases

`AutoHarnessLoop` takes the errors you record, groups them, writes a test case for each group and runs those cases as a gate. The hook below records every `chat` error with the message that caused it:

<!-- java-contract: src/main/java/com/example/tnsai/docs/EvaluationAutoHarnessExample.java -->
```java
public static final class FailureHook implements EvalHook {

    private final DefaultFailureMiner miner;
    private volatile String lastMessage = "";

    public FailureHook(DefaultFailureMiner miner) {
        this.miner = miner;
    }

    @Override
    public void onBeforeChat(EvalContext ctx, String message) {
        lastMessage = message;
    }

    @Override
    public void onError(EvalContext ctx, Throwable error, String phase) {
        miner.record(ctx.getAgentId(), phase, error.getClass().getName(),
            String.valueOf(error.getMessage()), Map.of("input", lastMessage));
    }
}

static boolean regress(DefaultFailureMiner miner, BenchmarkRunner runner) {
    AutoHarnessLoop loop = AutoHarnessLoop.builder()
        .miner(miner)
        .clusterer(new SimpleFailureClusterer())
        .generator(new DefaultEvalCaseGenerator())
        .runner(runner)                                   // runs the generated cases against your agent
        .passRateThreshold(0.9)
        .build();

    AutoHarnessLoop.LoopResult result = loop.run();       // clears the miner after the gate runs
    System.out.println(result.summary());
    return result.gateResult() == null || result.gatePassed();  // null: no failures recorded
}
```

`run()` goes through these steps:

1. Takes the recorded failures from the miner. With none, it returns at once: `gateResult()` is `null` and `gatePassed()` is `false`, so check for `null` as the example does.
2. Groups them. `SimpleFailureClusterer` groups by error type and phase.
3. Writes up to three cases per group with `DefaultEvalCaseGenerator`. Each case's input is the recorded `input` context value, or `Reproduce failure: <message>` when there is none. Its tags are `autoharness`, `regression` and the phase.
4. Runs the cases with your `BenchmarkRunner` and checks them against `passRateThreshold` (default 0.8) and `scoreThreshold` (default 0.7).
5. Clears the miner and returns a `LoopResult` with the failure, group and case counts, the gate result and a summary.

The generated cases have no expected answer, so choose evaluators for the runner that check whether the agent now answers without an error, not how good the answer is. `runUntilPass()` repeats `run()` up to `maxIterations` (default 5) and stops when the gate passes. The miner is empty after the first run, so unless new failures are recorded in between, the later runs find nothing and return without a gate result.

## Related

- [Evaluation](index.md)
- [Evaluators](evaluators.md)
- [Monitor agents](../observability/monitor.md)