tnsai-evaluation measures how well an agent answers. You run the agent over a dataset of test cases, score each answer with one or more evaluators, and gate the run on thresholds and on regressions against earlier runs.

To see how an agent runs rather than how well it answers, use Observability.

Write a dataset

A dataset is a JSON file with a list of cases:

{
  "name": "support",
  "description": "Billing and account questions",
  "version": "3",
  "cases": [
    {
      "id": "refund-window",
      "input": "How long do I have to ask for a refund?",
      "expectedOutput": "Refunds can be requested within 30 days of purchase.",
      "instructions": "Answer in one sentence.",
      "tags": ["billing"]
    }
  ]
}

Field	Use
`id`, `input`	Required. A blank value makes loading fail
`expectedOutput`	The reference answer that `response_quality` compares with
`expectedToolSequence`	The tool names `tool_selection` expects, in order
`instructions`	The constraints `instruction_following` checks
`tags`	Used by `filterByTag` and by `@EvalTest(tag)`
`metadata`	Kept on the test case, but not passed to evaluators by the runner

Load it with TestDataset.fromFile(path) or TestDataset.fromClasspath(resource). dataset.filterByTag("billing") returns a subset named support[tag=billing], so its baselines are stored apart from the full dataset's.

Run a benchmark

static boolean benchmark(Agent agent, Path datasetFile, Path baselineDir) throws IOException {
    TestDataset dataset = TestDataset.fromFile(datasetFile);

    BenchmarkRunner runner = BenchmarkRunner.builder()
        .evaluator(new ResponseQualityEvaluator())
        .evaluator(new InstructionFollowingEvaluator())
        .evaluator(new CostEfficiencyEvaluator(0.10, 30_000))
        .agentFunction(testCase -> BenchmarkRunner.AgentOutput.simple(
            agent.chat(testCase.input(), false)))        // no history between cases
        .build();
    BenchmarkResult result = runner.run(dataset);

    BaselineStore baselines = new BaselineStore(baselineDir);
    QualityGate gate = new QualityGate(QualityGateConfig.builder()
        .defaultThreshold(0.7)
        .evaluatorThreshold("cost_efficiency", 0.5)
        .maxRegressionPercent(5.0)
        .minPassRate(0.8)
        .build(), baselines);
    QualityGate.Verdict verdict = gate.evaluate(result);  // compares with the latest saved baseline
    System.out.println(verdict.summary());

    new EvaluationReport(result, verdict).exportJson(baselineDir.resolve("report.json"));
    if (verdict.passed()) {
        baselines.save(result);                           // the next run is compared with this one
    }
    return verdict.passed();
}

BenchmarkRunner calls the agent function once per case, measures its latency and passes the answer to every evaluator. build() throws IllegalStateException without an evaluator or an agent function.

Use agent.chat(input, false) so that one case's answer does not enter the next case's history.
An exception from the agent function becomes the answer ERROR: <message>, and an exception from an evaluator becomes a score of 0 for that evaluator. The run itself does not stop.
parallelism(n) runs cases on n threads, with a five-minute limit per case. The agent function must then be thread-safe.
The runner passes the case's input, expected output, expected tools and instructions to the evaluators, but not its metadata. Evaluators that read metadata, such as the RAG ones, need to be called directly.

AgentOutput.simple(answer) reports no tools, cost or tokens. To score tool use and cost, return them:

static BenchmarkRunner.AgentOutput run(SupportBot bot, TestCase testCase) {
    Reply reply = bot.answer(testCase.input());
    return BenchmarkRunner.AgentOutput.of(
        reply.text(), reply.tools(), reply.costUsd(), reply.inputTokens(), reply.outputTokens());
}

Read the result

BenchmarkResult holds one TestCaseResult per case and the aggregate scores.

Method	Returns
`totalCases()`, `duration()`	Size and wall time of the run
`passRate(threshold)`, `passedCases(threshold)`	Cases whose `overallScore()` is at least the threshold
`aggregateScores()`	`<evaluator>_mean`, `_median`, `_p95` and `_stddev` for each evaluator, plus `overall_mean`
`caseResults()`	Each case's answer, tools, latency, cost and `evaluationResults()`

A case's overallScore() is the mean of its evaluator scores. EvaluationReport adds evaluatorStats(), worstCases(n) and bestCases(n), and exportJson(path) writes the whole run, with the gate verdict, as JSON.

Gate the run

QualityGate.evaluate(result) returns a Verdict with passed(), failures(), warnings() and summary(). It fails the run when:

the pass rate at defaultThreshold is below minPassRate;
an evaluator's mean is below its threshold (evaluatorThreshold(name, value), or defaultThreshold);
with a BaselineStore, any score in the latest baseline dropped by more than maxRegressionPercent percent.

Setting	`QualityGateConfig.defaults()`
`defaultThreshold`	0.7
`minPassRate`	0.8
`maxRegressionPercent`	5.0

The regression check compares every stored score, including _stddev and _p95. A lower standard deviation is an improvement but still counts as a drop, so a more consistent run can fail the gate. A baseline that cannot be read becomes a warning, not a failure.

Keep baselines

BaselineStore writes one JSON file per saved run, in a directory named after the dataset. loadLatest(dataset) is the run the gate compares with, so save a run only after it passes; saving a failing run makes it the new reference. listBaselines(dataset) returns all saved runs, oldest first.

static void trend(BaselineStore baselines, String datasetName) throws IOException {
    TrendAnalyzer.TrendReport report = TrendAnalyzer.analyze(baselines.listBaselines(datasetName));
    report.metricTrends().forEach((metric, trend) ->
        System.out.println(metric + ": " + trend.firstValue() + " -> " + trend.lastValue()
            + " (" + trend.direction() + ")"));
    if (report.hasDegradations()) {
        System.out.println("Degrading: " + report.degradingMetrics());
    }
}

Each MetricTrend is IMPROVING, DEGRADING or STABLE (a change under 1%). Any decrease counts as degrading, including _stddev. With fewer than two baselines the report is empty.

Write an evaluator

An evaluator has a name() and turns an EvaluationInput into an EvaluationResult.

public static final class AnswerLengthEvaluator implements Evaluator {

    private final int maxWords;

    public AnswerLengthEvaluator(int maxWords) {
        this.maxWords = maxWords;
    }

    @Override
    public String name() {
        return "answer_length";
    }

    @Override
    public EvaluationResult evaluate(EvaluationInput input) {
        String response = input.agentResponse();
        if (response == null || response.isBlank() || response.startsWith("ERROR: ")) {
            return EvaluationResult.fail(name(), "No answer");
        }
        int words = response.trim().split("\\s+").length;
        double score = words <= maxWords ? 1.0 : (double) maxWords / words;
        return EvaluationResult.of(name(), score, words + " words", Map.of("words", (double) words));
    }
}

EvaluationResult.of clamps the score to 0 to 1. pass(name, details) scores 1 and fail(name, details) scores 0. Check for the ERROR: prefix when the agent function can throw; otherwise the runner's error text is scored like an answer.

The built-in evaluators are listed in Evaluators.

Evaluation