Run evaluations as JUnit tests
@EvalTest turns a dataset into JUnit 5 tests: one test per case, each named after the case's id. It is part of tnsai-evaluation, which brings junit-jupiter-api with it; your build still needs the JUnit engine to run the tests.
Write an eval test
static class SupportEvalTest {
static Agent agent; // build it in a @BeforeAll method
@EvalTest(dataset = "evals/support.json", tag = "billing", threshold = 0.6)
void answersBillingQuestions(TestCase testCase, EvalAssert eval) {
String answer = agent.chat(testCase.input(), false);
eval.assertEval(answer, new ResponseQualityEvaluator(), new PIILeakageEvaluator())
.passes();
}
}| Attribute | Default | Meaning |
|---|---|---|
dataset | required | A dataset file, looked up on the classpath first and then as a file path |
tag | all cases | Runs only the cases with this tag |
threshold | 0.7 | The lowest passing score for every evaluator |
The test method can take a TestCase, an EvalAssert, or both. The test method fails with an ExtensionConfigurationException when the dataset cannot be loaded or no case has the tag.
Assert on the scores
eval.assertEval(answer, evaluators...) runs each evaluator on the answer and returns an assertion:
| Method | Checks |
|---|---|
passes() | Every score is at least the threshold. The error lists each evaluator below it, with its score and details |
fails() | Every score is below the threshold |
withThreshold(value) | Replaces the threshold for this assertion |
results() | Returns the EvaluationResult list for your own checks |
The evaluation input holds the case's input as the user input, its expectedOutput, and the answer you pass. It does not carry the case's instructions, tools or metadata, so evaluators that need them score as if those were missing. assertEval(answer, metadata, evaluators...) adds metadata, and that metadata stays on the same EvalAssert for later calls in the method. For full control, build an EvaluationInput yourself, as in Call an evaluator directly.
Every evaluator scores higher for better answers, including bias and toxicity. To check that an answer is safe, use passes() with those evaluators, not fails().
Keep live tests out of the default build
An eval test calls the agent, and judge evaluators call a model. Run these tests in a separate Maven profile or tag them with JUnit's @Tag, so the regular build does not depend on a model being reachable.