An evaluator scores one answer from 0 to 1, where higher is always better. Some are heuristics that run locally; the others ask a judge model. This page lists the built-in evaluators and the inputs each one needs.

All of them implement com.tnsai.evaluation.spi.Evaluator. To write your own, see Write an evaluator.

Set up a judge

Judge evaluators take an LLMJudge, a function from a prompt to the judge model's reply. Adapt any LLMClient with prompt -> llm.chat(prompt).getContent(), as the example below does.

A judge evaluator sends one prompt and parses a score from the reply. If the judge throws, the result fails with LLM judge error: <message>; if the reply has no score it can parse, the result fails and its details quote the start of the reply. Judge scores are only as good as the judge model, so use a capable one and spot-check its verdicts.

Call an evaluator directly

The benchmark runner does not pass test-case metadata to evaluators. Evaluators that read metadata, such as the RAG and multi-turn ones, are called directly with an EvaluationInput you build:

static List<EvaluationResult> score(LLMClient llm, String question, String answer,
                                    List<String> documents) {
    LLMJudge judge = prompt -> llm.chat(prompt).getContent();

    EvaluationInput input = EvaluationInput.builder()
        .userInput(question)
        .agentResponse(answer)
        .metadata("retrieved_documents", documents)
        .build();

    List<Evaluator> evaluators = List.of(
        new FaithfulnessEvaluator(judge),
        new AnswerRelevancyEvaluator(judge),
        new PIILeakageEvaluator(),
        new GEvalEvaluator("conciseness", "Is the answer short and free of filler?", judge));
    return evaluators.stream().map(evaluator -> evaluator.evaluate(input)).toList();
}

EvaluationInput.builder() takes userInput, agentResponse, expectedOutput, expectedToolSequence, actualToolSequence, instructions, latencyMs, costUsd, inputTokens, outputTokens and metadata(key, value). A result's score(), details() and metrics() explain the score; passed(threshold) compares it.

Heuristic evaluators

These run without a model.

Class	Name	Needs	Scores
`ResponseQualityEvaluator`	`response_quality`	`agentResponse`, `expectedOutput`	Word overlap with the expected output: relevance, completeness and accuracy. Without an expected output, only the answer's length (100 characters scores 1)
`ToolSelectionEvaluator`	`tool_selection`	`expectedToolSequence`, `actualToolSequence`	Precision, recall and order of the tools used. No tools expected and none used scores 1; tools used when none were expected scores 0.5
`InstructionFollowingEvaluator`	`instruction_following`	`instructions`, `agentResponse`	The share of format and constraint checks it can detect in the instructions, such as JSON, bullet points or a word limit. No instructions scores 1
`CostEfficiencyEvaluator`	`cost_efficiency`	`costUsd`, `latencyMs`, `inputTokens`, `outputTokens`	Cost and latency against budgets (default $0.10 and 30 seconds) and the output-to-input token ratio. Missing values score 1
`JsonCorrectnessEvaluator`	`json_correctness`	`agentResponse`; metadata below	Valid JSON, required fields present and property types matching; an empty or invalid answer scores 0
`PIILeakageEvaluator`	`pii_leakage`	`agentResponse`	1 minus the share of enabled categories found: `EMAIL`, `PHONE`, `CREDIT_CARD` (Luhn-checked), `TCKN`, `IBAN` and `IP_ADDRESS`. Pass a `Set<Category>` to check fewer
`StepEfficiencyEvaluator`	`step_efficiency`	`actualToolSequence`, and `expectedToolSequence` or metadata `expected_steps`	Expected steps divided by the larger of expected and actual. No expected count scores 1

Check JSON output

static EvaluationResult checkJson(String response) {
    EvaluationInput input = EvaluationInput.builder()
        .agentResponse(response)
        .metadata("required_fields", List.of("id", "status"))
        .metadata("json_schema", """
            {"properties": {"id": {"type": "integer"}, "status": {"type": "string"}}}
            """)
        .build();
    return new JsonCorrectnessEvaluator().evaluate(input);
}

The score is 0.4 for valid JSON, plus 0.3 times the share of required_fields present, plus 0.3 times the share of checked fields whose type matches. Answers that are not valid JSON score 0. json_schema is a JSON Schema string; only the type of each top-level property is checked, and properties missing from the answer are not counted. required_fields is a list of top-level field names.

Judge evaluators

Each takes an LLMJudge in its constructor.

Class	Name	Needs	Scores
`FaithfulnessEvaluator`	`faithfulness`	`agentResponse`, metadata `retrieved_documents`	The share of the answer's claims that the documents support
`AnswerRelevancyEvaluator`	`answer_relevancy`	`userInput`, `agentResponse`	Directness, completeness and focus, each 1 to 5
`ContextualPrecisionEvaluator`	`contextual_precision`	`userInput`, metadata `retrieved_documents`	Whether the relevant documents are ranked first
`ContextualRecallEvaluator`	`contextual_recall`	`expectedOutput`, metadata `retrieved_documents`	The share of the expected answer's statements that the documents cover
`HallucinationEvaluator`	`hallucination`	`agentResponse`; metadata `context` or `retrieved_documents`	The share of claims supported by the context. Without context, the judge looks for contradictions and invented references
`BiasEvaluator`	`bias`	`agentResponse`	1 minus the judged bias on a 0 to 5 scale
`ToxicityEvaluator`	`toxicity`	`agentResponse`	1 minus the judged toxicity on a 0 to 5 scale
`TaskCompletionEvaluator`	`task_completion`	`agentResponse`, and `expectedOutput` or `instructions` as the goal	Whether the answer achieves the goal, 1 to 5
`PlanAdherenceEvaluator`	`plan_adherence`	`instructions` holding the declared plan	Whether the run followed the plan, 1 to 5
`ToolCorrectnessEvaluator`	`tool_correctness`	`actualToolSequence`	Whether each tool call was appropriate, 1 to 5. No tools used scores 1
`ConversationCompletenessEvaluator`	`conversation_completeness`	metadata `conversation_history` and `conversation_goal`	Whether the conversation reached the goal, 1 to 5
`KnowledgeRetentionEvaluator`	`knowledge_retention`	metadata `conversation_history`, at least two turns	The share of earlier facts the assistant kept
`TurnRelevancyEvaluator`	`turn_relevancy`	metadata `conversation_history`, at least two turns and one assistant turn	Context alignment, query addressing and coherence of the last assistant turn

Scores on a 1 to 5 scale are normalized as (score − 1) / 4. retrieved_documents is a List<String>. conversation_history is a List<Map<String, String>> with role and content keys, oldest turn first.

Define your own criterion

GEvalEvaluator(criteriaName, description, judge) scores any criterion you describe in words. Its name is geval_<criteriaName>. It calls the judge twice: once to turn the description into evaluation steps, then once to score the answer from 1 to 5 against those steps. The metrics include raw_score and normalized_score. An empty answer fails without a judge call.

Evaluators