# Evaluators An evaluator scores one answer from 0 to 1, where higher is always better. Some are heuristics that run locally; the others ask a judge model. This page lists the built-in evaluators and the inputs each one needs. All of them implement `com.tnsai.evaluation.spi.Evaluator`. To write your own, see [Write an evaluator](index.md#write-an-evaluator). ## Set up a judge Judge evaluators take an `LLMJudge`, a function from a prompt to the judge model's reply. Adapt any `LLMClient` with `prompt -> llm.chat(prompt).getContent()`, as the [example below](#call-an-evaluator-directly) does. A judge evaluator sends one prompt and parses a score from the reply. If the judge throws, the result fails with `LLM judge error: `; if the reply has no score it can parse, the result fails and its details quote the start of the reply. Judge scores are only as good as the judge model, so use a capable one and spot-check its verdicts. ## Call an evaluator directly The benchmark runner does not pass test-case metadata to evaluators. Evaluators that read metadata, such as the RAG and multi-turn ones, are called directly with an `EvaluationInput` you build: ```java static List score(LLMClient llm, String question, String answer, List documents) { LLMJudge judge = prompt -> llm.chat(prompt).getContent(); EvaluationInput input = EvaluationInput.builder() .userInput(question) .agentResponse(answer) .metadata("retrieved_documents", documents) .build(); List evaluators = List.of( new FaithfulnessEvaluator(judge), new AnswerRelevancyEvaluator(judge), new PIILeakageEvaluator(), new GEvalEvaluator("conciseness", "Is the answer short and free of filler?", judge)); return evaluators.stream().map(evaluator -> evaluator.evaluate(input)).toList(); } ``` `EvaluationInput.builder()` takes `userInput`, `agentResponse`, `expectedOutput`, `expectedToolSequence`, `actualToolSequence`, `instructions`, `latencyMs`, `costUsd`, `inputTokens`, `outputTokens` and `metadata(key, value)`. A result's `score()`, `details()` and `metrics()` explain the score; `passed(threshold)` compares it. ## Heuristic evaluators These run without a model. | Class | Name | Needs | Scores | |-------|------|-------|--------| | `ResponseQualityEvaluator` | `response_quality` | `agentResponse`, `expectedOutput` | Word overlap with the expected output: relevance, completeness and accuracy. Without an expected output, only the answer's length (100 characters scores 1) | | `ToolSelectionEvaluator` | `tool_selection` | `expectedToolSequence`, `actualToolSequence` | Precision, recall and order of the tools used. No tools expected and none used scores 1; tools used when none were expected scores 0.5 | | `InstructionFollowingEvaluator` | `instruction_following` | `instructions`, `agentResponse` | The share of format and constraint checks it can detect in the instructions, such as JSON, bullet points or a word limit. No instructions scores 1 | | `CostEfficiencyEvaluator` | `cost_efficiency` | `costUsd`, `latencyMs`, `inputTokens`, `outputTokens` | Cost and latency against budgets (default $0.10 and 30 seconds) and the output-to-input token ratio. Missing values score 1 | | `JsonCorrectnessEvaluator` | `json_correctness` | `agentResponse`; metadata below | Valid JSON, required fields present and property types matching; an empty or invalid answer scores 0 | | `PIILeakageEvaluator` | `pii_leakage` | `agentResponse` | 1 minus the share of enabled categories found: `EMAIL`, `PHONE`, `CREDIT_CARD` (Luhn-checked), `TCKN`, `IBAN` and `IP_ADDRESS`. Pass a `Set` to check fewer | | `StepEfficiencyEvaluator` | `step_efficiency` | `actualToolSequence`, and `expectedToolSequence` or metadata `expected_steps` | Expected steps divided by the larger of expected and actual. No expected count scores 1 | ### Check JSON output ```java static EvaluationResult checkJson(String response) { EvaluationInput input = EvaluationInput.builder() .agentResponse(response) .metadata("required_fields", List.of("id", "status")) .metadata("json_schema", """ {"properties": {"id": {"type": "integer"}, "status": {"type": "string"}}} """) .build(); return new JsonCorrectnessEvaluator().evaluate(input); } ``` The score is 0.4 for valid JSON, plus 0.3 times the share of `required_fields` present, plus 0.3 times the share of checked fields whose type matches. Answers that are not valid JSON score 0. `json_schema` is a JSON Schema string; only the `type` of each top-level property is checked, and properties missing from the answer are not counted. `required_fields` is a list of top-level field names. ## Judge evaluators Each takes an `LLMJudge` in its constructor. | Class | Name | Needs | Scores | |-------|------|-------|--------| | `FaithfulnessEvaluator` | `faithfulness` | `agentResponse`, metadata `retrieved_documents` | The share of the answer's claims that the documents support | | `AnswerRelevancyEvaluator` | `answer_relevancy` | `userInput`, `agentResponse` | Directness, completeness and focus, each 1 to 5 | | `ContextualPrecisionEvaluator` | `contextual_precision` | `userInput`, metadata `retrieved_documents` | Whether the relevant documents are ranked first | | `ContextualRecallEvaluator` | `contextual_recall` | `expectedOutput`, metadata `retrieved_documents` | The share of the expected answer's statements that the documents cover | | `HallucinationEvaluator` | `hallucination` | `agentResponse`; metadata `context` or `retrieved_documents` | The share of claims supported by the context. Without context, the judge looks for contradictions and invented references | | `BiasEvaluator` | `bias` | `agentResponse` | 1 minus the judged bias on a 0 to 5 scale | | `ToxicityEvaluator` | `toxicity` | `agentResponse` | 1 minus the judged toxicity on a 0 to 5 scale | | `TaskCompletionEvaluator` | `task_completion` | `agentResponse`, and `expectedOutput` or `instructions` as the goal | Whether the answer achieves the goal, 1 to 5 | | `PlanAdherenceEvaluator` | `plan_adherence` | `instructions` holding the declared plan | Whether the run followed the plan, 1 to 5 | | `ToolCorrectnessEvaluator` | `tool_correctness` | `actualToolSequence` | Whether each tool call was appropriate, 1 to 5. No tools used scores 1 | | `ConversationCompletenessEvaluator` | `conversation_completeness` | metadata `conversation_history` and `conversation_goal` | Whether the conversation reached the goal, 1 to 5 | | `KnowledgeRetentionEvaluator` | `knowledge_retention` | metadata `conversation_history`, at least two turns | The share of earlier facts the assistant kept | | `TurnRelevancyEvaluator` | `turn_relevancy` | metadata `conversation_history`, at least two turns and one assistant turn | Context alignment, query addressing and coherence of the last assistant turn | Scores on a 1 to 5 scale are normalized as (score − 1) / 4. `retrieved_documents` is a `List`. `conversation_history` is a `List>` with `role` and `content` keys, oldest turn first. ### Define your own criterion `GEvalEvaluator(criteriaName, description, judge)` scores any criterion you describe in words. Its name is `geval_`. It calls the judge twice: once to turn the description into evaluation steps, then once to score the answer from 1 to 5 against those steps. The metrics include `raw_score` and `normalized_score`. An empty answer fails without a judge call. ## Related - [Evaluation](index.md) - [Run evaluations as JUnit tests](junit.md) - [Evaluate traces and live runs](trace-eval.md) - [Retrieval strategies](../rag/retrieval-strategies.md)