Validate output and state

For checks on action arguments and return values, use @Contract. For input limits, see Validate input. For content checks on input and output, see Guardrails.

Cross-check answers across models

ParallelValidationExecutor sends one prompt to every participant at once, then picks or builds a single answer from the responses.

static String crossCheck(LLMClient first, LLMClient second, LLMClient third, String question) {
    ParallelValidationExecutor executor = ParallelValidationExecutor.builder()
        .addLLM(first)
        .addLLM(second)
        .addLLM(third)
        .aggregationStrategy(AggregationStrategy.CONSENSUS)
        .synthesisLLM(first)
        .timeout(Duration.ofSeconds(60))
        .build();

    ValidationResult result = executor.execute(question);
    System.out.println("Agreement " + result.getAgreementRatio()
        + ", " + result.getDivergences().size() + " divergences, "
        + result.getMetrics().getFailedAgents() + " failed");
    return result.getBestSolution();       // null when every participant failed
}

Participants are LLM clients (addLLM, llms) or agents (addAgent, agents); an agent answers through chat. You need at least one, or build() throws IllegalArgumentException. executeAsync returns a CompletableFuture<ValidationResult>.

Aggregation strategies

CONSENSUS and ENSEMBLE need synthesisLLM. Without it, CONSENSUS returns the highest-confidence response and ENSEMBLE returns the first one. AggregationStrategy.forTaskType(text) maps a task description to a strategy: "critical" or "quality" gives BEST_OF, "review" or "validation" gives CONSENSUS, "creative", "brainstorm" or "analysis" gives ENSEMBLE, and anything else gives VOTING.

How responses are compared

The comparison is lexical, not semantic:

• Two responses are similar when their word sets overlap (Jaccard similarity) by at least similarityThreshold.
• Confidence is a text heuristic. It starts at 0.5, rises for long or structured answers and for words such as "definitely", and falls for hedges such as "might". It does not come from the model.
• getAgreementRatio() mixes pairwise similarity with the share of key phrases all responses have in common.

ValidationResult also has getResponses(), getSuccessfulResponses(), getBestAgent(), getConsensusPoints(), getDivergences(), hasFullConsensus() and getMetrics(), which gives counts and timings.

Check state invariants

Mark a field with @State(invariants = ...) and put @InvariantCheck on the role or on the action. After the action body returns, the executor checks every invariant on the role's @State fields.

@InvariantCheck
public static class StockRole extends Role {

    @State(description = "Units in stock", invariants = {"value >= 0", "value <= 1000"})
    private int stock = 10;

    @Override
    public RoleIdentity getIdentity() {
        return new RoleIdentity("stock", "Track warehouse stock", "inventory");
    }

    @ActionSpec(type = ActionType.LOCAL, description = "Ship units from stock",
                precondition = "stock > 0")
    public int ship(int units) {
        stock -= units;
        return stock;
    }
}

Shipping 11 units leaves stock at -1. The action then fails with an ActionExecutionException in the VALIDATION category. The field keeps its new value: the check reports the violation, it does not roll the change back.

• value in an invariant stands for the field.
• With @InvariantCheck(strict = false), a violation is only logged and the action succeeds.
• A method-level @InvariantCheck overrides the class-level one.
• An invariant that reads like a sentence (more than three words and no ==, !=, >=, <=, && or ||) is skipped.
• Without tnsai-quality on the classpath, @InvariantCheck does nothing.

The executor checks only @State invariants. It never evaluates the precondition and postcondition strings on @ActionSpec; those describe the action to planners. The checkPreconditions, checkPostconditions and checkInvariants switches on @InvariantCheck are not read by the executor.

Expressions

Invariants and conditions use a small expression language:

• comparisons ==, !=, <, <=, >, >=, combined with &&, ||, ! and parentheses;
• numbers, quoted strings, true, false and null;
• field names, field.subfield, enum constant names such as ALIVE, and method calls such as isEmpty() or items.size().

Run the checker yourself

InvariantChecker evaluates the same expressions when you call it, including the @ActionSpec precondition and postcondition.

static void shipChecked(StockRole role, int units) {
    try {
        InvariantChecker.check(role, "ship");        // @ActionSpec precondition
        role.ship(units);
        InvariantChecker.checkAfter(role, "ship");   // postcondition, then @State invariants
    } catch (InvariantViolationException e) {
        System.out.println(e.getType() + " in " + e.getActionName() + ": " + e.getInvariant());
    }
}

static List<String> report(StockRole role) throws NoSuchMethodException {
    Method ship = StockRole.class.getMethod("ship", int.class);
    return new InvariantChecker(role).collectViolations(ship);   // never throws
}

new InvariantChecker(target) throws InvariantViolationException on a violation; new InvariantChecker(target, false) only logs it. The static check and checkAfter look the method up by name and always throw. The exception's getType() is PRECONDITION, POSTCONDITION, INVARIANT or STATE_INVARIANT.

Check state transitions with rules

RuleEngine checks rules of the form "effect if condition". When the condition holds, the state field must equal the effect.

static void checkTransition(Object cell) {
    RuleEngine engine = new RuleEngine(cell)
        .setStateField("status")
        .addRule("DEAD if neighbors < 2 (underpopulation)")
        .addRule("ALIVE if neighbors == 3 (reproduction)")
        .addRule("unchanged if neighbors == 2 (stasis)");

    for (RuleEngine.RuleViolation violation : engine.evaluateAllRules()) {
        System.out.println(violation.getMessage());
    }
}

static void enforce(Object cell, String[] rules) {
    RuleEngine.enforce(cell, rules, "status");   // throws RuleViolationException
}

• The state field defaults to state. evaluateAllRules(field) checks against another field for one call.
• The effect is compared with the field's value, or with an enum constant's name, ignoring case.
• An unchanged rule documents a case and never fails.
• Rules can also be written as field = VALUE when condition.
• A rule that does not parse is logged and skipped, and so is a condition that cannot be evaluated.
• Each RuleViolation has getRule(), getExpectedValue(), getActualValue() and getMessage().
• RuleEngine.from(target, rules) builds an engine from an array. enforce throws RuleEngine.RuleViolationException, whose getViolations() lists them.

The framework does not run rules for you. A rule in @State(invariants) reads like a sentence, so the invariant check skips it. Call RuleEngine after the state changes.

Related

• Contracts
• Enforce permissions and content rules
• Guardrails
• Evaluation