Learning and refinement

For typed output with field rules and retries, see StructuredOutputExecutor in Structured output.

Feedback

Feedback on an agent

With tnsai-intelligence on the classpath, every agent gets a feedback collector. Without it, the collector is a no-op.

FeedbackCollector feedback = agent.getFeedbackCollector();

feedback.recordUserFeedback("CORRECTION", "Use a formal tone", null);
feedback.recordUserFeedback("PREFERENCE", "Include code examples", Map.of("user", "u-42"));

The type is POSITIVE, NEGATIVE, CORRECTION or PREFERENCE, in any case; another value throws IllegalArgumentException.

Once the collector holds any feedback, the agent's system prompt gets a # Learned Behavior section. ## Rules lists the content of every CORRECTION, and ## Preferences lists the content of every PREFERENCE. The text is used as given; no LLM is called.

The agent also records each tool call: a success as POSITIVE, a failure as NEGATIVE with the error. These do not appear in the prompt.

The collector keeps feedback in memory for the agent's lifetime. Nothing is persisted.

Feedback

Feedback (package com.tnsai.learning) is a record: id, type, content, sessionId, timestamp and metadata. A null id or timestamp is filled in.

Feedback up = Feedback.thumbsUp("Clear explanation");      // POSITIVE
Feedback down = Feedback.thumbsDown("Too long");           // NEGATIVE
Feedback fix = Feedback.correction("Use a formal tone");   // CORRECTION
Feedback pref = Feedback.preference("Include examples");   // PREFERENCE

Feedback inSession = up.withSessionId("session-123");

FeedbackStore

FeedbackStore.inMemory() is the only implementation. Implement the interface to keep feedback elsewhere.

FeedbackLearner

FeedbackLearner asks an LLM to turn feedback into rules and preferences:

FeedbackStore store = FeedbackStore.inMemory();

FeedbackLearner learner = FeedbackLearner.builder()
    .llm(client)
    .feedbackStore(store)
    .build();

learner.recordFeedback(Feedback.correction("Use a formal tone"));
learner.recordFeedback(Feedback.thumbsDown("Response was too long"));
learner.recordFeedback(Feedback.preference("Include code examples"));

FeedbackLearner.LearningResult result = learner.learn();
result.promptAdjustments();
result.preferences();
result.goodExamples();

learn() reads every stored item:

A failed call leaves that list empty. feedbackAnalyzed() is the number of items read, and hasLearnings() is true when any list has entries.

AdaptivePromptEnricher caches a learner's result and appends it to a prompt:

AdaptivePromptEnricher enricher = new AdaptivePromptEnricher(learner, store);
String systemPrompt = enricher.enrich("You are a support assistant.");

enrich adds ## Learned Preferences with ### Rules and ### Preferences. It calls learn() again only when the store's count has changed. The agent's own collector does not use an enricher.

Norms

NormEngine (package com.tnsai.norms) checks actions against norms you declare. The agent does not call it; call it yourself before an action.

Declare norms on a class with @Norm (repeatable) or @Norms, from com.tnsai.annotations:

@Norm(type = NormType.PROHIBITION, action = "sharePersonalData",
      condition = "hasConsent == false",
      description = "Cannot share personal data without consent", priority = 10)
@Norm(type = NormType.OBLIGATION, action = "logAccess",
      description = "Must log all data access", priority = 5)
public class DataAgent { }

Build the engine from the class, or from NormEntry records:

NormEngine engine = NormEngine.fromAnnotations(DataAgent.class);

NormEngine other = NormEngine.of(
    new NormEntry(NormType.PROHIBITION, "hasConsent == false",
        "sharePersonalData", "No sharing without consent", 10));

engine.addNorm(new NormEntry(NormType.PROHIBITION, "",
    "deleteProduction", "Never delete production data", 100));

NormEntry is (type, condition, action, description, priority).

The engine does not interpret conditions. Each method takes a Predicate<String> that gets the condition text. ConditionEvaluator.evaluate(condition, state) in com.tnsai.planning evaluates these expressions against a state map:

Map<String, Object> state = Map.of("hasConsent", false);
Predicate<String> active = condition -> ConditionEvaluator.evaluate(condition, state);

NormEngine.CheckResult check = engine.checkAction("sharePersonalData", active);
if (check.isViolation()) {
    check.violations().forEach(v -> log.warn(v.reason()));
}

NormEngine.CheckResult missing = engine.checkObligations(Set.of("readData"), active);

A PERMISSION never overrides a prohibition; checkAction ignores permissions. CheckResult has violations(), isViolation() and isAllowed(). A NormViolation has norm(), action(), reason() and timestamp().

Refinement loop

RefinementLoop (package com.tnsai.refinement) checks an output, and while checks fail, asks the model to fix it:

RefinementLoop loop = RefinementLoop.builder()
    .task("Convert this Python module to TypeScript")
    .completionCriteria(CompletionCriteria.builder()
        .compilerCheck("tsc --noEmit $OUTPUT")
        .mustContain("export")
        .mustNotContain("def ")
        .build())
    .maxIterations(5)
    .timeout(Duration.ofMinutes(10))
    .build();

RefinementResult result = loop.execute(agent, firstDraft);

if (result.isSuccess()) {
    String code = result.finalOutput();
}

execute(agent, input) uses agent.chat; execute(llmClient, input) uses chat on the client. The input is the first output to check, not a request: pass a draft.

Each iteration:

1. Stops with TIMEOUT if the time limit has passed.
2. Runs the criteria on the current output.
3. Stops with SUCCESS if every required check passed.
4. Stops with STOPPED if a stop hook returns true.
5. Sends a prompt with the task, the current output, the failed checks and the passing ones, and takes the reply as the next output. If the call throws, it stops with ERROR.

After maxIterations checks without success, the status is MAX_ITERATIONS. The output from the last call is returned without being checked.

RefinementResult is a record: finalOutput(), iterationCount(), status(), totalDuration() and metrics(), plus isSuccess() and getFinalScore(). metrics() has totalIterations, totalDurationMs, totalTokens, scoreProgression (a double[]) and finalScore. A score is the share of checks that passed.

CompletionCriteria

The three command checks run sh -c with the output written to a temporary file; $OUTPUT in the command is replaced by its path. A command fails after 30 seconds. Its output is discarded; only the exit code counts.

llmCheck needs withLLM(client) earlier in the builder, or it throws. A reply without SCORE: fails the check.

Stop hooks

The hooks with counters keep state; build a new one for each loop.

Prompt optimizer

PromptOptimizer (package com.tnsai.prompt.optimizer) tries prompt variants against test cases and keeps the best:

PromptOptimizer optimizer = PromptOptimizer.builder()
    .llmClient(client)
    .maxIterations(5)
    .targetScore(0.9f)
    .build();

List<TestCase> cases = List.of(
    new TestCase("What is 2+2?", "4"),
    new TestCase("Capital of France?", "Paris"));

OptimizationResult result = optimizer.optimize("Answer the question:", cases);
result.getBestPrompt();
result.getScore();
result.wasImproved();

A prompt's score is the average over the test cases. Each case sends the prompt and the input in one call. A case whose call fails is left out.

Each round applies every strategy to the current best prompt, then asks the model to rewrite it from scores on the first three cases. Any variant that scores higher becomes the best. The loop stops at the target score, at maxIterations, or when the last three scores are within 0.01.

DefaultPromptEvaluator gives 1 for a match after normalizing. Otherwise it mixes keyword overlap (0.5), length similarity (0.2) and phrase overlap (0.3). Implement PromptEvaluator.evaluate(actual, expected) for another measure. Every round makes several calls per test case, so keep the cases few.

suggestStrategies(taskDescription) picks strategies from keywords. For example, "analyze" gives CHAIN_OF_THOUGHT and STRUCTURED_THINKING.

Related

• Reasoning: prompt strategies.
• Context: sessions, knowledge extraction and memory consolidation.
• Planning: planners.
• Structured output: typed replies with field rules.