# Learning and refinement

These classes are in `tnsai-intelligence`:

| Group | Classes | Use it to |
|---|---|---|
| [Feedback](#feedback) | `Feedback`, `FeedbackStore`, `FeedbackLearner`, `AdaptivePromptEnricher` | Collect feedback and turn it into prompt rules |
| [Norms](#norms) | `NormEngine`, `@Norm` | Check actions against obligations and prohibitions |
| [Refinement loop](#refinement-loop) | `RefinementLoop`, `CompletionCriteria`, `StopHook` | Revise an output until checks pass |
| [Prompt optimizer](#prompt-optimizer) | `PromptOptimizer`, `TestCase` | Score prompt variants against test cases |

For typed output with field rules and retries, see `StructuredOutputExecutor` in [Structured output](../agents/structured-output.md).

## Feedback

### Feedback on an agent

With `tnsai-intelligence` on the classpath, every agent gets a feedback collector. Without it, the collector is a no-op.

```java
FeedbackCollector feedback = agent.getFeedbackCollector();

feedback.recordUserFeedback("CORRECTION", "Use a formal tone", null);
feedback.recordUserFeedback("PREFERENCE", "Include code examples", Map.of("user", "u-42"));
```

The type is `POSITIVE`, `NEGATIVE`, `CORRECTION` or `PREFERENCE`, in any case; another value throws `IllegalArgumentException`.

Once the collector holds any feedback, the agent's system prompt gets a `# Learned Behavior` section. `## Rules` lists the content of every `CORRECTION`, and `## Preferences` lists the content of every `PREFERENCE`. The text is used as given; no LLM is called.

The agent also records each tool call: a success as `POSITIVE`, a failure as `NEGATIVE` with the error. These do not appear in the prompt.

The collector keeps feedback in memory for the agent's lifetime. Nothing is persisted.

### `Feedback`

`Feedback` (package `com.tnsai.learning`) is a record: `id`, `type`, `content`, `sessionId`, `timestamp` and `metadata`. A null `id` or `timestamp` is filled in.

```java
Feedback up = Feedback.thumbsUp("Clear explanation");      // POSITIVE
Feedback down = Feedback.thumbsDown("Too long");           // NEGATIVE
Feedback fix = Feedback.correction("Use a formal tone");   // CORRECTION
Feedback pref = Feedback.preference("Include examples");   // PREFERENCE

Feedback inSession = up.withSessionId("session-123");
```

### `FeedbackStore`

| Method | Returns |
|---|---|
| `save(feedback)` | |
| `getAll()`, `getByType(type)`, `getBySession(sessionId)` | `List<Feedback>` |
| `getRecent(limit)` | The last `limit` saved |
| `count()`, `clear()` | |

`FeedbackStore.inMemory()` is the only implementation. Implement the interface to keep feedback elsewhere.

### `FeedbackLearner`

`FeedbackLearner` asks an LLM to turn feedback into rules and preferences:

```java
FeedbackStore store = FeedbackStore.inMemory();

FeedbackLearner learner = FeedbackLearner.builder()
    .llm(client)
    .feedbackStore(store)
    .build();

learner.recordFeedback(Feedback.correction("Use a formal tone"));
learner.recordFeedback(Feedback.thumbsDown("Response was too long"));
learner.recordFeedback(Feedback.preference("Include code examples"));

FeedbackLearner.LearningResult result = learner.learn();
result.promptAdjustments();
result.preferences();
result.goodExamples();
```

| Builder | Default | Meaning |
|---|---|---|
| `llm(LLMClient)` | Required | Client for the extraction calls |
| `feedbackStore(FeedbackStore)` | Required | Where feedback is read and saved |
| `strategies(List<LearningStrategy>)` | All three | What to learn |
| `minFeedbackForLearning(int)` | `3` | Below this count, `learn()` returns an empty result |

`learn()` reads every stored item:

| `LearningStrategy` | Reads | Fills | LLM call |
|---|---|---|---|
| `PROMPT_ADJUSTMENT` | `CORRECTION`, `NEGATIVE` | `promptAdjustments()` | Yes; keeps lines starting `RULE:` |
| `PREFERENCE_LEARNING` | `PREFERENCE`, `CORRECTION` | `preferences()` | Yes; keeps lines starting `PREFERENCE:` |
| `EXAMPLE_COLLECTION` | `POSITIVE` | `goodExamples()` | No; the content as given |

A failed call leaves that list empty. `feedbackAnalyzed()` is the number of items read, and `hasLearnings()` is true when any list has entries.

`AdaptivePromptEnricher` caches a learner's result and appends it to a prompt:

```java
AdaptivePromptEnricher enricher = new AdaptivePromptEnricher(learner, store);
String systemPrompt = enricher.enrich("You are a support assistant.");
```

`enrich` adds `## Learned Preferences` with `### Rules` and `### Preferences`. It calls `learn()` again only when the store's count has changed. The agent's own collector does not use an enricher.

## Norms

`NormEngine` (package `com.tnsai.norms`) checks actions against norms you declare. The agent does not call it; call it yourself before an action.

Declare norms on a class with `@Norm` (repeatable) or `@Norms`, from `com.tnsai.annotations`:

```java
@Norm(type = NormType.PROHIBITION, action = "sharePersonalData",
      condition = "hasConsent == false",
      description = "Cannot share personal data without consent", priority = 10)
@Norm(type = NormType.OBLIGATION, action = "logAccess",
      description = "Must log all data access", priority = 5)
public class DataAgent { }
```

| `@Norm` | Default | Meaning |
|---|---|---|
| `type` | Required | `OBLIGATION`, `PROHIBITION` or `PERMISSION` |
| `action` | Required | The action name it applies to |
| `condition` | `""` | When the norm is active; blank means always |
| `description` | `""` | Used in violation messages |
| `priority` | `0` | Orders `getActiveObligations` |

Build the engine from the class, or from `NormEntry` records:

```java
NormEngine engine = NormEngine.fromAnnotations(DataAgent.class);

NormEngine other = NormEngine.of(
    new NormEntry(NormType.PROHIBITION, "hasConsent == false",
        "sharePersonalData", "No sharing without consent", 10));

engine.addNorm(new NormEntry(NormType.PROHIBITION, "",
    "deleteProduction", "Never delete production data", 100));
```

`NormEntry` is `(type, condition, action, description, priority)`.

The engine does not interpret conditions. Each method takes a `Predicate<String>` that gets the condition text. `ConditionEvaluator.evaluate(condition, state)` in `com.tnsai.planning` evaluates these expressions against a state map:

```java
Map<String, Object> state = Map.of("hasConsent", false);
Predicate<String> active = condition -> ConditionEvaluator.evaluate(condition, state);

NormEngine.CheckResult check = engine.checkAction("sharePersonalData", active);
if (check.isViolation()) {
    check.violations().forEach(v -> log.warn(v.reason()));
}

NormEngine.CheckResult missing = engine.checkObligations(Set.of("readData"), active);
```

| Method | Result |
|---|---|
| `checkAction(action, predicate)` | A violation for each active `PROHIBITION` on that action |
| `checkObligations(performedActions, predicate)` | A violation for each active `OBLIGATION` whose action is not in the set |
| `getActiveObligations(predicate)` | Active obligations, highest priority first |
| `getActiveProhibitions(predicate)`, `getActivePermissions(predicate)` | Active norms of that type |
| `getAllNorms()` | Every norm |

A `PERMISSION` never overrides a prohibition; `checkAction` ignores permissions. `CheckResult` has `violations()`, `isViolation()` and `isAllowed()`. A `NormViolation` has `norm()`, `action()`, `reason()` and `timestamp()`.

## Refinement loop

`RefinementLoop` (package `com.tnsai.refinement`) checks an output, and while checks fail, asks the model to fix it:

```java
RefinementLoop loop = RefinementLoop.builder()
    .task("Convert this Python module to TypeScript")
    .completionCriteria(CompletionCriteria.builder()
        .compilerCheck("tsc --noEmit $OUTPUT")
        .mustContain("export")
        .mustNotContain("def ")
        .build())
    .maxIterations(5)
    .timeout(Duration.ofMinutes(10))
    .build();

RefinementResult result = loop.execute(agent, firstDraft);

if (result.isSuccess()) {
    String code = result.finalOutput();
}
```

`execute(agent, input)` uses `agent.chat`; `execute(llmClient, input)` uses `chat` on the client. The input is the first output to check, not a request: pass a draft.

Each iteration:

1. Stops with `TIMEOUT` if the time limit has passed.
2. Runs the criteria on the current output.
3. Stops with `SUCCESS` if every required check passed.
4. Stops with `STOPPED` if a stop hook returns true.
5. Sends a prompt with the task, the current output, the failed checks and the passing ones, and takes the reply as the next output. If the call throws, it stops with `ERROR`.

After `maxIterations` checks without success, the status is `MAX_ITERATIONS`. The output from the last call is returned without being checked.

| Builder | Default | Meaning |
|---|---|---|
| `task(String)` | Required | Included in every revision prompt |
| `completionCriteria(CompletionCriteria)` | Required | The checks |
| `maxIterations(int)` | `10` | Values below 1 throw |
| `timeout(Duration)` | 30 minutes | Checked at the start of each iteration |
| `stopHook(StopHook)` | None | Can be called more than once |

`RefinementResult` is a record: `finalOutput()`, `iterationCount()`, `status()`, `totalDuration()` and `metrics()`, plus `isSuccess()` and `getFinalScore()`. `metrics()` has `totalIterations`, `totalDurationMs`, `totalTokens`, `scoreProgression` (a `double[]`) and `finalScore`. A score is the share of checks that passed.

### `CompletionCriteria`

| Builder | Passes when |
|---|---|
| `compilerCheck(cmd)`, `testCommand(cmd)`, `lintCheck(cmd)` | The command exits 0 |
| `mustContain(strings...)` | Every string appears, case-sensitive |
| `mustNotContain(strings...)` | No string appears |
| `matchesPattern(regex)` | The pattern is found |
| `wordCount(min, max)` | The word count is in range |
| `minLines(min)` | The output has at least `min` lines |
| `validJson()` | The output parses as JSON |
| `customCheck(name, predicate)`, `customCheck(name, predicate, failMessage)` | The `Predicate<String>` is true |
| `llmCheck(question, threshold)` | The model's `SCORE:` from 0 to 100, divided by 100, is at least `threshold` |

The three command checks run `sh -c` with the output written to a temporary file; `$OUTPUT` in the command is replaced by its path. A command fails after 30 seconds. Its output is discarded; only the exit code counts.

`llmCheck` needs `withLLM(client)` earlier in the builder, or it throws. A reply without `SCORE:` fails the check.

### Stop hooks

| `StopHook` | Stops when |
|---|---|
| `onScoreAbove(threshold)` | The score is at least `threshold` |
| `onNoImprovement(n)` | The score has not risen for `n` iterations in a row |
| `onCostExceeds(maxTokens)` | The estimated tokens of all outputs so far exceed `maxTokens` |
| `onCompileSuccess()` | A passed check's name contains `compile` |
| `any(hooks...)`, `all(hooks...)` | Any or all of the hooks would stop |

The hooks with counters keep state; build a new one for each loop.

## Prompt optimizer

`PromptOptimizer` (package `com.tnsai.prompt.optimizer`) tries prompt variants against test cases and keeps the best:

```java
PromptOptimizer optimizer = PromptOptimizer.builder()
    .llmClient(client)
    .maxIterations(5)
    .targetScore(0.9f)
    .build();

List<TestCase> cases = List.of(
    new TestCase("What is 2+2?", "4"),
    new TestCase("Capital of France?", "Paris"));

OptimizationResult result = optimizer.optimize("Answer the question:", cases);
result.getBestPrompt();
result.getScore();
result.wasImproved();
```

| Builder | Default | Meaning |
|---|---|---|
| `llmClient(LLMClient)` | Required | Runs the test cases and the rewrites |
| `maxIterations(int)` | `5` | Rounds after the first score |
| `targetScore(float)` | `0.9` | Stop once the best score reaches it |
| `candidateStrategies(List<PromptStrategy>)` | `CHAIN_OF_THOUGHT`, `CHAIN_OF_VERIFICATION`, `CONFIDENCE_WEIGHTED`, `STRUCTURED_THINKING` | Strategies to try |
| `evaluator(PromptEvaluator)` | `DefaultPromptEvaluator` | Scores a reply against the expected output |

A prompt's score is the average over the test cases. Each case sends the prompt and the input in one call. A case whose call fails is left out.

Each round applies every strategy to the current best prompt, then asks the model to rewrite it from scores on the first three cases. Any variant that scores higher becomes the best. The loop stops at the target score, at `maxIterations`, or when the last three scores are within 0.01.

`DefaultPromptEvaluator` gives 1 for a match after normalizing. Otherwise it mixes keyword overlap (0.5), length similarity (0.2) and phrase overlap (0.3). Implement `PromptEvaluator.evaluate(actual, expected)` for another measure. Every round makes several calls per test case, so keep the cases few.

`suggestStrategies(taskDescription)` picks strategies from keywords. For example, "analyze" gives `CHAIN_OF_THOUGHT` and `STRUCTURED_THINKING`.

## Related

- [Reasoning](reasoning.md): prompt strategies.
- [Context](context.md): sessions, knowledge extraction and memory consolidation.
- [Planning](planning.md): planners.
- [Structured output](../agents/structured-output.md): typed replies with field rules.