# Supervision and safeguards

`tnsai-coordination` has two kinds of safeguards for agent groups. A `SupervisorAgent` answers "may this agent do this?" through a list of guardrails, and an `AgentRecoveryManager` watches agent health. Separate utilities isolate failing agents, cap token spend, keep an episode log and run a task in a sandbox.

## Supervisor guardrails

`SupervisorAgent` in `com.tnsai.coordination.supervisor` wraps an `AgentGroup` and a list of `Guardrail`s. It does not intercept agent calls: your code asks `checkAction(agentId, action, context)` before it lets an agent act, and acts on the answer.

This `Guardrail` is about action permission. It is not `@InputGuardrail` and not the content-evaluation `Guardrail` used by tracing; see [Guardrails](../security/guardrails.md) for the three kinds.

A guardrail implements `getName()` and `check(agentId, action, context)`, and returns a `GuardrailResult(allowed, reason, severity)`:

| Factory | `allowed` | `severity` |
|---|---|---|
| `GuardrailResult.allow()` | true | `INFO` |
| `GuardrailResult.warning(reason)` | true | `WARNING` |
| `GuardrailResult.blocked(reason)` | false | `BLOCK` |

`checkAction` runs the guardrails in order:

- With no guardrails, it allows.
- The first result that is not allowed is returned at once, and later guardrails do not run.
- A guardrail that throws counts as a block with reason `Guardrail error: …`.
- A warning is logged and evaluation continues. If every guardrail allows, the result is the last guardrail's, so a warning from an earlier guardrail is not in the returned result.

`addGuardrail(guardrail)` appends one, `removeGuardrail(name)` removes every guardrail with that name, and `getGuardrails()` returns a copy of the list.

### Human approval

`HITLApproval` is a guardrail that asks a person. Create it with a set of action patterns and a callback, and optionally a name (default `hitl-approval`). An action matches when it contains any pattern, ignoring case, so the pattern `deploy` also matches `redeploy-cache`. Non-matching actions are allowed.

For a matching action, the guardrail calls the callback with an `ApprovalRequest(agentId, action, context, approval)` and then blocks the calling thread until the `approval` future completes. Complete it with `true` to allow or `false` to block. There is no timeout, so make sure every request gets an answer. A callback exception or an exceptionally completed future blocks the action. `addPattern` and `removePattern` change the patterns at runtime.

<!-- java-contract: src/main/java/com/example/tnsai/docs/SupervisionExample.java -->
```java
package com.example.tnsai.docs;

import com.tnsai.agents.groups.AgentGroup;
import com.tnsai.coordination.supervisor.AgentRecoveryManager;
import com.tnsai.coordination.supervisor.AgentRecoveryManager.HealthStatus;
import com.tnsai.coordination.supervisor.AgentRecoveryManager.RecoveryAction;
import com.tnsai.coordination.supervisor.AgentRecoveryManager.RecoveryPolicy;
import com.tnsai.coordination.supervisor.AgentRecoveryManager.RecoveryResult;
import com.tnsai.coordination.supervisor.Guardrail;
import com.tnsai.coordination.supervisor.Guardrail.GuardrailResult;
import com.tnsai.coordination.supervisor.HITLApproval;
import com.tnsai.coordination.supervisor.SupervisorAgent;

import java.time.Duration;
import java.util.List;
import java.util.Map;
import java.util.Set;

/** Compilation contract for multi-agent/supervision.md supervisor and recovery. */
public final class SupervisionExample {

    private SupervisionExample() {
    }

    /** Blocks any action whose context carries a cost above a fixed cap. */
    static final class CostCap implements Guardrail {
        @Override
        public String getName() {
            return "cost-cap";
        }

        @Override
        public GuardrailResult check(String agentId, String action, Map<String, Object> context) {
            Object cost = context.get("cost");
            if (cost instanceof Number n && n.doubleValue() > 100.0) {
                return GuardrailResult.blocked("Cost " + n + " is over the cap");
            }
            return GuardrailResult.allow();
        }
    }

    static boolean mayDeploy(AgentGroup group) {
        HITLApproval approval = new HITLApproval(Set.of("deploy", "delete"), request -> {
            // Ask a person here; complete the future with their answer.
            System.out.println(request.agentId() + " wants to " + request.action());
            request.approval().complete(Boolean.TRUE);
        });

        SupervisorAgent supervisor = new SupervisorAgent(group, List.of(new CostCap(), approval));

        GuardrailResult result = supervisor.checkAction(
            "release-agent", "deploy", Map.of("target", "production", "cost", 40));
        if (!result.allowed()) {
            System.out.println("Blocked: " + result.reason() + " (" + result.severity() + ")");
        }
        return result.allowed();
    }

    static AgentRecoveryManager monitor(SupervisorAgent supervisor) {
        AgentRecoveryManager recovery = new AgentRecoveryManager(supervisor, RecoveryPolicy.builder()
            .maxRetries(3)
            .escalateAfter(2)
            .cooldown(Duration.ofSeconds(10))
            .defaultAction(RecoveryAction.REPLACE)
            .build());

        recovery.addHealthCheck("worker-1", agent -> !agent.chat("ping").isBlank());
        recovery.addListener(new AgentRecoveryManager.RecoveryListener() {
            @Override
            public void onHealthCheckFailed(HealthStatus status) {
                System.out.println("Unhealthy: " + status.agentId() + " - " + status.reason());
            }

            @Override
            public void onRecoveryAttempt(RecoveryResult result) {
                System.out.println(result.actionTaken() + ": " + result.detail());
            }

            @Override
            public void onEscalation(String agentId, int failedAttempts) {
                System.out.println("Escalate " + agentId + " after " + failedAttempts + " attempts");
            }
        });

        recovery.start(Duration.ofSeconds(30));   // call stop() when done
        return recovery;
    }
}
```

## Health checks and recovery

`AgentRecoveryManager` takes the supervisor and a `RecoveryPolicy`. Register a `HealthCheck` per agent id with `addHealthCheck(agentId, check)`; a check is a predicate on the `Agent`. `start(interval)` runs every check on a background thread at a fixed rate until `stop()`. `runHealthChecks()` runs them once and returns a `HealthStatus` per agent. Agents that are no longer members of the group are skipped.

A failed or throwing check, or a call to `triggerRecovery(agentId, reason)`, starts a recovery attempt:

1. Within the cooldown of the agent's previous attempt, nothing happens and a failed `RecoveryResult` is returned.
2. Otherwise the action is chosen from the agent's retry count: `EJECT` once it reaches `maxRetries`, `ESCALATE` once it reaches `escalateAfter`, and the policy's `defaultAction` before that.
3. The retry count goes up by one, and the result is added to the history and sent to listeners.

| `RecoveryPolicy` builder | Default |
|---|---|
| `maxRetries` | 3 |
| `escalateAfter` | 3 |
| `cooldown` | 5 seconds |
| `defaultAction` | `RESTART` |

With the defaults, `maxRetries` and `escalateAfter` are equal, so an agent is ejected without being escalated. Set `escalateAfter` below `maxRetries` to escalate first.

| `RecoveryAction` | Effect |
|---|---|
| `EJECT` | Removes the agent from the group, with its health check and retry count |
| `REPLACE` | The same removal; adding a replacement is up to you |
| `RESTART`, `RESTART_CLEAN` | Reported as successful but change nothing; react to them in a listener |
| `ESCALATE` | Changes nothing and calls `onEscalation` on the listeners |

A `RecoveryListener` has `onHealthCheckFailed`, `onRecoveryAttempt` and `onEscalation`, and all three must be implemented. The retry count is not reset when an agent becomes healthy again; call `resetRetryCounter(agentId)` for that. `getRecoveryHistory()` and `getRetryCount(agentId)` report what happened.

## Production safeguards

The utilities in `com.tnsai.coordination.production`, and `SandboxedExecution`, are independent of the supervisor. Use them around your own agent calls.

### Failure isolation

`FailureIsolator` keeps a circuit breaker per agent id. `execute(agentId, action, fallback)` runs the action:

- A success resets the failure count and closes the circuit.
- A failure below `failureThreshold` is rethrown.
- The failure that reaches the threshold opens the circuit and returns the degraded value instead of throwing.
- While the circuit is `OPEN`, calls return the degraded value without running the action.
- After `recoveryTimeout` the circuit is `HALF_OPEN` and calls run the action again. A success closes it; a failure opens it again.

| `DegradationStrategy` | Degraded value |
|---|---|
| `FALLBACK` (default) | The fallback supplier's value, or null without one |
| `SKIP` | null |
| `CACHED` | The agent's last successful result, or null |
| `FAIL_FAST` | Throws `FailureIsolator.CircuitOpenException` |

Builder defaults are a threshold of 3 failures and a 30-second recovery timeout. `state(agentId)`, `failureCount(agentId)` and `reset(agentId)` inspect and reset a circuit.

### Token budgets

`TokenBudget` tracks token use per agent against an allocation and a global budget. It records and reports; it does not stop anything.

- `allocate(agentId, tokens)` sets an agent's allocation. Allocating again replaces it and starts the agent's count from zero.
- `consume(agentId, tokens)` adds to the agent's and the global count. It throws `IllegalStateException` for an agent with no allocation, and it records usage beyond the budget, logging a warning from 90% usage.
- `canConsume(agentId, tokens)` is true when both the agent's allocation and the global budget have room. Check it before each call.
- `remaining`, `globalRemaining`, `usagePercent`, `globalUsagePercent` and `allAllocations` report the state.

### Episodic memory

`EpisodicMemory(maxEpisodes)` is a bounded log of `Episode(taskId, agentId, taskType, outcome, metadata, timestamp)` records. `Episode.of(...)` stamps the current time. When the log is full, `record` drops the oldest episode. The queries return the newest episodes first, up to a limit:

- `querySimilar(taskType, limit)`: the task type contains the text, ignoring case;
- `queryByAgent(agentId, limit)`;
- `queryByOutcome(outcome, limit)`: the outcome equals the text, ignoring case.

### Sandboxed execution

`SandboxedExecution.execute(agent, task, config)` runs `agent.chat(task)` on its own thread and waits up to the config's timeout. It returns a `SandboxResult(result, completed, duration, error)`. A timeout, interruption or exception gives `completed = false` with an error message; it does not throw.

`SandboxConfig(timeout, maxMemoryBytes, allowedActions, readOnly)` has `defaults()` of 30 seconds, 0, no actions and not read-only. Only `timeout` and `readOnly` have an effect. `readOnly` prefixes the task with `[READ-ONLY MODE]`; it is an instruction to the model, not an enforced restriction. To run untrusted code in isolation, see [Sandbox](../security/sandbox.md).

<!-- java-contract: src/main/java/com/example/tnsai/docs/ProductionHardeningExample.java -->
```java
package com.example.tnsai.docs;

import com.tnsai.agents.Agent;
import com.tnsai.coordination.production.EpisodicMemory;
import com.tnsai.coordination.production.EpisodicMemory.Episode;
import com.tnsai.coordination.production.FailureIsolator;
import com.tnsai.coordination.production.FailureIsolator.DegradationStrategy;
import com.tnsai.coordination.production.TokenBudget;
import com.tnsai.coordination.supervisor.SandboxedExecution;
import com.tnsai.coordination.supervisor.SandboxedExecution.SandboxConfig;
import com.tnsai.coordination.supervisor.SandboxedExecution.SandboxResult;

import java.time.Duration;
import java.util.List;
import java.util.Map;
import java.util.Set;

/** Compilation contract for multi-agent/supervision.md production safeguards. */
public final class ProductionHardeningExample {

    private ProductionHardeningExample() {
    }

    static String answer(Agent agent, String question) {
        FailureIsolator isolator = FailureIsolator.builder()
            .failureThreshold(3)
            .recoveryTimeout(Duration.ofSeconds(30))
            .degradationStrategy(DegradationStrategy.FALLBACK)
            .build();

        TokenBudget budget = new TokenBudget(100_000);
        budget.allocate(agent.getId(), 30_000);

        EpisodicMemory memory = new EpisodicMemory(1_000);

        if (!budget.canConsume(agent.getId(), 2_000)) {
            return "Budget exhausted";
        }

        String reply = isolator.execute(agent.getId(),
            () -> agent.chat(question),
            () -> "The assistant is unavailable, try again later.");

        budget.consume(agent.getId(), 2_000);   // record what the call used
        memory.record(Episode.of("q-1", agent.getId(), "faq", "success", Map.of("chars", "" + reply.length())));

        List<Episode> recent = memory.queryByAgent(agent.getId(), 10);
        System.out.println(recent.size() + " episodes, circuit " + isolator.state(agent.getId())
            + ", " + budget.remaining(agent.getId()) + " tokens left");
        return reply;
    }

    static SandboxResult dryRun(Agent agent, String task) {
        SandboxResult result = SandboxedExecution.execute(agent, task,
            new SandboxConfig(Duration.ofSeconds(20), 0, Set.of(), true));
        if (!result.completed()) {
            System.out.println("Failed after " + result.duration() + ": " + result.error());
        }
        return result;
    }
}
```

## Related

- [Guardrails](../security/guardrails.md)
- [Run code in a sandbox](../security/sandbox.md)
- [Shared state and trust](shared-state.md)
- [Fan-out and agent pools](fan-out.md)