# Sampling

Production agents emit many events per second: chat chunks, tool calls, memory writes, hook invocations. Shipping all of them is expensive, and keeping only errors makes debugging hard. The sampling SPI in `tnsai-quality` (`com.tnsai.quality.sampling`) decides per event whether to capture, drop or down-sample, from "capture everything" in development to "errors always, a share of the rest" in production.

The SPI is `EventSamplingPolicy`. Default implementations (`AlwaysCapturePolicy`, `HeadBasedSamplingPolicy`, `PriorityBasedSamplingPolicy`, `ErrorAlwaysPolicy`) ship with the framework. Operators wire a policy into their event publisher via the `SamplingAgentEventPublisher` decorator.

## Quick start

`SamplingAgentEventPublisher` implements `com.tnsai.observability.events.AgentEventPublisher` (the logging fan-out SPI). That is a different type from `com.tnsai.events.TnsAIEventPublisher`, which is what `Agent.setEventPublisher(...)` accepts. Wrap the observability publisher, and do not pass the sampler to the agent.

<!-- java-contract: src/main/java/com/example/tnsai/docs/SamplingPublisherExample.java -->
```java
package com.example.tnsai.docs;

import com.tnsai.observability.events.AgentEventPublisher;
import com.tnsai.observability.events.Slf4jAgentEventPublisher;
import com.tnsai.quality.sampling.EventSamplingPolicy;
import com.tnsai.quality.sampling.SamplingAgentEventPublisher;
import com.tnsai.quality.sampling.SamplingPolicies;

public final class SamplingPublisherExample {
    private SamplingPublisherExample() {}

    public static AgentEventPublisher productionSampler() {
        EventSamplingPolicy policy = SamplingPolicies.production();
        AgentEventPublisher slf4j = new Slf4jAgentEventPublisher();
        return new SamplingAgentEventPublisher(slf4j, policy);
    }
}
```

## Decision shape

`SamplingDecision` is a sealed interface with three variants. Java 21 exhaustive `switch` makes the publisher's dispatch compile-checked:

| Decision | Meaning |
|---|---|
| `Capture` | Capture at full fidelity. No downstream rate adjustment needed. |
| `Drop` | Discard the event. Never reaches the underlying publisher. |
| `CaptureDownsampled(rate)` | Capture, but record the configured rate so downstream aggregators (Prometheus, Datadog) can extrapolate volume (`count += 1 / rate`). |

`Capture.INSTANCE` and `Drop.INSTANCE` are reusable singletons, so most policies don't allocate a fresh decision per call.

## Default policies

### `AlwaysCapturePolicy`

The dev / single-tenant default: every event captured. Use the singleton: `AlwaysCapturePolicy.INSTANCE`.

```java
EventSamplingPolicy verbose = AlwaysCapturePolicy.INSTANCE;
```

### `HeadBasedSamplingPolicy`

Coin-flip per event at a fixed rate. Stateless, lock-free, the simplest production policy. Returns `CaptureDownsampled(rate)` when captured.

```java
EventSamplingPolicy p = new HeadBasedSamplingPolicy(0.10);   // 10%
```

Trade-off: no per-session affinity. A captured chat may have some chunks captured and others not.

### `PriorityBasedSamplingPolicy`

Per-`EventLevel` rate. Default rates encode the production observability rule of thumb: never drop anything that's already a problem:

| Level | Default rate |
|---|---|
| `CRITICAL` | 1.0 (always) |
| `ERROR` | 1.0 (always) |
| `WARN` | 1.0 (always) |
| `INFO` | 0.1 (10%) |
| `DEBUG` | 0.01 (1%) |

```java
// Defaults
EventSamplingPolicy p = PriorityBasedSamplingPolicy.withDefaults();

// Custom rates per level
Map<EventLevel, Double> overrides = Map.of(
        EventLevel.INFO, 0.5,
        EventLevel.DEBUG, 0.0           // hard-drop DEBUG
);
EventSamplingPolicy p = new PriorityBasedSamplingPolicy(overrides);
```

### `ErrorAlwaysPolicy`

Simpler than `PriorityBasedSamplingPolicy`: one switch on level, no per-level map:

- `WARN` / `ERROR` / `CRITICAL` → always `Capture`
- `INFO` / `DEBUG` → coin-flip at `infoDebugRate`

```java
EventSamplingPolicy p = new ErrorAlwaysPolicy(0.05);     // 5% INFO/DEBUG
EventSamplingPolicy d = ErrorAlwaysPolicy.withDefault(); // 10% INFO/DEBUG
```

Use when "is this a problem?" is the only distinction that matters for capture.

### `RuleBasedSamplingPolicy`

Ordered list of declarative rules, first match wins. Each rule combines an event-kind glob, a minimum level, an optional tenant filter, and an outcome (`CAPTURE` / `DROP` / `SAMPLE_AT_RATE`).

Glob shapes:

- `"*"`: matches any kind
- `"chat.*"`: matches one component after the prefix (`chat.chunk` ✓, `chat.tool.invoked` ✗)
- `"tool.invoked"`: exact match

```java
EventSamplingPolicy p = RuleBasedSamplingPolicy.builder()
        .capture("tool.invoked")                      // always capture
        .drop("internal.heartbeat")                   // hard-drop noise
        .sample("chat.chunk", 0.001)                  // 0.1% of chunks
        .capture("*", EventLevel.WARN, null)          // WARN+ everywhere
        .defaultDrop()                                // anything unmatched
        .build();
```

### `AdaptiveSamplingPolicy`

Wraps a delegate policy and **degrades gracefully under measured backpressure**. The consumer supplies a `BackpressureSignal`, a `double` in `[0.0, 1.0]` representing "fraction of capacity in use" (queue depth, ack lag, Resilience4j RateLimiter saturation, …). The policy reads it and:

- below `lowWaterMark` (default 0.5) → delegate runs unchanged
- between low and high → linearly downsample the delegate's `Capture` decisions
- at or above `highWaterMark` (default 0.9) → drop everything below `criticalLevel` (default `ERROR`)

Critical-and-above events always survive, regardless of pressure.

```java
BackpressureSignal queueSignal = () -> queueDepth.get() / (double) MAX_DEPTH;

EventSamplingPolicy p = new AdaptiveSamplingPolicy(
        SamplingPolicies.production(),     // delegate
        queueSignal);                       // sane defaults: 0.5 / 0.9 / ERROR

// Or with custom thresholds:
EventSamplingPolicy p2 = new AdaptiveSamplingPolicy(
        delegate, queueSignal,
        0.4, 0.85, EventLevel.WARN);
```

Tracks `droppedDueToPressureCount()` for monitoring how often pressure forced a drop above what the delegate would have done.

## Named factories: `SamplingPolicies`

Three shapes for the most common operator intents:

| Factory | Returns | When to use |
|---|---|---|
| `SamplingPolicies.verbose()` | `AlwaysCapturePolicy.INSTANCE` | dev / staging / single tenant |
| `SamplingPolicies.defaults()` | `PriorityBasedSamplingPolicy.withDefaults()` | production with INFO/DEBUG distinction |
| `SamplingPolicies.production()` | `ErrorAlwaysPolicy.withDefault()` | production with strict "errors-always" |

```java
EventSamplingPolicy p = SamplingPolicies.production();
```

## Per-tenant policies

`TenantPolicySamplingPolicy` dispatches each call to a per-tenant policy based on `EventContext.tenantId`. A multi-tenant agent group can apply different rates per tenant without the publisher knowing tenant boundaries.

Resolution order:

1. `EventContext.tenantId` from `SamplingInput.context`
2. Configured default policy, the fallback for null / blank / unmatched

```java
EventSamplingPolicy dispatcher = TenantPolicySamplingPolicy.builder()
        .tenant("acme",   SamplingPolicies.verbose())            // power user — keep everything
        .tenant("globex", new ErrorAlwaysPolicy(0.001))          // chatty — aggressive sampling
        .defaultPolicy(SamplingPolicies.production())            // everyone else
        .build();
```

The dispatch happens on every call (no per-thread cache) so a single agent group serving multiple tenants doesn't leak rates across boundaries.

## Wiring into the publisher

`SamplingAgentEventPublisher` is a decorator over the `AgentEventPublisher` SPI in `tnsai-core`. Each publish call builds a `SamplingInput` from `event.eventType()` + `EventLevel` + `EventContext`, asks the policy, and publishes only when `SamplingDecision.shouldPublish()` is true.

```java
AgentEventPublisher base    = new Slf4jAgentEventPublisher();
AgentEventPublisher sampled = new SamplingAgentEventPublisher(base, policy);
```

Constructor variants:

| Constructor | Use when |
|---|---|
| `(delegate, policy)` | bare `publish(event)` calls fall back to `EventContext.empty()` |
| `(delegate, policy, contextSupplier)` | per-call tenant / agent context flows in via the supplier |

Both `publish` overloads (level-aware and bare) route through the policy. Bare `publish(event)` defaults to `EventLevel.INFO`. Null event passes straight through to the delegate without consulting the policy.

## LLM calls

`SamplingLLMCallPublisher` applies the same policies to `LLMCallLog` events from [`CapturingLLMClient`](llm-calls.md). Each call is sampled as event kind `llm.called`, at level `ERROR` when it failed and `INFO` otherwise, with the call's own `EventContext`, so `ErrorAlwaysPolicy` keeps every failed call and tenant policies see the tenant.

```java
LLMCallPublisher sampled = new SamplingLLMCallPublisher(
        new RedactingLLMCallPublisher(sink, redactor),   // redact inside
        SamplingPolicies.production());                  // sample outside

LLMClient observed = new CapturingLLMClient(client, JsonLLMPricingRegistry.defaultRegistry(), sampled);
```

With sampling outside, the redactor only runs on calls that will ship.

## Composition with redaction

`SamplingAgentEventPublisher` and `RedactingAgentEventPublisher` are independent decorators, and both implement the same SPI. Stack order chooses which runs first:

```java
// Order A: sample first, redact what survives.
//   Cheaper when drop rate is high — redactor only runs on captured events.
AgentEventPublisher pub = new RedactingAgentEventPublisher(
        new SamplingAgentEventPublisher(base, policy), redactor);

// Order B: redact every event, then sample.
//   Uniform redaction cost; slightly easier to debug ("what would have been emitted?").
AgentEventPublisher pub = new SamplingAgentEventPublisher(
        new RedactingAgentEventPublisher(base, redactor), policy);
```

Pick order A in production (where drop rate is meaningful) and order B when debugging redaction patterns.

## Rate limiting (last-resort backstop)

Sampling makes per-event capture/drop decisions. A **rate limiter** is a per-tenant volume cap that protects the aggregator when sampling alone isn't enough. The two compose: sampling reduces volume cheaply, the limiter bounds the survivors.

```java
// Per-tenant token bucket backed by Resilience4j. The no-argument constructor allows 1000 events a second.
Resilience4jEventRateLimiter limiter = new Resilience4jEventRateLimiter(1000, Duration.ofSeconds(1));
limiter.registerTenant("noisy-tenant", RateLimiterConfig.custom()      // io.github.resilience4j.ratelimiter
        .limitForPeriod(100)
        .limitRefreshPeriod(Duration.ofSeconds(1))
        .timeoutDuration(Duration.ZERO)
        .build());

AgentEventPublisher slf4j   = new Slf4jAgentEventPublisher();
AgentEventPublisher rate    = new RateLimitedAgentEventPublisher(slf4j, limiter);
AgentEventPublisher sampled = new SamplingAgentEventPublisher(rate, policy);
```

**Stack order matters.** `sampled → rate → sink` (above) lets the cheap sampling decision run first, so the rate limiter only sees what survived. Reverse the stack (`rate → sampled → sink`) when the limiter's per-tenant cap matters more than sampling cost.

The limiter returns a `Decision`:

- `ACCEPT` → publish
- `DROP` → silently swallow (with a counter increment for monitoring)
- `DEFER` → opt-in deferred path. The default decorator drops, so override `onDefer` to queue

`Resilience4jEventRateLimiter#flushDropCounters()` returns a per-tenant snapshot (and clears) so a periodic flusher can emit one `ratelimit.dropped` aggregate event per minute instead of suppressing the signal entirely.

## SPI summary

| Type | Role |
|---|---|
| `EventSamplingPolicy` | The SPI, with the single method `sample(SamplingInput) → SamplingDecision`. |
| `SamplingDecision` | Sealed interface: `Capture`, `Drop`, `CaptureDownsampled(rate)`. |
| `SamplingInput` | Per-call frozen view: `eventKind`, `level`, `context`. |
| `AlwaysCapturePolicy` | Capture every event. The dev default. |
| `HeadBasedSamplingPolicy` | Coin-flip at fixed rate. |
| `PriorityBasedSamplingPolicy` | Per-`EventLevel` rates with sensible defaults. |
| `ErrorAlwaysPolicy` | WARN+ always, INFO/DEBUG at one rate. |
| `RuleBasedSamplingPolicy` | Declarative rule list (glob + level + tenant + outcome), first match wins. |
| `AdaptiveSamplingPolicy` | Wraps a delegate and downsamples linearly under backpressure. |
| `SamplingPolicies` | Named factories: `verbose`, `defaults`, `production`. |
| `TenantPolicySamplingPolicy` | Per-tenant dispatcher. |
| `SamplingAgentEventPublisher` | `AgentEventPublisher` decorator that consults the policy. |
| `EventRateLimiter` | Per-tenant token-bucket SPI returning `ACCEPT` / `DROP` / `DEFER`. |
| `Resilience4jEventRateLimiter` | Default limiter with per-tenant configs and drop counters. |
| `RateLimitedAgentEventPublisher` | Decorator that consults the limiter before delegating. |

## Trade-offs

- **No per-session affinity**: every policy decides per event in isolation. A captured session may have some chunks dropped.
- **`CaptureDownsampled` extrapolation cost**: downstream aggregators must know to scale counters by `1 / rate`. If your aggregator doesn't support per-event rates, use `Capture` (no rate) policies only.
- **Statistical variance**: at low rates over short windows, observed capture counts vary widely. Plan dashboards over windows large enough that the law of large numbers settles.
- **Cost vs. fidelity**: aggressive sampling reduces aggregator cost but loses signal for rare bugs. Production agents typically run `SamplingPolicies.production()` (10% INFO/DEBUG) until cost forces tighter rates.

## Related

- [LLM calls](llm-calls.md): the `LLMCallLog` events that `SamplingLLMCallPublisher` samples
- [Monitor agents](monitor.md): logs, traces, metrics and health checks
- [Redaction](../security/redaction.md): remove personal data before events leave the process
- [Cost](../llm/cost.md#tenant-budgets): per-tenant budgets