# Observe LLM calls

Capture every LLM call as a typed `LLMCallLog` event: prompt, response, token usage, cost, streaming timing, errors and the full context. `CapturingLLMClient` wraps any `LLMClient` and publishes one event per request, so every provider works without changes. For OpenTelemetry spans that follow the GenAI conventions, see [LLM spans](#llm-spans).

## Quick start

Wrap any `LLMClient` with `CapturingLLMClient`. The default publisher writes one structured SLF4J line per call. `AgentBuilder.build()` requires `principal` and `authorityScope`; `liabilitySink` is optional.

<!-- java-contract: src/main/java/com/example/tnsai/docs/LlmObservabilityBuilderExample.java -->
```java
package com.example.tnsai.docs;

import com.tnsai.accountability.AuthorityScope;
import com.tnsai.accountability.LiabilitySink;
import com.tnsai.agents.Agent;
import com.tnsai.agents.AgentBuilder;
import com.tnsai.identity.AgentPrincipal;
import com.tnsai.llm.LLMClient;
import com.tnsai.llm.LLMClientFactory;
import com.tnsai.llm.observability.CapturingLLMClient;
import com.tnsai.llm.observability.JsonLLMPricingRegistry;
import com.tnsai.llm.observability.Slf4jLLMCallPublisher;
import com.tnsai.roles.Role;

public final class LlmObservabilityBuilderExample {
    private LlmObservabilityBuilderExample() {}

    public static Agent build(
        Role role,
        AgentPrincipal principal,
        LiabilitySink sink,
        AuthorityScope scope
    ) {
        LLMClient observed = new CapturingLLMClient(
            LLMClientFactory.create("openai", "gpt-4o", 0.7f),
            JsonLLMPricingRegistry.defaultRegistry(),
            new Slf4jLLMCallPublisher());
        return AgentBuilder.create()
            .role(role)
            .llm(observed)
            .principal(principal)
            .liabilitySink(sink)
            .authorityScope(scope)
            .build();
    }
}
```

Every `chat` and `streamChat` call now logs to the `com.tnsai.llm.callLog` logger:

```
INFO  com.tnsai.llm.callLog - llm.call provider=openai model=gpt-4o elapsedMs=842 \
  promptTokens=312 completionTokens=89 cachedTokens=0 totalTokens=401 \
  costUSD=0.00168 pricingTable=2026-05 finishReason=STOP streamed=false tools=2
```

Failures log at `WARN` with `retryAttempt`, `httpStatus` and `errorClass`. The original exception is re-thrown after capture.

## What gets captured

`LLMCallLog` is a typed record carrying:

| Field | Description |
|---|---|
| `callId` | UUID, the key for joining a call to downstream events |
| `startedAt` / `completedAt` / `elapsed` | Wall-clock timing |
| `provider` / `model` / `endpoint` | Routing |
| `prompt` | Messages, system prompt, parameters, prompt-cache markers |
| `tools` | `ToolSurface`: names, schemas, SHA-256 hash for cache correlation |
| `response` | Content, tool calls, reasoning content |
| `usage` | Prompt, completion, cached, reasoning and total tokens |
| `cost` | `CostEstimate`: prompt, completion, cached discount, reasoning and total USD, plus `pricingTableVersion` |
| `finishReason` | STOP, LENGTH, TOOL_CALL, CONTENT_FILTER |
| `streamMetrics` | Time to first token and chunk count for streaming calls |
| `error` | `ErrorInfo` for failed calls |
| `context` | Full `EventContext`: tenant, agent, role, capability, session |
| `retryAttempt` | Retry counter |

## Pricing registry

`JsonLLMPricingRegistry.defaultRegistry()` loads the bundled rate card from `/pricing/2026-05.json`. It covers `openai` (GPT-4o, GPT-4o mini, o1-preview), `anthropic` (Claude Sonnet 4, Opus 4, Haiku 4.5), `google` (Gemini 2.0 Flash and Pro), `mistral` (Large, Small), `groq` (Llama 3.3 70B, Mixtral 8x7B), `cohere` (Command R+, R) and `ollama` (a wildcard at zero for local models).

Build your own rate card for negotiated pricing or new models. Rates are USD per 1K tokens, and the reasoning rate is `null` for models without one:

```java
LLMPricingRegistry custom = InMemoryLLMPricingRegistry.builder("contract-2026-05")
        .put("openai", "gpt-4o", 0.0015, 0.0005, 0.006, null)   // prompt, cached, completion, reasoning
        .build();

LLMClient observed = new CapturingLLMClient(base, custom, new Slf4jLLMCallPublisher());
```

`call.cost().pricingTableVersion()` records which table produced the estimate, so historical costs don't shift when rates change.

## Streaming capture

For streaming calls, the wrapper fills `StreamMetrics`:

```java
public record StreamMetrics(
    Instant firstChunkAt,
    Duration timeToFirstToken,
    long chunkCount,
    Duration interChunkP50,
    Duration interChunkP99
) {}
```

`timeToFirstToken` is the metric to graph for user-perceived latency. The wrapper does not compute inter-chunk percentiles; `interChunkP50` and `interChunkP99` are always `Duration.ZERO`.

## Tool surface hashing

When the call advertises tools, `ToolSurface` carries the names and JSON schemas, plus a SHA-256 hash of their canonical sorted-key form:

```java
public record ToolSurface(
    List<String> toolNames,
    List<String> toolSchemas,
    String surfaceHash
) {}
```

The same `surfaceHash` across calls means the same tool set, which is what prompt caching needs. Use it to find cacheable trajectories in your dashboards.

## Custom publisher

`LLMCallPublisher` is a single-method functional interface. Implement it to send calls to your own sink:

```java
public final class MapPublisher implements LLMCallPublisher {
    private final Consumer<Map<String, Object>> sink;   // your exporter

    public MapPublisher(Consumer<Map<String, Object>> sink) {
        this.sink = sink;
    }

    @Override
    public void publish(LLMCallLog call) {
        try {
            sink.accept(Map.of(
                    "callId", call.callId(),
                    "provider", call.provider(),
                    "model", call.model(),
                    "tenant", Objects.requireNonNullElse(call.context().tenantId(), "default"),
                    "totalTokens", call.usage().totalTokens(),
                    "costUSD", call.cost().totalUSD()));
        } catch (RuntimeException e) {
            // never let a sink failure reach the agent
        }
    }
}
```

`publish` must not throw: observability failures must never block the agent. To send only a share of calls, wrap your publisher in `SamplingLLMCallPublisher`; see [Sampling](sampling.md#llm-calls).

## Cost attribution

`LLMCallLog.context()` carries the full `EventContext`. Aggregate cost in your sink along any of its dimensions:

- **Per tenant**: billing
- **Per agent**: which agent spends the most
- **Per role**: which role's allocation is tight
- **Per capability**: chatty versus terse capabilities
- **Per session**: per-conversation cost for end-user billing

Group context propagates, so cost splits across multi-agent group members the same way.

## Limits

- `Slf4jLLMCallPublisher` never logs raw prompt or response text, because both can carry personal data. To ship content, use a separate publisher behind [redaction](../security/redaction.md).
- The wrapper captures the text `chat` path and `streamChat`. Multimodal `chat(List<ContentPart>, …)` and `streamChatWithSpec` pass through without capture.
- `usage().promptTokens()` is zero when the provider returns no usage block (some local Ollama models). The cost estimate is then zero too, which means "no usage data", not a free call.
- `endpoint` is set when you pass it to the four-argument constructor, and is an empty string otherwise.

## Observers

`ObservableLLMClient` (`com.tnsai.llm.observability`) is a lighter hook. It calls one or more `LLMObserver`s around each call and builds no `LLMCallLog`:

```java
LLMMetrics metrics = new LLMMetrics();
LLMClient observed = new ObservableLLMClient(client, metrics, PromptLogger.withPIIFiltering());
```

`null` observers and `LLMObserver.NOOP` are dropped. To combine observers yourself, use `ObservableLLMClient.CompositeObserver.of(...)`.

### `LLMObserver`

Every hook has an empty default, so override only the ones you need:

| Hook | When it runs | Arguments |
|---|---|---|
| `onRequest` | Before the request is sent | client, message, systemPrompt, history, tools |
| `onResponse` | After a response | client, response, latencyMs |
| `onError` | When the request fails | client, exception, latencyMs |
| `onStreamChunk` | For each streamed piece | client, chunk, chunkIndex |
| `onStreamComplete` | When the stream ends | client, totalChunks, latencyMs |
| `onStreamError` | When the stream fails | client, exception, chunksReceived, latencyMs |

### `PromptLogger`

`PromptLogger` logs each prompt and reply to a separate SLF4J logger, so you can route content apart from the class's own log. Its name is the `com.tnsai` prefix followed by `.diagnostics.content.PromptLogger`:

```java
PromptLogger logger = PromptLogger.builder()
    .filterPII(true)                      // Default true
    .logLevel(PromptLogger.LogLevel.INFO) // DEBUG, INFO or WARN. Default INFO.
    .maxContentLength(500)                // Default 200 characters
    .logFullContent(false)                // true turns off truncation
    .build();
```

`PromptLogger.withPIIFiltering()` and `PromptLogger.withoutFiltering()` give the defaults with filtering on or off.

With filtering on, the text is redacted before it is cut to length:

| Found | Replaced with |
|---|---|
| Email address | `[REDACTED_EMAIL]` |
| Phone number | `[REDACTED_PHONE]` |
| Card number | `[REDACTED_CC]` |
| US Social Security number | `[REDACTED_SSN]` |
| API key or token | `[REDACTED_KEY]` |
| IP address | `[REDACTED_IP]` |

Each call gets an 8-character request id. During the call, the logger puts `llm.requestId`, `llm.provider` and `llm.model` in the SLF4J MDC, adds `llm.latencyMs` on the response, and removes them afterwards.

```
LLM Request [3f2a9c1e] OpenAI/gpt-4o: "My email is [REDACTED_EMAIL]"
LLM Response [3f2a9c1e] OpenAI/gpt-4o (245ms): "Noted."
```

### `LLMMetrics`

`LLMMetrics` counts calls, estimated tokens, latency and estimated cost:

| Method | Returns |
|---|---|
| `getReport()` | `LLMMetrics.Report`: request, response and error totals, input and output tokens, `totalEstimatedCost`, average and p50, p95 and p99 latency, `successRate()`, `errorRate()` |
| `getMetricsByProvider()` | `Map<String, ProviderMetrics>`, keyed `<provider>/<model>`, such as `OpenAI/gpt-4o` |
| `reset()` | Clears every counter |

Tokens are estimated at four characters each. The cost uses the model's `getInputCostPer1KTokens()` and `getOutputCostPer1KTokens()`, and counts as zero when the client has no price.

## LLM spans

`LlmInstrumentation` (`com.tnsai.quality.observability`, in `tnsai-quality`) wraps a call in an OpenTelemetry span named `<provider>.<operation> <model>`, with the GenAI semantic-convention attributes. Tools such as Jaeger, Zipkin and Grafana Tempo read them without custom mapping.

```java
LlmInstrumentation llm = new LlmInstrumentation(telemetry.getTracer());

ChatResponse response = llm.traceChat("openai", "gpt-4o",
        () -> client.chat(message));

ChatResponse tuned = llm.traceChat("openai", "gpt-4o", 4096, 0.7,   // maxTokens, temperature (either may be null)
        () -> client.chat(message));

Stream<String> stream = llm.traceStreamChat("anthropic", "claude-sonnet-4",
        () -> client.streamChat(message));

List<float[]> vectors = llm.traceEmbeddings("openai", "text-embedding-3-small", inputs.size(),
        () -> embed(inputs));                                        // your embedding call
```

- `traceChat` sets `gen_ai.response.finish_reasons` to `stop` when the supplier returns and to `error` when it throws; it does not read the provider's finish reason. Exceptions are recorded on the span and re-thrown.
- `traceStreamChat` ends its span as soon as the supplier returns the stream, with a `stream_started` event. It does not measure the stream itself.
- `traceEmbeddings` adds `gen_ai.request.embedding_inputs`.
- These methods do not make the span current, so `Span.current()` inside the supplier is not the LLM span.

To manage the span yourself, start it with `spanBuilder` and record results on it:

```java
Span span = llm.spanBuilder("openai", "gpt-4o")
    .operation("chat")                    // default "chat"
    .maxTokens(4096)
    .temperature(0.7)
    .topP(0.9)
    .agent("agent-001", "ResearchAgent")
    .messageLength(500)
    .toolUseEnabled(true)
    .start();
try {
    // ... call the model
    llm.recordResponse(span, 150, 50, "stop");      // prompt tokens, completion tokens, finish reason
    llm.recordToolUse(span, 1);
    llm.recordToolCallEvent(span, "brave_search", true);
} finally {
    span.end();
}
```

`recordTokenUsage(span, prompt, completion)` sets only the token counts, including their total.

GenAI attributes:

| Attribute | Description |
|---|---|
| `gen_ai.system` | Provider (`openai`, `anthropic`, …) |
| `gen_ai.request.model` | Model name |
| `gen_ai.request.max_tokens` | Max tokens requested |
| `gen_ai.request.temperature` | Temperature |
| `gen_ai.request.top_p` | Top-p |
| `gen_ai.usage.prompt_tokens` | Prompt tokens |
| `gen_ai.usage.completion_tokens` | Completion tokens |
| `gen_ai.usage.total_tokens` | Total tokens |
| `gen_ai.response.finish_reasons` | Finish reason |
| `gen_ai.operation.name` | `chat`, `stream_chat`, `embeddings` or your `operation(...)` |

TnsAI attributes:

| Attribute | Description |
|---|---|
| `tnsai.agent.id` | Agent id |
| `tnsai.agent.name` | Agent name |
| `tnsai.message.length` | Input message length |
| `tnsai.tool_use.enabled` | Whether tool use is on |
| `tnsai.tool_calls.count` | Number of tool calls |

`recordToolCallEvent` adds a `tool_call_success` or `tool_call_failure` event with `tool.name` and `tool.success`.

## Related

- [Monitor agents](monitor.md): logs, traces, metrics and health checks
- [Sampling](sampling.md): send a share of calls to a high-volume sink
- [Cost](../llm/cost.md): `CostAwareLLMClient` and `BudgetManager` for spend limits
- [Providers](../llm/providers.md): every built-in provider works with the wrapper unchanged
- [Redaction](../security/redaction.md): remove personal data before calls leave the process