Observe LLM calls

Quick start

Wrap any LLMClient with CapturingLLMClient. The default publisher writes one structured SLF4J line per call. AgentBuilder.build() requires principal and authorityScope; liabilitySink is optional.

package com.example.tnsai.docs;

import com.tnsai.accountability.AuthorityScope;
import com.tnsai.accountability.LiabilitySink;
import com.tnsai.agents.Agent;
import com.tnsai.agents.AgentBuilder;
import com.tnsai.identity.AgentPrincipal;
import com.tnsai.llm.LLMClient;
import com.tnsai.llm.LLMClientFactory;
import com.tnsai.llm.observability.CapturingLLMClient;
import com.tnsai.llm.observability.JsonLLMPricingRegistry;
import com.tnsai.llm.observability.Slf4jLLMCallPublisher;
import com.tnsai.roles.Role;

public final class LlmObservabilityBuilderExample {
    private LlmObservabilityBuilderExample() {}

    public static Agent build(
        Role role,
        AgentPrincipal principal,
        LiabilitySink sink,
        AuthorityScope scope
    ) {
        LLMClient observed = new CapturingLLMClient(
            LLMClientFactory.create("openai", "gpt-4o", 0.7f),
            JsonLLMPricingRegistry.defaultRegistry(),
            new Slf4jLLMCallPublisher());
        return AgentBuilder.create()
            .role(role)
            .llm(observed)
            .principal(principal)
            .liabilitySink(sink)
            .authorityScope(scope)
            .build();
    }
}

Every chat and streamChat call now logs to the com.tnsai.llm.callLog logger:

INFO  com.tnsai.llm.callLog - llm.call provider=openai model=gpt-4o elapsedMs=842 \
  promptTokens=312 completionTokens=89 cachedTokens=0 totalTokens=401 \
  costUSD=0.00168 pricingTable=2026-05 finishReason=STOP streamed=false tools=2

Failures log at WARN with retryAttempt, httpStatus and errorClass. The original exception is re-thrown after capture.

What gets captured

LLMCallLog is a typed record carrying:

Pricing registry

JsonLLMPricingRegistry.defaultRegistry() loads the bundled rate card from /pricing/2026-05.json. It covers openai (GPT-4o, GPT-4o mini, o1-preview), anthropic (Claude Sonnet 4, Opus 4, Haiku 4.5), google (Gemini 2.0 Flash and Pro), mistral (Large, Small), groq (Llama 3.3 70B, Mixtral 8x7B), cohere (Command R+, R) and ollama (a wildcard at zero for local models).

Build your own rate card for negotiated pricing or new models. Rates are USD per 1K tokens, and the reasoning rate is null for models without one:

LLMPricingRegistry custom = InMemoryLLMPricingRegistry.builder("contract-2026-05")
        .put("openai", "gpt-4o", 0.0015, 0.0005, 0.006, null)   // prompt, cached, completion, reasoning
        .build();

LLMClient observed = new CapturingLLMClient(base, custom, new Slf4jLLMCallPublisher());

call.cost().pricingTableVersion() records which table produced the estimate, so historical costs don't shift when rates change.

Streaming capture

For streaming calls, the wrapper fills StreamMetrics:

public record StreamMetrics(
    Instant firstChunkAt,
    Duration timeToFirstToken,
    long chunkCount,
    Duration interChunkP50,
    Duration interChunkP99
) {}

timeToFirstToken is the metric to graph for user-perceived latency. The wrapper does not compute inter-chunk percentiles; interChunkP50 and interChunkP99 are always Duration.ZERO.

Tool surface hashing

When the call advertises tools, ToolSurface carries the names and JSON schemas, plus a SHA-256 hash of their canonical sorted-key form:

public record ToolSurface(
    List<String> toolNames,
    List<String> toolSchemas,
    String surfaceHash
) {}

The same surfaceHash across calls means the same tool set, which is what prompt caching needs. Use it to find cacheable trajectories in your dashboards.

Custom publisher

LLMCallPublisher is a single-method functional interface. Implement it to send calls to your own sink:

public final class MapPublisher implements LLMCallPublisher {
    private final Consumer<Map<String, Object>> sink;   // your exporter

    public MapPublisher(Consumer<Map<String, Object>> sink) {
        this.sink = sink;
    }

    @Override
    public void publish(LLMCallLog call) {
        try {
            sink.accept(Map.of(
                    "callId", call.callId(),
                    "provider", call.provider(),
                    "model", call.model(),
                    "tenant", Objects.requireNonNullElse(call.context().tenantId(), "default"),
                    "totalTokens", call.usage().totalTokens(),
                    "costUSD", call.cost().totalUSD()));
        } catch (RuntimeException e) {
            // never let a sink failure reach the agent
        }
    }
}

publish must not throw: observability failures must never block the agent. To send only a share of calls, wrap your publisher in SamplingLLMCallPublisher; see Sampling.

Cost attribution

LLMCallLog.context() carries the full EventContext. Aggregate cost in your sink along any of its dimensions:

• Per tenant: billing
• Per agent: which agent spends the most
• Per role: which role's allocation is tight
• Per capability: chatty versus terse capabilities
• Per session: per-conversation cost for end-user billing

Group context propagates, so cost splits across multi-agent group members the same way.

Limits

• Slf4jLLMCallPublisher never logs raw prompt or response text, because both can carry personal data. To ship content, use a separate publisher behind redaction.
• The wrapper captures the text chat path and streamChat. Multimodal chat(List<ContentPart>, …) and streamChatWithSpec pass through without capture.
• usage().promptTokens() is zero when the provider returns no usage block (some local Ollama models). The cost estimate is then zero too, which means "no usage data", not a free call.
• endpoint is set when you pass it to the four-argument constructor, and is an empty string otherwise.

Observers

ObservableLLMClient (com.tnsai.llm.observability) is a lighter hook. It calls one or more LLMObservers around each call and builds no LLMCallLog:

LLMMetrics metrics = new LLMMetrics();
LLMClient observed = new ObservableLLMClient(client, metrics, PromptLogger.withPIIFiltering());

null observers and LLMObserver.NOOP are dropped. To combine observers yourself, use ObservableLLMClient.CompositeObserver.of(...).

LLMObserver

Every hook has an empty default, so override only the ones you need:

PromptLogger

PromptLogger logs each prompt and reply to a separate SLF4J logger, so you can route content apart from the class's own log. Its name is the com.tnsai prefix followed by .diagnostics.content.PromptLogger:

PromptLogger logger = PromptLogger.builder()
    .filterPII(true)                      // Default true
    .logLevel(PromptLogger.LogLevel.INFO) // DEBUG, INFO or WARN. Default INFO.
    .maxContentLength(500)                // Default 200 characters
    .logFullContent(false)                // true turns off truncation
    .build();

PromptLogger.withPIIFiltering() and PromptLogger.withoutFiltering() give the defaults with filtering on or off.

With filtering on, the text is redacted before it is cut to length:

Each call gets an 8-character request id. During the call, the logger puts llm.requestId, llm.provider and llm.model in the SLF4J MDC, adds llm.latencyMs on the response, and removes them afterwards.

LLM Request [3f2a9c1e] OpenAI/gpt-4o: "My email is [REDACTED_EMAIL]"
LLM Response [3f2a9c1e] OpenAI/gpt-4o (245ms): "Noted."

LLMMetrics

LLMMetrics counts calls, estimated tokens, latency and estimated cost:

Tokens are estimated at four characters each. The cost uses the model's getInputCostPer1KTokens() and getOutputCostPer1KTokens(), and counts as zero when the client has no price.

LLM spans

LlmInstrumentation (com.tnsai.quality.observability, in tnsai-quality) wraps a call in an OpenTelemetry span named <provider>.<operation> <model>, with the GenAI semantic-convention attributes. Tools such as Jaeger, Zipkin and Grafana Tempo read them without custom mapping.

LlmInstrumentation llm = new LlmInstrumentation(telemetry.getTracer());

ChatResponse response = llm.traceChat("openai", "gpt-4o",
        () -> client.chat(message));

ChatResponse tuned = llm.traceChat("openai", "gpt-4o", 4096, 0.7,   // maxTokens, temperature (either may be null)
        () -> client.chat(message));

Stream<String> stream = llm.traceStreamChat("anthropic", "claude-sonnet-4",
        () -> client.streamChat(message));

List<float[]> vectors = llm.traceEmbeddings("openai", "text-embedding-3-small", inputs.size(),
        () -> embed(inputs));                                        // your embedding call

• traceChat sets gen_ai.response.finish_reasons to stop when the supplier returns and to error when it throws; it does not read the provider's finish reason. Exceptions are recorded on the span and re-thrown.
• traceStreamChat ends its span as soon as the supplier returns the stream, with a stream_started event. It does not measure the stream itself.
• traceEmbeddings adds gen_ai.request.embedding_inputs.
• These methods do not make the span current, so Span.current() inside the supplier is not the LLM span.

To manage the span yourself, start it with spanBuilder and record results on it:

Span span = llm.spanBuilder("openai", "gpt-4o")
    .operation("chat")                    // default "chat"
    .maxTokens(4096)
    .temperature(0.7)
    .topP(0.9)
    .agent("agent-001", "ResearchAgent")
    .messageLength(500)
    .toolUseEnabled(true)
    .start();
try {
    // ... call the model
    llm.recordResponse(span, 150, 50, "stop");      // prompt tokens, completion tokens, finish reason
    llm.recordToolUse(span, 1);
    llm.recordToolCallEvent(span, "brave_search", true);
} finally {
    span.end();
}

recordTokenUsage(span, prompt, completion) sets only the token counts, including their total.

GenAI attributes:

TnsAI attributes:

recordToolCallEvent adds a tool_call_success or tool_call_failure event with tool.name and tool.success.

Related

• Monitor agents: logs, traces, metrics and health checks
• Sampling: send a share of calls to a high-volume sink
• Cost: CostAwareLLMClient and BudgetManager for spend limits
• Providers: every built-in provider works with the wrapper unchanged
• Redaction: remove personal data before calls leave the process