# Monitor agents

The `tnsai-quality` module gives you four signals for a running agent: structured logs, OpenTelemetry traces and metrics, a Prometheus endpoint and health checks. It also ships a structured trace model (`AgentTrace`) that records each step of a conversation and the scores attached to it.

Add the module next to the BOM:

```xml
<dependency>
  <groupId>com.tnsai</groupId>
  <artifactId>tnsai-quality</artifactId>
</dependency>
```

## Logs, traces, metrics and health

This example covers all four signals:

<!-- java-contract: src/main/java/com/example/tnsai/docs/ObservabilityTelemetryExample.java -->
```java
package com.example.tnsai.docs;

import com.tnsai.quality.observability.AgentLogger;
import com.tnsai.quality.observability.OpenTelemetryMetrics;
import com.tnsai.quality.observability.OpenTelemetryTracer;
import com.tnsai.quality.observability.PrometheusMetricsExporter;
import com.tnsai.quality.observability.TnsAITelemetry;
import com.tnsai.quality.observability.health.HealthCheckResult;
import com.tnsai.quality.observability.health.HealthIndicator;
import com.tnsai.quality.observability.health.HealthRegistry;
import com.tnsai.quality.observability.health.ResourceHealthIndicator;
import java.io.IOException;
import java.util.Map;
import java.util.function.Supplier;

/** Compilation contract for observability/monitor.md. */
public final class ObservabilityTelemetryExample {

    private ObservabilityTelemetryExample() {
    }

    static void log(String userId) {
        AgentLogger logger = AgentLogger.forAgent("agent-001");
        logger.info("Processing request")
            .with("userId", userId)
            .with("action", "search")
            .log();
        logger.logActionComplete("search", 1_200, true);
    }

    static String traceAndMeasure(Supplier<String> work) {
        TnsAITelemetry telemetry = TnsAITelemetry.builder()
            .serviceName("my-agent-service")
            .otlpEndpoint("http://localhost:4317")      // OTLP gRPC collector
            .build();
        OpenTelemetryTracer tracer = new OpenTelemetryTracer(telemetry.getTracer());
        OpenTelemetryMetrics metrics = new OpenTelemetryMetrics(telemetry.getMeter());

        long start = System.currentTimeMillis();
        try (OpenTelemetryTracer.SpanScope scope = tracer.startSpan("process-request")) {
            scope.setAttribute("request.type", "search");
            String result = work.get();
            scope.setSuccess();
            metrics.recordActionSuccess("search", System.currentTimeMillis() - start);
            return result;
        } finally {
            telemetry.shutdown();
        }
    }

    static PrometheusMetricsExporter prometheus() throws IOException {
        PrometheusMetricsExporter exporter = PrometheusMetricsExporter.builder()
            .fromEnvironment()                           // TNSAI_PROMETHEUS_PORT, default 9090
            .build();
        exporter.start();                                // serves /metrics
        exporter.recordLlmCall("openai", "gpt-4o", 1_000, 500, 2_300);
        return exporter;
    }

    static Map<String, Object> health() {
        HealthRegistry registry = HealthRegistry.getInstance();
        registry.register(new ResourceHealthIndicator());          // heap, CPU, threads, deadlocks
        registry.register(HealthIndicator.of("cache", () ->
            HealthCheckResult.up().withDetail("entries", 42)));
        return registry.getHealthSummary();             // status, timestamp, components
    }
}
```

- `AgentLogger` writes SLF4J entries with key-value fields. Its helpers are `logActionStart`, `logActionComplete`, `logActionError`, `logLLMCall` and `timed(name, supplier)`.
- `TnsAITelemetry` exports over OTLP. Without builder calls it reads `OTEL_SERVICE_NAME` (default `tnsai-agent`), `OTEL_EXPORTER_OTLP_ENDPOINT` (default `http://localhost:4317`) and `TNSAI_TELEMETRY_ENABLED` (default `true`).
- `OpenTelemetryTracer.trace(name, supplier)` wraps one call in a span without a `try` block.
- `HealthRegistry.checkAll()` runs every indicator concurrently with a 30-second timeout. `getQuickStatus()` uses a 2-second timeout per indicator. `HealthCheckResult.isHealthy()` is true for `UP` and `DEGRADED`.
- `ResourceHealthIndicator` reports `DEGRADED` above 85% heap use or a load average of 0.9 per processor by default, and `DOWN` on a thread deadlock. Pass your own thresholds to `new ResourceHealthIndicator(memory, cpu)`.

## OpenTelemetry metrics

`OpenTelemetryMetrics` registers these instruments on the meter you pass in:

| Metric | Type | Recorded by |
|--------|------|-------------|
| `tnsai.agent.actions.total` | Counter | `recordActionSuccess`, `recordActionFailure` |
| `tnsai.agent.actions.success` | Counter | `recordActionSuccess` |
| `tnsai.agent.actions.failed` | Counter | `recordActionFailure(type, durationMs, errorType)` |
| `tnsai.agent.action.duration` | Histogram (ms) | both action methods |
| `tnsai.llm.calls.total` | Counter | `recordLlmCall`, `recordLlmCallFailure` |
| `tnsai.llm.calls.failed` | Counter | `recordLlmCallFailure(provider, model, latencyMs, errorType)` |
| `tnsai.llm.tokens.prompt` | Counter | `recordLlmCall(provider, model, prompt, completion, latencyMs)` |
| `tnsai.llm.tokens.completion` | Counter | `recordLlmCall` |
| `tnsai.llm.latency` | Histogram (ms) | both LLM methods |
| `tnsai.agent.messages.sent` | Counter | `recordMessageSent(agentId, messageType)` |
| `tnsai.agent.messages.received` | Counter | `recordMessageReceived(agentId, messageType)` |
| `tnsai.agent.active` | Gauge | `setActiveAgents(count)` |

## Prometheus endpoint

`PrometheusMetricsExporter` serves the Prometheus text format at `/metrics`. The port is 9090 unless you set `port(...)` or `TNSAI_PROMETHEUS_PORT` with `fromEnvironment()`. `PrometheusMetricsExporter.createAndStart()` builds from the environment and starts in one call; `stop()` shuts the server down.

The argument order differs from `OpenTelemetryMetrics` for failures: `recordActionFailure(type, errorType, durationMs)` and `recordLlmCallFailure(provider, model, errorType, latencyMs)`.

| Metric | Type | Labels |
|--------|------|--------|
| `tnsai_agent_actions_total` | Counter | `status` |
| `tnsai_agent_actions_success_total` | Counter | none |
| `tnsai_agent_actions_failed_total` | Counter | `error_type` |
| `tnsai_llm_calls_total` | Counter | `provider`, `model`, `status` |
| `tnsai_llm_calls_failed_total` | Counter | `provider`, `model`, `error_type` |
| `tnsai_llm_prompt_tokens_total` | Counter | `provider`, `model` |
| `tnsai_llm_completion_tokens_total` | Counter | `provider`, `model` |
| `tnsai_agent_active` | Gauge | none |
| `tnsai_agent_action_duration_seconds` | Histogram | `action_type` |
| `tnsai_llm_latency_seconds` | Histogram | `provider`, `model` |

Every 15 seconds the exporter copies the active-agent gauge from the `Metrics` singleton. Add your own series with `getOrCreateCounter`, `getOrCreateGauge` and `getOrCreateHistogram(name, help, labelNames...)`.

## Instrument an agent

`AgentInstrumentation` wraps agent calls in spans and records metrics, without changing the agent:

```java
AgentInstrumentation instrumentation = new AgentInstrumentation(telemetry);
instrumentation.registerAgent(agent);             // increments tnsai.agent.active

String reply = instrumentation.traceChat(agent, "Hello", () -> agent.chat("Hello"));
Object hits = instrumentation.traceToolCall(agent, "search", args, () -> tool.execute(args));

instrumentation.unregisterAgent(agent);           // decrements it
```

| Span | Attributes |
|------|-----------|
| `agent.chat` | `agent.id`, `agent.name`, `message.length`, `message.preview` |
| `agent.action` (`traceAction`) | `agent.id`, `agent.name`, `action.name`, `action.parameters.count` |
| `agent.tool_call` (`traceToolCall`) | `agent.id`, `tool.name`, `tool.arguments.count` |
| `agent.llm_call` (`traceLlmCall`) | `agent.id`, `gen_ai.system`, `gen_ai.request.model`, `gen_ai.operation.name` |

For spans on the LLM call itself, with GenAI semantic-convention attributes, see [LLM calls](llm-calls.md).

## Structured traces

The `com.tnsai.quality.tracing` package records one unit of work as an `AgentTrace`. A trace holds a tree of observations and the scores attached to them:

<!-- java-contract: src/main/java/com/example/tnsai/docs/ObservabilityTracingExample.java -->
```java
package com.example.tnsai.docs;

import com.tnsai.quality.tracing.AgentTrace;
import com.tnsai.quality.tracing.Observation;
import com.tnsai.quality.tracing.Score;
import com.tnsai.quality.tracing.ScoreSource;
import com.tnsai.quality.tracing.TraceContext;
import java.util.Map;

/** Compilation contract for observability/monitor.md. */
public final class ObservabilityTracingExample {

    private ObservabilityTracingExample() {
    }

    static AgentTrace record(String question, String answer) {
        AgentTrace trace = AgentTrace.builder("research-agent")
            .sessionId("session-456")
            .metadata(Map.of("env", "production"))
            .build();
        TraceContext.setTrace(trace);
        try {
            Observation search = trace.startSpan("search-tool")
                .input(question)
                .output("3 results found")
                .build();
            trace.addObservation(search);

            try (TraceContext.Scope scope = TraceContext.scoped(search.id())) {
                Observation generation = trace.startGeneration("llm-call")   // parent: search
                    .model("gpt-4o")
                    .inputTokens(1_500)
                    .outputTokens(800)
                    .prompt(question)
                    .completion(answer)
                    .build();
                trace.addObservation(generation);
                trace.addScore(Score.numeric("relevance", 0.92, ScoreSource.MODEL)
                    .forObservation(generation.id()));
            }
            trace.addScore(Score.bool("pii-free", true, ScoreSource.HEURISTIC));
            return trace;
        } finally {
            TraceContext.clear();
        }
    }
}
```

- An `Observation` is a `SPAN` (a timed step), a `GENERATION` (an LLM call with model, tokens, cost, prompt and completion) or an `EVENT` (a point in time). Build one directly with `Observation.span`, `generation` or `event`.
- `trace.startSpan` and `startGeneration` set the parent from `TraceContext`, a thread-local stack. `TraceContext.scoped(id)` pushes an id and pops it on close. Call `TraceContext.clear()` at the end of each request.
- Read the tree back with `rootObservations()`, `childrenOf(parentId)` and `scoresFor(observationId)`. On a generation, `totalTokens()` adds input and output tokens and `durationMs()` measures start to end.
- A `Score` is `numeric`, `bool` (stored as 1.0 or 0.0) or `categorical`. Its source is `HUMAN`, `MODEL` or `HEURISTIC`. `forObservation(id)` binds a copy to one observation.

To score traces automatically, see [Trace evaluation](../evaluation/trace-eval.md). To check traced text for PII or other content rules, see [Guardrails](../security/guardrails.md).

## Related

- [LLM calls](llm-calls.md)
- [Sampling](sampling.md)
- [Agent events](../agents/events.md)
- [MCP observability](../mcp/observability.md)
- [Trace evaluation](../evaluation/trace-eval.md)