The tnsai-quality module gives you four signals for a running agent: structured logs, OpenTelemetry traces and metrics, a Prometheus endpoint and health checks. It also ships a structured trace model (AgentTrace) that records each step of a conversation and the scores attached to it.

Add the module next to the BOM:

<dependency>
  <groupId>com.tnsai</groupId>
  <artifactId>tnsai-quality</artifactId>
</dependency>

Logs, traces, metrics and health

This example covers all four signals:

package com.example.tnsai.docs;

import com.tnsai.quality.observability.AgentLogger;
import com.tnsai.quality.observability.OpenTelemetryMetrics;
import com.tnsai.quality.observability.OpenTelemetryTracer;
import com.tnsai.quality.observability.PrometheusMetricsExporter;
import com.tnsai.quality.observability.TnsAITelemetry;
import com.tnsai.quality.observability.health.HealthCheckResult;
import com.tnsai.quality.observability.health.HealthIndicator;
import com.tnsai.quality.observability.health.HealthRegistry;
import com.tnsai.quality.observability.health.ResourceHealthIndicator;
import java.io.IOException;
import java.util.Map;
import java.util.function.Supplier;

/** Compilation contract for observability/monitor.md. */
public final class ObservabilityTelemetryExample {

    private ObservabilityTelemetryExample() {
    }

    static void log(String userId) {
        AgentLogger logger = AgentLogger.forAgent("agent-001");
        logger.info("Processing request")
            .with("userId", userId)
            .with("action", "search")
            .log();
        logger.logActionComplete("search", 1_200, true);
    }

    static String traceAndMeasure(Supplier<String> work) {
        TnsAITelemetry telemetry = TnsAITelemetry.builder()
            .serviceName("my-agent-service")
            .otlpEndpoint("http://localhost:4317")      // OTLP gRPC collector
            .build();
        OpenTelemetryTracer tracer = new OpenTelemetryTracer(telemetry.getTracer());
        OpenTelemetryMetrics metrics = new OpenTelemetryMetrics(telemetry.getMeter());

        long start = System.currentTimeMillis();
        try (OpenTelemetryTracer.SpanScope scope = tracer.startSpan("process-request")) {
            scope.setAttribute("request.type", "search");
            String result = work.get();
            scope.setSuccess();
            metrics.recordActionSuccess("search", System.currentTimeMillis() - start);
            return result;
        } finally {
            telemetry.shutdown();
        }
    }

    static PrometheusMetricsExporter prometheus() throws IOException {
        PrometheusMetricsExporter exporter = PrometheusMetricsExporter.builder()
            .fromEnvironment()                           // TNSAI_PROMETHEUS_PORT, default 9090
            .build();
        exporter.start();                                // serves /metrics
        exporter.recordLlmCall("openai", "gpt-4o", 1_000, 500, 2_300);
        return exporter;
    }

    static Map<String, Object> health() {
        HealthRegistry registry = HealthRegistry.getInstance();
        registry.register(new ResourceHealthIndicator());          // heap, CPU, threads, deadlocks
        registry.register(HealthIndicator.of("cache", () ->
            HealthCheckResult.up().withDetail("entries", 42)));
        return registry.getHealthSummary();             // status, timestamp, components
    }
}

AgentLogger writes SLF4J entries with key-value fields. Its helpers are logActionStart, logActionComplete, logActionError, logLLMCall and timed(name, supplier).
TnsAITelemetry exports over OTLP. Without builder calls it reads OTEL_SERVICE_NAME (default tnsai-agent), OTEL_EXPORTER_OTLP_ENDPOINT (default http://localhost:4317) and TNSAI_TELEMETRY_ENABLED (default true).
OpenTelemetryTracer.trace(name, supplier) wraps one call in a span without a try block.
HealthRegistry.checkAll() runs every indicator concurrently with a 30-second timeout. getQuickStatus() uses a 2-second timeout per indicator. HealthCheckResult.isHealthy() is true for UP and DEGRADED.
ResourceHealthIndicator reports DEGRADED above 85% heap use or a load average of 0.9 per processor by default, and DOWN on a thread deadlock. Pass your own thresholds to new ResourceHealthIndicator(memory, cpu).

OpenTelemetry metrics

OpenTelemetryMetrics registers these instruments on the meter you pass in:

Metric	Type	Recorded by
`tnsai.agent.actions.total`	Counter	`recordActionSuccess`, `recordActionFailure`
`tnsai.agent.actions.success`	Counter	`recordActionSuccess`
`tnsai.agent.actions.failed`	Counter	`recordActionFailure(type, durationMs, errorType)`
`tnsai.agent.action.duration`	Histogram (ms)	both action methods
`tnsai.llm.calls.total`	Counter	`recordLlmCall`, `recordLlmCallFailure`
`tnsai.llm.calls.failed`	Counter	`recordLlmCallFailure(provider, model, latencyMs, errorType)`
`tnsai.llm.tokens.prompt`	Counter	`recordLlmCall(provider, model, prompt, completion, latencyMs)`
`tnsai.llm.tokens.completion`	Counter	`recordLlmCall`
`tnsai.llm.latency`	Histogram (ms)	both LLM methods
`tnsai.agent.messages.sent`	Counter	`recordMessageSent(agentId, messageType)`
`tnsai.agent.messages.received`	Counter	`recordMessageReceived(agentId, messageType)`
`tnsai.agent.active`	Gauge	`setActiveAgents(count)`

Prometheus endpoint

PrometheusMetricsExporter serves the Prometheus text format at /metrics. The port is 9090 unless you set port(...) or TNSAI_PROMETHEUS_PORT with fromEnvironment(). PrometheusMetricsExporter.createAndStart() builds from the environment and starts in one call; stop() shuts the server down.

The argument order differs from OpenTelemetryMetrics for failures: recordActionFailure(type, errorType, durationMs) and recordLlmCallFailure(provider, model, errorType, latencyMs).

Metric	Type	Labels
`tnsai_agent_actions_total`	Counter	`status`
`tnsai_agent_actions_success_total`	Counter	none
`tnsai_agent_actions_failed_total`	Counter	`error_type`
`tnsai_llm_calls_total`	Counter	`provider`, `model`, `status`
`tnsai_llm_calls_failed_total`	Counter	`provider`, `model`, `error_type`
`tnsai_llm_prompt_tokens_total`	Counter	`provider`, `model`
`tnsai_llm_completion_tokens_total`	Counter	`provider`, `model`
`tnsai_agent_active`	Gauge	none
`tnsai_agent_action_duration_seconds`	Histogram	`action_type`
`tnsai_llm_latency_seconds`	Histogram	`provider`, `model`

Every 15 seconds the exporter copies the active-agent gauge from the Metrics singleton. Add your own series with getOrCreateCounter, getOrCreateGauge and getOrCreateHistogram(name, help, labelNames...).

Instrument an agent

AgentInstrumentation wraps agent calls in spans and records metrics, without changing the agent:

AgentInstrumentation instrumentation = new AgentInstrumentation(telemetry);
instrumentation.registerAgent(agent);             // increments tnsai.agent.active

String reply = instrumentation.traceChat(agent, "Hello", () -> agent.chat("Hello"));
Object hits = instrumentation.traceToolCall(agent, "search", args, () -> tool.execute(args));

instrumentation.unregisterAgent(agent);           // decrements it

Span	Attributes
`agent.chat`	`agent.id`, `agent.name`, `message.length`, `message.preview`
`agent.action` (`traceAction`)	`agent.id`, `agent.name`, `action.name`, `action.parameters.count`
`agent.tool_call` (`traceToolCall`)	`agent.id`, `tool.name`, `tool.arguments.count`
`agent.llm_call` (`traceLlmCall`)	`agent.id`, `gen_ai.system`, `gen_ai.request.model`, `gen_ai.operation.name`

For spans on the LLM call itself, with GenAI semantic-convention attributes, see LLM calls.

Structured traces

The com.tnsai.quality.tracing package records one unit of work as an AgentTrace. A trace holds a tree of observations and the scores attached to them:

package com.example.tnsai.docs;

import com.tnsai.quality.tracing.AgentTrace;
import com.tnsai.quality.tracing.Observation;
import com.tnsai.quality.tracing.Score;
import com.tnsai.quality.tracing.ScoreSource;
import com.tnsai.quality.tracing.TraceContext;
import java.util.Map;

/** Compilation contract for observability/monitor.md. */
public final class ObservabilityTracingExample {

    private ObservabilityTracingExample() {
    }

    static AgentTrace record(String question, String answer) {
        AgentTrace trace = AgentTrace.builder("research-agent")
            .sessionId("session-456")
            .metadata(Map.of("env", "production"))
            .build();
        TraceContext.setTrace(trace);
        try {
            Observation search = trace.startSpan("search-tool")
                .input(question)
                .output("3 results found")
                .build();
            trace.addObservation(search);

            try (TraceContext.Scope scope = TraceContext.scoped(search.id())) {
                Observation generation = trace.startGeneration("llm-call")   // parent: search
                    .model("gpt-4o")
                    .inputTokens(1_500)
                    .outputTokens(800)
                    .prompt(question)
                    .completion(answer)
                    .build();
                trace.addObservation(generation);
                trace.addScore(Score.numeric("relevance", 0.92, ScoreSource.MODEL)
                    .forObservation(generation.id()));
            }
            trace.addScore(Score.bool("pii-free", true, ScoreSource.HEURISTIC));
            return trace;
        } finally {
            TraceContext.clear();
        }
    }
}

An Observation is a SPAN (a timed step), a GENERATION (an LLM call with model, tokens, cost, prompt and completion) or an EVENT (a point in time). Build one directly with Observation.span, generation or event.
trace.startSpan and startGeneration set the parent from TraceContext, a thread-local stack. TraceContext.scoped(id) pushes an id and pops it on close. Call TraceContext.clear() at the end of each request.
Read the tree back with rootObservations(), childrenOf(parentId) and scoresFor(observationId). On a generation, totalTokens() adds input and output tokens and durationMs() measures start to end.
A Score is numeric, bool (stored as 1.0 or 0.0) or categorical. Its source is HUMAN, MODEL or HEURISTIC. forObservation(id) binds a copy to one observation.

To score traces automatically, see Trace evaluation. To check traced text for PII or other content rules, see Guardrails.

Monitor agents

Logs, traces, metrics and health

OpenTelemetry metrics

Prometheus endpoint

Instrument an agent

Structured traces

Related