Skip to content
tnsaiDocs
GitHub

Streaming

Stream a reply token by token from an agent, or read typed chunks straight from an LLMClient.

CallToolsReturns
agent.streamChat(message)Not offered to the modelStream<String> of text tokens
agent.streamChatWithTools(message, handler)Run by the agent, up to 10 roundsThe full reply; chunks go to the handler
llm.streamChatWithSpec(request)Not runStream<ChatChunk>
llm.streamChatWithHandler(request, handler)Not runNothing; chunks go to the handler

For lifecycle events such as run start, tool start and tool end, use agent.chatWithEvents. See Events.

package com.example.tnsai.docs;

import com.tnsai.agents.Agent;
import com.tnsai.llm.ChatChunk;
import com.tnsai.llm.ChatRequest;
import com.tnsai.llm.LLMClient;

public final class StreamingExample {
    private StreamingExample() {}

    // Text only. The agent's tools are not offered to the model.
    static void tokens(Agent agent) {
        agent.streamChat("Explain relativity").forEach(System.out::print);
    }

    // The agent runs requested tools and streams again, up to 10 rounds.
    static String withTools(Agent agent) {
        return agent.streamChatWithTools("Find the TnsAI release notes", chunk -> {
            if (chunk.isContent()) {
                System.out.print(chunk.getContent());
            } else if (chunk.isToolCall()) {
                chunk.getToolCall().ifPresent(call -> System.out.println("\n[tool] " + call.getName()));
            }
        });
    }

    // Straight from a client: typed chunks, no agent, no tool execution.
    static void chunks(LLMClient llm) {
        ChatRequest request = ChatRequest.builder().message("Tell me a story").build();
        llm.streamChatWithSpec(request).forEach(chunk -> {
            switch (chunk.getType()) {
                case START -> System.out.println("model: " + chunk.getModel().orElse("?"));
                case CONTENT -> System.out.print(chunk.getContent());
                case TOOL_CALL -> System.out.println("\n[tool call] " + chunk.getContent());
                case DONE -> System.out.println("\nfinish: " + chunk.getFinishReason().orElse(ChatChunk.FinishReason.STOP));
                case ERROR -> System.err.println("error: " + chunk.getContent());
            }
        });
    }
}

Tool rounds

streamChatWithTools streams one reply. If the model asked for tools, the agent runs them, adds the results to memory and streams again. It stops at a reply without tool calls, or after 10 rounds with a warning in the log.

The handler gets the first START chunk, every CONTENT and TOOL_CALL chunk, and one DONE chunk with STOP at the end. It does not get ERROR chunks: a failed model call ends the round with no text, and the method returns an empty string.

Client streaming

streamChatWithSpec and streamChatWithHandler only report what the model sends. A TOOL_CALL chunk carries the call; running it is up to you.

With tools in the request, streamChatWithHandler does not stream: it makes one chat call and sends the reply as one CONTENT chunk, then one TOOL_CALL chunk per call, then DONE with TOOL_CALLS or STOP. It turns any exception into an ERROR chunk, except an LLMCapabilityException, which it throws.

The default streamChatWithSpec wraps the client's text stream in START, CONTENT and DONE chunks. A provider may override it with richer chunks.

ChatChunk

TypeCarries
STARTgetModel(), getRequestId()
CONTENTgetContent(), and getTokenCount() and getChunkIndex() when known
TOOL_CALLgetToolCall(), and getContent() as name(arguments)
DONEgetFinishReason(): STOP, LENGTH, TOOL_CALLS, CONTENT_FILTER or ERROR
ERRORgetContent(), the error message

isStart(), isContent(), isToolCall(), isDone() and isError() test the type. The static factories start, content, toolCall, done and error build chunks for tests or custom clients.

Per-call retrieval context

The streaming, event and action calls each have an overload that takes a ChatRetrievalContext. It scopes one call: tenant and session, run id, cancellation token, deadline and metadata filters. Fields you leave empty get the agent's tenant, session and name; fields you set are kept. chatWithEvents(message, sessionId) takes no context.

package com.example.tnsai.docs;

import com.tnsai.actions.model.ActionRequest;
import com.tnsai.agents.Agent;
import com.tnsai.cancellation.CancellationToken;
import com.tnsai.rag.ChatRetrievalContext;

import java.time.Duration;
import java.util.Map;

public final class ActionRetrievalContextExample {

    private ActionRetrievalContextExample() {}

    public static void dispatch(Agent agent) {
        CancellationToken cancellation = CancellationToken.create();
        ChatRetrievalContext context = ChatRetrievalContext.builder()
            .scope("tenant-a", "conversation-42")
            .runId("run-7")
            .simulationState("review")
            .currentEpisode("episode-3")
            .sourceRevision("kb-2026-08-12")
            .cancellationToken(cancellation)
            .timeout(Duration.ofSeconds(3))
            .metadataFilters(Map.of("tier", "approved"))
            .build();

        agent.streamChatWithTools("Summarize the evidence", chunk -> {}, context);
        agent.streamChatWithRetrievedContext(
            "Fenced outbound request", "Original user message", chunk -> {}, context
        );
        agent.chatWithEvents("Summarize the evidence", event -> {}, context);

        agent.executeAction(
            "answer", Map.of("question", "What changed?"), context
        );
        agent.executeAction(
            ActionRequest.of("answer", Map.of("question", "What changed?")),
            context
        );
        agent.executeActionOnRole(
            "researcher", "answer", Map.of("question", "What changed?"), context
        );
        agent.executeActionOnRole(
            "researcher",
            ActionRequest.of("answer", Map.of("question", "What changed?")),
            context
        );
    }
}

Cancellation is cooperative. A retriever that blocks on its own backend has to apply the deadline itself. For what the context does to knowledge retrieval, see Ground an action.

AsyncAgent

com.tnsai.agents.async.AsyncAgent is an interface with no implementation in the framework. To use it, implement it yourself.