Providers

Create a client

LLMClientFactory picks the client class from a provider key and reads the API key from the environment:

import com.tnsai.llm.LLMClient;
import com.tnsai.llm.LLMClientFactory;
import com.tnsai.llm.ChatResponse;

LLMClient client = LLMClientFactory.create("openai", "gpt-4o", 0.7f);
ChatResponse response = client.chat("What is quantum computing?");

The provider key is case-insensitive. A null key means ollama. A blank model, or a key the factory does not know, throws IllegalArgumentException.

LLMClientFactory.fromAnnotation(MyRole.class) builds the client from the role's @RoleSpec. It returns null when the class has no @RoleSpec or no model.

Provider keys

Model names change often. Use the model string your provider documents.

Keys and endpoints

A client takes its API key from the constructor, or else from the provider's <PROVIDER>_API_KEY variable, such as OPENAI_API_KEY or ANTHROPIC_API_KEY. The base URL comes from the constructor, then <PROVIDER>_BASE_URL, then the provider default. Configuration reference lists every variable.

The local servers need no key:

Ollama switches to Ollama Cloud when it has a key or an ollama.com URL.

Construct a client directly

Construct the class yourself for a key, endpoint or setting the factory does not take:

import com.tnsai.llm.providers.AnthropicClient;
import com.tnsai.llm.providers.AzureOpenAIClient;
import com.tnsai.llm.providers.OllamaClient;
import com.tnsai.llm.providers.OpenAIClient;

// Model, temperature, topP, maxTokens.
LLMClient openai = new OpenAIClient("gpt-4o", 0.7f, 0.95f, 4096);

// Base URL (null for the default), model, temperature, maxTokens, API key.
LLMClient anthropic = new AnthropicClient(
    null, "claude-sonnet-4-20250514", 0.7f, 4096, myAnthropicKey);

// Base URL, model, temperature, maxTokens, API key (null for a local server).
LLMClient ollama = new OllamaClient("http://gpu-server:11434", "llama3", 0.7f, 4096, null);

// Azure has factories instead of a public constructor.
LLMClient azure = AzureOpenAIClient.withApiKey(
    "https://myresource.openai.azure.com", myAzureKey, "my-gpt4o-deployment");

AzureOpenAIClient.fromEnv() reads the endpoint, key and deployment from the environment. withAzureAD(endpoint, token, deploymentId) signs in with a Microsoft Entra ID token.

Streaming

streamChat returns a Stream<String> of text. It holds an HTTP connection open, so close it with try-with-resources:

import com.tnsai.llm.ChatRequest;
import java.util.stream.Stream;

try (Stream<String> tokens = client.streamChat(ChatRequest.of("Tell me a story"))) {
    tokens.forEach(System.out::print);
}

ChatRequest.builder() takes message, messageParts, systemPrompt, history, tools and temperature. With tools, streamChatWithHandler first checks that the model supports tool calling.

Errors and retries

A failed call throws LLMException (com.tnsai.exceptions), a RuntimeException.

Most clients retry network failures, such as a refused connection or a socket timeout, up to three times. The backoff starts at one second and doubles, with a little jitter.

An HTTP error response is not retried. It is thrown at once as a typed LLMException that carries the HTTP status, the provider's error code and any Retry-After value. Wrap the client in a circuit breaker, or retry in your own code, when you want more.

Some clients differ:

Circuit breaker

CircuitBreakerClient (com.tnsai.llm.resilience) stops calling a provider that keeps failing:

import com.tnsai.llm.resilience.CircuitBreakerClient;
import java.time.Duration;

CircuitBreakerClient resilient = CircuitBreakerClient.builder()
    .client(openai)
    .failureThreshold(3)
    .recoveryTimeout(Duration.ofSeconds(60))
    .build();

// Or the defaults: 5 failures, 30 seconds.
CircuitBreakerClient withDefaults = CircuitBreakerClient.wrap(openai);

getState(), getConsecutiveFailures() and reset() inspect and reset the breaker. getMetrics() returns a CircuitBreakerMetrics record with request, success, failure and rejection counts, the number of state transitions, and successRate().

JSON mode

JsonModeClient (com.tnsai.llm.format) makes a client answer in JSON:

import com.tnsai.llm.format.JsonModeClient;

JsonModeClient json = JsonModeClient.wrap(openai);
ChatResponse response = json.chat("List 3 programming languages");

OpenAIClient, MiniMaxClient and ZhipuAIClient have a native JSON mode, and JsonModeClient uses it. All three take jsonObject(). Only OpenAIClient takes jsonSchema(...), and only for gpt-4o and gpt-4-turbo models. In every other case, JsonModeClient adds JSON instructions, with the schema if there is one, to the prompt.

Parse into a type

chatAs asks for JSON and maps the answer onto a class with Jackson. It strips a fenced json block first:

record LanguageList(List<String> languages) {}

LanguageList list = json.chatAs(LanguageList.class, "List 3 programming languages");
LanguageList withSystem = json.chatAs(LanguageList.class, "List 3 languages",
    Optional.of("You are a helpful assistant"));

A reply that does not parse throws JsonModeClient.JsonParseException. getRawContent() returns what the model sent.

Builder and formats

import com.tnsai.llm.format.ResponseFormat;

JsonModeClient person = JsonModeClient.builder()
    .client(openai)                          // Required
    .schemaFromClass("Person", Person.class) // Or .responseFormat(...)
    .forcePromptEngineering(false)           // true skips native JSON mode
    .build();

The builder also takes your own Jackson objectMapper.

SchemaGenerator.generateSchema(Class) builds a JSON Schema from a class's fields, and lists only primitive fields as required. SchemaGenerator.generateRecordSchema(Class) takes a record and lists every component as required.

Capabilities

client.getCapabilities() returns the model's LLMCapabilities:

knowsInputWindow() is false when getMaxInputTokens() is a placeholder for a model the client does not list. Treat the window as unknown then.

meetsRequirements(requiresVision, requiresTools, minContextTokens) checks three needs at once. To fail early instead, call:

LLMCapabilities caps = client.getCapabilities();
caps.requireToolCalling(); // ToolCallNotSupportedException
caps.requireStreaming();   // LLMCapabilityException
caps.requireVision();      // LLMCapabilityException

Images, audio and video

The multimodal chat overload takes a list of ContentParts from com.tnsai.models.chat:

import com.tnsai.models.chat.ContentPart;
import com.tnsai.models.chat.ImagePart;
import com.tnsai.models.chat.TextPart;

List<ContentPart> parts = List.of(
    new TextPart("What do you see in this image?"),
    ImagePart.fromBase64(base64Png, "image/png"));

if (client.getCapabilities().supportsVision()) {
    ChatResponse answer = client.chat(parts,
        Optional.empty(), Optional.empty(), Optional.empty());
}

A client without multimodal support throws UnsupportedOperationException from this overload. Check the capabilities first. For speech to text, see Audio.

Related

• Routing: pick a client per request.
• LLM calls: observe and log every call.
• Configuration reference: every provider variable.