# Providers

`tnsai-llm` ships 31 provider clients. Each one implements `LLMClient`, so moving an agent to another provider changes the client, not the agent.

## Create a client

`LLMClientFactory` picks the client class from a provider key and reads the API key from the environment:

```java
import com.tnsai.llm.LLMClient;
import com.tnsai.llm.LLMClientFactory;
import com.tnsai.llm.ChatResponse;

LLMClient client = LLMClientFactory.create("openai", "gpt-4o", 0.7f);
ChatResponse response = client.chat("What is quantum computing?");
```

| Overload | Adds |
|---|---|
| `create(provider, model, temperature)` | |
| `create(provider, model, temperature, maxTokens)` | An output token cap. `null` keeps the provider default. |
| `create(provider, model, temperature, maxTokens, topP)` | Nucleus sampling. `null` keeps the provider default. |
| `create(provider, model, temperature, maxTokens, topP, settings)` | `LLMClientSettings`. Settings other than the defaults are accepted for `openai` and `ollama` only. |

The provider key is case-insensitive. A `null` key means `ollama`. A blank model, or a key the factory does not know, throws `IllegalArgumentException`.

`LLMClientFactory.fromAnnotation(MyRole.class)` builds the client from the role's `@RoleSpec`. It returns `null` when the class has no `@RoleSpec` or no model.

## Provider keys

| Provider | Keys | Class |
|---|---|---|
| OpenAI | `openai` | `OpenAIClient` |
| Anthropic | `anthropic`, `claude` | `AnthropicClient` |
| Google Gemini | `gemini`, `google` | `GeminiClient` |
| Google Vertex AI | `vertexai`, `vertex-ai`, `vertex` | `VertexAIClient` |
| xAI Grok | `xai`, `grok`, `xai-grok` | `XAIGrokClient` |
| Mistral | `mistral` | `MistralClient` |
| Cohere | `cohere` | `CohereClient` |
| DeepSeek | `deepseek` | `DeepSeekClient` |
| Perplexity | `perplexity`, `pplx` | `PerplexityClient` |
| Groq | `groq` | `GroqClient` |
| Cerebras | `cerebras` | `CerebrasClient` |
| NVIDIA NIM | `nvidia`, `nvidia-nim`, `nim` | `NvidiaNIMClient` |
| Together AI | `together`, `togetherai`, `together-ai` | `TogetherAIClient` |
| Fireworks AI | `fireworks`, `fireworksai`, `fireworks-ai` | `FireworksAIClient` |
| DeepInfra | `deepinfra`, `deep-infra` | `DeepInfraClient` |
| Replicate | `replicate` | `ReplicateClient` |
| Hugging Face | `huggingface`, `hf` | `HuggingFaceClient` |
| OpenRouter | `openrouter` | `OpenRouterClient` |
| AWS Bedrock | `bedrock`, `aws` | `BedrockClient` |
| Azure OpenAI | `azure`, `azure-openai`, `azureopenai` | `AzureOpenAIClient` |
| IBM watsonx.ai | `watsonx`, `ibm` | `WatsonxClient` |
| Databricks | `databricks`, `mosaic` | `DatabricksClient` |
| Alibaba Qwen | `qwen`, `dashscope`, `alibaba` | `QwenCloudClient` |
| Tencent Hunyuan | `hunyuan`, `tencent`, `tencent-hunyuan` | `TencentHunyuanClient` |
| Zhipu AI | `zhipu`, `glm` | `ZhipuAIClient` |
| MiniMax | `minimax` | `MiniMaxClient` |
| 01.AI Yi | `yi`, `01ai`, `01.ai` | `YiClient` |
| Ollama | `ollama` | `OllamaClient` |
| LM Studio | `lmstudio`, `lm-studio` | `LMStudioClient` |
| llama.cpp server | `llamacpp`, `llama-cpp`, `llama.cpp` | `LlamaCppServerClient` |
| vLLM | `vllm` | `VLLMClient` |

Model names change often. Use the model string your provider documents.

## Keys and endpoints

A client takes its API key from the constructor, or else from the provider's `<PROVIDER>_API_KEY` variable, such as `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`. The base URL comes from the constructor, then `<PROVIDER>_BASE_URL`, then the provider default. [Configuration reference](../reference/configuration.md) lists every variable.

The local servers need no key:

| Provider | Default base URL | Override |
|---|---|---|
| Ollama | `http://localhost:11434` | `OLLAMA_BASE_URL` |
| LM Studio | `http://localhost:1234/v1` | `LMSTUDIO_BASE_URL` |
| llama.cpp server | `http://localhost:8080/v1` | `LLAMACPP_BASE_URL` |
| vLLM | `http://localhost:8000/v1` | `VLLM_BASE_URL` |

Ollama switches to Ollama Cloud when it has a key or an `ollama.com` URL.

## Construct a client directly

Construct the class yourself for a key, endpoint or setting the factory does not take:

```java
import com.tnsai.llm.providers.AnthropicClient;
import com.tnsai.llm.providers.AzureOpenAIClient;
import com.tnsai.llm.providers.OllamaClient;
import com.tnsai.llm.providers.OpenAIClient;

// Model, temperature, topP, maxTokens.
LLMClient openai = new OpenAIClient("gpt-4o", 0.7f, 0.95f, 4096);

// Base URL (null for the default), model, temperature, maxTokens, API key.
LLMClient anthropic = new AnthropicClient(
    null, "claude-sonnet-4-20250514", 0.7f, 4096, myAnthropicKey);

// Base URL, model, temperature, maxTokens, API key (null for a local server).
LLMClient ollama = new OllamaClient("http://gpu-server:11434", "llama3", 0.7f, 4096, null);

// Azure has factories instead of a public constructor.
LLMClient azure = AzureOpenAIClient.withApiKey(
    "https://myresource.openai.azure.com", myAzureKey, "my-gpt4o-deployment");
```

`AzureOpenAIClient.fromEnv()` reads the endpoint, key and deployment from the environment. `withAzureAD(endpoint, token, deploymentId)` signs in with a Microsoft Entra ID token.

## Streaming

`streamChat` returns a `Stream<String>` of text. It holds an HTTP connection open, so close it with try-with-resources:

```java
import com.tnsai.llm.ChatRequest;
import java.util.stream.Stream;

try (Stream<String> tokens = client.streamChat(ChatRequest.of("Tell me a story"))) {
    tokens.forEach(System.out::print);
}
```

| Method | Gives you |
|---|---|
| `streamChat(ChatRequest)` | Text pieces |
| `streamChatWithSpec(ChatRequest)` | `ChatChunk` objects: `START`, `CONTENT`, `TOOL_CALL`, `DONE` or `ERROR` |
| `streamChatWithHandler(ChatRequest, Consumer<ChatChunk>)` | The same chunks, pushed to your callback |

`ChatRequest.builder()` takes `message`, `messageParts`, `systemPrompt`, `history`, `tools` and `temperature`. With tools, `streamChatWithHandler` first checks that the model supports tool calling.

## Errors and retries

A failed call throws `LLMException` (`com.tnsai.exceptions`), a `RuntimeException`.

Most clients retry network failures, such as a refused connection or a socket timeout, up to three times. The backoff starts at one second and doubles, with a little jitter.

An HTTP error response is not retried. It is thrown at once as a typed `LLMException` that carries the HTTP status, the provider's error code and any `Retry-After` value. Wrap the client in a [circuit breaker](#circuit-breaker), or retry in your own code, when you want more.

Some clients differ:

| Client | Retries |
|---|---|
| `OllamaClient` | Every I/O error |
| `AzureOpenAIClient` | Rate limits and HTTP 408, 425, 429, 500, 502, 503, 504 and 529, three times from one second. A content-filter rejection is not retried. |
| `BedrockClient` | None |

### Circuit breaker

`CircuitBreakerClient` (`com.tnsai.llm.resilience`) stops calling a provider that keeps failing:

```java
import com.tnsai.llm.resilience.CircuitBreakerClient;
import java.time.Duration;

CircuitBreakerClient resilient = CircuitBreakerClient.builder()
    .client(openai)
    .failureThreshold(3)
    .recoveryTimeout(Duration.ofSeconds(60))
    .build();

// Or the defaults: 5 failures, 30 seconds.
CircuitBreakerClient withDefaults = CircuitBreakerClient.wrap(openai);
```

| State | Behaviour |
|---|---|
| `CLOSED` | Calls go through. `failureThreshold` failures in a row open the circuit. |
| `OPEN` | Calls throw `CircuitOpenException` without reaching the provider, until `recoveryTimeout` has passed. |
| `HALF_OPEN` | One probe call goes through. Success closes the circuit, and failure opens it again. |

`getState()`, `getConsecutiveFailures()` and `reset()` inspect and reset the breaker. `getMetrics()` returns a `CircuitBreakerMetrics` record with request, success, failure and rejection counts, the number of state transitions, and `successRate()`.

## JSON mode

`JsonModeClient` (`com.tnsai.llm.format`) makes a client answer in JSON:

```java
import com.tnsai.llm.format.JsonModeClient;

JsonModeClient json = JsonModeClient.wrap(openai);
ChatResponse response = json.chat("List 3 programming languages");
```

`OpenAIClient`, `MiniMaxClient` and `ZhipuAIClient` have a native JSON mode, and `JsonModeClient` uses it. All three take `jsonObject()`. Only `OpenAIClient` takes `jsonSchema(...)`, and only for `gpt-4o` and `gpt-4-turbo` models. In every other case, `JsonModeClient` adds JSON instructions, with the schema if there is one, to the prompt.

### Parse into a type

`chatAs` asks for JSON and maps the answer onto a class with Jackson. It strips a fenced `json` block first:

```java
record LanguageList(List<String> languages) {}

LanguageList list = json.chatAs(LanguageList.class, "List 3 programming languages");
LanguageList withSystem = json.chatAs(LanguageList.class, "List 3 languages",
    Optional.of("You are a helpful assistant"));
```

A reply that does not parse throws `JsonModeClient.JsonParseException`. `getRawContent()` returns what the model sent.

### Builder and formats

```java
import com.tnsai.llm.format.ResponseFormat;

JsonModeClient person = JsonModeClient.builder()
    .client(openai)                          // Required
    .schemaFromClass("Person", Person.class) // Or .responseFormat(...)
    .forcePromptEngineering(false)           // true skips native JSON mode
    .build();
```

The builder also takes your own Jackson `objectMapper`.

| `ResponseFormat` | Asks for |
|---|---|
| `text()` | Plain text |
| `jsonObject()` | Any valid JSON object. The `wrap` default. |
| `jsonSchema(name, Map)` | JSON that matches the schema, in strict mode |
| `jsonSchema(name, Map, strict)` | The same, with strict mode chosen |
| `jsonSchema(name, Class)` | A schema generated from the class |

`SchemaGenerator.generateSchema(Class)` builds a JSON Schema from a class's fields, and lists only primitive fields as required. `SchemaGenerator.generateRecordSchema(Class)` takes a record and lists every component as required.

## Capabilities

`client.getCapabilities()` returns the model's `LLMCapabilities`:

| Group | Methods |
|---|---|
| Features | `supportsStreaming()`, `supportsVision()`, `supportsFunctionCalling()`, `supportsStructuredOutput()`, `supportsSystemPrompt()`, `supportsParallelFunctionCalls()` |
| Extras | `supportsCodeExecution()`, `supportsWebBrowsing()` and `supportsReasoning()`, `false` unless the client says otherwise. `supportsFileUpload()`, true when `FILE` is a supported modality. |
| Limits | `getMaxInputTokens()`, `getMaxOutputTokens()`, `getContextWindow()`, `knowsInputWindow()` |
| Input types | `getSupportedModalities()`, `supportsModality(Modality)`. `Modality` is `TEXT`, `IMAGE`, `AUDIO`, `VIDEO` or `FILE`. |
| Identity and price | `getProviderName()`, `getModelId()`, `getModelVersion()`, `getInputCostPer1KTokens()`, `getOutputCostPer1KTokens()`, `getEstimatedLatencyMs()` |

`knowsInputWindow()` is `false` when `getMaxInputTokens()` is a placeholder for a model the client does not list. Treat the window as unknown then.

`meetsRequirements(requiresVision, requiresTools, minContextTokens)` checks three needs at once. To fail early instead, call:

```java
LLMCapabilities caps = client.getCapabilities();
caps.requireToolCalling(); // ToolCallNotSupportedException
caps.requireStreaming();   // LLMCapabilityException
caps.requireVision();      // LLMCapabilityException
```

## Images, audio and video

The multimodal `chat` overload takes a list of `ContentPart`s from `com.tnsai.models.chat`:

| Part | Create with |
|---|---|
| `TextPart` | `new TextPart(text)` |
| `ImagePart` | `ImagePart.fromBase64(data, mimeType)` |
| `AudioPart` | `fromFile`, `fromBytes`, `fromBase64`, `fromUrl` |
| `VideoPart` | `fromFile`, `fromBytes`, `fromBase64`, `fromUrl` |

```java
import com.tnsai.models.chat.ContentPart;
import com.tnsai.models.chat.ImagePart;
import com.tnsai.models.chat.TextPart;

List<ContentPart> parts = List.of(
    new TextPart("What do you see in this image?"),
    ImagePart.fromBase64(base64Png, "image/png"));

if (client.getCapabilities().supportsVision()) {
    ChatResponse answer = client.chat(parts,
        Optional.empty(), Optional.empty(), Optional.empty());
}
```

A client without multimodal support throws `UnsupportedOperationException` from this overload. Check the capabilities first. For speech to text, see [Audio](audio.md).

## Related

- [Routing](routing.md): pick a client per request.
- [LLM calls](../observability/llm-calls.md): observe and log every call.
- [Configuration reference](../reference/configuration.md): every provider variable.