# Routing

A router holds several clients and picks one per request. Every router implements `LLMRouter`, which extends `LLMClient`, so you pass a router anywhere a client goes, including `AgentBuilder.llm(...)`. The routers live in `tnsai-llm`, in the package `com.tnsai.llm.routing`.

| Router | Picks | Moves to the next client on failure |
|---|---|---|
| `FallbackRouter` | The clients in the order you list them | Yes |
| `CostBasedRouter` | The lowest declared input price first | Yes |
| `LatencyBasedRouter` | The lowest average latency first | Yes |
| `CapabilityRouter` | The clients that meet a capability requirement | Yes |
| `RoundRobinRouter` | The next client in rotation | Yes |
| `TaskBasedRouter` | The client mapped to the message's task type | No |

Each router needs at least one client, or it throws `IllegalArgumentException`. When every client fails, the router throws `LLMException` with the last failure as its cause.

## Prices and latencies come from the model table

`CostBasedRouter`, `LatencyBasedRouter` and `CapabilityRouter` read `client.getCapabilities()`. The provider clients answer from a built-in table of known models, matched on the model name, so a model the table does not know gets conservative defaults:

- no price, so `CostBasedRouter` tries it last;
- an estimated latency of 1000 ms until it has been measured;
- no vision, tool calling or structured output, and a 4096-token input window.

Check what a client declares before you rely on a router:

```java
LLMCapabilities caps = client.getCapabilities();
caps.getInputCostPer1KTokens();   // Optional<Double>
caps.getEstimatedLatencyMs();     // Optional<Long>
caps.supportsVision();
```

See [Providers: capabilities](providers.md#capabilities) for the full interface.

## `FallbackRouter`

Tries the clients in order and moves on when one throws.

```java
import com.tnsai.llm.providers.AnthropicClient;
import com.tnsai.llm.providers.GroqClient;
import com.tnsai.llm.providers.OpenAIClient;
import com.tnsai.llm.routing.FallbackRouter;
import com.tnsai.llm.routing.LLMRouter;

LLMRouter router = FallbackRouter.of(
    new OpenAIClient("gpt-4o"),
    new AnthropicClient("claude-sonnet-4-20250514"),
    new GroqClient("llama-3.3-70b-versatile"));

ChatResponse response = router.chat("Hello");
// OpenAI first, then Anthropic, then Groq.
```

`of(...)` takes clients or a `List<LLMClient>`. The builder adds a filter:

```java
FallbackRouter router = FallbackRouter.builder()
    .addClient(new OpenAIClient("gpt-4o"))
    .addClient(new AnthropicClient("claude-sonnet-4-20250514"))
    .requireFunctionCalling()   // or requireVision(), or requireCapability(predicate)
    .build();
```

Clients that fail the filter are skipped. If none passes, the router has nothing to try and throws `LLMException`.

## `CostBasedRouter`

Sorts the clients by declared input price per 1K tokens, cheapest first, and falls back to the next one on failure. Clients without a price go last.

```java
import com.tnsai.llm.routing.CostBasedRouter;

CostBasedRouter router = CostBasedRouter.of(
    new OpenAIClient("gpt-4o"),                      // 0.0025 per 1K input tokens
    new OpenAIClient("gpt-4o-mini"),                 // 0.00015
    new GroqClient("llama-3.3-70b-versatile"));      // 0.00059
// Tries gpt-4o-mini, then Groq, then gpt-4o.
```

The builder takes `addClient`, `requireCapability`, `requireVision` and `requireFunctionCalling`. As with `FallbackRouter`, a filter that matches no client leaves nothing to try.

`getCostStats()` returns a `CostStats` record:

| Component | Meaning |
|---|---|
| `totalEstimatedCost` | Estimated spend across all clients |
| `costPerProvider` | Estimated spend per `provider/model` |
| `totalInputTokens` | Estimated input tokens |
| `totalOutputTokens` | Estimated output tokens |

The token counts are estimates at four characters per token, not the provider's usage figures. For billed usage, see [Cost](cost.md).

## `LatencyBasedRouter`

Measures each call and sends the next request to the client with the lowest average over its last 20 calls. Before a client has been measured, its estimated latency from the model table stands in. A failed call is recorded as its duration plus 5000 ms, which pushes the client down the order.

```java
import com.tnsai.llm.routing.LatencyBasedRouter;

LatencyBasedRouter router = LatencyBasedRouter.builder()
    .addClient(new GroqClient("llama-3.3-70b-versatile"))
    .addClient(new OpenAIClient("gpt-4o"))
    .maxLatencyMs(500)
    .build();
```

`maxLatencyMs` leaves out clients whose average is above the limit. The builder also takes `requireCapability`, `requireVision` and `requireFunctionCalling`. If the filters leave no client, the router uses all of them.

`getLatencyStats()` returns `avgLatencyPerProvider`, `fastestProvider` and `fastestLatencyMs`.

## `CapabilityRouter`

Keeps the clients that meet a requirement you set on the builder, in the order you added them, and falls back down that list on failure. It does not look at the request: a message with an image does not select a vision model by itself. Set the requirement for the workload, or pick a client explicitly.

```java
import com.tnsai.llm.providers.OllamaClient;
import com.tnsai.llm.routing.CapabilityRouter;

CapabilityRouter router = CapabilityRouter.builder()
    .addClient(new OllamaClient("llama3.2"))
    .addClient(new OpenAIClient("gpt-4o"))
    .requireVision()   // or requireFunctionCalling(), requireStructuredOutput(), defaultRequirement(predicate)
    .build();
// Only gpt-4o declares vision, so every request goes there.
```

If no client meets the requirement, the router uses all of them.

To pick a client yourself:

| Method | Returns the first client that |
|---|---|
| `selectVisionCapable()` | Supports vision |
| `selectToolCapable()` | Supports function calling |
| `selectStructuredOutputCapable()` | Supports structured output |
| `selectWithMinContext(int minTokens)` | Has at least `minTokens` of input window |
| `selectByCapability(predicate)` | Matches the predicate |

Each returns `Optional<LLMClient>`. `selectByCapability` is on every router.

## `RoundRobinRouter`

Starts each request at the next client in rotation. On failure it tries the following clients, once each.

```java
import com.tnsai.llm.providers.GeminiClient;
import com.tnsai.llm.routing.RoundRobinRouter;

RoundRobinRouter router = RoundRobinRouter.of(
    new OpenAIClient("gpt-4o"),
    new AnthropicClient("claude-sonnet-4-20250514"),
    new GeminiClient("gemini-2.5-flash"));
```

`of(...)` takes clients or a list. The builder takes `addClient`.

## `TaskBasedRouter`

Classifies each message into a `TaskType` and sends it to the client mapped to that type. It sends one request and does not fall back: if that client throws, the exception reaches you. Wrap the mapped clients in a `FallbackRouter` when you need failover per task.

```java
import com.tnsai.llm.routing.TaskBasedRouter;
import com.tnsai.llm.routing.TaskType;

TaskBasedRouter router = TaskBasedRouter.builder()
    .forTask(TaskType.CODING, new AnthropicClient("claude-sonnet-4-20250514"))
    .forTask(TaskType.MATH, new OpenAIClient("o1"))
    .forTask(TaskType.FAST, new GroqClient("llama-3.3-70b-versatile"))
    .defaultClient(new OpenAIClient("gpt-4o-mini"))
    .build();

router.chat("Write a Python function to sort a list");  // CODING, Anthropic
router.chat("Solve: integral of x squared dx");          // MATH, o1
router.chat("What is 2+2?");                             // MATH: "2+2" matches the equation pattern
router.chat("Write a poem about autumn");                // CREATIVE at 0.2, below 0.3, so the default client
```

### Builder

| Method | Meaning | Default |
|---|---|---|
| `forTask(type, client)` | Maps a task type to a client | None |
| `defaultClient(client)` | The client for unmapped types and low confidence. `build()` throws `IllegalStateException` without it. | Required |
| `classifier(classifier)` | A custom `TaskClassifier` | `TaskClassifier.defaultClassifier()` |
| `confidenceThreshold(threshold)` | Below this confidence, the message is routed as `GENERAL` | `0.3` |
| `withCommonSetup(coding, reasoning, fast, general)` | Maps `CODING`; `MATH` and `REASONING`; `FAST` and `CHAT`; and sets the default | None |

`GENERAL` goes to the default client unless you map it with `forTask`.

### Other methods

```java
// Skip classification.
ChatResponse response = router.chatAs(TaskType.CODING, "Explain this concept");

// The client a type maps to, or the default client.
LLMClient coding = router.getClientForTask(TaskType.CODING);

// Classify without sending.
TaskClassifier.ClassificationResult result = router.classifyTask("Write a poem");
```

`chat` passes the `content` of each history message to the classifier. `streamChat` classifies the same way.

### Statistics

`getTaskStats()` returns a `TaskRoutingStats` record:

| Member | Meaning |
|---|---|
| `requestsPerTask()` | Requests per task type |
| `tokensPerTask()` | Estimated tokens per task type, at four characters per token |
| `estimatedSavings()` | Estimated savings against sending every token at 0.003 per 1K |
| `mostCommonTask()` | `Optional<TaskType>` |
| `taskDistribution()` | Percentage per task type |
| `routingStats()` | The `RoutingStats` below |

`resetStats()` clears the task counters and the routing counters.

## Task types

| `TaskType` | Example keywords |
|---|---|
| `CODING` | code, function, class, debug, refactor, python, java, algorithm, api |
| `CREATIVE` | write a story, poem, fiction, marketing, slogan, blog post |
| `MATH` | calculate, solve, equation, integral, derivative, theorem |
| `ANALYSIS` | analyze, summarize, extract, compare, report, metrics |
| `TRANSLATION` | translate, convert to, in english, in french |
| `CHAT` | hello, hi, what is, explain, tell me, why |
| `FAST` | quick, simple, brief, yes or no, one word |
| `REASONING` | think step by step, reason, logic, deduce, infer |
| `VISION` | image, picture, photo, screenshot, diagram |
| `GENERAL` | None. The fallback type. |

Each type also has Turkish keywords, such as `kod`, `şiir` and `çevir`. `getKeywords()` returns the full set.

Keywords match as lowercase substrings, so a short keyword can match inside a longer word: `hi` matches "this", and `go` matches "algorithm". `matches(text)` reports whether any keyword occurs, and `matchScore(text)` counts them. `TaskType.fromId("coding")` returns `CODING`, and an unknown id returns `GENERAL`.

## `TaskClassifier`

`TaskClassifier.defaultClassifier()` scores every task type and picks the highest:

| Signal | Points |
|---|---|
| A fenced code block | `CODING` +8 |
| A declaration keyword, such as `def `, `function `, `public ` or `import ` | `CODING` +6 |
| A source file extension, such as `.java` or `.py` | `CODING` +4 |
| An arithmetic expression or equation, such as `2+2` or `x^2` | `MATH` +6 |
| A math function name, such as `integral`, `sqrt` or `sin` | `MATH` +8 |
| A reference to an image, such as "this image" or "the photo" | `VISION` +10 |
| Each custom pattern that matches | +5 |
| Each built-in keyword that occurs | +2 |
| Each custom keyword that occurs | +3 |
| A history message, of the last three, that contains a keyword of the type | +1.5 |

Confidence is the best score divided by 10, capped at 1.0. Below the classifier's own minimum, 0.1 by default, the result is the default type.

```java
import com.tnsai.llm.routing.TaskClassifier;
import com.tnsai.llm.routing.TaskClassifier.ClassificationResult;

TaskClassifier classifier = TaskClassifier.defaultClassifier();

TaskType type = classifier.classify("Write a Python function to sort a list");   // CODING

ClassificationResult result = classifier.classifyWithConfidence("Solve x^2 + 2x + 1 = 0");
result.taskType();     // MATH
result.confidence();   // 0.8: equation 6, "solve" 2
result.reason();       // "Pattern and keyword match"
result.isConfident(0.5);

List<String> history = List.of("I'm working on a Java service", "Help me debug it");
TaskType withHistory = classifier.classify("What's wrong here?", history);
```

`classify` and `classifyWithConfidence` take an optional history list. An empty message returns the default type with confidence 0.

A custom classifier:

```java
TaskClassifier classifier = TaskClassifier.builder()
    .addKeywords(TaskType.CODING, Set.of("backend", "frontend", "docker"))
    .addPattern(TaskType.MATH, Pattern.compile("\\d+%"))
    .setDefaultType(TaskType.CHAT)
    .setMinConfidenceThreshold(0.2)
    .build();
```

## `RoutingStats`

Every router returns `getStats()`, and `resetStats()` clears it.

| Component | Meaning |
|---|---|
| `totalRequests` | Requests received |
| `successfulRequests` | Requests that returned a response |
| `failedRequests` | Requests where every client failed |
| `requestsPerProvider` | Attempts per `provider/model` |
| `failuresPerProvider` | Failed attempts per `provider/model` |
| `avgLatencyPerProvider` | Average latency per `provider/model`, filled by the fallback and latency routers |

## Related

- [Providers](providers.md): the clients a router holds, and the circuit breaker.
- [Cost](cost.md): billed usage and budgets.
- [LLM calls](../observability/llm-calls.md): observe each call.