# Caching

TnsAI has two kinds of caching, both in `tnsai-llm`, package `com.tnsai.llm.cache`:

| | `CachedLLMClient` | `PromptCachingClient` |
|---|---|---|
| What it caches | Whole responses, in your JVM | The prompt prefix, on Anthropic's side |
| Saves | The API call | Input-token cost and latency |
| Matches | Similar prompts, by embedding similarity | An identical prefix |
| Providers | Any `LLMClient` | Anthropic |

## Response cache: `CachedLLMClient`

`CachedLLMClient` wraps any `LLMClient`. Before a call it looks up the prompt in a `SemanticCache`. On a hit it returns the stored text without calling the provider.

```java
import com.tnsai.llm.LLMClient;
import com.tnsai.llm.cache.CachedLLMClient;
import com.tnsai.llm.cache.InMemorySemanticCache;
import com.tnsai.llm.cache.OllamaEmbeddingProvider;

LLMClient cached = CachedLLMClient.wrap(baseClient)
    .withCache(InMemorySemanticCache.builder()
        .embeddingProvider(new OllamaEmbeddingProvider())
        .ttlSeconds(3600)
        .maxEntries(10_000)
        .build())
    .highThreshold(0.95)
    .lowThreshold(0.70)
    .build();
```

| `wrap(client)` builder | Default | Meaning |
|---|---|---|
| `withCache(SemanticCache)` | `InMemorySemanticCache.builder().build()` | Where responses are stored |
| `highThreshold(double)` | `0.95` | Minimum similarity for a streaming hit |
| `lowThreshold(double)` | `0.70` | Minimum similarity for a `chat` hit |
| `cacheStreaming(boolean)` | `true` | Use the cache for `streamChat` |

### When the cache is used

- `chat` with tools or history goes straight to the provider. It is not looked up and not stored.
- `chat` without them returns the cached text when the best match scores at least `lowThreshold`. There is no extra check between the two thresholds: a match at 0.75 is served like a match at 0.99.
- `streamChat` returns a cached response only at `highThreshold` or above, as a one-element stream. On a miss it streams from the provider and stores the full text when you close the stream. Unlike `chat`, it does not skip calls with tools or history.
- With `cacheStreaming(false)`, `streamChat` neither looks up nor stores.
- `streamChatWithSpec` always goes to the provider.

A cached `ChatResponse` carries the text and no token usage.

The system prompt is part of the key when the cache has `includeSystemPrompt(true)`, which is the default. The same question under a different system prompt is a miss.

Pick `lowThreshold` with care. A lower value means more hits, and more answers to a question that was only similar.

### Embedding providers

The cache compares prompts as vectors from an `EmbeddingProvider`:

| Provider | Matches | Use |
|---|---|---|
| `HashEmbeddingProvider` (default) | Only the same text, after trimming and lowercasing | Tests and development |
| `OllamaEmbeddingProvider` | Similar meaning | Real similarity matching |

With the default provider, the cache behaves as an exact-match cache.

`OllamaEmbeddingProvider` uses the model `nomic-embed-text`. It reads `OLLAMA_BASE_URL` (else `http://localhost:11434`) and `OLLAMA_API_KEY`. Pass a model, or a base URL, model and key, to its constructors. Pull the model on the Ollama server first.

To use another embedding service, implement `EmbeddingProvider`: `embed`, `embedBatch`, `getDimension` and `getModelName`. `EmbeddingProvider.cosineSimilarity(a, b)` is available as a static helper.

### `InMemorySemanticCache`

| Builder | Default | Meaning |
|---|---|---|
| `embeddingProvider(EmbeddingProvider)` | `new HashEmbeddingProvider()` | Turns prompts into vectors |
| `ttlSeconds(long)` | `3600` | Entry lifetime, counted from when it was stored |
| `maxEntries(int)` | `10000` | Size limit |
| `includeSystemPrompt(boolean)` | `true` | Make the system prompt part of the key |

A lookup checks an exact hash of the prompt first, then the similarity of every entry. Reading an entry does not extend its lifetime. When the cache is full, the entry read least recently is evicted. Expired entries are removed every 100 stores.

### Manage the cache

| Method | Meaning |
|---|---|
| `getCacheStats()` | `SemanticCache.CacheStats`: `hits`, `misses`, `puts`, `evictions`, `currentSize`, `hitRate` |
| `clearCache()` | Remove every entry |
| `warmCache(Map<String, String>)` | Store prompt and response pairs up front, with no system prompt |
| `getCache()` | The `SemanticCache` |
| `getDelegate()` | The wrapped client |

### Your own store

Implement `SemanticCache` to keep responses somewhere else, such as Redis:

| Method | Returns |
|---|---|
| `put(prompt, response)`, `put(prompt, systemPrompt, response)` | `CacheEntry` |
| `findSimilar(prompt, minSimilarity)`, `findSimilar(prompt, systemPrompt, minSimilarity)` | `Optional<CacheEntry>` |
| `findAllSimilar(prompt, minSimilarity, maxResults)` | `List<SimilarityResult>` |
| `getByHash(hash)` | `Optional<CacheEntry>` |
| `remove(id)` | `boolean` |
| `clear()`, `size()`, `getStats()`, `resetStats()` | |

`InMemorySemanticCache` is the built-in implementation.

## Prompt caching on Anthropic

Anthropic can cache a request prefix (system prompt, tools, early messages) and charge less to read it again. The request marks the prefix with `cache_control` breakpoints, at most four per request.

### On the client

`AnthropicClient` marks the system prompt and the last user messages of the history itself:

```java
import com.tnsai.llm.providers.AnthropicClient;

AnthropicClient claude = AnthropicClient.builder()
    .model("claude-sonnet-4-20250514")
    .withPromptCaching()
    .cacheLastNTurns(3)
    .build();
```

`withPromptCaching()` adds the prompt-caching beta header and turns the markers on. `cacheLastNTurns` defaults to 3 and is capped at 3, because the system prompt takes one breakpoint. Caching is off by default.

### `PromptCachingClient`

`PromptCachingClient` wraps a client and marks tools and history:

```java
import com.tnsai.llm.cache.PromptCachingClient;

PromptCachingClient client = PromptCachingClient.builder()
    .client(claude)
    .cacheTools(true)
    .cacheHistoryBreakpoints(2)
    .build();
```

| Builder | Default | Meaning |
|---|---|---|
| `client(LLMClient)` | Required | The client to wrap. `build()` throws `IllegalStateException` without it. |
| `cacheTools(boolean)` | `true` | Mark the last tool definition |
| `cacheHistoryBreakpoints(int)` | `4` | History breakpoints, capped at 4 |
| `cacheSystemPrompt(boolean)` | `true` | Stored and reported only. The system prompt is passed through unchanged. |
| `minTokensForCaching(int)` | `1024` | Stored only. Values below 1024 become 1024. |

When tools are marked, the tool marker takes one of the four breakpoints and history gets the rest. History breakpoints are spread evenly, never on the first or last message.

Only `AnthropicClient` turns these markers into `cache_control` on the wire. To cache the system prompt, enable it on `AnthropicClient` as shown above.

### Statistics

| Method | Meaning |
|---|---|
| `getTotalCacheReadTokens()` | Sum of cache-read tokens reported in responses |
| `getTotalCacheCreationTokens()` | Sum of cache-write tokens |
| `getRequestCount()` | `chat` calls made through the wrapper |
| `getCacheHitRate()` | `1 / requests` once any cache read was seen, else 0. It is not a per-request rate. |
| `getEstimatedSavings()` | `(read × 0.9 − write × 0.25) / (read + write)` |
| `resetStats()` | Reset the counters |

Streaming calls are not counted. `isCacheSystemPromptEnabled()`, `isCacheToolsEnabled()`, `getCacheHistoryBreakpoints()` and `getDelegate()` return the configuration.

Each `ChatResponse` also reports its own cache usage:

```java
ChatResponse response = client.chat("Hello", systemPrompt, history, tools);

if (response.hasCacheUsage()) {
    response.getCacheReadInputTokens().ifPresent(t -> System.out.println("Read: " + t));
    response.getCacheCreationInputTokens().ifPresent(t -> System.out.println("Written: " + t));
}
```

For cache prices and lifetimes, see Anthropic's [prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) documentation.

## Related

- [Providers](providers.md): `AnthropicClient` and the other clients.
- [Cost](cost.md): token cost tracking.