	`CachedLLMClient`	`PromptCachingClient`
What it caches	Whole responses, in your JVM	The prompt prefix, on Anthropic's side
Saves	The API call	Input-token cost and latency
Matches	Similar prompts, by embedding similarity	An identical prefix
Providers	Any `LLMClient`	Anthropic

Response cache: `CachedLLMClient`

CachedLLMClient wraps any LLMClient. Before a call it looks up the prompt in a SemanticCache. On a hit it returns the stored text without calling the provider.

import com.tnsai.llm.LLMClient;
import com.tnsai.llm.cache.CachedLLMClient;
import com.tnsai.llm.cache.InMemorySemanticCache;
import com.tnsai.llm.cache.OllamaEmbeddingProvider;

LLMClient cached = CachedLLMClient.wrap(baseClient)
    .withCache(InMemorySemanticCache.builder()
        .embeddingProvider(new OllamaEmbeddingProvider())
        .ttlSeconds(3600)
        .maxEntries(10_000)
        .build())
    .highThreshold(0.95)
    .lowThreshold(0.70)
    .build();

`wrap(client)` builder	Default	Meaning
`withCache(SemanticCache)`	`InMemorySemanticCache.builder().build()`	Where responses are stored
`highThreshold(double)`	`0.95`	Minimum similarity for a streaming hit
`lowThreshold(double)`	`0.70`	Minimum similarity for a `chat` hit
`cacheStreaming(boolean)`	`true`	Use the cache for `streamChat`

When the cache is used

chat with tools or history goes straight to the provider. It is not looked up and not stored.
chat without them returns the cached text when the best match scores at least lowThreshold. There is no extra check between the two thresholds: a match at 0.75 is served like a match at 0.99.
streamChat returns a cached response only at highThreshold or above, as a one-element stream. On a miss it streams from the provider and stores the full text when you close the stream. Unlike chat, it does not skip calls with tools or history.
With cacheStreaming(false), streamChat neither looks up nor stores.
streamChatWithSpec always goes to the provider.

A cached ChatResponse carries the text and no token usage.

The system prompt is part of the key when the cache has includeSystemPrompt(true), which is the default. The same question under a different system prompt is a miss.

Pick lowThreshold with care. A lower value means more hits, and more answers to a question that was only similar.

Embedding providers

The cache compares prompts as vectors from an EmbeddingProvider:

Provider	Matches	Use
`HashEmbeddingProvider` (default)	Only the same text, after trimming and lowercasing	Tests and development
`OllamaEmbeddingProvider`	Similar meaning	Real similarity matching

With the default provider, the cache behaves as an exact-match cache.

OllamaEmbeddingProvider uses the model nomic-embed-text. It reads OLLAMA_BASE_URL (else http://localhost:11434) and OLLAMA_API_KEY. Pass a model, or a base URL, model and key, to its constructors. Pull the model on the Ollama server first.

To use another embedding service, implement EmbeddingProvider: embed, embedBatch, getDimension and getModelName. EmbeddingProvider.cosineSimilarity(a, b) is available as a static helper.

`InMemorySemanticCache`

Builder	Default	Meaning
`embeddingProvider(EmbeddingProvider)`	`new HashEmbeddingProvider()`	Turns prompts into vectors
`ttlSeconds(long)`	`3600`	Entry lifetime, counted from when it was stored
`maxEntries(int)`	`10000`	Size limit
`includeSystemPrompt(boolean)`	`true`	Make the system prompt part of the key

A lookup checks an exact hash of the prompt first, then the similarity of every entry. Reading an entry does not extend its lifetime. When the cache is full, the entry read least recently is evicted. Expired entries are removed every 100 stores.

Manage the cache

Method	Meaning
`getCacheStats()`	`SemanticCache.CacheStats`: `hits`, `misses`, `puts`, `evictions`, `currentSize`, `hitRate`
`clearCache()`	Remove every entry
`warmCache(Map<String, String>)`	Store prompt and response pairs up front, with no system prompt
`getCache()`	The `SemanticCache`
`getDelegate()`	The wrapped client

Your own store

Implement SemanticCache to keep responses somewhere else, such as Redis:

Method	Returns
`put(prompt, response)`, `put(prompt, systemPrompt, response)`	`CacheEntry`
`findSimilar(prompt, minSimilarity)`, `findSimilar(prompt, systemPrompt, minSimilarity)`	`Optional<CacheEntry>`
`findAllSimilar(prompt, minSimilarity, maxResults)`	`List<SimilarityResult>`
`getByHash(hash)`	`Optional<CacheEntry>`
`remove(id)`	`boolean`
`clear()`, `size()`, `getStats()`, `resetStats()`

InMemorySemanticCache is the built-in implementation.

Prompt caching on Anthropic

Anthropic can cache a request prefix (system prompt, tools, early messages) and charge less to read it again. The request marks the prefix with cache_control breakpoints, at most four per request.

On the client

AnthropicClient marks the system prompt and the last user messages of the history itself:

import com.tnsai.llm.providers.AnthropicClient;

AnthropicClient claude = AnthropicClient.builder()
    .model("claude-sonnet-4-20250514")
    .withPromptCaching()
    .cacheLastNTurns(3)
    .build();

withPromptCaching() adds the prompt-caching beta header and turns the markers on. cacheLastNTurns defaults to 3 and is capped at 3, because the system prompt takes one breakpoint. Caching is off by default.

`PromptCachingClient`

PromptCachingClient wraps a client and marks tools and history:

import com.tnsai.llm.cache.PromptCachingClient;

PromptCachingClient client = PromptCachingClient.builder()
    .client(claude)
    .cacheTools(true)
    .cacheHistoryBreakpoints(2)
    .build();

Builder	Default	Meaning
`client(LLMClient)`	Required	The client to wrap. `build()` throws `IllegalStateException` without it.
`cacheTools(boolean)`	`true`	Mark the last tool definition
`cacheHistoryBreakpoints(int)`	`4`	History breakpoints, capped at 4
`cacheSystemPrompt(boolean)`	`true`	Stored and reported only. The system prompt is passed through unchanged.
`minTokensForCaching(int)`	`1024`	Stored only. Values below 1024 become 1024.

When tools are marked, the tool marker takes one of the four breakpoints and history gets the rest. History breakpoints are spread evenly, never on the first or last message.

Only AnthropicClient turns these markers into cache_control on the wire. To cache the system prompt, enable it on AnthropicClient as shown above.

Statistics

Method	Meaning
`getTotalCacheReadTokens()`	Sum of cache-read tokens reported in responses
`getTotalCacheCreationTokens()`	Sum of cache-write tokens
`getRequestCount()`	`chat` calls made through the wrapper
`getCacheHitRate()`	`1 / requests` once any cache read was seen, else 0. It is not a per-request rate.
`getEstimatedSavings()`	`(read × 0.9 − write × 0.25) / (read + write)`
`resetStats()`	Reset the counters

Streaming calls are not counted. isCacheSystemPromptEnabled(), isCacheToolsEnabled(), getCacheHistoryBreakpoints() and getDelegate() return the configuration.

Each ChatResponse also reports its own cache usage:

ChatResponse response = client.chat("Hello", systemPrompt, history, tools);

if (response.hasCacheUsage()) {
    response.getCacheReadInputTokens().ifPresent(t -> System.out.println("Read: " + t));
    response.getCacheCreationInputTokens().ifPresent(t -> System.out.println("Written: " + t));
}

For cache prices and lifetimes, see Anthropic's prompt caching documentation.

Providers: AnthropicClient and the other clients.
Cost: token cost tracking.