Cost

Track a client's cost

Wrap a client with CostAwareLLMClient. Each call is recorded as a UsageRecord, priced from ModelPricing.

import com.tnsai.llm.cost.CostAwareLLMClient;
import com.tnsai.llm.cost.InMemoryCostTracker;

CostAwareLLMClient tracked = CostAwareLLMClient.wrap(client);

// Or with every option:
CostAwareLLMClient tracked = CostAwareLLMClient.builder()
    .client(client)
    .costTracker(InMemoryCostTracker.builder().maxRecords(10_000).build())
    .budgetManager(budget)
    .build();

wrap(client, budget) is the short form with a budget. InMemoryCostTracker.maxRecords defaults to 0, which keeps every record. Over the limit, the oldest record is dropped.

What gets recorded

• Model: the record uses the wrapped client's getModel().
• Tokens: chat takes token counts from the response. When the response has none, both input and output are estimated from the response length, divided by estimatedInputTokensPerChar.
• Streaming: each streamed chunk counts as one output token, and input is estimated from the prompt length. The record is written when you close the stream.
• Price: ModelPricing.forModel matches the model ID exactly, else by the longest registered prefix. gpt-4o-2024-08-06 is priced as gpt-4o. A model with no match costs 0.

Read the totals

From the client:

From the tracker:

import com.tnsai.llm.cost.CostTracker;
import java.math.BigDecimal;
import java.time.Instant;
import java.time.temporal.ChronoUnit;

CostTracker tracker = tracked.getCostTracker();

BigDecimal total = tracker.getTotalCost();
BigDecimal gpt4o = tracker.getCostByModel().get("gpt-4o");
BigDecimal lastHour = tracker.getTotalCost(
    Instant.now().minus(1, ChronoUnit.HOURS), Instant.now());

Time ranges include from and exclude to.

CostStatistics holds the record count, token totals (input, output, cached), input, output and total cost, the average cost and latency per request, the most expensive request, cost by model and by provider, and the time span.

Record usage yourself

Record calls made outside a wrapped client:

import com.tnsai.llm.cost.UsageRecord;

tracker.record(UsageRecord.builder()
    .modelId("gpt-4o")
    .inputTokens(1000)
    .outputTokens(500)
    .build());

modelId is required. build() prices the record, and getProvider() comes from the pricing entry unless you set provider. The builder also takes cachedInputTokens, latencyMs, timestamp, id and metadata. Read the cost with getInputCost(), getOutputCost(), getCachedInputCost() and getTotalCost().

Limit a client's spend: BudgetManager

import com.tnsai.llm.cost.BudgetManager;

BudgetManager budget = BudgetManager.builder()
    .limit("50.00")
    .daily()
    .defaultAlertThresholds()
    .onAlert(alert -> System.out.println(alert))
    .build();

When a call is blocked

Before a chat(String, …) call, CostAwareLLMClient estimates the cost: the prompt, system prompt and history length for input, and the client's max tokens (else 1000) for output. With a hard limit, it throws com.tnsai.llm.cost.BudgetExceededException when that estimate would take spend over the limit. The exception carries getCurrentSpend(), getLimit(), getEstimatedCost() and getRemaining().

Only that chat overload checks. Multimodal chat, chat with a temperature, and the streaming methods record spend but never block. With estimateBeforeRequest(false), nothing blocks. A model with no price estimates to 0, so it is blocked only once recorded spend is already over the limit.

Status

getStatus() returns a BudgetStatus(state, currentSpend, limit, remaining, usagePercent, periodStart, remainingTime).

The period restarts on the first budget call after it ends, which clears the spend and the triggered alerts. reset() restarts it at once. canSpend(amount) checks without recording. trySpend(amount) records only when the amount fits.

Model prices

ModelPricing holds about 50 models, in USD per million tokens. A sample:

The table also covers older OpenAI, Anthropic and Gemini models, Groq, Mistral, OpenRouter, ZhipuAI and MiniMax. A model ID is priced only if it starts with a registered ID, so claude-sonnet-4 alone has no price.

import com.tnsai.llm.cost.ModelPricing;
import java.math.BigDecimal;
import java.util.Optional;

Optional<ModelPricing> pricing = ModelPricing.forModel("gpt-4o");
BigDecimal cost = pricing
    .map(p -> p.calculateTotalCost(1000, 500))
    .orElse(BigDecimal.ZERO);

// Add or replace a price, per million tokens.
ModelPricing.registerPricing("my-model", "MyProvider",
    new BigDecimal("1.00"), new BigDecimal("2.00"));

ModelPricing also has calculateInputCost, calculateOutputCost, calculateCachedInputCost, calculateTotalCost(input, cached, output), getInputPricePerMillion, getOutputPricePerMillion, getCachedInputPricePerMillion and supportsCachedPricing.

Tenant budgets

Tenant budgets keep spend in a shared store, so one cap can cover many clients and agents. You record spend into the store and check it before each call.

Budgets and scopes

import com.tnsai.quality.cost.BudgetPolicy;
import com.tnsai.quality.cost.CostBudget;
import java.math.BigDecimal;
import java.time.Duration;

CostBudget tenantCap = CostBudget.tenantWide("acme",
    new BigDecimal("500.00"), Duration.ofDays(1), BudgetPolicy.HARD_STOP);

CostBudget summarizerCap = CostBudget.forAgent("acme", "summarizer-agent",
    new BigDecimal("50.00"), Duration.ofDays(1), BudgetPolicy.SOFT_WARN);

CostBudget(tenantId, agentId, capabilityId, periodBudgetUSD, period, onExceed) takes an Optional agent and capability. The budget and period must be positive, and the tenant must not be blank.

Spend is recorded against a CostScope: CostScope.tenant(t), CostScope.agent(t, a) or CostScope.full(t, a, c). A budget sees every spend inside its scope. A tenant budget counts all of the tenant's agents, and an agent budget counts all of that agent's capabilities.

Record and evaluate

import com.tnsai.quality.cost.BudgetState;
import com.tnsai.quality.cost.CostBudgetEvaluator;
import com.tnsai.quality.cost.CostBudgetStore;
import com.tnsai.quality.cost.CostScope;
import com.tnsai.quality.cost.InMemoryCostBudgetStore;

CostBudgetStore store = new InMemoryCostBudgetStore(Duration.ofDays(31));
CostBudgetEvaluator evaluator = new CostBudgetEvaluator(store);

// After each call, for example from an LLMCallLog named call:
store.record(CostScope.full("acme", "summarizer-agent", "extractive-summary"),
    call.cost().totalUSD());

// Before the next call:
BudgetState state = evaluator.evaluate(tenantCap);
switch (state.status()) {
    case HEALTHY -> { }
    case WARNING -> System.out.println("At " + (int) (state.fractionConsumed() * 100) + "%");
    case EXHAUSTED -> throw new IllegalStateException("acme is over its daily cap");
}

InMemoryCostBudgetStore keeps entries for maxAge, so set it at least as long as your longest budget period. A budget period is a rolling window: the last 24 hours for Duration.ofDays(1), not a calendar day.

BudgetState holds budget, spentUSD, remainingUSD, fractionConsumed, status and evaluatedAt. The status is HEALTHY below 80%, WARNING from 80% to 100%, and EXHAUSTED at 100% or more.

To share spend across processes, implement CostBudgetStore: record(scope, amountUSD), currentSpend(scope, period) and clear().

Enforce with a hook

CostBudgetEnforcer is a Hook<PreAction> that checks a budget before LLM actions:

import com.tnsai.hooks.HookDispatcher;
import com.tnsai.hooks.HookFailurePolicy;
import com.tnsai.quality.cost.CostBudgetEnforcer;
import java.util.Map;

Map<String, CostBudget> budgetByTenant = Map.of("acme", tenantCap);

CostBudgetEnforcer enforcer = new CostBudgetEnforcer(store,
    event -> event.context().tenantId().map(budgetByTenant::get));

HookDispatcher dispatcher = new HookDispatcher(Duration.ofSeconds(5), HookFailurePolicy.FAIL_CLOSED);
dispatcher.register(enforcer);

• Matching: it handles only PreAction events of type ActionType.LLM, at priority -100, before other hooks.
• Resolver: the second argument maps an event to its budget. With no budget, the call is allowed.
• Estimate: an optional third argument, a Function<PreAction, BigDecimal>, estimates the call's cost. The two-argument form uses 0, so a call is caught only once recorded spend reaches the cap.
• Trigger: the policy applies when spend plus the estimate reaches the budget.

TnsAI does not dispatch PreAction events itself. Call dispatcher.dispatch(event) before each LLM call and stop on a Block. The default HookDispatcher() is fail-open: a hook that throws or times out allows the call. Use FAIL_CLOSED for enforcement.

Related

• LLM calls: LLMCallLog and its cost estimate.
• Caching: cut cost with response and prompt caching.
• Routing: pick a client by cost.