External LLM Configuration

Register a source on the bridge:

SCOPBridge bridge = SCOPBridge.getInstance();

// One JSON file per agent, re-read on every resolution
bridge.llmConfigurationSource(LLMConfigurationSource.folder(Path.of("/etc/tnsai/llm")));

// Or: TNSAI_LLM_<AGENT>_<FIELD> from System properties and the environment
bridge.llmConfigurationSource(LLMConfigurationSource.environment());

// Or: explicitly nothing, so the annotations stand alone
bridge.llmConfigurationSource(LLMConfigurationSource.noOp());

A folder source reads <folder>/<agentName>.json:

{
  "provider": "OPENAI",
  "model": "gpt-4o",
  "temperature": 0.0,
  "maxTokens": 2048
}

The keys are provider, model, temperature, maxTokens, endpoint and apiKeyEnv — every one optional. A key that is not in that set is an error, not a no-op, so a misspelled temprature fails loudly instead of quietly doing nothing.

The examples on this page are plain snippets, not compiled contract fixtures. These types are not in a published artifact yet, and the unreleased-main contract lane compiles against a pinned framework commit that predates them.

Where a value can come from

Four layers contribute, lowest precedence first:

1. Playground — @AgentSpec(llm = @LLMSpec(...)) on the playground class
2. Agent — @AgentSpec(llm = @LLMSpec(...)) on the agent class
3. Role — @RoleSpec(llm = @LLMSpec(...)) on the role class
4. External source — the registered LLMConfigurationSource

They do not all combine the same way. There are two rules, not one, and reading them as a single "higher layer overrides lower" rule is how you end up sending a prompt to the wrong provider.

Routing is taken as a unit

provider, model, endpoint and apiKeyEnv are routing. They come as a set from the highest-precedence annotation tier that declares a model. No other tier contributes any part of the routing — not its provider, not its endpoint.

A tier that names a routing field but no model contributes nothing, and logs a warning naming what it dropped.

@AgentSpec(llm = @LLMSpec(provider = Provider.OPENAI, model = "gpt-4o", apiKeyEnv = "OPENAI_API_KEY"))
public class SupportAgent extends Agent { }

@RoleSpec(llm = @LLMSpec(model = "llama3.2"))
public class SupportRole extends Role { }

// Role declares a model, so Role owns the whole routing set:
//   model    = llama3.2
//   provider = OLLAMA   <- @LLMSpec's own default, NOT the agent's OPENAI
//   endpoint = the Ollama default

The reason is worth stating plainly, because the alternative looks more helpful and is not: a model belongs to a provider, an endpoint speaks one provider's wire format, and a key is issued by one provider. Merging the four fields across tiers written by different authors produces a configuration nobody wrote. In the example above, a per-field merge would take the role's local model name and post it to api.openai.com under the agent's key.

If no layer supplies a model, resolution yields an empty result and warns. That is unchanged.

Tuning merges field by field

temperature and maxTokens are tuning. They merge independently across every tier, with higher precedence winning per field. A tier that declares only a temperature now contributes it, which is what makes a role able to say "be deterministic" without restating the model.

The external source is the only layer that overlays every field individually, routing included. It is one operator configuring one deployment, so requiring them to restate the provider in order to change a temperature is the duplication this layer removes.

temperature = 0.0 is a value, not a blank

Zero is a legitimate temperature and zero is a legitimate maxTokens. A plain float cannot tell "not overridden" from "set to zero", so every field is carried as an Optional, OptionalDouble or OptionalInt:

LLMOverrides deterministic = LLMOverrides.builder().temperature(0.0).build();
// temperature is present and 0.0 — it survives the merge and reaches the client

Write "temperature": 0.0 in a source file and it applies.

A member set to its own default reads as unwritten

@LLMSpec's members have real defaults, not sentinels: provider defaults to OLLAMA, temperature to 0.7f, maxTokens to 4096. Only model, endpoint and apiKeyEnv default to the empty string. Every annotated class therefore carries a full set of values whether or not its author wrote any of them, and annotation retention keeps no record of which members were actually typed.

So a member counts as declared only when it differs from the default @LLMSpec itself declares. The consequence: writing temperature = 0.7f explicitly is indistinguishable from not writing it, and a lower tier's temperature will win.

This is harmless for routing, because routing comes whole from one tier and no competing value can reach the configuration. It is not harmless for tuning, which still merges. If you need a value that happens to equal the default, set it from a source rather than an annotation.

When a source cannot be read

Nothing falls back. Answering with a model other than the configured one is worse than not answering, so LLMConfigurationException is thrown for:

• a file that cannot be read, or is not valid JSON
• a JSON value of the wrong type
• an unknown key
• a blank string value
• a file larger than 1 MiB
• a maxTokens that is fractional or outside int range
• a temperature outside 0.0–2.0
• a provider the bridge cannot route

A missing file is not an error. The source returns nothing and the annotations stand.

Validation applies to sources only. Annotation values are never re-validated: @LLMSpec(maxTokens = -1) is an established way to say "let the provider decide", and rejecting a value that is already compiled and shipped would break working deployments.

Which agent a source is asked about

The lookup key is the agent's own getName(), falling back to the simple class name. Pass it explicitly through the four-argument resolveLLMSpec overload when you need a different one.

Two things to know about folder lookup:

• Case sensitivity is the filesystem's, not ours. Researcher.json and researcher.json are one entry on macOS and two on Linux. A configuration that works on a developer's Mac can stop applying in a Linux container.
• An agent name that cannot be a file name — a path separator, .., a NUL — resolves to nothing and logs. It is never resolved against the filesystem.

Environment source keys

LLMConfigurationSource.environment() reads TNSAI_LLM_<AGENT>_<FIELD>, where <AGENT> is the agent name upper-cased with every non-alphanumeric character replaced by _, and <FIELD> is one of PROVIDER, MODEL, TEMPERATURE, MAX_TOKENS, ENDPOINT, API_KEY_ENV:

TNSAI_LLM_RESEARCHER_MODEL=gpt-4o
TNSAI_LLM_RESEARCHER_TEMPERATURE=0.0

Because the key is normalised, agent names differing only in punctuation — data-analyst and data_analyst — map to the same variable.

Values are read on every call for System properties and environment variables. They are not re-read from a .env file, which is loaded once per JVM: editing it, including removing an override, takes effect only after a restart.

Trust boundary

A configuration source is trusted infrastructure, not user input. Give it the same trust as the deployment's own configuration.

apiKeyEnv names any environment variable or System property the process can see, and endpoint decides where that value is sent as a bearer token, along with the request body. Whoever can write the configuration folder can therefore read and exfiltrate a secret that has nothing to do with LLMs, without any access to the code.

Do not point a source at a directory that users, uploads or a web request can write to, and do not build a feature that lets an end user supply these fields.

A source can also supply a model for an agent whose annotations declare none, which means it can enable LLM dispatch for an agent the developer never configured for it.

Each resolution logs the lookup key, which fields the source contributed, and the endpoint and apiKeyEnv values when it sets them. None of those is itself a secret — apiKeyEnv is a variable name, not its value.

Limitations

• An endpoint pin is ignored on the Core SPI client path. When tnsai-llm is on the classpath, client creation carries neither endpoint nor apiKeyEnv, so a configured endpoint is dropped and traffic goes to the provider default. A warning is logged. Tracked as TAN-6202.
• HUGGINGFACE has no default endpoint of its own. A source naming it is refused, because it would otherwise fall back to the Ollama endpoint carrying whatever apiKeyEnv came with it. Annotations are not checked, so @LLMSpec(provider = Provider.HUGGINGFACE) still misroutes. Pre-existing.
• maxTokens = 0 does not mean the same thing everywhere. Through the Core SPI path it means "let the provider decide". On the HTTP fallback path for Anthropic it becomes a hardcoded 4096.
• Provider matching is case-insensitive and normalised with Locale.ROOT, so "openai" resolves to OPENAI.

Related

• Providers — the wired clients and their default endpoints
• Routing — choosing between models per request
• SCOP Bridge — where resolution runs
• Configuration Reference — environment variables