LLM
An agent talks to a model through an LLMClient. Pick a provider client, then add routing, caching and cost tracking around it as your app needs them.
- Providers: the shipped clients, their constructors and keys, and how to add your own.
- Routing: pick a model per request by size, cost or capability.
- Configuration: change an agent's provider, model or tuning from a JSON file or the environment, without recompiling.
- Caching: cache responses, exactly or by meaning.
- Cost: track spend per agent and session, and enforce budgets.
- LLM calls: capture every LLM call as an
LLMCallLogevent with prompt, response, usage, cost and timing. - Audio: speech-to-text and translation with
WhisperClient.
For text to speech, see Images and speech.