Multimodal Tools

Pairs with the older OpenAI-only MEDIA_TOOLS (openai_tts, whisper_transcribe) — register both when you want OpenAI plus a non-OpenAI fallback in the same agent.

Quick start

Register the toolkits the same way as any other built-in tool. The framework instantiates the backing POJO, scans for @Tool methods, and exposes each as a separate function the LLM can call:

import com.tnsai.agents.AgentBuilder;
import com.tnsai.enums.BuiltInTool;
import com.tnsai.llm.providers.OpenAIClient;

Agent artist = AgentBuilder.create()
    .llm(new OpenAIClient("gpt-4o"))
    .role(myRole)
    .builtInTools(
        BuiltInTool.IMAGE_GEN_TOOLS,        // dalle3_generate, flux_generate, stability_generate
        BuiltInTool.TEXT_TO_SPEECH_TOOLS,   // elevenlabs_tts, cartesia_tts, deepgram_tts
        BuiltInTool.SPEECH_TO_TEXT_TOOLS    // deepgram_transcribe, assemblyai_transcribe, replicate_whisper
    )
    .build();

String response = artist.chat("Draw me a watercolor painting of a giraffe on Mars");
// LLM picks dalle3_generate, calls it; the agent reply embeds the URL.

Each @Tool reads its API key from a process environment variable on first call. Missing keys throw IllegalStateException with the exact variable name in the message.

Image generation — IMAGE_GEN_TOOLS

dalle3_generate and flux_generate return provider-hosted URLs. DALL-E URLs expire roughly an hour after generation — fetch and restage if you need persistence. FLUX URLs live for the duration documented by Replicate.

stability_generate returns image bytes, which the tool encodes inline as a data:image/png;base64,… data URI. The same pattern applies to every TTS method below — uniform shape so the agent doesn't need per-provider plumbing.

// Each method validates parameters before the HTTP call so a bad LLM
// argument fails fast with a clear message instead of a provider 4xx.
String json = new ImageGenTools().dalle3Generate(
    "A watercolor giraffe on Mars",
    "1024x1024",      // 1024x1024 | 1792x1024 | 1024x1792
    "hd",             // standard | hd
    1                 // DALL-E 3 supports n=1 only; tool rejects n>1 upfront
);

Defaults are provider-appropriate: DALL-E 1024x1024 standard, FLUX schnell model with 1:1 aspect ratio in webp, Stability core model with 1:1 aspect ratio.

Text-to-speech — TEXT_TO_SPEECH_TOOLS

All three return the same envelope:

{
  "provider": "elevenlabs",
  "model": "eleven_multilingual_v2",
  "audio_uri": "data:audio/mpeg;base64,SUQzBAAAAAAAJ...",
  "audio_bytes": 45920
}

audio_uri is ready for <audio src=…> playback or attachment to a message channel without a separate hosting step. The 2000-character per-call cap is enforced before the HTTP request — it's the most conservative of the three providers' limits, so the LLM gets a clean validation error instead of an upstream 4xx.

ElevenLabs defaults to the well-known "Rachel" voice (21m00Tcm4TlvDq8ikWAM); override via voiceId. Deepgram picks the voice through the model parameter directly (aura-asteria-en default, aura-orion-en for male voices, etc.).

Speech-to-text — SPEECH_TO_TEXT_TOOLS

Uniform output shape:

{
  "provider": "deepgram",
  "model": "nova-2",
  "text": "The full transcript...",
  "language": "en",
  "confidence": 0.97
}

All three accept MP3, WAV, M4A, OGG, FLAC, and (where the upstream supports it) WebM. The 50 MB shared cap is enforced before the upload; the OpenAI-side path in MEDIA_TOOLS#whisperTranscribe keeps its own 25 MB cap because the upstream rejects larger files.

assemblyai_transcribe is internally upload → submit → poll; the loop has a 5-minute hard cap so a @Tool call returns deterministically. Longer transcripts should batch through AssemblyAI's webhook flow, which is out of scope for the synchronous tool surface.

Environment variables

Every key is read on first call, not at agent build time — registering a toolkit with no keys present is fine; the IllegalStateException only fires when the agent actually invokes that specific method.

Cost notes

April 2026 list pricing — providers may change. Check each provider's current pricing page before relying on these numbers in production:

Cost is not yet emitted as a ToolEvent attribute — when that ships, multimodal spend will flow through the same Cost Governance layer as LLM calls and surface as tnsai_multimodal_cost_usd_total{provider, model, modality} in Prometheus.

Implementation notes

• No vendor SDKs. Every method uses OkHttp + Jackson directly against the provider REST API. Keeps the tnsai-tools transitive graph lean and avoids version-conflict diamond problems with consumers who already pull the same SDK at a different version.
• Test seam via package-private constructors. Each toolkit has a test-only constructor that swaps the provider base URL for a MockWebServer and shares a single OkHttpClient. The public no-arg constructor (the only one BuiltInTool ever calls) wires the real upstream.
• Per-provider input validation. Each method checks size / model / aspect-ratio enums before the HTTP call so a bad LLM argument fails fast with a message naming the allowed values, not an opaque upstream 4xx.

Future work

• MediaStore SPI. A pluggable store (LocalFilesystemMediaStore, S3MediaStore, InMemoryMediaStore, ExpiringUrlPassthroughStore) so that DALL-E URLs can be persisted past their 1-hour TTL and TTS bytes can be served from a stable URL instead of inline data URIs.
• ToolEvent.cost_usd attribute. Once the tool-event hook system carries cost, multimodal spend flows into CostBudget enforcement and a tnsai_multimodal_cost_usd_total Prometheus metric.
• Async job pattern. Video generation, 8K image variants, and long-form audiobook TTS exceed the synchronous @Tool model. A future surface returns {job_id, status: "pending"} and lets the agent poll or subscribe via webhook.
• ContentModerationHook. Optional pre-call hook for local models (Stable Diffusion, Piper) that ship without server-side moderation. Hosted providers already enforce their own policy and surface violations as standard error codes.
• Additional providers. Imagen, Ideogram, Piper local TTS, and a Replicate-platform meta-tool were in the original scope; not shipped yet.
• Integration tests behind RUN_INTEGRATION_TESTS. The current test suite uses MockWebServer for canned responses; an opt-in env-flag tier would call live providers when API keys are present.

See also

• Catalog — full shipped toolkit inventory; multimodal entries to be added under media and image categories
• Audio & Speech — direct-call WhisperClient for non-agent transcription workflows (no @Tool wrapping)
• Cost Governance — budget layer that multimodal spend will join once ToolEvent.cost_usd is wired
• Custom Tools — write your own @Tool methods following the same function-shape POJO pattern