Images and speech

MEDIA_TOOLS adds the OpenAI pair: openai_tts and whisper_transcribe. Register it next to the others when you want OpenAI and another provider in the same agent.

Register the toolkits

Pass the toolkits to builtInTools(...). AgentBuilder.build() requires principal and authorityScope, and liabilitySink is optional.

package com.example.tnsai.docs;

import com.tnsai.accountability.AuthorityScope;
import com.tnsai.accountability.LiabilitySink;
import com.tnsai.agents.Agent;
import com.tnsai.agents.AgentBuilder;
import com.tnsai.enums.BuiltInTool;
import com.tnsai.identity.AgentPrincipal;
import com.tnsai.llm.providers.OpenAIClient;
import com.tnsai.roles.Role;

public final class ToolsMultimodalBuilderExample {
    private ToolsMultimodalBuilderExample() {}

    public static Agent build(
        Role myRole,
        AgentPrincipal principal,
        LiabilitySink sink,
        AuthorityScope scope
    ) {
        return AgentBuilder.create()
            .llm(new OpenAIClient("gpt-4o"))
            .role(myRole)
            .builtInTools(
                BuiltInTool.IMAGE_GEN_TOOLS,
                BuiltInTool.TEXT_TO_SPEECH_TOOLS,
                BuiltInTool.SPEECH_TO_TEXT_TOOLS
            )
            .principal(principal)
            .liabilitySink(sink)
            .authorityScope(scope)
            .build();
    }
}

The agent then calls the tools from a chat turn:

Agent artist = ToolsMultimodalBuilderExample.build(myRole, principal, sink, scope);
String reply = artist.chat("Draw a watercolor painting of a giraffe on Mars");

Every tool returns a JSON string. Each tool checks its arguments before the HTTP call, so a bad value fails with IllegalArgumentException naming the allowed values.

Images: IMAGE_GEN_TOOLS

DALL-E and FLUX return provider-hosted URLs. DALL-E URLs expire about an hour after generation, so download the image if you need to keep it. Stability returns the image itself, and urls holds it as a data:image/png;base64,… URI.

Speech: TEXT_TO_SPEECH_TOOLS

ElevenLabs and Cartesia take a voiceId next to the model. Deepgram picks the voice through model.

All three return the same shape:

{
  "provider": "elevenlabs",
  "model": "eleven_multilingual_v2",
  "audio_uri": "data:audio/mpeg;base64,SUQzBAAAAAAAJ...",
  "audio_bytes": 45920
}

audio_uri plays directly in an <audio> element. Input is capped at 2000 characters per call.

Transcription: SPEECH_TO_TEXT_TOOLS

Each tool takes audioPath, the absolute path to a local file of at most 50 MB. Supported formats are MP3, WAV, M4A, OGG and FLAC.

deepgram_transcribe takes a language code, default en. assemblyai_transcribe takes an optional languageCode.

assemblyai_transcribe uploads the file, submits a job and polls it. Polling starts at two seconds, backs off to ten, and gives up after 300 seconds.

The OpenAI whisper_transcribe in MEDIA_TOOLS has its own 25 MB cap, and openai_tts caps input at 4096 characters.

API keys

A tool reads its key from the environment when it is called, not when the agent is built. You can register a toolkit without its keys. A call without the key throws IllegalStateException, and the message names the missing variable.

Related

• Catalog: every built-in toolkit.
• Audio: WhisperClient for transcription outside an agent.
• Custom tools: write your own @Tool methods.