WhisperClient sends audio to an OpenAI-compatible Whisper endpoint. It transcribes speech to text, and translates speech in another language into English text. It lives in tnsai-llm, in the package com.tnsai.llm.audio.

For text to speech, and for transcription through Deepgram, AssemblyAI or Replicate, see Images and speech.

Create a client

import com.tnsai.llm.audio.WhisperClient;

WhisperClient whisper = new WhisperClient();

The no-arg constructor reads OPENAI_API_KEY and uses the model whisper-1. Without the key it throws IllegalStateException.

Use the builder for another key, model or endpoint, such as a self-hosted Whisper-compatible server:

WhisperClient whisper = WhisperClient.builder()
    .model("whisper-1")
    .apiKey(System.getenv("MY_WHISPER_KEY"))
    .baseUrl("https://whisper.internal.example.com/v1")
    .build();

Setting	Default
`model`	`whisper-1`
`apiKey`	`OPENAI_API_KEY`
`baseUrl`	`OPENAI_BASE_URL`, else `https://api.openai.com/v1`

Transcribe

import com.tnsai.llm.audio.WhisperClient.ResponseFormat;
import com.tnsai.llm.audio.WhisperClient.TranscriptionOptions;
import com.tnsai.llm.audio.WhisperClient.TranscriptionResult;
import com.tnsai.models.chat.AudioPart;
import java.io.File;
import java.util.List;

// File in, text out.
String text = whisper.transcribe(new File("speech.mp3"));

// With options.
TranscriptionResult result = whisper.transcribe(
    AudioPart.fromFile(new File("meeting.wav")),
    TranscriptionOptions.builder()
        .language("en")
        .responseFormat(ResponseFormat.VERBOSE_JSON)
        .timestampGranularities(List.of("word", "segment"))
        .temperature(0.0f)
        .prompt("Technical meeting about AI architecture")
        .build());

String transcript = result.getText();
result.getLanguage().ifPresent(lang -> System.out.println("Detected: " + lang));
result.getDuration().ifPresent(seconds -> System.out.println("Duration: " + seconds + "s"));

transcribe(AudioPart) uses the default options.

`TranscriptionOptions`

Option	Type	Default	Meaning
`language`	`String`	Detected	ISO-639-1 code, such as `en`, `tr` or `fr`
`prompt`	`String`	None	Text that guides style or continues an earlier segment
`responseFormat`	`ResponseFormat`	`JSON`	`JSON`, `TEXT` or `VERBOSE_JSON`
`temperature`	`float`	Server default	Sampling temperature. `0.0` is the most deterministic.
`timestampGranularities`	`List<String>`	Empty	`word`, `segment` or both. Needs `VERBOSE_JSON`.

transcribe parses the response as JSON unless the format is TEXT, so SRT and VTT fail there. Use translate for subtitle formats.

`TranscriptionResult`

Method	Returns	Meaning
`getText()`	`String`	The transcript
`getLanguage()`	`Optional<String>`	Detected language. `VERBOSE_JSON` only.
`getDuration()`	`Optional<Double>`	Audio length in seconds. `VERBOSE_JSON` only.
`getSegments()`	`List<Map<String, Object>>`	Segment timestamps
`getWords()`	`List<Map<String, Object>>`	Word timestamps
`hasSegments()`	`boolean`	Whether segment timestamps are present
`hasWords()`	`boolean`	Whether word timestamps are present

Translate to English

Translation detects the source language and always returns English text.

import com.tnsai.llm.audio.WhisperClient.ResponseFormat;
import com.tnsai.llm.audio.WhisperClient.TranslationOptions;
import com.tnsai.models.chat.AudioPart;
import java.io.File;

String english = whisper.translate(new File("turkish_speech.mp3"));

String subtitles = whisper.translate(
    AudioPart.fromFile(new File("german_lecture.wav")),
    TranslationOptions.builder()
        .responseFormat(ResponseFormat.SRT)
        .prompt("Academic lecture on physics")
        .build());

TranslationOptions takes prompt, responseFormat (default JSON) and temperature. With JSON or VERBOSE_JSON, translate returns the text field. With TEXT, SRT or VTT, it returns the response body as is.

Audio input

WhisperClient takes an AudioPart from tnsai-core (com.tnsai.models.chat):

AudioPart fromFile = AudioPart.fromFile(new File("speech.wav"));
AudioPart fromBase64 = AudioPart.fromBase64(base64String, "audio/mp3");
AudioPart fromBytes = AudioPart.fromBytes(rawBytes, "audio/wav");

AudioPart.fromUrl(...) exists, but WhisperClient cannot send it. Download the file and pass its bytes.

fromFile picks the MIME type from the file extension. MP3, MP4, MPEG, MPGA, M4A, WAV, WebM, OGG and FLAC are recognised. Files are limited to 25 MB.

Errors and retries

Every failure is retried up to three times, with a backoff that starts at one second. That includes rate limits, server errors and an oversized file. After the last attempt, transcribe and translate throw LLMException (com.tnsai.exceptions) with the cause attached.

import com.tnsai.exceptions.LLMException;

try {
    String text = whisper.transcribe(new File("speech.mp3"));
} catch (LLMException e) {
    System.err.println("Transcription failed: " + e.getMessage());
}

Check the file size before you call, so an oversized file does not wait through the retries.

Images and speech: text to speech and other transcription providers, as agent tools.
Providers: the chat clients.