# Audio

`WhisperClient` sends audio to an OpenAI-compatible Whisper endpoint. It transcribes speech to text, and translates speech in another language into English text. It lives in `tnsai-llm`, in the package `com.tnsai.llm.audio`.

For text to speech, and for transcription through Deepgram, AssemblyAI or Replicate, see [Images and speech](../tools/images-and-speech.md).

## Create a client

```java
import com.tnsai.llm.audio.WhisperClient;

WhisperClient whisper = new WhisperClient();
```

The no-arg constructor reads `OPENAI_API_KEY` and uses the model `whisper-1`. Without the key it throws `IllegalStateException`.

Use the builder for another key, model or endpoint, such as a self-hosted Whisper-compatible server:

```java
WhisperClient whisper = WhisperClient.builder()
    .model("whisper-1")
    .apiKey(System.getenv("MY_WHISPER_KEY"))
    .baseUrl("https://whisper.internal.example.com/v1")
    .build();
```

| Setting | Default |
|---|---|
| `model` | `whisper-1` |
| `apiKey` | `OPENAI_API_KEY` |
| `baseUrl` | `OPENAI_BASE_URL`, else `https://api.openai.com/v1` |

## Transcribe

```java
import com.tnsai.llm.audio.WhisperClient.ResponseFormat;
import com.tnsai.llm.audio.WhisperClient.TranscriptionOptions;
import com.tnsai.llm.audio.WhisperClient.TranscriptionResult;
import com.tnsai.models.chat.AudioPart;
import java.io.File;
import java.util.List;

// File in, text out.
String text = whisper.transcribe(new File("speech.mp3"));

// With options.
TranscriptionResult result = whisper.transcribe(
    AudioPart.fromFile(new File("meeting.wav")),
    TranscriptionOptions.builder()
        .language("en")
        .responseFormat(ResponseFormat.VERBOSE_JSON)
        .timestampGranularities(List.of("word", "segment"))
        .temperature(0.0f)
        .prompt("Technical meeting about AI architecture")
        .build());

String transcript = result.getText();
result.getLanguage().ifPresent(lang -> System.out.println("Detected: " + lang));
result.getDuration().ifPresent(seconds -> System.out.println("Duration: " + seconds + "s"));
```

`transcribe(AudioPart)` uses the default options.

### `TranscriptionOptions`

| Option | Type | Default | Meaning |
|---|---|---|---|
| `language` | `String` | Detected | ISO-639-1 code, such as `en`, `tr` or `fr` |
| `prompt` | `String` | None | Text that guides style or continues an earlier segment |
| `responseFormat` | `ResponseFormat` | `JSON` | `JSON`, `TEXT` or `VERBOSE_JSON` |
| `temperature` | `float` | Server default | Sampling temperature. `0.0` is the most deterministic. |
| `timestampGranularities` | `List<String>` | Empty | `word`, `segment` or both. Needs `VERBOSE_JSON`. |

`transcribe` parses the response as JSON unless the format is `TEXT`, so `SRT` and `VTT` fail there. Use `translate` for subtitle formats.

### `TranscriptionResult`

| Method | Returns | Meaning |
|---|---|---|
| `getText()` | `String` | The transcript |
| `getLanguage()` | `Optional<String>` | Detected language. `VERBOSE_JSON` only. |
| `getDuration()` | `Optional<Double>` | Audio length in seconds. `VERBOSE_JSON` only. |
| `getSegments()` | `List<Map<String, Object>>` | Segment timestamps |
| `getWords()` | `List<Map<String, Object>>` | Word timestamps |
| `hasSegments()` | `boolean` | Whether segment timestamps are present |
| `hasWords()` | `boolean` | Whether word timestamps are present |

## Translate to English

Translation detects the source language and always returns English text.

```java
import com.tnsai.llm.audio.WhisperClient.ResponseFormat;
import com.tnsai.llm.audio.WhisperClient.TranslationOptions;
import com.tnsai.models.chat.AudioPart;
import java.io.File;

String english = whisper.translate(new File("turkish_speech.mp3"));

String subtitles = whisper.translate(
    AudioPart.fromFile(new File("german_lecture.wav")),
    TranslationOptions.builder()
        .responseFormat(ResponseFormat.SRT)
        .prompt("Academic lecture on physics")
        .build());
```

`TranslationOptions` takes `prompt`, `responseFormat` (default `JSON`) and `temperature`. With `JSON` or `VERBOSE_JSON`, `translate` returns the `text` field. With `TEXT`, `SRT` or `VTT`, it returns the response body as is.

## Audio input

`WhisperClient` takes an `AudioPart` from `tnsai-core` (`com.tnsai.models.chat`):

```java
AudioPart fromFile = AudioPart.fromFile(new File("speech.wav"));
AudioPart fromBase64 = AudioPart.fromBase64(base64String, "audio/mp3");
AudioPart fromBytes = AudioPart.fromBytes(rawBytes, "audio/wav");
```

`AudioPart.fromUrl(...)` exists, but `WhisperClient` cannot send it. Download the file and pass its bytes.

`fromFile` picks the MIME type from the file extension. MP3, MP4, MPEG, MPGA, M4A, WAV, WebM, OGG and FLAC are recognised. Files are limited to 25 MB.

## Errors and retries

Every failure is retried up to three times, with a backoff that starts at one second. That includes rate limits, server errors and an oversized file. After the last attempt, `transcribe` and `translate` throw `LLMException` (`com.tnsai.exceptions`) with the cause attached.

```java
import com.tnsai.exceptions.LLMException;

try {
    String text = whisper.transcribe(new File("speech.mp3"));
} catch (LLMException e) {
    System.err.println("Transcription failed: " + e.getMessage());
}
```

Check the file size before you call, so an oversized file does not wait through the retries.

## Related

- [Images and speech](../tools/images-and-speech.md): text to speech and other transcription providers, as agent tools.
- [Providers](providers.md): the chat clients.