# Embedding benchmark

`tnsai-evaluation` includes a retrieval benchmark that compares embedding models on Turkish and English text. Use it to choose an [embedding provider](../rag/embedding-prefixes.md#install-an-embedding-provider) before you install one. It does not change the default embedding function.

The benchmark runs as a test in the framework source, so you need a checkout of the framework repository and a local Ollama server.

## Run it

```bash
ollama pull qllama/multilingual-e5-small
ollama pull bge-m3
mvn -pl tnsai-evaluation -am -Pembedding-bench test
```

The `embedding-bench` profile enables the live test; a plain `mvn test` never calls a model server. Set `OLLAMA_BASE_URL` to use a server other than `http://localhost:11434`. The results are written to `tnsai-evaluation/target/embedding-bench-results.md`.

## What it measures

| Part | Contents |
|------|----------|
| Data | `embedding-bench/tr-en-pairs.json`: 50 facts, each written in Turkish and in English, plus 10 cross-language questions |
| Index | All 100 passages together, so the translation of the right fact competes with it |
| Metrics | recall@5, MRR and median embedding latency for each language slice, and storage per document (dimensions × 4 bytes) |

| Model row | Embeds with |
|-----------|-------------|
| `token-bag-384` | A bag-of-tokens baseline that needs no model server |
| `me5-small` | `qllama/multilingual-e5-small`, 384 dimensions, with `query:` and `passage:` prefixes |
| `bge-m3@384` | `bge-m3` cut to 384 dimensions with `EmbeddingFunctions.matryoshka(..., 384)` |
| `bge-m3` | `bge-m3` at its native 1,024 dimensions |

`EmbeddingBenchRecommendation.decide(scores)` names the best 384-dimension row as the installable default and the row with the highest mean recall@5 as the quality option. It always reports the production default as `hash (unchanged)`: the benchmark only recommends, it never switches the model.

## Published results

One measured run is kept in `tnsai-evaluation/src/main/resources/embedding-bench/RESULTS.md`, with the date and server it ran on. Results depend on the hardware and model versions, so run the benchmark on your own setup before you decide.

## Related

- [Embeddings](../rag/embedding-prefixes.md)
- [Evaluation](index.md)