# Bilingual embedding bench

`tnsai-evaluation` ships an opt-in multilingual retrieval bench
(`EmbeddingBenchRunner`, `@since 0.14.0` on `TnsAI@c28d8783`). It is
in Maven Central `0.14.1`. Default `mvn test` stays offline and
does not call a model host.

This harness does **not** change the production hash embedding default.

## Run

Requires a local Ollama daemon and:

```bash
ollama pull qllama/multilingual-e5-small
ollama pull bge-m3
mvn -pl tnsai-evaluation -am -Pembedding-bench test
```

The live IT is gated by `@EnabledIfSystemProperty(embedding.bench=true)`,
which the `embedding-bench` Maven profile sets. Override the host with
`OLLAMA_BASE_URL` (default `http://localhost:11434`).

## Protocol

| Piece | Contract |
|---|---|
| Fixture | `embedding-bench/tr-en-pairs.json` — 50 shared facts × {TR, EN} plus 10 CROSS probes |
| Index | Union of gold documents (100 passages). A translation of the same fact is a live distractor |
| Metrics | recall@5, MRR, p50 embed latency, `dim * 4` bytes/doc |
| Rows | `token-bag-384` (always on), `me5-small`, `bge-m3@384` (`EmbeddingFunctions.matryoshka(..., 384)`), `bge-m3` native 1024-d |

`EmbeddingBenchRecommendation` always reports production default
`hash (unchanged)`. An installable default is recommended only from
measured scores; this ticket does not ship a process-wide model.

## Measured table

One dated local-Ollama run lives in the framework tree:

`tnsai-evaluation/src/test/resources/embedding-bench/RESULTS.md`

Re-run the profile to refresh `target/embedding-bench-results.md`. Do not
treat a single laptop run as a universal ranking.

## Not this page

- Changing the process-wide `EmbeddingFunction` — still TAN-2998 / the
  installable provider path on [Strategies](../capabilities/rag/strategies.md#tnsai-0140-declarative-embedding-provider)