Embedding benchmark
tnsai-evaluation includes a retrieval benchmark that compares embedding models on Turkish and English text. Use it to choose an embedding provider before you install one. It does not change the default embedding function.
The benchmark runs as a test in the framework source, so you need a checkout of the framework repository and a local Ollama server.
Run it
ollama pull qllama/multilingual-e5-small
ollama pull bge-m3
mvn -pl tnsai-evaluation -am -Pembedding-bench testThe embedding-bench profile enables the live test; a plain mvn test never calls a model server. Set OLLAMA_BASE_URL to use a server other than http://localhost:11434. The results are written to tnsai-evaluation/target/embedding-bench-results.md.
What it measures
| Part | Contents |
|---|---|
| Data | embedding-bench/tr-en-pairs.json: 50 facts, each written in Turkish and in English, plus 10 cross-language questions |
| Index | All 100 passages together, so the translation of the right fact competes with it |
| Metrics | recall@5, MRR and median embedding latency for each language slice, and storage per document (dimensions × 4 bytes) |
| Model row | Embeds with |
|---|---|
token-bag-384 | A bag-of-tokens baseline that needs no model server |
me5-small | qllama/multilingual-e5-small, 384 dimensions, with query: and passage: prefixes |
bge-m3@384 | bge-m3 cut to 384 dimensions with EmbeddingFunctions.matryoshka(..., 384) |
bge-m3 | bge-m3 at its native 1,024 dimensions |
EmbeddingBenchRecommendation.decide(scores) names the best 384-dimension row as the installable default and the row with the highest mean recall@5 as the quality option. It always reports the production default as hash (unchanged): the benchmark only recommends, it never switches the model.
Published results
One measured run is kept in tnsai-evaluation/src/main/resources/embedding-bench/RESULTS.md, with the date and server it ran on. Results depend on the hardware and model versions, so run the benchmark on your own setup before you decide.