Bilingual embedding bench
tnsai-evaluation ships an opt-in multilingual retrieval bench (EmbeddingBenchRunner, @since 0.14.0 on TnsAI@c28d8783). It is in Maven Central 0.14.1. Default mvn test stays offline and does not call a model host.
This harness does not change the production hash embedding default.
Run
Requires a local Ollama daemon and:
ollama pull qllama/multilingual-e5-small
ollama pull bge-m3
mvn -pl tnsai-evaluation -am -Pembedding-bench testThe live IT is gated by @EnabledIfSystemProperty(embedding.bench=true),
which the embedding-bench Maven profile sets. Override the host with
OLLAMA_BASE_URL (default http://localhost:11434).
Protocol
| Piece | Contract |
|---|---|
| Fixture | embedding-bench/tr-en-pairs.json — 50 shared facts × {TR, EN} plus 10 CROSS probes |
| Index | Union of gold documents (100 passages). A translation of the same fact is a live distractor |
| Metrics | recall@5, MRR, p50 embed latency, dim * 4 bytes/doc |
| Rows | token-bag-384 (always on), me5-small, bge-m3@384 (EmbeddingFunctions.matryoshka(..., 384)), bge-m3 native 1024-d |
EmbeddingBenchRecommendation always reports production default
hash (unchanged). An installable default is recommended only from
measured scores; this ticket does not ship a process-wide model.
Measured table
One dated local-Ollama run lives in the framework tree:
tnsai-evaluation/src/test/resources/embedding-bench/RESULTS.md
Re-run the profile to refresh target/embedding-bench-results.md. Do not
treat a single laptop run as a universal ranking.
Not this page
- Changing the process-wide
EmbeddingFunction— still TAN-2998 / the installable provider path on Strategies
Advanced Evaluation
This guide covers the specialized evaluator families introduced in 0.3.0: RAG evaluators for retrieval-augmented generation pipelines, multi-turn evaluators for conversation quality, safety evaluators for harmful content detection, and the trace-eval bridge that connects observability with evaluation.
Payments
TnsAI agents talk to other agents. Sometimes the other agent charges. The PaymentBroker SPI is the framework's answer to "how does an agent settle a payment without you, the human, in the loop?" — same surface for free internal calls (PaymentBroker.noOp()), real on-chain micropayments (X402PaymentBroker), or your own custom rail.