Skip to content
tnsaiDocs
GitHub

Selective re-embed index

com.tnsai.intelligence.rag.vector.SelectiveReembedIndex is a local vector index that re-embeds only chunks whose content hash changed. It is a standalone index you use directly: Role RAG does not use it, it does not implement com.tnsai.memory.advanced.VectorIndex, and nothing opens it from an environment URL or @VectorMemory.provider. After a one-file edit, only that file's new hashes are embedded again.

SelectiveReembedIndex index = new SelectiveReembedIndex(embeddings);
ReindexStats first = index.syncSource("a.md", List.of("alpha", "shared"));
ReindexStats again = index.syncSource("a.md", List.of("alpha changed", "shared"));
List<ScoredChunk> hits = index.search("alpha", 3);

syncSource(sourceId, chunks) replaces one source's ordered chunk texts. Each chunk is keyed by SHA-256 of its UTF-8 bytes:

  1. a hash that already has a vector is reused
  2. a new hash calls EmbeddingFunction.embed once and stores that vector
  3. a hash dropped from this source is removed only when no other source still owns it
  4. consecutive chunks on the same source become neighbors, which neighbors(contentHash) returns

ReindexStats reports considered, reused, embedded, and removed. Duplicate payloads across sources share one stored embedding. size() is the unique-hash count. uniqueEmbeddingRatio() is unique / naive, and 1.0 means no sharing.

search(query, topK) embeds the query and ranks unique stored vectors by cosine similarity. Hits are ScoredChunk records (contentHash, sourceId, text, score). Dimension is pinned on the first embed, and a later mismatched length fails loud.

  • Remote vector stores: the Qdrant and pgvector VectorIndex adapters.
  • KnowledgeBase API: InMemoryKnowledgeBase, a different store.
  • Pipeline: FILE ingest, which reuses a file's chunks while its ingestion fingerprint is unchanged.
  • Embeddings: the EmbeddingFunction this index calls.