Selective re-embed index

This is the laptop / default ingest path. Qdrant and pgvector stay the production com.tnsai.memory.advanced.VectorIndex backends. This type does not implement VectorIndex, and nothing in Role RAG opens it from an env URL or @VectorMemory.provider.

Storage vs a naive full reindex is the unique-hash set, not a marketing 97%. Recompute cost on a one-file edit is that file's new hashes, not the whole corpus.

What it does

SelectiveReembedIndex index = new SelectiveReembedIndex(embeddings);
ReindexStats first = index.syncSource("a.md", List.of("alpha", "shared"));
ReindexStats again = index.syncSource("a.md", List.of("alpha changed", "shared"));
List<ScoredChunk> hits = index.search("alpha", 3);

syncSource(sourceId, chunks) replaces one source's ordered chunk texts. Each chunk is keyed by SHA-256 of its UTF-8 bytes:

1. a hash that already has a vector is reused
2. a new hash calls EmbeddingFunction.embed once and stores that vector
3. a hash dropped from this source is removed only when no other source still owns it
4. consecutive chunks on the same source become neighbors

ReindexStats reports considered, reused, embedded, and removed. Duplicate payloads across sources share one stored embedding. size() is the unique-hash count. uniqueEmbeddingRatio() is unique / naive; 1.0 means no sharing.

search(query, topK) embeds the query and ranks unique stored vectors by cosine similarity. Hits are ScoredChunk records (contentHash, sourceId, text, score). Dimension is pinned on the first embed; a later mismatched length fails loud.

Not this page

• Qdrant and pgvector — production VectorIndex adapters
• Knowledge Base — InMemoryKnowledgeBase is a different store
• Server FILE incremental skip — SHA-256 of whole files on the indexer path, not this type

Related

• Pipeline — FILE ingest and Server FileIndexer
• Embeddings — process-wide EmbeddingFunction