Server RAG pipeline

How it fits together

Directory  -->  FileIndexer  -->  FileIngestionService  -->  InMemoryKnowledgeBase + BM25Stream
                                                                  |
Query  -->  HybridRetriever  -->  [BM25Stream, weight 0.6]    -+  RRF  -->  Results
                             -->  [TF-IDF search, weight 0.4]  -+

SessionManager.getRag(sessionId) creates a session's RagService on first use, and removing the session clears it. Indexing and clear() run under one lock, so only one index operation runs at a time per session; searches don't take that lock.

RagService

RagService rag = sessionManager.getRag("my-session");

rag.indexDirectory(Path.of("/project/src"), progress -> {
    System.out.printf("Indexed %d/%d: %s%n",
        progress.indexedFiles(), progress.totalFiles(), progress.currentFile());
});

List<SearchResult> results = rag.search("authentication middleware", 5);
String prompt = rag.buildContextPrompt("How does auth work?", 5);
IndexStatus status = rag.getStatus();

getStatus() reports the file count, chunk count, last index time and indexed path.

Context prompt

buildContextPrompt(query, topK) searches and wraps the results in the same untrusted-context fence that agent RAG uses. Each result's ID, metadata and content are XML-escaped inside <tnsai-memory>, and the query follows unchanged:

<tnsai-memory>
<evidence>
<document-id>...</document-id>
<provenance>
<entry><key>file</key><value>src/auth/Middleware.java</value></entry>
</provenance>
<content>...</content>
</evidence>
</tnsai-memory>

<tnsai-user-message>
How does auth work?
</tnsai-user-message>

With no results, or if rendering fails, the query is returned as is.

Ground an agent's chat

attachChatTo(agent) binds the session corpus to an agent as live chat knowledge, so chat grounds on the user's original message rather than on a rewritten prompt. The server's WebSocket handler calls it before each streamed reply.

The agent must declare MEMORY chat knowledge named RagService.CHAT_SOURCE ("session-knowledge") that searches with RagService.CHAT_TOP_K (5); otherwise it refuses the binding. The binding holds the corpus instead of copying it, so documents indexed later are visible without rebinding. While the corpus is empty, the call does nothing.

Document management

String docId = rag.addDocument("Custom knowledge content",
    Map.of("source", "user", "topic", "deployment"));

List<RagService.DocumentInfo> docs = rag.listDocuments();
Optional<Document> doc = rag.getDocument(docId);
boolean removed = rag.removeDocument(docId);

rag.clear();

listDocuments() lists only documents added with addDocument, as DocumentInfo(id, preview, contentLength, metadata) with a 100-character preview. Added documents and indexed chunks are searched together. clear() empties the knowledge base, the BM25 index and the fingerprints, and resets the status.

FileIndexer

FileIndexer is the server adapter over com.tnsai.intelligence.rag.binding.FileIngestionService. Discovery, .gitignore and .tnsignore, the default skip lists, format detection, extraction and chunking follow the same contract as declarative @Knowledge(type = FILE), and chunks come from NormalizedDocumentChunker. See File formats.

new FileIndexer() and new FileIngestionService() share FileIngestionService.Limits.defaults(). Pass the same Limits record to both constructors when you tighten a deployment.

Formats

There is no separate extension list. Candidates follow DocumentFormat and AUTO detection on File formats. include and exclude on a KnowledgeSourceConfig still run before the format check.

Ignore files and default skips

Each directory may carry .gitignore and .tnsignore. Rules from a nested file apply under that directory, not as if they were written at the corpus root.

.gitignore and .tnsignore themselves are never ingested.

These directory names are skipped by default, case-insensitively: .aws, .git, .gnupg, .gradle, .hg, .idea, .kube, .next, .ssh, .svn, .vscode, __pycache__, build, coverage, dist, node_modules, out, target, vendor. Nine of them (.aws, .git, .gnupg, .hg, .kube, .ssh, .svn, node_modules, vendor) also match as any component of a path.

These file names are skipped anywhere: .env, .env.*, .git-credentials, .netrc, .npmrc, .pypirc, id_dsa, id_ecdsa, id_ed25519, id_rsa.

Ignore files are capped at 64 KiB and 1,024 compiled selectors.

Admission caps

FileIngestionService.Limits is one policy object:

A single file is also bounded at 16 MiB of raw input. Symbolic links are never followed. An empty regular file has no special skip rule; format detection still decides.

Incremental indexing

Each file has a pipeline fingerprint built from its content, format, extractor, chunker and selectors. A file whose fingerprint is unchanged keeps its previous chunks.

An index run stages the whole new generation, then swaps it in with KnowledgeBase.replaceDocuments and the matching BM25Stream.replaceDocuments. If the BM25 swap fails, the knowledge-base swap is undone. InMemoryKnowledgeBase and BM25Stream hold their write locks across the swap, so readers never see part of a generation. A store that keeps the interface default removes and then adds, and readers can see a mix.

• A rejected source, or a failure while ingesting or committing, leaves the previous index untouched.
• A file that fails to read is not treated as deleted. The readable files are committed and the failed file keeps its previous chunks until a later run replaces them.
• Files that are gone, renamed or outside the indexed root lose their chunks.

Call fileIndexer.clearHashes() to drop the fingerprints and force a full re-index.

CodeChunker

com.tnsai.server.rag.CodeChunker chunks one source file on its own. It delegates to the same source-code extractor and NormalizedDocumentChunker as FILE ingest, so its symbol boundaries, IDs, limits and provenance match. FileIndexer does not call it. Symbol units and metadata are on File formats.

BM25Stream

BM25Stream ranks by Okapi BM25 with K1 = 1.2 and B = 0.75. It holds at most 10,000 documents by default, and documents expire one hour after last access.

Queries and documents are processed the same way:

1. Tokenize: lowercase, replace everything except a-z, 0-9 and _ with spaces, split, and drop one-character tokens.
2. Remove stop words: 56 common English words.
3. Stem: strip one of 15 suffixes: -ies, -ing, -tion, -sion, -ment, -ness, -able, -ous, -ful, -less, -ly, -ed, -er, -es, -s. Words shorter than four characters are not stemmed.
4. Expand synonyms, at query time only, in both directions:

HybridRetriever

HybridRetriever fuses several streams with Reciprocal Rank Fusion. It asks each available stream for 2 × topK results and scores each document:

score(doc) = sum over streams of weight(stream) / (60 + rank(doc, stream) + 1)

rank starts at 0. A stream that throws is logged and skipped. Results are sorted by fused score, and at most three chunks per file metadata value are kept, so one large file cannot fill the result list.

HybridRetriever retriever = HybridRetriever.builder()
    .stream(bm25Stream, 0.6)
    .stream(new KnowledgeBaseStream(knowledgeBase), 0.4)
    .build();

List<SearchResult> results = retriever.retrieve("authentication flow", 10);

KnowledgeBaseStream wraps KnowledgeBase.search, which in InMemoryKnowledgeBase is TF-IDF; it uses no embeddings. A builder with no streams throws IllegalStateException.

Related

• KnowledgeBase API: the store behind the pipeline.
• File formats: what ingestion reads and how it chunks.
• RAG: declarative @Knowledge sources for agents.
• Retrieval strategies: hybrid retrieval in declarative RAG.