RAG Pipeline

Architecture Overview

The RAG pipeline has three stages: indexing (scanning files and splitting them into chunks), storage (keeping chunks in an in-memory knowledge base with BM25 and vector indexes), and retrieval (finding the most relevant chunks for a user's query using hybrid search).

Directory  -->  FileIndexer  -->  FileIngestionService  -->  KnowledgeBase (in-memory)
                                                                  |
User Query  -->  HybridRetriever  -->  [BM25Stream 60%]          -+  RRF  -->  Results
                                  -->  [VectorStream 40%]         -+

Each session gets its own RagService, lazily created by SessionManager.getRag(sessionId). The service is thread-safe: indexing is serialized via a ReentrantLock, while reads (search) run concurrently.

RagService

The central orchestrator for a session's RAG pipeline.

RagService rag = sessionManager.getRag("my-session");

// Index a directory
rag.indexDirectory(Path.of("/project/src"), progress -> {
    System.out.printf("Indexed %d/%d: %s%n",
        progress.indexedFiles(), progress.totalFiles(), progress.currentFile());
});

// Search
List<SearchResult> results = rag.search("authentication middleware", 5);

// Build augmented prompt (auto-prepends context)
String prompt = rag.buildContextPrompt("How does auth work?", 5);

// Document management
String docId = rag.addDocument("Custom knowledge...", Map.of("source", "manual"));
rag.removeDocument(docId);
List<RagService.DocumentInfo> docs = rag.listDocuments();

The hybrid retriever is configured at construction with BM25 at 60% weight and the vector knowledge base at 40%:

this.hybridRetriever = HybridRetriever.builder()
    .stream(bm25Stream, 0.6)
    .stream(new KnowledgeBaseStream(knowledgeBase), 0.4)
    .build();

FileIndexer

FileIndexer is the Server adapter over com.tnsai.intelligence.rag.binding.FileIngestionService. Discovery, .gitignore / .tnsignore, default skip lists, format detection, extraction, and chunking are the same contract as declarative @KnowledgeSource(type = FILE). The class does not call CodeChunker. Chunks come from NormalizedDocumentChunker — see File formats.

This path is in TnsAI 0.14.0 (TnsAI@b7f3a098, TAN-5806) and ships in Maven Central 0.14.1. Maven Central 0.13.0 used a Server-local walk.

new FileIndexer() and new FileIngestionService() share FileIngestionService.Limits.defaults(). Pass the same Limits record to both constructors when you tighten a deployment.

Formats

The walk does not have a private "28 extensions" list. Candidates follow DocumentFormat / AUTO on File formats. include / exclude on a KnowledgeSourceConfig still run before the format check.

Ignore files and default skips

Each directory may carry .gitignore and .tnsignore. Rules from a nested file apply under that directory, not as if they were written at the corpus root.

.gitignore and .tnsignore themselves are never ingested.

Default directory names (case-insensitive) are also skipped: .aws, .git, .gnupg, .gradle, .hg, .idea, .kube, .next, .ssh, .svn, .vscode, __pycache__, build, coverage, dist, node_modules, out, target, vendor. A subset (.git, .ssh, node_modules, vendor, and the other secret/VCS ancestors) also matches when it appears as any path component.

Default file names skipped anywhere: .env, .env.*, .git-credentials, .netrc, .npmrc, .pypirc, id_dsa, id_ecdsa, id_ed25519, id_rsa.

Ignore files are capped at 64 KiB and 1 024 compiled selectors.

Admission caps

FileIngestionService.Limits is one policy object:

A single file is also bounded at 16 MiB raw input (ExtractionContext). There is no 512 KB skip threshold.

Symbolic links are never followed. An empty regular file is not a special skip rule — format detection still decides.

Incremental indexing

Reuse is ingestionFingerprint (content hash plus format, extractor, chunker, and selectors), not a content-only SHA-256. Unchanged fingerprints keep the previous chunks.

FILE ingest stages a complete generation before mutating TF-IDF or BM25 (TAN-5620). A source-level format or extraction failure discards the staged replacement — file N cannot publish chunks from files 1..N-1 of the new attempt. FileIngestionService.IngestionResult carries fingerprints (Map<String, String>); the previous eight-argument form is gone. The atomic swap is KnowledgeBase.replaceDocuments. InMemoryKnowledgeBase and BM25Stream hold their write locks across that generation. The interface default walks remove-then-add sequentially — concurrent readers can see a mix unless the store overrides.

Call fileIndexer.clearHashes() to drop the fingerprint map and force a full re-index.

CodeChunker

com.tnsai.server.rag.CodeChunker is the Server facade for one-file source chunking. TnsAI 0.14.0 (TnsAI@a652e847, TAN-5618) delegates to SourceCodeContentExtractor and NormalizedDocumentChunker, so callers share symbol boundaries, IDs, limits, and provenance with declarative FILE ingest.

FileIndexer still does not call CodeChunker. Live directory ingest goes through FileIngestionService and the same extractor. Do not treat a leftover regex language table as the live path.

Symbol units and metadata: File formats.

BM25Stream

The BM25Stream provides keyword-based search using the Okapi BM25 algorithm, which is the same ranking function used by search engines like Elasticsearch. It scores documents based on how well their terms match the query, accounting for term frequency and document length.

Parameters

These BM25 parameters control how the scoring behaves. The defaults work well for code search and rarely need tuning.

Text Processing Pipeline

Before scoring, queries and documents go through a text processing pipeline that normalizes, tokenizes, and stems terms. This improves recall by matching different forms of the same word.

1. Tokenization: Lowercase, strip non-alphanumeric (except _), split on whitespace, drop tokens with 1 character or fewer
2. Stop word removal: 50 common English stop words
3. Stemming: Suffix-stripping rules for 14 suffixes (-ies, -ing, -tion, -sion, -ment, -ness, -able, -ous, -ful, -less, -ly, -ed, -er, -es, -s)
4. Synonym expansion (query-time only): 20 coding-domain synonym pairs

Synonym Pairs

At query time, common coding abbreviations are expanded to their full forms (and vice versa) so that searching for "auth" also finds documents containing "authentication".

HybridRetriever

The HybridRetriever combines results from multiple search strategies (like BM25 keyword search and vector similarity search) into a single ranked list. This hybrid approach gives better results than either method alone because keyword search finds exact term matches while vector search captures semantic similarity.

Fusion Algorithm

The retriever merges results using Reciprocal Rank Fusion (RRF), which combines rankings without needing normalized scores. For each document appearing in any stream's results:

score(doc) = SUM over streams: weight(stream) / (K + rank(doc, stream) + 1)

Where K = 60 (the RRF constant). Documents are then sorted by fused score.

Diversification

To prevent a single large file from dominating search results, the retriever limits output to a maximum of 3 chunks per source file. This ensures the agent sees context from multiple relevant files.

HybridRetriever retriever = HybridRetriever.builder()
    .stream(bm25Stream, 0.6)       // 60% weight
    .stream(vectorStream, 0.4)      // 40% weight
    .build();

List<SearchResult> results = retriever.retrieve("authentication flow", 10);

Context Prompt Format

When the agent asks a question, RagService.buildContextPrompt searches for relevant code and prepends it to the user's query. This gives the LLM the codebase context it needs to answer accurately.

[Relevant code context]
--- file: src/auth/Middleware.java (lines 15-45) ---
public class AuthMiddleware {
    private final TokenValidator validator;
    ...
}

--- file: src/auth/TokenValidator.java (lines 1-30) ---
public class TokenValidator {
    ...
}

[User question]
How does the authentication middleware work?

If no context is found (empty knowledge base or no matches), the original query is returned unchanged.

Document Management API

Beyond automatic directory indexing, you can manually add, list, and remove documents in the knowledge base. This is useful for injecting custom knowledge (like deployment procedures or domain-specific documentation) that is not part of the codebase.

// Add a document with metadata
String docId = rag.addDocument("Custom knowledge content",
    Map.of("source", "user", "topic", "deployment"));

// List documents (returns preview, length, metadata)
List<RagService.DocumentInfo> docs = rag.listDocuments();
// DocumentInfo(id, preview(100chars), contentLength, metadata)

// Get a specific document
Optional<Document> doc = rag.getDocument(docId);

// Remove
boolean removed = rag.removeDocument(docId);

// Clear everything
rag.clear();

Documents added via addDocument are tracked separately and appear in listDocuments(). Both manually added documents and file-indexed chunks are searchable through the same hybrid retriever.