# RAG Pipeline

The server provides a per-session Retrieval-Augmented Generation pipeline that indexes local codebases, chunks source files by language boundaries, and retrieves relevant context using hybrid BM25 + vector search with Reciprocal Rank Fusion.

## Architecture Overview

The RAG pipeline has three stages: indexing (scanning files and splitting them into chunks), storage (keeping chunks in an in-memory knowledge base with BM25 and vector indexes), and retrieval (finding the most relevant chunks for a user's query using hybrid search).

```
Directory  -->  FileIndexer  -->  FileIngestionService  -->  KnowledgeBase (in-memory)
                                                                  |
User Query  -->  HybridRetriever  -->  [BM25Stream 60%]          -+  RRF  -->  Results
                                  -->  [VectorStream 40%]         -+
```

Each session gets its own `RagService`, lazily created by `SessionManager.getRag(sessionId)`. The service is thread-safe: indexing is serialized via a `ReentrantLock`, while reads (search) run concurrently.

## RagService

The central orchestrator for a session's RAG pipeline.

```java
RagService rag = sessionManager.getRag("my-session");

// Index a directory
rag.indexDirectory(Path.of("/project/src"), progress -> {
    System.out.printf("Indexed %d/%d: %s%n",
        progress.indexedFiles(), progress.totalFiles(), progress.currentFile());
});

// Search
List<SearchResult> results = rag.search("authentication middleware", 5);

// Build augmented prompt (auto-prepends context)
String prompt = rag.buildContextPrompt("How does auth work?", 5);

// Document management
String docId = rag.addDocument("Custom knowledge...", Map.of("source", "manual"));
rag.removeDocument(docId);
List<RagService.DocumentInfo> docs = rag.listDocuments();
```

The hybrid retriever is configured at construction with BM25 at 60% weight and the vector knowledge base at 40%:

```java
this.hybridRetriever = HybridRetriever.builder()
    .stream(bm25Stream, 0.6)
    .stream(new KnowledgeBaseStream(knowledgeBase), 0.4)
    .build();
```

## FileIndexer

`FileIndexer` is the Server adapter over
`com.tnsai.intelligence.rag.binding.FileIngestionService`. Discovery,
`.gitignore` / `.tnsignore`, default skip lists, format detection,
extraction, and chunking are the same contract as declarative
`@KnowledgeSource(type = FILE)`. The class does **not** call
`CodeChunker`. Chunks come from `NormalizedDocumentChunker` — see
[File formats](formats.md).

This path is in TnsAI 0.14.0 (`TnsAI@b7f3a098`,
TAN-5806) and ships in Maven Central `0.14.1`. Maven Central `0.13.0`
used a Server-local walk.

`new FileIndexer()` and `new FileIngestionService()` share
`FileIngestionService.Limits.defaults()`. Pass the same `Limits`
record to both constructors when you tighten a deployment.

### Formats

The walk does not have a private "28 extensions" list. Candidates
follow `DocumentFormat` / `AUTO` on [File formats](formats.md).
`include` / `exclude` on a `KnowledgeSourceConfig` still run before
the format check.

### Ignore files and default skips

Each directory may carry `.gitignore` and `.tnsignore`. Rules from a
nested file apply **under that directory**, not as if they were
written at the corpus root.

| Pattern | Meaning |
|---|---|
| leading `/` | Anchored to the ignore file's directory. `/build` in `src/.gitignore` does not match `docs/build`. |
| trailing `/` | Directory tree only. `tmp/` skips the `tmp` directory and its children; a regular file named `tmp` stays. |
| unanchored name (`*.log`, `target`) | Matches at any depth under that ignore file. |
| `#` | Comment. |
| `!` | Unsupported. The ignore file is a configuration error. |

`.gitignore` and `.tnsignore` themselves are never ingested.

Default directory names (case-insensitive) are also skipped:
`.aws`, `.git`, `.gnupg`, `.gradle`, `.hg`, `.idea`, `.kube`,
`.next`, `.ssh`, `.svn`, `.vscode`, `__pycache__`, `build`,
`coverage`, `dist`, `node_modules`, `out`, `target`, `vendor`.
A subset (`.git`, `.ssh`, `node_modules`, `vendor`, and the other
secret/VCS ancestors) also matches when it appears as any path
component.

Default file names skipped anywhere: `.env`, `.env.*`,
`.git-credentials`, `.netrc`, `.npmrc`, `.pypirc`, `id_dsa`,
`id_ecdsa`, `id_ed25519`, `id_rsa`.

Ignore files are capped at 64 KiB and 1 024 compiled selectors.

### Admission caps

`FileIngestionService.Limits` is one policy object:

| Field | Default | Meaning |
|---|---:|---|
| `maxDecodedBytes` | 64 MiB | Aggregate decoded-character budget for the source |
| `maxFiles` | 4 096 | Regular files that may be admitted |
| `maxEntries` | 8 192 | Walk entries (files and directories) |
| `maxDepth` | 64 | Maximum relative path depth |

A single file is also bounded at 16 MiB raw input
(`ExtractionContext`). There is no 512 KB skip threshold.

Symbolic links are never followed. An empty regular file is not a
special skip rule — format detection still decides.

### Incremental indexing

Reuse is `ingestionFingerprint` (content hash plus format, extractor,
chunker, and selectors), not a content-only SHA-256. Unchanged
fingerprints keep the previous chunks.

FILE ingest **stages** a complete generation before mutating TF-IDF or
BM25 (TAN-5620). A source-level format or extraction failure discards
the staged replacement — file *N* cannot publish chunks from files
`1..N-1` of the new attempt. `FileIngestionService.IngestionResult`
carries `fingerprints` (`Map<String, String>`); the previous
eight-argument form is gone. The atomic swap is
`KnowledgeBase.replaceDocuments`. `InMemoryKnowledgeBase` and
`BM25Stream` hold their write locks across that generation. The
interface default walks remove-then-add sequentially — concurrent
readers can see a mix unless the store overrides.

Call `fileIndexer.clearHashes()` to drop the fingerprint map and force
a full re-index.

## CodeChunker

`com.tnsai.server.rag.CodeChunker` is the Server facade for one-file
source chunking. TnsAI 0.14.0 (`TnsAI@a652e847`, TAN-5618) delegates
to `SourceCodeContentExtractor` and `NormalizedDocumentChunker`, so
callers share symbol boundaries, IDs, limits, and provenance with
declarative FILE ingest.

`FileIndexer` still does **not** call `CodeChunker`. Live directory
ingest goes through `FileIngestionService` and the same extractor.
Do not treat a leftover regex language table as the live path.

Symbol units and metadata: [File formats](formats.md).

## BM25Stream

The `BM25Stream` provides keyword-based search using the Okapi BM25 algorithm, which is the same ranking function used by search engines like Elasticsearch. It scores documents based on how well their terms match the query, accounting for term frequency and document length.

### Parameters

These BM25 parameters control how the scoring behaves. The defaults work well for code search and rarely need tuning.

| Parameter | Value | Description |
|-----------|-------|-------------|
| K1 | 1.2 | Term frequency saturation |
| B | 0.75 | Document length normalization |

### Text Processing Pipeline

Before scoring, queries and documents go through a text processing pipeline that normalizes, tokenizes, and stems terms. This improves recall by matching different forms of the same word.

1. **Tokenization**: Lowercase, strip non-alphanumeric (except `_`), split on whitespace, drop tokens with 1 character or fewer
2. **Stop word removal**: 50 common English stop words
3. **Stemming**: Suffix-stripping rules for 14 suffixes (`-ies`, `-ing`, `-tion`, `-sion`, `-ment`, `-ness`, `-able`, `-ous`, `-ful`, `-less`, `-ly`, `-ed`, `-er`, `-es`, `-s`)
4. **Synonym expansion** (query-time only): 20 coding-domain synonym pairs

### Synonym Pairs

At query time, common coding abbreviations are expanded to their full forms (and vice versa) so that searching for "auth" also finds documents containing "authentication".

| Term | Synonyms |
|------|----------|
| db | database |
| auth | authentication, authorization |
| config | configuration |
| perf | performance |
| impl | implementation |
| req | request |
| res | response |
| err | error |
| msg | message |
| fn | function |
| param | parameter |
| repo | repository |
| env | environment |
| async | asynchronous |
| sync | synchronous |

## HybridRetriever

The `HybridRetriever` combines results from multiple search strategies (like BM25 keyword search and vector similarity search) into a single ranked list. This hybrid approach gives better results than either method alone because keyword search finds exact term matches while vector search captures semantic similarity.

### Fusion Algorithm

The retriever merges results using Reciprocal Rank Fusion (RRF), which combines rankings without needing normalized scores. For each document appearing in any stream's results:

```
score(doc) = SUM over streams: weight(stream) / (K + rank(doc, stream) + 1)
```

Where `K = 60` (the RRF constant). Documents are then sorted by fused score.

### Diversification

To prevent a single large file from dominating search results, the retriever limits output to a maximum of 3 chunks per source file. This ensures the agent sees context from multiple relevant files.

```java
HybridRetriever retriever = HybridRetriever.builder()
    .stream(bm25Stream, 0.6)       // 60% weight
    .stream(vectorStream, 0.4)      // 40% weight
    .build();

List<SearchResult> results = retriever.retrieve("authentication flow", 10);
```

## Context Prompt Format

When the agent asks a question, `RagService.buildContextPrompt` searches for relevant code and prepends it to the user's query. This gives the LLM the codebase context it needs to answer accurately.

```
[Relevant code context]
--- file: src/auth/Middleware.java (lines 15-45) ---
public class AuthMiddleware {
    private final TokenValidator validator;
    ...
}

--- file: src/auth/TokenValidator.java (lines 1-30) ---
public class TokenValidator {
    ...
}

[User question]
How does the authentication middleware work?
```

If no context is found (empty knowledge base or no matches), the original query is returned unchanged.

## Document Management API

Beyond automatic directory indexing, you can manually add, list, and remove documents in the knowledge base. This is useful for injecting custom knowledge (like deployment procedures or domain-specific documentation) that is not part of the codebase.

```java
// Add a document with metadata
String docId = rag.addDocument("Custom knowledge content",
    Map.of("source", "user", "topic", "deployment"));

// List documents (returns preview, length, metadata)
List<RagService.DocumentInfo> docs = rag.listDocuments();
// DocumentInfo(id, preview(100chars), contentLength, metadata)

// Get a specific document
Optional<Document> doc = rag.getDocument(docId);

// Remove
boolean removed = rag.removeDocument(docId);

// Clear everything
rag.clear();
```

Documents added via `addDocument` are tracked separately and appear in `listDocuments()`. Both manually added documents and file-indexed chunks are searchable through the same hybrid retriever.