# Server RAG pipeline

`tnsai-server` gives each session its own retrieval pipeline over a local directory. It indexes files through the same ingestion service as declarative `@Knowledge(type = FILE)`, keeps the chunks in memory, and ranks them by fusing a BM25 stream with the knowledge base's TF-IDF search.

## How it fits together

```text
Directory  -->  FileIndexer  -->  FileIngestionService  -->  InMemoryKnowledgeBase + BM25Stream
                                                                  |
Query  -->  HybridRetriever  -->  [BM25Stream, weight 0.6]    -+  RRF  -->  Results
                             -->  [TF-IDF search, weight 0.4]  -+
```

`SessionManager.getRag(sessionId)` creates a session's `RagService` on first use, and removing the session clears it. Indexing and `clear()` run under one lock, so only one index operation runs at a time per session; searches don't take that lock.

## RagService

```java
RagService rag = sessionManager.getRag("my-session");

rag.indexDirectory(Path.of("/project/src"), progress -> {
    System.out.printf("Indexed %d/%d: %s%n",
        progress.indexedFiles(), progress.totalFiles(), progress.currentFile());
});

List<SearchResult> results = rag.search("authentication middleware", 5);
String prompt = rag.buildContextPrompt("How does auth work?", 5);
IndexStatus status = rag.getStatus();
```

`getStatus()` reports the file count, chunk count, last index time and indexed path.

### Context prompt

`buildContextPrompt(query, topK)` searches and wraps the results in the same untrusted-context fence that agent RAG uses. Each result's ID, metadata and content are XML-escaped inside `<tnsai-memory>`, and the query follows unchanged:

```text
<tnsai-memory>
<evidence>
<document-id>...</document-id>
<provenance>
<entry><key>file</key><value>src/auth/Middleware.java</value></entry>
</provenance>
<content>...</content>
</evidence>
</tnsai-memory>

<tnsai-user-message>
How does auth work?
</tnsai-user-message>
```

With no results, or if rendering fails, the query is returned as is.

### Ground an agent's chat

`attachChatTo(agent)` binds the session corpus to an agent as live chat knowledge, so chat grounds on the user's original message rather than on a rewritten prompt. The server's WebSocket handler calls it before each streamed reply.

The agent must declare MEMORY chat knowledge named `RagService.CHAT_SOURCE` (`"session-knowledge"`) that searches with `RagService.CHAT_TOP_K` (5); otherwise it refuses the binding. The binding holds the corpus instead of copying it, so documents indexed later are visible without rebinding. While the corpus is empty, the call does nothing.

### Document management

```java
String docId = rag.addDocument("Custom knowledge content",
    Map.of("source", "user", "topic", "deployment"));

List<RagService.DocumentInfo> docs = rag.listDocuments();
Optional<Document> doc = rag.getDocument(docId);
boolean removed = rag.removeDocument(docId);

rag.clear();
```

`listDocuments()` lists only documents added with `addDocument`, as `DocumentInfo(id, preview, contentLength, metadata)` with a 100-character preview. Added documents and indexed chunks are searched together. `clear()` empties the knowledge base, the BM25 index and the fingerprints, and resets the status.

## FileIndexer

`FileIndexer` is the server adapter over `com.tnsai.intelligence.rag.binding.FileIngestionService`. Discovery, `.gitignore` and `.tnsignore`, the default skip lists, format detection, extraction and chunking follow the same contract as declarative `@Knowledge(type = FILE)`, and chunks come from `NormalizedDocumentChunker`. See [File formats](file-formats.md).

`new FileIndexer()` and `new FileIngestionService()` share `FileIngestionService.Limits.defaults()`. Pass the same `Limits` record to both constructors when you tighten a deployment.

### Formats

There is no separate extension list. Candidates follow `DocumentFormat` and `AUTO` detection on [File formats](file-formats.md). `include` and `exclude` on a `KnowledgeSourceConfig` still run before the format check.

### Ignore files and default skips

Each directory may carry `.gitignore` and `.tnsignore`. Rules from a nested file apply under that directory, not as if they were written at the corpus root.

| Pattern | Meaning |
|---|---|
| leading `/` | Anchored to the ignore file's directory. `/build` in `src/.gitignore` does not match `docs/build`. |
| trailing `/` | Directory tree only. `tmp/` skips the `tmp` directory and its children. A regular file named `tmp` stays. |
| unanchored name (`*.log`, `target`) | Matches at any depth under that ignore file. |
| `#` | Comment. |
| `!` | Unsupported. The ignore file is a configuration error. |

`.gitignore` and `.tnsignore` themselves are never ingested.

These directory names are skipped by default, case-insensitively: `.aws`, `.git`, `.gnupg`, `.gradle`, `.hg`, `.idea`, `.kube`, `.next`, `.ssh`, `.svn`, `.vscode`, `__pycache__`, `build`, `coverage`, `dist`, `node_modules`, `out`, `target`, `vendor`. Nine of them (`.aws`, `.git`, `.gnupg`, `.hg`, `.kube`, `.ssh`, `.svn`, `node_modules`, `vendor`) also match as any component of a path.

These file names are skipped anywhere: `.env`, `.env.*`, `.git-credentials`, `.netrc`, `.npmrc`, `.pypirc`, `id_dsa`, `id_ecdsa`, `id_ed25519`, `id_rsa`.

Ignore files are capped at 64 KiB and 1,024 compiled selectors.

### Admission caps

`FileIngestionService.Limits` is one policy object:

| Field | Default | Meaning |
|---|---:|---|
| `maxDecodedBytes` | 64 MiB | Aggregate decoded-character budget for the source |
| `maxFiles` | 4,096 | Regular files that may be admitted |
| `maxEntries` | 8,192 | Walk entries (files and directories) |
| `maxDepth` | 64 | Maximum relative path depth |

A single file is also bounded at 16 MiB of raw input. Symbolic links are never followed. An empty regular file has no special skip rule; format detection still decides.

### Incremental indexing

Each file has a pipeline fingerprint built from its content, format, extractor, chunker and selectors. A file whose fingerprint is unchanged keeps its previous chunks.

An index run stages the whole new generation, then swaps it in with `KnowledgeBase.replaceDocuments` and the matching `BM25Stream.replaceDocuments`. If the BM25 swap fails, the knowledge-base swap is undone. `InMemoryKnowledgeBase` and `BM25Stream` hold their write locks across the swap, so readers never see part of a generation. A store that keeps the interface default removes and then adds, and readers can see a mix.

- A rejected source, or a failure while ingesting or committing, leaves the previous index untouched.
- A file that fails to read is not treated as deleted. The readable files are committed and the failed file keeps its previous chunks until a later run replaces them.
- Files that are gone, renamed or outside the indexed root lose their chunks.

Call `fileIndexer.clearHashes()` to drop the fingerprints and force a full re-index.

## CodeChunker

`com.tnsai.server.rag.CodeChunker` chunks one source file on its own. It delegates to the same source-code extractor and `NormalizedDocumentChunker` as FILE ingest, so its symbol boundaries, IDs, limits and provenance match. `FileIndexer` does not call it. Symbol units and metadata are on [File formats](file-formats.md).

## BM25Stream

`BM25Stream` ranks by Okapi BM25 with `K1 = 1.2` and `B = 0.75`. It holds at most 10,000 documents by default, and documents expire one hour after last access.

Queries and documents are processed the same way:

1. **Tokenize:** lowercase, replace everything except `a-z`, `0-9` and `_` with spaces, split, and drop one-character tokens.
2. **Remove stop words:** 56 common English words.
3. **Stem:** strip one of 15 suffixes: `-ies`, `-ing`, `-tion`, `-sion`, `-ment`, `-ness`, `-able`, `-ous`, `-ful`, `-less`, `-ly`, `-ed`, `-er`, `-es`, `-s`. Words shorter than four characters are not stemmed.
4. **Expand synonyms**, at query time only, in both directions:

| Term | Expands to |
|------|----------|
| db | database |
| auth | authentication, authorization |
| config | configuration |
| perf | performance |
| impl | implementation |
| req | request |
| res | response |
| err | error |
| msg | message |
| fn | function |
| param | parameter |
| repo | repository |
| env | environment |
| async | asynchronous |
| sync | synchronous |

## HybridRetriever

`HybridRetriever` fuses several streams with Reciprocal Rank Fusion. It asks each available stream for `2 × topK` results and scores each document:

```text
score(doc) = sum over streams of weight(stream) / (60 + rank(doc, stream) + 1)
```

`rank` starts at 0. A stream that throws is logged and skipped. Results are sorted by fused score, and at most three chunks per `file` metadata value are kept, so one large file cannot fill the result list.

```java
HybridRetriever retriever = HybridRetriever.builder()
    .stream(bm25Stream, 0.6)
    .stream(new KnowledgeBaseStream(knowledgeBase), 0.4)
    .build();

List<SearchResult> results = retriever.retrieve("authentication flow", 10);
```

`KnowledgeBaseStream` wraps `KnowledgeBase.search`, which in `InMemoryKnowledgeBase` is TF-IDF; it uses no embeddings. A builder with no streams throws `IllegalStateException`.

## Related

- [KnowledgeBase API](knowledge-base.md): the store behind the pipeline.
- [File formats](file-formats.md): what ingestion reads and how it chunks.
- [RAG](index.md): declarative `@Knowledge` sources for agents.
- [Retrieval strategies](retrieval-strategies.md): hybrid retrieval in declarative RAG.