# Searching an agent's run files

**0.18.6.** `IncrementalTextFilesRetriever` (`com.tnsai.rag.files`, in
`tnsai-core`) lets an agent's chat search the text files that agent writes
in a run: its `.history` and `.log` files and any other text file under
`<run>/episode-N/<agent>/`. Each line is one piece of evidence, ranked by
keyword relevance (BM25) against the request's turn message, or its query
when it has none. The retriever follows the files as they grow.

## Giving it to a chat

The retriever is a `RetrievalEngine.Retriever`. A host gives it to a chat as
an undeclared binding, a source no `@Knowledge` declares (see
[Exact owner and `isOwnedBy`](../../integrate/scop.md#exact-owner-and-isownedby)):

```java
var retriever = IncrementalTextFilesRetriever.builder(runFolder, "Alice")
        .interpreter(hostInterpreter)            // what the host's records mean
        .masker(Masker.hidingOutside(runFolder)) // opt-in; masking is off by default
        .build();
var binding = ChatKnowledgeBinding.undeclared(
        agent, "Alice", "agent-files", 5, retriever); // owner, owner id, source name, topK
```

| Builder member | Default |
|---|---|
| `sourceName(String)` | `"agent-files"` (`DEFAULT_SOURCE_NAME`) |
| `scope(Scope)` | `Scope.RUN` |
| `interpreter(RecordInterpreter)` | `RecordInterpreter.NONE`: every record is a note |
| `masker(Masker)` | `Masker.noOp()`: nothing is hidden |

## What a record is

A host tells the retriever what its records mean with a `RecordInterpreter`.
Its `read(file, line, record)` returns a `Reading`: a note, a message the
agent was `told` or `said`, or a record `leftOut`. Without an interpreter
every record is a note.

- With a host interpreter, records of being told the turn message are not
  returned, and a message several files record is given once.
- A note several files record is given once with any interpreter.
- Log records at `DEBUG` and `TRACE` are left out.

## Line format

A host that keeps per-agent `.history` and `.log` files writes them with
the logback patterns of `AgentRecordFormat`. The episode state goes under
the `episode-state` MDC key and the conversation under `conversation-id`.
Every line carries a conversation field before the writer's name, `-` on a
line written outside any conversation, so a history line still ends with
`] <agent> <message>`:

```text
2026-10-21 18:00:00 [running] [conv-1] Alice has started episode 1.
2026-10-21 18:00:00 INFO  [running] [-] Alice chat started
```

A message of several lines is written with two spaces before each line
after its first. `RecordParser` parses a line only from its start, so an
indented line continues the record before it and never starts one: a line
in a message that looks like a record, such as one of another conversation,
stays in the record it was written in. `RecordParser` also reads lines
written without the conversation field.

**Known limit.** Only a message's own lines are indented. Lines written
before 0.18.6, and the stack trace logback appends after a message, are
not, so a line in them that looks like a record of another conversation is
read as one.

A reader that splits these lines by position, such as a tail view, has to
skip the new field; see [Migration](../../help/migration.md).

## Episodes and conversations

The request's current episode is the decimal `N` of `episode-N` in its
`ChatRetrievalContext.CURRENT_EPISODE_FILTER` metadata filter; any other
value is rejected.

- `Scope.RUN`, the default, searches every episode of the run.
  `Scope.EPISODE` searches only the current episode and finds nothing
  without one. A record of another episode is evidence whatever
  conversation it was written in, so `Scope.RUN` suits a single-user host;
  a host whose run serves several users uses `Scope.EPISODE`.
- A `.history` or `.log` record is evidence only for a request with both a
  current episode and a conversation id. A request without either gets no
  such record from any episode; its other files are still searched.
- With both, a record of the current episode is evidence when it was
  written in the request's conversation or outside any (`-`).
- A line without the conversation field in the current episode does not
  enter evidence and is reported with one WARN per file. A line before a
  file's first record counts as one. A run written before 0.18.6 has no
  field on any line, so its other episodes are searched and its current
  episode's records are not.
- Other files are not filtered by conversation.

## Files

The run folder is taken by its real path, so a run folder that is itself a
symbolic link is followed. Links below it are not followed, and a file whose
real path lies outside the run folder is left out with one WARN. A file with
a NUL byte in its first 8 KiB is left out.

## Masking

Evidence text passes through the builder's `Masker` before it is indexed,
so what the masker hides is neither returned nor searched. A record is
masked whole before it is cut into lines and pieces, so a key block over
several of its lines is hidden whole. See
[Masking local paths and credentials](../../security/redaction.md#masking-local-paths-and-credentials)
for `Masker.hidingOutside` and `PathRedactor`.

**Known limit.** Masking sees one read at a time. A record whose lines
arrive in separate reads, such as one still being written when a search
reads it, or one across the 1 MiB block a long unread stretch is read in, is
masked in parts, so a key block over those lines can be left partly visible.

## Cost

The retriever has no file, piece or line cap: its index holds every line of
the agent's files in the run. Each search reads only what was appended since
the last one, and reads a file again from its start when it shrank, was
replaced or was rewritten. A piece of evidence holds at most
`MAX_PIECE_CHARS` (2000) characters of a line, a longer line being cut into
pieces, and a result holds at most the request's `topK` pieces.