# File formats and ingestion

`Knowledge.Type.FILE` says **where** documents come from (a path).
`DocumentFormat` says **how** their bytes are interpreted. They are not
the same field.

`@Knowledge.format()` defaults to `DocumentFormat.AUTO`. AUTO is
resolved independently for every regular, non-symlink file under the
root. An explicit format is strict: every remaining candidate must use
one of that format's registered extensions (except `TEXT`, which
accepts any file name and decodes UTF-8).

`include` / `exclude` run **before** the format check. An excluded file
cannot trigger an explicit-format mismatch. Unsupported AUTO candidates
that survive the selectors are skipped. A directory that contains files
but no usable candidate is a configuration error.

Nested `.gitignore` / `.tnsignore` and `FileIngestionService.Limits`
are the same contract on declarative FILE sources and Server
`FileIndexer`. Details: [Pipeline: ignore and admission](pipeline.md#ignore-files-and-default-skips).

`FILE` and `URL` have bundled loaders. `DATABASE` is an SPI key with no
framework loader, so such a source is skipped with a warning unless an
application module registers one. `MEMORY` and `GRAPH` load nothing, so no
loader serves them. `format` applies to FILE and HTTP/HTTPS URL sources.

<!-- java-contract: src/main/java/com/example/tnsai/docs/FileFormatAnnotationsExample.java -->
```java
@RoleSpec(knowledge = @Knowledge(
    name = "mixed",
    path = "knowledge",
    format = DocumentFormat.AUTO,
    search = @Search(queryParam = "question")))
```

```java
@RoleSpec(knowledge = @Knowledge(
    name = "xml-corpus",
    path = "data",
    format = DocumentFormat.XML,
    include = {"**/*.xml"},
    search = @Search(queryParam = "question")))
```

```java
@RoleSpec(knowledge = @Knowledge(
    name = "code",
    path = "src",
    format = DocumentFormat.SOURCE_CODE,
    include = {"**/*.java", "**/*.kt"},
    search = @Search(queryParam = "question")))
```

Each Role's action names its source with
`@ActionSpec(knowledge = @Knowledge(name = ...))`. The full classes are in the
contract fixture.

A single file path is ingested as one document. A directory is walked:
`*` / `?` do not cross directories, and `**` matches zero or more segments.
Include is evaluated first, and any matching exclude drops the path.

## Format matrix

Aliases come from `DocumentFormat`. Media types are what
`DefaultContentDetector` reports. PDF / Office backends stay
`optional` on `tnsai-rag`. They are **not** bundled just because you
depend on `tnsai-rag`. EPUB is claimed without those libraries (JDK zip
+ jsoup). A missing extractor for a claimed binary format fails when
that format is requested.

| Format | Extensions | Media type | Bundled decode | Extractor | Chunk unit |
|---|---|---|---|---|---|
| `AUTO` | (per file) | (detected) | n/a | detector + format extractor | (detected) |
| `TEXT` | `txt` (any name accepted) | `text/plain` | yes | `StructuredTextContentExtractor` | small file kept whole, else line/char groups |
| `MARKDOWN` | `md`, `markdown` | `text/markdown` | yes | same | heading/block path when present |
| `HTML` | `html`, `htm` | `text/html` | yes | same | element path |
| `CSV` | `csv` | `text/csv` | yes | same | row groups (max 4096 columns) |
| `TSV` | `tsv` | `text/tab-separated-values` | yes | same | row groups |
| `JSON` | `json` | `application/json` | yes | same | structural path |
| `JSONL` | `jsonl`, `ndjson` | `application/x-ndjson` | yes | same | line records |
| `YAML` | `yaml`, `yml` | `application/yaml` | yes | same | structural path |
| `XML` | `xml` | `application/xml` | yes | same | element path |
| `SOURCE_CODE` | `java`, `kt`, `kts`, `scala`, `groovy`, `js`, `jsx`, `ts`, `tsx`, `py`, `rb`, `php`, `go`, `rs`, `c`, `h`, `cc`, `cpp`, `cxx`, `hpp`, `cs`, `swift`, `sh`, `bash`, `zsh`, `sql`, `css` | `text/plain` | yes | `SourceCodeContentExtractor` | symbol blocks (class/function/method), with fallback file/residual spans |
| `PDF` | `pdf` | `application/pdf` | **optional** PDFBox | `OfficeDocumentContentExtractor` | page |
| `DOCX` | `docx` | OOXML word | **optional** POI | same | paragraph/table |
| `XLSX` | `xlsx` | OOXML sheet | **optional** POI | same | sheet/row |
| `PPTX` | `pptx` | OOXML slides | **optional** POI | same | slide |
| `EPUB` | `epub` | `application/epub+zip` | yes (zip+jsoup) | same | spine chapter |

`SOURCE_CODE` is a **family**. Language is not a `DocumentFormat`
value. The language is detected after the format is selected, and
`SourceCodeContentExtractor` with `NormalizedDocumentChunker` emits
**stable symbol blocks**.

Detection order: `languageHint` metadata, then extension, then
shebang / content hints. Known keys include `java`, `kotlin`,
`javascript`, `typescript`, `python`, `go`, `rust`, plus the other
family extensions (`ruby`, `php`, `scala`, `c`, `cpp`, `csharp`,
`swift`, `shell`, `sql`, `css`). Unrecognized content becomes
`source`.

| Language / path | `parser` metadata | Notes |
|---|---|---|
| Java on a full JDK | `jdk.compiler` | Nested types, methods, signatures, `package`, `imports`, `extends` / `implements`, method `calls`. A parse failure moves to `structural-fallback`. |
| TypeScript, JavaScript, Kotlin, Python, Go, Rust on a full JDK | `structural-fallback` | Declarations, braces, strings, comments. Python uses indentation. |
| Other family languages on a full JDK | `structural-fallback` | Language metadata is still recorded. A file with no recognized symbol uses a `generic-fallback` block. |

Provider discovery and `TEXT`, `MARKDOWN`, and `JSONL` extraction work on a
Java 21 JRE without `jdk.compiler`. `SOURCE_CODE` remains available there and
uses the bounded JRE structural adapter. That adapter records
`parser=structural`, `fallback=true`, and
`fallbackReason=jdk-compiler-unavailable`. This is distinct from the full-JDK
non-Java `structural-fallback` path and from a `generic-fallback` block emitted
when no structural symbol is recognized.

Fallback use and reason are always visible. Document-level keys include
`language`, `parser`, `fallback`, and `fallbackReason`. Symbol blocks
add:

| Key | Meaning |
|---|---|
| `symbol` | Simple name (`<init>` constructors become the type name) |
| `symbolKind` | e.g. `class` / `method` / `file` / `residual` |
| `parentSymbol` | Enclosing type when present |
| `signature` | Bounded declaration text |
| `parser` | `jdk.compiler`, `structural-fallback`, `structural`, or `generic-fallback`, depending on the path above |
| `lineStart` / line provenance | Survives indexing |

`NormalizedDocumentChunker` keeps those blocks as retrieval units
instead of whole-file / line groups. The 100-line / 16 KiB limits apply
to every emitted block, including oversized symbol blocks.

Declarative `@Knowledge(format = SOURCE_CODE)` and Server
`CodeChunker` share extraction, symbol boundaries, content, language,
and line provenance, but not the exact ID namespace. Declarative FILE
ingest prefixes the path with `source.name`, while `CodeChunker` uses the path
directly. `FileIndexer` does **not** call `CodeChunker`. It uses
`FileIngestionService`, which loads the same extractor. See
[Pipeline: CodeChunker](pipeline.md#codechunker).

## Limits and failures

Default extract bounds (`ExtractionContext`): 16 MiB raw input, 16 MiB
normalized text, 65 536 blocks. Shared walk caps
(`FileIngestionService.Limits.defaults()`): 64 MiB
decoded aggregate, 4 096 files, 8 192 walk entries, depth 64. Shared
chunker (`NormalizedDocumentChunker`): 100 lines / 16 KiB per generic
chunk, 1024 chunks / 16 MiB per document.

- Explicit format + wrong extension → binding/config error before
  ingest.
- AUTO + unknown type → skip that file. Empty usable set → error.
- Ingest/extract failures throw
  `com.tnsai.rag.DocumentProcessingException`.
  Catch this type, not `IllegalArgumentException`, for a missing
  backend or a corrupt file. The public message has source name, URI,
  format, and a framework detail. Document bytes are never in it.

| Stage | When |
|---|---|
| `UNSUPPORTED_FORMAT` | No `ContentExtractor` is installed for the requested format |
| `VALIDATION` | Size / identity / returned-format checks fail |
| `EXTRACTION` | Provider throw, timeout, interrupt, null result, or unreadable explicit office file (PDF/DOCX/XLSX/PPTX/EPUB) when the office extractor is installed |
| `DETECTION` / `CHUNKING` | Detection or chunker failures on the same type |

- Binary format declared, extractor/backend absent →
  `UNSUPPORTED_FORMAT`. Never treat those bytes as UTF-8.
- Extract timeout default: 30 s (`ContentExtractorRegistry`) →
  `EXTRACTION`.
- A provider that throws an `Error` during extraction is contained
  like an exception: the result is `EXTRACTION` with `<provider> failed`,
  and the original throwable is not attached. Only a direct
  `VirtualMachineError` (such as `OutOfMemoryError` or
  `StackOverflowError`) escapes the registry unwrapped. `ThreadDeath` is contained like any
  other non-fatal `Error`.
- When the registry is built (`discover()` or `of(...)`), a
  `VirtualMachineError` in a provider failure's cause chain (up to 256
  links) is rethrown. Any other provider failure becomes a `VALIDATION`
  failure that keeps the original throwable as its cause.
- `ContentExtractorRegistry.discover()` and `FileIngestionService`
  share one `META-INF/services` snapshot. The office
  extractor is not hand-appended beside ServiceLoader, so the advertised
  format set is the ingest set.

### PDF and OOXML caps

The PDFBox and POI extractors in `tnsai-rag` refuse
an input past a cap. They do not truncate it. Through ingestion the
refusal is an `EXTRACTION` failure with the detail
`<provider> failed`, and the cap reason is not in the message. A
caller that invokes the extractor directly gets
`IllegalArgumentException` with the reason.

PDF (PDFBox):

- More than `min(10,000, maxBlocks)` pages → refused.
- A page with more than `min(maxBlockChars, 1,048,576)` glyphs →
  refused. The default `maxBlockChars` is already 1,048,576, so
  under the default bounds a page has no new limit.

DOCX, XLSX and PPTX (POI) are pre-scanned before POI opens the
package. The package is refused when it:

- has more than 1,000 zip entries, a duplicate entry, or no
  `[Content_Types].xml`
- nests XML deeper than 256
- passes the XML node cap, the XML text cap, or the 1,000,000-node
  heap cap (node and text caps scale with `maxBlocks` and the total
  block-character bound)
- inflates past 8 × `maxInputBytes` (128 MiB under the default)
- has a part declared as XML that is not well-formed
- lists one sheet twice in its workbook, or names its glossary more
  than once

A part POI parses more than once counts toward the caps once per
parse. Under the heap cap, spreadsheets beyond about 3,100 rows × 50
columns are refused. A legacy `.xls` under an XLSX label and a
default-password-encrypted XLSX are refused. POI's `POIXMLException`
and `RecordFormatException` surface as `IllegalArgumentException`, so
they also reach ingestion as `EXTRACTION`.

**Limitation:** PDFs are not checked for stream
decompression size before loading. A crafted PDF can still exhaust
memory, and the page and glyph caps do not bound it. Do not treat
these caps as a memory bound for untrusted PDFs.

## Fingerprints and reindex

Each indexed file gets `ingestionFingerprint` (SHA-256 of source
identity + content hash + declared/detected format + extractor id/
version + include/exclude + chunker id/version). Content bytes alone
do not decide reuse. After a parser, chunker, selector, or extractor
version change, rebuild the index. Stale chunks stay until you
re-ingest.

## Related

- [KnowledgeBase API](knowledge-base.md): loading files outside a Role.
- [Pipeline](pipeline.md): chunkers and the ingestion fingerprint.
- [Install retrieval providers](providers.md): adding a `ContentExtractor`.
- [Strategies](retrieval-strategies.md)
- [Smart segmentation](segmentation.md): analysis splitter, not FILE ingest