# File formats and ingestion

`KnowledgeType.FILE` says **where** documents come from (a path).
`DocumentFormat` says **how** their bytes are interpreted. They are not
the same field.

`@KnowledgeSource.format()` defaults to `DocumentFormat.AUTO`. AUTO is
resolved independently for every regular, non-symlink file under the
root. An explicit format is strict: every remaining candidate must use
one of that format's registered extensions (except `TEXT`, which
accepts any file name and decodes UTF-8).

`include` / `exclude` run **before** the format check. An excluded file
cannot trigger an explicit-format mismatch. Unsupported AUTO candidates
that survive the selectors are skipped; a directory that contains files
but no usable candidate is a configuration error.

Nested `.gitignore` / `.tnsignore` and `FileIngestionService.Limits`
are the same contract on declarative FILE sources and Server
`FileIndexer`. Details: [Pipeline — ignore and admission](pipeline.md#ignore-files-and-default-skips).

Only `KnowledgeType.FILE` has a bundled loader. `URL`, `DATABASE`, and
`MEMORY` are SPI keys with no framework loader — those sources are
skipped with a warning unless an application module registers one.

<!-- java-contract: src/main/java/com/example/tnsai/docs/FileFormatAnnotationsExample.java -->
```java
@KnowledgeSource(name = "mixed", path = "knowledge", format = DocumentFormat.AUTO)
```

```java
@KnowledgeSource(
    name = "xml-corpus",
    path = "data",
    format = DocumentFormat.XML,
    include = {"**/*.xml"})
```

```java
@KnowledgeSource(
    name = "code",
    path = "src",
    format = DocumentFormat.SOURCE_CODE,
    include = {"**/*.java", "**/*.kt"})
```

A single file path is ingested as one document. A directory is walked;
`*` / `?` do not cross directories, `**` matches zero or more segments.
Include is evaluated first; any matching exclude drops the path.

## Format matrix

Aliases come from `DocumentFormat`. Media types are what
`DefaultContentDetector` reports. PDF / Office backends stay
`optional` on `tnsai-intelligence` — they are **not** bundled just
because you depend on `tnsai-intelligence`. EPUB is claimed without
those libraries (JDK zip + jsoup). A missing extractor for a claimed
binary format fails when that format is requested.

| Format | Extensions | Media type | Bundled decode | Extractor | Chunk unit |
|---|---|---|---|---|---|
| `AUTO` | (per file) | (detected) | n/a | detector + format extractor | (detected) |
| `TEXT` | `txt` (any name accepted) | `text/plain` | yes | `StructuredTextContentExtractor` | small file kept whole; else line/char groups |
| `MARKDOWN` | `md`, `markdown` | `text/markdown` | yes | same | heading/block path when present |
| `HTML` | `html`, `htm` | `text/html` | yes | same | element path |
| `CSV` | `csv` | `text/csv` | yes | same | row groups (max 4096 columns) |
| `TSV` | `tsv` | `text/tab-separated-values` | yes | same | row groups |
| `JSON` | `json` | `application/json` | yes | same | structural path |
| `JSONL` | `jsonl`, `ndjson` | `application/x-ndjson` | yes | same | line records |
| `YAML` | `yaml`, `yml` | `application/yaml` | yes | same | structural path |
| `XML` | `xml` | `application/xml` | yes | same | element path |
| `SOURCE_CODE` | `java`, `kt`, `kts`, `scala`, `groovy`, `js`, `jsx`, `ts`, `tsx`, `py`, `rb`, `php`, `go`, `rs`, `c`, `h`, `cc`, `cpp`, `cxx`, `hpp`, `cs`, `swift`, `sh`, `bash`, `zsh`, `sql`, `css` | `text/plain` | yes | `SourceCodeContentExtractor` | symbol blocks (class/function/method); fallback file/residual spans |
| `PDF` | `pdf` | `application/pdf` | **optional** PDFBox | `OfficeDocumentContentExtractor` | page |
| `DOCX` | `docx` | OOXML word | **optional** POI | same | paragraph/table |
| `XLSX` | `xlsx` | OOXML sheet | **optional** POI | same | sheet/row |
| `PPTX` | `pptx` | OOXML slides | **optional** POI | same | slide |
| `EPUB` | `epub` | `application/epub+zip` | yes (zip+jsoup) | same | spine chapter |

`SOURCE_CODE` is a **family**. Language is not a `DocumentFormat`
value. TnsAI 0.14.0 (`TnsAI@a652e847`, TAN-5618 / PR #140) detects
it after the format is selected and emits **stable symbol blocks**
through `SourceCodeContentExtractor` + `NormalizedDocumentChunker`.
The type ships on the `v0.14.0` tag and remains in Maven Central `0.14.1`.

Detection order: `languageHint` metadata, then extension, then
shebang / content hints. Known keys include `java`, `kotlin`,
`javascript`, `typescript`, `python`, `go`, `rust`, plus the other
family extensions (`ruby`, `php`, `scala`, `c`, `cpp`, `csharp`,
`swift`, `shell`, `sql`, `css`). Unrecognized content becomes
`source`.

| Language / path | `parser` metadata | Notes |
|---|---|---|
| Java on a full JDK | `jdk.compiler` | Nested types, methods, signatures, `package`, `imports`, `extends` / `implements`, method `calls`. A parse failure moves to `structural-fallback`. |
| TypeScript, JavaScript, Kotlin, Python, Go, Rust on a full JDK | `structural-fallback` | Declarations, braces, strings, comments; Python uses indentation. |
| Other family languages on a full JDK | `structural-fallback` | Language metadata is still recorded; a file with no recognized symbol uses a `generic-fallback` block. |

Provider discovery and `TEXT`, `MARKDOWN`, and `JSONL` extraction work on a
Java 21 JRE without `jdk.compiler`. `SOURCE_CODE` remains available there and
uses the bounded JRE structural adapter. That adapter records
`parser=structural`, `fallback=true`, and
`fallbackReason=jdk-compiler-unavailable`. This is distinct from the full-JDK
non-Java `structural-fallback` path and from a `generic-fallback` block emitted
when no structural symbol is recognized.

Fallback use and reason are always visible. Document-level keys include
`language`, `parser`, `fallback`, and `fallbackReason`. Symbol blocks
add:

| Key | Meaning |
|---|---|
| `symbol` | Simple name (`<init>` constructors become the type name) |
| `symbolKind` | e.g. `class` / `method` / `file` / `residual` |
| `parentSymbol` | Enclosing type when present |
| `signature` | Bounded declaration text |
| `parser` | `jdk.compiler`, `structural-fallback`, `structural`, or `generic-fallback`, depending on the path above |
| `lineStart` / line provenance | Survives indexing |

`NormalizedDocumentChunker` keeps those blocks as retrieval units
instead of whole-file / line groups. The 100-line / 16 KiB limits apply
to every emitted block, including oversized symbol blocks.

Declarative `@KnowledgeSource(format = SOURCE_CODE)` and Server
`CodeChunker` share extraction, symbol boundaries, content, language,
and line provenance, but not the exact ID namespace. Declarative FILE
ingest prefixes the path with `source.name`; `CodeChunker` uses the path
directly. `FileIndexer` does **not** call `CodeChunker`; it uses
`FileIngestionService`, which loads the same extractor. See
[Pipeline — CodeChunker](pipeline.md#codechunker).

## Limits and failures

Default extract bounds (`ExtractionContext`): 16 MiB raw input, 16 MiB
normalized text, 65 536 blocks. Shared walk caps
(`FileIngestionService.Limits.defaults()`, TnsAI 0.14.0): 64 MiB
decoded aggregate, 4 096 files, 8 192 walk entries, depth 64. Shared
chunker (`NormalizedDocumentChunker`): 100 lines / 16 KiB per generic
chunk, 1024 chunks / 16 MiB per document.

- Explicit format + wrong extension → binding/config error before
  ingest.
- AUTO + unknown type → skip that file; empty usable set → error.
- Ingest/extract failures throw
  `com.tnsai.rag.DocumentProcessingException` (`@since 0.13.0`).
  Catch this type — not `IllegalArgumentException` — for a missing
  backend or a corrupt file. The public message has source name, URI,
  format, and a framework detail. Document bytes are never in it.

| Stage | When |
|---|---|
| `UNSUPPORTED_FORMAT` | No `ContentExtractor` is installed for the requested format |
| `VALIDATION` | Size / identity / returned-format checks fail |
| `EXTRACTION` | Provider throw, timeout, interrupt, null result, or unreadable explicit office (PDF/DOCX/XLSX/PPTX/EPUB) after the office extractor is on the SPI (TnsAI 0.14.0, TAN-5812) |
| `DETECTION` / `CHUNKING` | Detection or chunker failures on the same type |

- Binary format declared, extractor/backend absent →
  `UNSUPPORTED_FORMAT`. Never treat those bytes as UTF-8.
- Extract timeout default: 30 s (`ContentExtractorRegistry`) →
  `EXTRACTION`.
- `ContentExtractorRegistry.discover()` and `FileIngestionService`
  share one `META-INF/services` snapshot (TAN-5812). The office
  extractor is not hand-appended beside ServiceLoader, so the advertised
  format set is the ingest set.

## Fingerprints and reindex

Each indexed file gets `ingestionFingerprint` (SHA-256 of source
identity + content hash + declared/detected format + extractor id/
version + include/exclude + chunker id/version). Content bytes alone
do not decide reuse. After a parser, chunker, selector, or extractor
version change, rebuild the index — stale chunks stay until you
re-ingest.

## Related

- [Knowledge Base](knowledge-base.md)
- [Pipeline](pipeline.md)
- [Strategies](strategies.md)
- [Smart segmentation](segmentation.md) — analysis splitter, not FILE ingest