File formats and ingestion

@KnowledgeSource.format() defaults to DocumentFormat.AUTO. AUTO is resolved independently for every regular, non-symlink file under the root. An explicit format is strict: every remaining candidate must use one of that format's registered extensions (except TEXT, which accepts any file name and decodes UTF-8).

include / exclude run before the format check. An excluded file cannot trigger an explicit-format mismatch. Unsupported AUTO candidates that survive the selectors are skipped; a directory that contains files but no usable candidate is a configuration error.

Nested .gitignore / .tnsignore and FileIngestionService.Limits are the same contract on declarative FILE sources and Server FileIndexer. Details: Pipeline — ignore and admission.

Only KnowledgeType.FILE has a bundled loader. URL, DATABASE, and MEMORY are SPI keys with no framework loader — those sources are skipped with a warning unless an application module registers one.

@KnowledgeSource(name = "mixed", path = "knowledge", format = DocumentFormat.AUTO)

@KnowledgeSource(
    name = "xml-corpus",
    path = "data",
    format = DocumentFormat.XML,
    include = {"**/*.xml"})

@KnowledgeSource(
    name = "code",
    path = "src",
    format = DocumentFormat.SOURCE_CODE,
    include = {"**/*.java", "**/*.kt"})

A single file path is ingested as one document. A directory is walked; * / ? do not cross directories, ** matches zero or more segments. Include is evaluated first; any matching exclude drops the path.

Format matrix

Aliases come from DocumentFormat. Media types are what DefaultContentDetector reports. PDF / Office backends stay optional on tnsai-intelligence — they are not bundled just because you depend on tnsai-intelligence. EPUB is claimed without those libraries (JDK zip + jsoup). A missing extractor for a claimed binary format fails when that format is requested.

SOURCE_CODE is a family. Language is not a DocumentFormat value. TnsAI 0.14.0 (TnsAI@a652e847, TAN-5618 / PR #140) detects it after the format is selected and emits stable symbol blocks through SourceCodeContentExtractor + NormalizedDocumentChunker. The type ships on the v0.14.0 tag and remains in Maven Central 0.14.1.

Detection order: languageHint metadata, then extension, then shebang / content hints. Known keys include java, kotlin, javascript, typescript, python, go, rust, plus the other family extensions (ruby, php, scala, c, cpp, csharp, swift, shell, sql, css). Unrecognized content becomes source.

Provider discovery and TEXT, MARKDOWN, and JSONL extraction work on a Java 21 JRE without jdk.compiler. SOURCE_CODE remains available there and uses the bounded JRE structural adapter. That adapter records parser=structural, fallback=true, and fallbackReason=jdk-compiler-unavailable. This is distinct from the full-JDK non-Java structural-fallback path and from a generic-fallback block emitted when no structural symbol is recognized.

Fallback use and reason are always visible. Document-level keys include language, parser, fallback, and fallbackReason. Symbol blocks add:

NormalizedDocumentChunker keeps those blocks as retrieval units instead of whole-file / line groups. The 100-line / 16 KiB limits apply to every emitted block, including oversized symbol blocks.

Declarative @KnowledgeSource(format = SOURCE_CODE) and Server CodeChunker share extraction, symbol boundaries, content, language, and line provenance, but not the exact ID namespace. Declarative FILE ingest prefixes the path with source.name; CodeChunker uses the path directly. FileIndexer does not call CodeChunker; it uses FileIngestionService, which loads the same extractor. See Pipeline — CodeChunker.

Limits and failures

Default extract bounds (ExtractionContext): 16 MiB raw input, 16 MiB normalized text, 65 536 blocks. Shared walk caps (FileIngestionService.Limits.defaults(), TnsAI 0.14.0): 64 MiB decoded aggregate, 4 096 files, 8 192 walk entries, depth 64. Shared chunker (NormalizedDocumentChunker): 100 lines / 16 KiB per generic chunk, 1024 chunks / 16 MiB per document.

• Explicit format + wrong extension → binding/config error before ingest.
• AUTO + unknown type → skip that file; empty usable set → error.
• Ingest/extract failures throw com.tnsai.rag.DocumentProcessingException (@since 0.13.0). Catch this type — not IllegalArgumentException — for a missing backend or a corrupt file. The public message has source name, URI, format, and a framework detail. Document bytes are never in it.

• Binary format declared, extractor/backend absent → UNSUPPORTED_FORMAT. Never treat those bytes as UTF-8.
• Extract timeout default: 30 s (ContentExtractorRegistry) → EXTRACTION.
• ContentExtractorRegistry.discover() and FileIngestionService share one META-INF/services snapshot (TAN-5812). The office extractor is not hand-appended beside ServiceLoader, so the advertised format set is the ingest set.

Fingerprints and reindex

Each indexed file gets ingestionFingerprint (SHA-256 of source identity + content hash + declared/detected format + extractor id/ version + include/exclude + chunker id/version). Content bytes alone do not decide reuse. After a parser, chunker, selector, or extractor version change, rebuild the index — stale chunks stay until you re-ingest.

Related

• Knowledge Base
• Pipeline
• Strategies
• Smart segmentation — analysis splitter, not FILE ingest