File formats and ingestion

@Knowledge.format() defaults to DocumentFormat.AUTO. AUTO is resolved independently for every regular, non-symlink file under the root. An explicit format is strict: every remaining candidate must use one of that format's registered extensions (except TEXT, which accepts any file name and decodes UTF-8).

include / exclude run before the format check. An excluded file cannot trigger an explicit-format mismatch. Unsupported AUTO candidates that survive the selectors are skipped. A directory that contains files but no usable candidate is a configuration error.

Nested .gitignore / .tnsignore and FileIngestionService.Limits are the same contract on declarative FILE sources and Server FileIndexer. Details: Pipeline: ignore and admission.

FILE and URL have bundled loaders. DATABASE is an SPI key with no framework loader, so such a source is skipped with a warning unless an application module registers one. MEMORY and GRAPH load nothing, so no loader serves them. format applies to FILE and HTTP/HTTPS URL sources.

@RoleSpec(knowledge = @Knowledge(
    name = "mixed",
    path = "knowledge",
    format = DocumentFormat.AUTO,
    search = @Search(queryParam = "question")))

@RoleSpec(knowledge = @Knowledge(
    name = "xml-corpus",
    path = "data",
    format = DocumentFormat.XML,
    include = {"**/*.xml"},
    search = @Search(queryParam = "question")))

@RoleSpec(knowledge = @Knowledge(
    name = "code",
    path = "src",
    format = DocumentFormat.SOURCE_CODE,
    include = {"**/*.java", "**/*.kt"},
    search = @Search(queryParam = "question")))

Each Role's action names its source with @ActionSpec(knowledge = @Knowledge(name = ...)). The full classes are in the contract fixture.

A single file path is ingested as one document. A directory is walked: * / ? do not cross directories, and ** matches zero or more segments. Include is evaluated first, and any matching exclude drops the path.

Format matrix

Aliases come from DocumentFormat. Media types are what DefaultContentDetector reports. PDF / Office backends stay optional on tnsai-rag. They are not bundled just because you depend on tnsai-rag. EPUB is claimed without those libraries (JDK zip

• jsoup). A missing extractor for a claimed binary format fails when that format is requested.

SOURCE_CODE is a family. Language is not a DocumentFormat value. The language is detected after the format is selected, and SourceCodeContentExtractor with NormalizedDocumentChunker emits stable symbol blocks.

Detection order: languageHint metadata, then extension, then shebang / content hints. Known keys include java, kotlin, javascript, typescript, python, go, rust, plus the other family extensions (ruby, php, scala, c, cpp, csharp, swift, shell, sql, css). Unrecognized content becomes source.

Provider discovery and TEXT, MARKDOWN, and JSONL extraction work on a Java 21 JRE without jdk.compiler. SOURCE_CODE remains available there and uses the bounded JRE structural adapter. That adapter records parser=structural, fallback=true, and fallbackReason=jdk-compiler-unavailable. This is distinct from the full-JDK non-Java structural-fallback path and from a generic-fallback block emitted when no structural symbol is recognized.

Fallback use and reason are always visible. Document-level keys include language, parser, fallback, and fallbackReason. Symbol blocks add:

NormalizedDocumentChunker keeps those blocks as retrieval units instead of whole-file / line groups. The 100-line / 16 KiB limits apply to every emitted block, including oversized symbol blocks.

Declarative @Knowledge(format = SOURCE_CODE) and Server CodeChunker share extraction, symbol boundaries, content, language, and line provenance, but not the exact ID namespace. Declarative FILE ingest prefixes the path with source.name, while CodeChunker uses the path directly. FileIndexer does not call CodeChunker. It uses FileIngestionService, which loads the same extractor. See Pipeline: CodeChunker.

Limits and failures

Default extract bounds (ExtractionContext): 16 MiB raw input, 16 MiB normalized text, 65 536 blocks. Shared walk caps (FileIngestionService.Limits.defaults()): 64 MiB decoded aggregate, 4 096 files, 8 192 walk entries, depth 64. Shared chunker (NormalizedDocumentChunker): 100 lines / 16 KiB per generic chunk, 1024 chunks / 16 MiB per document.

• Explicit format + wrong extension → binding/config error before ingest.
• AUTO + unknown type → skip that file. Empty usable set → error.
• Ingest/extract failures throw com.tnsai.rag.DocumentProcessingException. Catch this type, not IllegalArgumentException, for a missing backend or a corrupt file. The public message has source name, URI, format, and a framework detail. Document bytes are never in it.

• Binary format declared, extractor/backend absent → UNSUPPORTED_FORMAT. Never treat those bytes as UTF-8.
• Extract timeout default: 30 s (ContentExtractorRegistry) → EXTRACTION.
• A provider that throws an Error during extraction is contained like an exception: the result is EXTRACTION with <provider> failed, and the original throwable is not attached. Only a direct VirtualMachineError (such as OutOfMemoryError or StackOverflowError) escapes the registry unwrapped. ThreadDeath is contained like any other non-fatal Error.
• When the registry is built (discover() or of(...)), a VirtualMachineError in a provider failure's cause chain (up to 256 links) is rethrown. Any other provider failure becomes a VALIDATION failure that keeps the original throwable as its cause.
• ContentExtractorRegistry.discover() and FileIngestionService share one META-INF/services snapshot. The office extractor is not hand-appended beside ServiceLoader, so the advertised format set is the ingest set.

PDF and OOXML caps

The PDFBox and POI extractors in tnsai-rag refuse an input past a cap. They do not truncate it. Through ingestion the refusal is an EXTRACTION failure with the detail <provider> failed, and the cap reason is not in the message. A caller that invokes the extractor directly gets IllegalArgumentException with the reason.

PDF (PDFBox):

• More than min(10,000, maxBlocks) pages → refused.
• A page with more than min(maxBlockChars, 1,048,576) glyphs → refused. The default maxBlockChars is already 1,048,576, so under the default bounds a page has no new limit.

DOCX, XLSX and PPTX (POI) are pre-scanned before POI opens the package. The package is refused when it:

• has more than 1,000 zip entries, a duplicate entry, or no [Content_Types].xml
• nests XML deeper than 256
• passes the XML node cap, the XML text cap, or the 1,000,000-node heap cap (node and text caps scale with maxBlocks and the total block-character bound)
• inflates past 8 × maxInputBytes (128 MiB under the default)
• has a part declared as XML that is not well-formed
• lists one sheet twice in its workbook, or names its glossary more than once

A part POI parses more than once counts toward the caps once per parse. Under the heap cap, spreadsheets beyond about 3,100 rows × 50 columns are refused. A legacy .xls under an XLSX label and a default-password-encrypted XLSX are refused. POI's POIXMLException and RecordFormatException surface as IllegalArgumentException, so they also reach ingestion as EXTRACTION.

Limitation: PDFs are not checked for stream decompression size before loading. A crafted PDF can still exhaust memory, and the page and glyph caps do not bound it. Do not treat these caps as a memory bound for untrusted PDFs.

Fingerprints and reindex

Each indexed file gets ingestionFingerprint (SHA-256 of source identity + content hash + declared/detected format + extractor id/ version + include/exclude + chunker id/version). Content bytes alone do not decide reuse. After a parser, chunker, selector, or extractor version change, rebuild the index. Stale chunks stay until you re-ingest.

Related

• KnowledgeBase API: loading files outside a Role.
• Pipeline: chunkers and the ingestion fingerprint.
• Install retrieval providers: adding a ContentExtractor.
• Strategies
• Smart segmentation: analysis splitter, not FILE ingest