Smart document segmentation
com.tnsai.intelligence.rag.segment.SmartDocumentSegmenter splits a long paper on headings and returns com.tnsai.intelligence.rag.segment.DocumentSegment records. Both are @since 0.14.0 in TnsAI 0.14.0 (TnsAI@609a661d, TAN-3651). They ship in Maven Central 0.14.1.
This is an analysis API. FILE ingest still chunks through
NormalizedDocumentChunker. Do not treat these sections as retrieval
units.
What it does
SmartDocumentSegmenter segmenter = new SmartDocumentSegmenter();
List<DocumentSegment> sections = segmenter.segment(Path.of("paper.md"));segment(String) accepts already-extracted Markdown or prose.
segment(Path) reads .md as UTF-8. PDF and DOCX go through the
installed ContentExtractorRegistry. A missing office backend fails
loud (DocumentProcessingException). Other extensions are also read
as UTF-8.
Each section:
- starts at an ATX (
#–######), numbered (3.1 Title), orSection 3.1: Titleheading; text before the first heading isPreamble - keeps
Figure/Fig./Tablecaption lines with that section - collects
Section/Sec./§/Figure/Tablementions from the body (crossRefs); numbered section ids resolve to the later heading title when present - sets
cumulativeSummaryto a first-sentence digest of prior sections (about 160 characters each)
DocumentSegment fields: index, sectionId (empty on the
preamble), sectionTitle, body, crossRefs, cumulativeSummary,
figures.
Related
- File formats — extractors used for PDF/DOCX
- Pipeline — FILE ingest chunker (different path)
Vectorless reasoning retrieval
Retrieval.Strategy.REASONING descends a document tree with a model choosing the branch. It does not score fragments, so there is no embedding and no similarity on the path. The strategy and the TreeNavigator SPI are @since 0.14.0 in TnsAI 0.14.0 (TnsAI@c2bb5306). They are in Maven Central 0.14.1.
Selective re-embed index
com.tnsai.intelligence.rag.vector.SelectiveReembedIndex is a local vector index that re-embeds only chunks whose content hash changed. It is @since 0.14.0 in TnsAI 0.14.0 (TnsAI@1291fe9e, TAN-3660 / PR #150). It ships in Maven Central 0.14.1.