Skip to content
tnsaiDocs
GitHub

Smart document segmentation

com.tnsai.intelligence.rag.segment.SmartDocumentSegmenter splits a long paper on headings and returns com.tnsai.intelligence.rag.segment.DocumentSegment records.

This is an analysis API. FILE ingest still chunks through NormalizedDocumentChunker. Do not treat these sections as retrieval units.

Segment a document

SmartDocumentSegmenter segmenter = new SmartDocumentSegmenter();
List<DocumentSegment> sections = segmenter.segment(Path.of("paper.md"));

segment(String) accepts already-extracted Markdown or prose. segment(Path) reads .md as UTF-8. PDF and DOCX go through the installed ContentExtractorRegistry. A missing office backend fails loud (DocumentProcessingException). Other extensions are also read as UTF-8.

Each section:

  1. starts at an ATX (# to ######), numbered (3.1 Title), or Section 3.1: Title heading (text before the first heading is Preamble)
  2. keeps Figure / Fig. / Table caption lines with that section
  3. collects Section / Sec. / § / Figure / Table mentions from the body (crossRefs), and numbered section ids resolve to the later heading title when present
  4. sets cumulativeSummary to a first-sentence digest of prior sections (about 160 characters each)

DocumentSegment fields: index, sectionId (empty on the preamble), sectionTitle, body, crossRefs, cumulativeSummary, figures.