# Smart document segmentation

`com.tnsai.intelligence.rag.segment.SmartDocumentSegmenter` splits a
long paper on headings and returns
`com.tnsai.intelligence.rag.segment.DocumentSegment` records. Both are
`@since 0.14.0` in TnsAI 0.14.0 (`TnsAI@609a661d`,
TAN-3651). They ship in Maven Central `0.14.1`.

This is an **analysis** API. FILE ingest still chunks through
`NormalizedDocumentChunker`. Do not treat these sections as retrieval
units.

## What it does

```java
SmartDocumentSegmenter segmenter = new SmartDocumentSegmenter();
List<DocumentSegment> sections = segmenter.segment(Path.of("paper.md"));
```

`segment(String)` accepts already-extracted Markdown or prose.
`segment(Path)` reads `.md` as UTF-8. PDF and DOCX go through the
installed `ContentExtractorRegistry`. A missing office backend fails
loud (`DocumentProcessingException`). Other extensions are also read
as UTF-8.

Each section:

1. starts at an ATX (`#`–`######`), numbered (`3.1 Title`), or
   `Section 3.1: Title` heading; text before the first heading is
   `Preamble`
2. keeps `Figure` / `Fig.` / `Table` caption lines with that section
3. collects `Section` / `Sec.` / `§` / `Figure` / `Table` mentions
   from the body (`crossRefs`); numbered section ids resolve to the
   later heading title when present
4. sets `cumulativeSummary` to a first-sentence digest of **prior**
   sections (about 160 characters each)

`DocumentSegment` fields: `index`, `sectionId` (empty on the
preamble), `sectionTitle`, `body`, `crossRefs`, `cumulativeSummary`,
`figures`.

## Related

- [File formats](formats.md) — extractors used for PDF/DOCX
- [Pipeline](pipeline.md) — FILE ingest chunker (different path)