# Smart document segmentation

`com.tnsai.intelligence.rag.segment.SmartDocumentSegmenter` splits a
long paper on headings and returns
`com.tnsai.intelligence.rag.segment.DocumentSegment` records.

This is an **analysis** API. FILE ingest still chunks through
`NormalizedDocumentChunker`. Do not treat these sections as retrieval
units.

## Segment a document

```java
SmartDocumentSegmenter segmenter = new SmartDocumentSegmenter();
List<DocumentSegment> sections = segmenter.segment(Path.of("paper.md"));
```

`segment(String)` accepts already-extracted Markdown or prose.
`segment(Path)` reads `.md` as UTF-8. PDF and DOCX go through the
installed `ContentExtractorRegistry`. A missing office backend fails
loud (`DocumentProcessingException`). Other extensions are also read
as UTF-8.

Each section:

1. starts at an ATX (`#` to `######`), numbered (`3.1 Title`), or
   `Section 3.1: Title` heading (text before the first heading is
   `Preamble`)
2. keeps `Figure` / `Fig.` / `Table` caption lines with that section
3. collects `Section` / `Sec.` / `§` / `Figure` / `Table` mentions
   from the body (`crossRefs`), and numbered section ids resolve to the
   later heading title when present
4. sets `cumulativeSummary` to a first-sentence digest of **prior**
   sections (about 160 characters each)

`DocumentSegment` fields: `index`, `sectionId` (empty on the
preamble), `sectionTitle`, `body`, `crossRefs`, `cumulativeSummary`,
`figures`.

## Related

- [File formats](file-formats.md): extractors used for PDF/DOCX
- [Pipeline](pipeline.md): FILE ingest chunker (different path)