Recursive Chunker
Deterministic, non-LLM chunker that splits each record's text by trying a list
of separators from coarsest (e.g. \n\n) to finest (e.g. " "), recursively
re-splitting any piece that exceeds the size budget (measured in words). After
splitting, adjacent small pieces are merged back together up to the chunk size
so chunks stay as full as possible. Falls back to a fixed word-window split
with overlap when no separator works.
This is the standard "recursive character text splitter" pattern — a good
default for general prose. For semantic splitting use AI::Chunker; for
markdown-aware splitting use AI::MarkdownChunker.
Parameters
Field on each incoming JSON record holding the text to chunk. Leave empty to chunk the full record.
Field name written on each outgoing record holding the chunk text.
Defaults to chunk when left empty.
Target maximum number of words per chunk (a "word" is any run of
non-whitespace characters). Defaults to 200 when left empty.
Overlap (words) used when the splitter falls back to fixed word-window
splitting because no separator could break a chunk down further. Defaults
to 0.
Ordered list of separator strings tried from first to last. The chunker
picks the first separator actually present in the text, splits on it, and
recurses with the remaining separators on any piece still over the size
budget. An explicit empty-string entry ("") forces a fixed word-window
split at that point. Leave the list empty to use the default ["\n\n", "\n", ". ", " "] (paragraph → line → sentence → word).
Input
JSONL file with one record per line, or a plain-text .txt file. A .txt
file is treated as a single document and produces one JSONL row per chunk
(chunk text under the output attribute).
Output
JSONL file with one line per produced chunk.
Number of input records that were processed.
Number of chunk records that were produced.