Fixed Chunker
Deterministic, non-LLM chunker that splits each record's text into fixed-size
word windows with optional overlap. Cuts always fall on whitespace boundaries,
so formatting and punctuation inside a chunk are preserved byte-for-byte.
Cheap, fast, and reproducible — use this when content boundaries don't matter
and you just want consistently sized chunks. For semantic splitting use AI::Chunker; for markdown-aware splitting
use AI::MarkdownChunker; for separator-based splitting use AI::RecursiveChunker.
Parameters
Field on each incoming JSON record holding the text to chunk. Leave empty
to chunk the full record (serialized to JSON). The source attribute is
removed from each output record. Ignored for .txt inputs.
Field name written on each outgoing record holding the chunk text.
Defaults to chunk when left empty.
Number of words per chunk (a "word" is any run of non-whitespace
characters). Defaults to 200 when left empty.
Number of words shared between consecutive chunks. Higher overlap
preserves context across boundaries at the cost of duplicated content.
Values greater than or equal to the chunk size are clamped to chunk size −
1 so the chunker always makes forward progress. Defaults to 0.
Input
JSONL file with one record per line, or a plain-text .txt file. A .txt
file is treated as a single document and produces one JSONL row per chunk
(chunk text under the output attribute).
Output
JSONL file with one line per produced chunk. The original record's fields are preserved (minus the source attribute, which is removed) with the chunk text written to the output attribute.
Number of input records that were processed.
Number of chunk records that were produced.