Markdown Chunker
Deterministic chunker that splits markdown text along heading boundaries while
respecting a maximum chunk size. Optionally prepends each chunk with the
heading hierarchy it sits under so the chunk is interpretable in isolation.
Sections larger than the chunk size are re-split recursively over
["\n\n", "\n", ". ", " "], with the heading prefix re-attached to each
resulting piece.
Use this for markdown documentation, articles, or anything where the heading
structure carries meaning. For arbitrary prose use AI::RecursiveChunker;
for fixed-size character windows use AI::FixedChunker; for LLM-driven
semantic splitting use AI::Chunker.
Parameters
Field on each incoming JSON record holding the markdown text. Leave empty to chunk the full record.
Field name written on each outgoing record holding the chunk text.
Defaults to chunk when left empty.
Maximum number of characters per chunk before an oversized section is
re-split recursively. Defaults to 1500.
Deepest heading level (1 = #, 2 = ##, …) treated as a chunk boundary.
Headings deeper than this are kept inside their parent chunk. With the
default 3, #, ##, and ### start new chunks while #### and deeper
are folded into the parent section. Values are clamped to the range 1–6.
When enabled (the default), each chunk is prefixed with the chain of parent headings so the chunk is self-contained. When disabled, each chunk still starts with its own immediate heading line — only the ancestor breadcrumb is dropped.
Input
JSONL file containing markdown text, or a raw .md/.markdown file. A
raw markdown file is treated as a single document and produces one JSONL
row per chunk (chunk text under the output attribute).
Output
JSONL file with one line per produced chunk.
Number of input records that were processed.
Number of chunk records that were produced.