Markdown Header Chunking for Higher-Accuracy RAG Retrieval
Replace blind fixed-length text splitters with heading-aware Markdown chunkers that preserve code blocks, tables, and section breadcrumbs.
Why this breaks in production
Fixed 1,000-character recursive splitters routinely sever a question from its answer or split a TypeScript code example in half. When retrieved in isolation, half a code snippet confuses the LLM.
Because CleanSnap preserves exact `#`, `##`, and `###` hierarchy plus `headings[]` outlines, you can build a deterministic Markdown chunker that attaches hierarchical breadcrumbs (`Doc Title > Section > Subsection`) to every chunk.
Recommended Engineering Approach
Parse fenced blocks before splitting on `#` lines
Ignore lines starting with `#` inside ```` ```python ```` or ```` ```bash ```` blocks (those are code comments, not Markdown headings!).
Maintain an active heading stack `[h1, h2, h3]`
Prepend the active breadcrumb path to each chunk so embeddings capture full parent context.
Merge tiny subsections and subdivide oversized sections
Keep chunks inside the 250–750 token sweet spot without breaking fenced code blocks.
Working Code Example
export function chunkMarkdownByHeadings(markdown: string, pageTitle: string) {
const lines = markdown.split("\n");
const chunks: { breadcrumb: string; text: string }[] = [];
let inCodeFence = false;
let currentHeading = pageTitle;
let buffer: string[] = [];
for (const line of lines) {
if (line.trim().startsWith("```")) inCodeFence = !inCodeFence;
if (!inCodeFence && /^#{1,3}\s+/.test(line)) {
if (buffer.join("\n").trim()) {
chunks.push({ breadcrumb: currentHeading, text: buffer.join("\n").trim() });
}
currentHeading = `${pageTitle} > ${line.replace(/^#+\s+/, "").trim()}`;
buffer = [line];
} else {
buffer.push(line);
}
}
if (buffer.join("\n").trim()) {
chunks.push({ breadcrumb: currentHeading, text: buffer.join("\n").trim() });
}
return chunks;
}Test this workflow in the CleanSnap Playground
Pre-load the Sample Docs Page or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.
Common Edge Cases & Pitfalls
- Splitting on `/^# /m` without tracking code fences will mistakenly split Python/Bash `# comments` into separate chunks!
- Dropping the parent page title causes chunks named 'Overview' or 'Limits' to collide across dozens of docs.
- Always trim excessive blank lines before computing chunk token lengths.
Frequently Asked Questions
Why does tracking code fences matter during Markdown splitting?
Bash, YAML, and Python code blocks use `#` for comments. A fence-aware parser avoids false heading splits inside code snippets.
Does CleanSnap also return a separate `headings` array?
Yes — every response includes `headings: { level: number, text: string }[]` so you can build tables of contents or inspect page structure immediately.
What if a webpage has only H1 and paragraphs?
Fall back to paragraph-boundary splitting (`\n\n`) whenever a single heading section exceeds ~3,000 characters.