CleanSnapv1 API
Intermediate·6 min read·Topic: markdown header chunking rag

Markdown Header Chunking for Higher-Accuracy RAG Retrieval

Replace blind fixed-length text splitters with heading-aware Markdown chunkers that preserve code blocks, tables, and section breadcrumbs.

Why this breaks in production

Fixed 1,000-character recursive splitters routinely sever a question from its answer or split a TypeScript code example in half. When retrieved in isolation, half a code snippet confuses the LLM.

Because CleanSnap preserves exact `#`, `##`, and `###` hierarchy plus `headings[]` outlines, you can build a deterministic Markdown chunker that attaches hierarchical breadcrumbs (`Doc Title > Section > Subsection`) to every chunk.

Recommended Engineering Approach

1

Parse fenced blocks before splitting on `#` lines

Ignore lines starting with `#` inside ```` ```python ```` or ```` ```bash ```` blocks (those are code comments, not Markdown headings!).

2

Maintain an active heading stack `[h1, h2, h3]`

Prepend the active breadcrumb path to each chunk so embeddings capture full parent context.

3

Merge tiny subsections and subdivide oversized sections

Keep chunks inside the 250–750 token sweet spot without breaking fenced code blocks.

Working Code Example

markdown-header-chunker.tsts
export function chunkMarkdownByHeadings(markdown: string, pageTitle: string) {
  const lines = markdown.split("\n");
  const chunks: { breadcrumb: string; text: string }[] = [];
  let inCodeFence = false;
  let currentHeading = pageTitle;
  let buffer: string[] = [];

  for (const line of lines) {
    if (line.trim().startsWith("```")) inCodeFence = !inCodeFence;

    if (!inCodeFence && /^#{1,3}\s+/.test(line)) {
      if (buffer.join("\n").trim()) {
        chunks.push({ breadcrumb: currentHeading, text: buffer.join("\n").trim() });
      }
      currentHeading = `${pageTitle} > ${line.replace(/^#+\s+/, "").trim()}`;
      buffer = [line];
    } else {
      buffer.push(line);
    }
  }

  if (buffer.join("\n").trim()) {
    chunks.push({ breadcrumb: currentHeading, text: buffer.join("\n").trim() });
  }
  return chunks;
}
TRY IT LIVE — ZERO LOGIN

Test this workflow in the CleanSnap Playground

Pre-load the Sample Docs Page or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.

Run Sample Docs Page

Common Edge Cases & Pitfalls

  • Splitting on `/^# /m` without tracking code fences will mistakenly split Python/Bash `# comments` into separate chunks!
  • Dropping the parent page title causes chunks named 'Overview' or 'Limits' to collide across dozens of docs.
  • Always trim excessive blank lines before computing chunk token lengths.

Frequently Asked Questions

Why does tracking code fences matter during Markdown splitting?

Bash, YAML, and Python code blocks use `#` for comments. A fence-aware parser avoids false heading splits inside code snippets.

Does CleanSnap also return a separate `headings` array?

Yes — every response includes `headings: { level: number, text: string }[]` so you can build tables of contents or inspect page structure immediately.

What if a webpage has only H1 and paragraphs?

Fall back to paragraph-boundary splitting (`\n\n`) whenever a single heading section exceeds ~3,000 characters.

Related Developer Guides