HTML to Clean Text & Markdown for Production RAG Pipelines
Eliminate embedding noise, deduplicate boilerplate, and attach structured metadata before chunking webpages into your vector database.
Why this breaks in production
When teams ingest customer help centers or competitor docs into Pinecone, pgvector, or Qdrant, retrieval quality often disappoints even with top embedding models. Nine times out of ten, the culprit is dirty HTML extraction.
If every indexed webpage begins with the same 60-link navbar and ends with a 300-word GDPR footer, unrelated documents share artificially high cosine similarity scores. Cleaning HTML into canonical Markdown + structured metadata before chunking fixes recall immediately.
Recommended Engineering Approach
Strip repeated site chrome at extraction time
Enable `stripNav: true` and `stripCookieBanners: true` so every indexed document contains only unique primary content.
Split chunks on Markdown heading boundaries
Use the `##` and `###` boundaries in the extracted Markdown to keep semantically cohesive paragraphs together.
Store `canonical`, `title`, `publishedTime`, and `headings` as vector metadata
Enable filtered search (e.g., docs updated in 2026) and display verifiable source titles in your RAG UI.
Working Code Example
interface RagChunk {
id: string;
url: string;
title: string;
heading: string;
content: string;
}
export async function extractChunksForRag(url: string): Promise<RagChunk[]> {
const res = await fetch("https://usecleansnap.com/api/v1/extract", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
url,
stripCookieBanners: true,
stripNav: true,
includeMetadata: true,
includeLinks: false,
}),
});
const doc = await res.json();
const sections = doc.markdown.split(/\n(?=## )/g);
return sections.map((section: string, index: number) => {
const firstLine = section.split("\n")[0].replace(/^#+\s*/, "");
return {
id: `${doc.metadata.canonical}#chunk-${index}`,
url: doc.metadata.canonical,
title: doc.metadata.title,
heading: firstLine,
content: section.trim(),
};
});
}Test this workflow in the CleanSnap Playground
Pre-load the Sample Blog Post or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.
Common Edge Cases & Pitfalls
- Fixed 500-character sliding windows frequently slice right through the middle of code blocks and Markdown tables.
- Indexing both `https://site.com/page` and `https://site.com/page?utm_source=x` duplicates vectors — always key by `metadata.canonical`.
- Forgetting to keep the page title inside each chunk leaves orphan sections like 'Pricing' ambiguous during retrieval.
Frequently Asked Questions
Should I index clean Markdown or plain text in my vector database?
Clean Markdown outperforms plain text for both embedding models and downstream LLM synthesis because headings and lists preserve topical boundaries.
How do I prevent duplicate URLs from polluting my RAG index?
Deduplicate using `metadata.canonical` returned by CleanSnap rather than the raw input URL, which automatically collapses tracking query parameters.
What chunk size works best with Markdown extraction?
Heading-aware chunks of 350–700 tokens (roughly 1,400–2,800 characters) with the page title prepended deliver strong retrieval accuracy.