CleanSnapv1 API
Intermediate·7 min read·Topic: html to clean text rag pipeline

HTML to Clean Text & Markdown for Production RAG Pipelines

Eliminate embedding noise, deduplicate boilerplate, and attach structured metadata before chunking webpages into your vector database.

Why this breaks in production

When teams ingest customer help centers or competitor docs into Pinecone, pgvector, or Qdrant, retrieval quality often disappoints even with top embedding models. Nine times out of ten, the culprit is dirty HTML extraction.

If every indexed webpage begins with the same 60-link navbar and ends with a 300-word GDPR footer, unrelated documents share artificially high cosine similarity scores. Cleaning HTML into canonical Markdown + structured metadata before chunking fixes recall immediately.

Recommended Engineering Approach

1

Strip repeated site chrome at extraction time

Enable `stripNav: true` and `stripCookieBanners: true` so every indexed document contains only unique primary content.

2

Split chunks on Markdown heading boundaries

Use the `##` and `###` boundaries in the extracted Markdown to keep semantically cohesive paragraphs together.

3

Store `canonical`, `title`, `publishedTime`, and `headings` as vector metadata

Enable filtered search (e.g., docs updated in 2026) and display verifiable source titles in your RAG UI.

Working Code Example

rag-ingest-url.tsts
interface RagChunk {
  id: string;
  url: string;
  title: string;
  heading: string;
  content: string;
}

export async function extractChunksForRag(url: string): Promise<RagChunk[]> {
  const res = await fetch("https://usecleansnap.com/api/v1/extract", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      url,
      stripCookieBanners: true,
      stripNav: true,
      includeMetadata: true,
      includeLinks: false,
    }),
  });
  const doc = await res.json();
  const sections = doc.markdown.split(/\n(?=## )/g);

  return sections.map((section: string, index: number) => {
    const firstLine = section.split("\n")[0].replace(/^#+\s*/, "");
    return {
      id: `${doc.metadata.canonical}#chunk-${index}`,
      url: doc.metadata.canonical,
      title: doc.metadata.title,
      heading: firstLine,
      content: section.trim(),
    };
  });
}
TRY IT LIVE — ZERO LOGIN

Test this workflow in the CleanSnap Playground

Pre-load the Sample Blog Post or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.

Run Sample Blog Post

Common Edge Cases & Pitfalls

  • Fixed 500-character sliding windows frequently slice right through the middle of code blocks and Markdown tables.
  • Indexing both `https://site.com/page` and `https://site.com/page?utm_source=x` duplicates vectors — always key by `metadata.canonical`.
  • Forgetting to keep the page title inside each chunk leaves orphan sections like 'Pricing' ambiguous during retrieval.

Frequently Asked Questions

Should I index clean Markdown or plain text in my vector database?

Clean Markdown outperforms plain text for both embedding models and downstream LLM synthesis because headings and lists preserve topical boundaries.

How do I prevent duplicate URLs from polluting my RAG index?

Deduplicate using `metadata.canonical` returned by CleanSnap rather than the raw input URL, which automatically collapses tracking query parameters.

What chunk size works best with Markdown extraction?

Heading-aware chunks of 350–700 tokens (roughly 1,400–2,800 characters) with the page title prepended deliver strong retrieval accuracy.

Related Developer Guides