Parsing Canonical URLs (`rel="canonical"`) Correctly in Node.js
Stop indexing duplicate URLs (`?utm_source=...`, trailing slashes, AMP pages) by extracting and resolving canonical links accurately.
Why this breaks in production
Users and search APIs constantly feed UTM-tagged, ref-tagged, or shortened URLs into web scrapers and RAG pipelines. Indexing by raw input URL causes the exact same article to appear 5 times in your database.
Resolving `<link rel="canonical" href="...">` against the final post-redirect URL gives you a single stable primary key for caching, deduplication, and attribution.
Recommended Engineering Approach
Follow redirects first to establish the true base URL
A shortened link (`bit.ly/...`) resolves to the target host; relative canonical paths must resolve against that `finalUrl`.
Match `<link>` tags regardless of attribute order
Support both `<link rel="canonical" href="...">` and `<link href="..." rel="canonical">`.
Fallback to `finalUrl` when canonical is missing
Always return a valid normalized string so downstream caches never receive `undefined`.
Working Code Example
const res = await fetch("https://usecleansnap.com/api/v1/extract", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
url: "https://docs.example.com/guide?utm_source=hn&ref=sidebar",
includeMetadata: true,
}),
});
const { metadata, finalUrl } = await res.json();
const dedupKey = metadata.canonical || finalUrl;
console.log("Canonical deduplication key:", dedupKey);Test this workflow in the CleanSnap Playground
Pre-load the Sample Blog Post or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.
Common Edge Cases & Pitfalls
- Some misconfigured CMS plugins emit relative canonicals like `/blog/post` — always run `new URL(canonical, finalUrl)`.
- Single-quoted or unquoted `rel=canonical` attributes are valid HTML5 and break strict double-quote regexes.
- If a syndicated post sets its canonical to an external origin, store both `finalUrl` and `metadata.canonical` for auditability.
Frequently Asked Questions
Does CleanSnap always populate `metadata.canonical`?
Yes — CleanSnap resolves the page's `<link rel='canonical'>` if present, and falls back to `finalUrl` otherwise.
Why is canonical deduplication critical for RAG?
Duplicate chunks crowd out diverse top-K results during vector retrieval, lowering answer quality.
Does CleanSnap return the post-redirect URL too?
Yes — `sourceUrl` preserves your input and `finalUrl` records the final resolved destination after HTTP redirects.