CleanSnapv1 API
All 25 Developer GuidesMetadata & Parsing
Beginner·4 min read·Topic: parse canonical url rel link node

Parsing Canonical URLs (`rel="canonical"`) Correctly in Node.js

Stop indexing duplicate URLs (`?utm_source=...`, trailing slashes, AMP pages) by extracting and resolving canonical links accurately.

Why this breaks in production

Users and search APIs constantly feed UTM-tagged, ref-tagged, or shortened URLs into web scrapers and RAG pipelines. Indexing by raw input URL causes the exact same article to appear 5 times in your database.

Resolving `<link rel="canonical" href="...">` against the final post-redirect URL gives you a single stable primary key for caching, deduplication, and attribution.

Recommended Engineering Approach

1

Follow redirects first to establish the true base URL

A shortened link (`bit.ly/...`) resolves to the target host; relative canonical paths must resolve against that `finalUrl`.

2

Match `<link>` tags regardless of attribute order

Support both `<link rel="canonical" href="...">` and `<link href="..." rel="canonical">`.

3

Fallback to `finalUrl` when canonical is missing

Always return a valid normalized string so downstream caches never receive `undefined`.

Working Code Example

canonical-cache-key.tsts
const res = await fetch("https://usecleansnap.com/api/v1/extract", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    url: "https://docs.example.com/guide?utm_source=hn&ref=sidebar",
    includeMetadata: true,
  }),
});

const { metadata, finalUrl } = await res.json();
const dedupKey = metadata.canonical || finalUrl;
console.log("Canonical deduplication key:", dedupKey);
TRY IT LIVE — ZERO LOGIN

Test this workflow in the CleanSnap Playground

Pre-load the Sample Blog Post or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.

Run Sample Blog Post

Common Edge Cases & Pitfalls

  • Some misconfigured CMS plugins emit relative canonicals like `/blog/post` — always run `new URL(canonical, finalUrl)`.
  • Single-quoted or unquoted `rel=canonical` attributes are valid HTML5 and break strict double-quote regexes.
  • If a syndicated post sets its canonical to an external origin, store both `finalUrl` and `metadata.canonical` for auditability.

Frequently Asked Questions

Does CleanSnap always populate `metadata.canonical`?

Yes — CleanSnap resolves the page's `<link rel='canonical'>` if present, and falls back to `finalUrl` otherwise.

Why is canonical deduplication critical for RAG?

Duplicate chunks crowd out diverse top-K results during vector retrieval, lowering answer quality.

Does CleanSnap return the post-redirect URL too?

Yes — `sourceUrl` preserves your input and `finalUrl` records the final resolved destination after HTTP redirects.

Related Developer Guides