CleanSnapv1 API
All 25 Developer GuidesMetadata & Parsing
Intermediate·5 min read·Topic: extract internal outbound link graph scraper

Extracting Clean Internal & Outbound Link Graphs from Webpages

Separate high-signal contextual links inside `<article>` from noisy navbar/footer links and classify them as `internal` vs `outbound`.

Why this breaks in production

When an AI agent researches a topic or an SEO tool audits internal linking, extracting every `<a>` tag on the page returns 80% noise (Login, Privacy Policy, Twitter footer icons, Careers).

Extracting links *after* stripping `<nav>`, `<footer>`, `<aside>`, and cookie modals isolates genuine editorial citations and contextual internal links.

Recommended Engineering Approach

1

Strip boilerplate containers before scanning `<a>` elements

With `stripNav: true` and `includeLinks: true`, CleanSnap harvests only links inside the main content area.

2

Filter out non-navigable schemes and anchor jumps

Ignore `#section`, `javascript:void(0)`, `mailto:`, and `tel:` links, then resolve relative paths against `finalUrl`.

3

Classify internal vs outbound by root/subdomain host comparison

Compare each link's hostname against `baseHost` to label links cleanly as `internal` or `outbound`.

Working Code Example

inspect-link-graph.tsts
const res = await fetch("https://usecleansnap.com/api/v1/extract", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    url: "https://example.com/blog/ai-agents",
    stripNav: true,
    includeLinks: true,
  }),
});

const { links } = await res.json();
const internal = links.filter((l: { type: string }) => l.type === "internal");
const outbound = links.filter((l: { type: string }) => l.type === "outbound");

console.log(`Internal: ${internal.length}, Outbound citations: ${outbound.length}`);
TRY IT LIVE — ZERO LOGIN

Test this workflow in the CleanSnap Playground

Pre-load the Sample Blog Post or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.

Run Sample Blog Post

Common Edge Cases & Pitfalls

  • Forgetting to deduplicate links wastes agent crawl budgets when an article links to the same resource 4 times.
  • Links wrapping images with empty text need a graceful fallback (such as the resolved URL) so anchor labels aren't blank.
  • Subdomains (`docs.example.com` vs `example.com`) should be handled thoughtfully when building internal crawl queues.

Frequently Asked Questions

How many links does CleanSnap return per page?

CleanSnap returns up to 40 deduplicated contextual links (`{ text, href, type }`) from the cleaned content region.

Can I turn off link extraction to shrink JSON payload size?

Yes — set `includeLinks: false` to omit the `links` array from the response.

How do multi-hop AI agents use the link graph?

An agent reads the main page Markdown, inspects `links` for relevant docs or citations, and queues 2–3 follow-up extractions automatically.

Related Developer Guides