Extracting Clean Internal & Outbound Link Graphs from Webpages
Separate high-signal contextual links inside `<article>` from noisy navbar/footer links and classify them as `internal` vs `outbound`.
Why this breaks in production
When an AI agent researches a topic or an SEO tool audits internal linking, extracting every `<a>` tag on the page returns 80% noise (Login, Privacy Policy, Twitter footer icons, Careers).
Extracting links *after* stripping `<nav>`, `<footer>`, `<aside>`, and cookie modals isolates genuine editorial citations and contextual internal links.
Recommended Engineering Approach
Strip boilerplate containers before scanning `<a>` elements
With `stripNav: true` and `includeLinks: true`, CleanSnap harvests only links inside the main content area.
Filter out non-navigable schemes and anchor jumps
Ignore `#section`, `javascript:void(0)`, `mailto:`, and `tel:` links, then resolve relative paths against `finalUrl`.
Classify internal vs outbound by root/subdomain host comparison
Compare each link's hostname against `baseHost` to label links cleanly as `internal` or `outbound`.
Working Code Example
const res = await fetch("https://usecleansnap.com/api/v1/extract", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
url: "https://example.com/blog/ai-agents",
stripNav: true,
includeLinks: true,
}),
});
const { links } = await res.json();
const internal = links.filter((l: { type: string }) => l.type === "internal");
const outbound = links.filter((l: { type: string }) => l.type === "outbound");
console.log(`Internal: ${internal.length}, Outbound citations: ${outbound.length}`);Test this workflow in the CleanSnap Playground
Pre-load the Sample Blog Post or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.
Common Edge Cases & Pitfalls
- Forgetting to deduplicate links wastes agent crawl budgets when an article links to the same resource 4 times.
- Links wrapping images with empty text need a graceful fallback (such as the resolved URL) so anchor labels aren't blank.
- Subdomains (`docs.example.com` vs `example.com`) should be handled thoughtfully when building internal crawl queues.
Frequently Asked Questions
How many links does CleanSnap return per page?
CleanSnap returns up to 40 deduplicated contextual links (`{ text, href, type }`) from the cleaned content region.
Can I turn off link extraction to shrink JSON payload size?
Yes — set `includeLinks: false` to omit the `links` array from the response.
How do multi-hop AI agents use the link graph?
An agent reads the main page Markdown, inspects `links` for relevant docs or citations, and queues 2–3 follow-up extractions automatically.