How to Strip Navbars, Sidebars & Footers When Web Scraping
Combine semantic HTML5 tag excision, chrome class patterns, and `<article>`/`<main>` landmark prioritization to isolate clean body copy.
Why this breaks in production
SaaS marketing sites and documentation portals often contain 80+ navigation links across sticky top bars, multi-column megamenus, documentation tree sidebars, and 6-column footers.
If your scraper extracts the whole `<body>`, navigation text frequently exceeds the actual article length — causing LLMs to hallucinate features listed in unrelated navbar links.
Recommended Engineering Approach
Excise semantic chrome tags first
Remove `<nav>`, `<footer>`, and `<aside>` blocks using balanced tag matching before converting HTML to Markdown.
Filter common ad, breadcrumb, and popup class names
Target classes and IDs matching `sidebar`, `breadcrumb`, `newsletter-popup`, `ad-slot`, `sponsored`, and `social-share`.
Prioritize `<article>` then `<main>` with a minimum text threshold
Only lock onto `<article>` or `<main>` if its plain-text length exceeds ~160 characters — protecting against sites that misuse `<article>` for tiny cards.
Working Code Example
const res = await fetch("https://usecleansnap.com/api/v1/extract", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
url: "https://docs.example.com/api-reference",
stripNav: true,
stripCookieBanners: true,
}),
});
const data = await res.json();
console.log("Stripped blocks:", data.stats.strippedElements);
console.log("Token savings:", data.stats.tokenSavingsPercent + "%");Test this workflow in the CleanSnap Playground
Pre-load the Sample Docs Page or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.
Common Edge Cases & Pitfalls
- Blindly deleting `<header>` globally can delete an `<article><header><h1>Title</h1></header></article>` — only strip site-level headers outside the article.
- Some SPA docs put the real content inside `<div id='content'>` without `<main>`, so always keep a `<body>` fallback after stripping chrome.
- Verify whether you actually need footer links (e.g., finding a Terms of Service URL) — CleanSnap lets you flip `stripNav: false` per request.
Frequently Asked Questions
What happens if a page has no `<article>` or `<main>` tag?
CleanSnap still strips `<nav>`, `<footer>`, `<aside>`, scripts, styles, and common sidebar/ad containers, then extracts cleanly from `<body>`.
Can I keep navigation links when mapping a site?
Yes. Pass `stripNav: false` and `includeLinks: true` to harvest the full internal link graph.
Does stripping navigation affect heading extraction?
It improves it! Removing sidebar `<h4>Navigation</h4>` widgets keeps your `headings` outline strictly focused on the page's real sections.