How to Hide & Strip Cookie Banners in Puppeteer (or Skip Headless Chrome Entirely)
Remove OneTrust, Cookiebot, TrustArc, and custom GDPR overlays cleanly before extracting text or feeding pages to an LLM.
Why this breaks in production
Every EU-compliant site ships a consent management platform (CMP) like OneTrust, Cookiebot, Didomi, or Quantcast. In headless Puppeteer runs, these modals inject 200–600 words of legal boilerplate directly into document.body.innerText and often trap scroll focus.
Clicking 'Accept All' is brittle because button labels change by locale ('Accepter tout', 'Alle akzeptieren') and shadow DOM containers resist basic querySelector clicks. Removing the consent nodes at the DOM/HTML layer — or offloading extraction to a specialized API — is dramatically faster and deterministic.
Recommended Engineering Approach
Match known CMP containers by ID, class, and ARIA role
Target common identifiers (`onetrust-banner-sdk`, `CybotCookiebotDialog`, `truste-consent-track`, `[role='dialog'][aria-label*='cookie' i]`) before reading page text.
Block CMP vendor scripts at the network layer if using Puppeteer
Abort requests to `cdn.cookielaw.org`, `consent.cookiebot.com`, and `consent.trustarc.com` via `page.setRequestInterception(true)` so banners never hydrate.
Use CleanSnap API for SSR pages without launching Chromium
For docs, blogs, and SaaS pages that already render HTML server-side, call `/api/v1/extract` with `stripCookieBanners: true` to excise balanced consent containers in ~300ms.
Working Code Example
// Option A: Strip cookie banners in Puppeteer before reading DOM
await page.evaluate(() => {
const selectors = [
"#onetrust-banner-sdk",
"#CybotCookiebotDialog",
"[id*='cookie' i][role='dialog']",
"[class*='cookie-banner' i]",
"[class*='gdpr-modal' i]",
];
document.querySelectorAll(selectors.join(",")).forEach((el) => el.remove());
});
// Option B: Zero-browser CleanSnap API call (10x faster on SSR/SSG pages)
const res = await fetch("https://usecleansnap.com/api/v1/extract", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
url: "https://example.com/docs",
stripCookieBanners: true,
stripNav: true,
}),
});
const { markdown, stats } = await res.json();
console.log("Stripped:", stats.strippedElements);Test this workflow in the CleanSnap Playground
Pre-load the Sample Docs Page or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.
Common Edge Cases & Pitfalls
- Naive regex removal of `<div class='cookie...'>` breaks when consent banners contain nested `<div>` elements — always use balanced tag matching or DOM removal.
- Some CMPs inject `overflow: hidden` onto `<body>`; if taking screenshots in Puppeteer, reset `document.body.style.overflow = 'auto'`.
- Do not match the word `cookie` across the entire body text or you will accidentally delete baking recipes and HTTP cookie tutorials — inspect element attributes only.
Frequently Asked Questions
Why does clicking 'Accept All' fail in Puppeteer across European sites?
Consent buttons are frequently localized into 20+ languages, embedded inside cross-origin iframes (Sourcepoint, Didomi), or delayed by 1–2 seconds after `domcontentloaded`.
Does CleanSnap remove cookie banners without headless Chrome?
Yes. For server-rendered consent banners embedded in HTML, CleanSnap removes balanced consent blocks automatically when `stripCookieBanners: true` is passed.
How many LLM tokens does a typical GDPR modal waste?
Between 180 and 550 tokens per URL — which compounds rapidly when an AI agent inspects 10–20 URLs per research task.