CleanSnapv1 API
All 25 Developer GuidesCookie & Clutter Removal
Advanced·7 min read·Topic: handle dynamic consent modals playwright

Handling Dynamic Consent Modals in Playwright Without Flaky Waits

Eliminate flaky timeout errors caused by late-loading CMP banners in Playwright automation and hybrid AI scrapers.

Why this breaks in production

Unlike server-rendered banners, modern consent managers often load asynchronously 1,500ms after page hydration. A standard Playwright script that checks for a modal immediately after `page.goto` misses it, only for the modal to pop up mid-scrape and block clicks.

Playwright's `page.addLocatorHandler` combined with network route blocking solves dynamic overlays cleanly — while a fast-path HTTP check via CleanSnap avoids booting Playwright entirely on 80% of URLs.

Recommended Engineering Approach

1

Route-block top CMP script domains before navigation

Use `page.route()` to abort known consent CDN scripts so the modal never renders in the first place.

2

Register a `page.addLocatorHandler` safety net

If a first-party consent dialog still appears dynamically, Playwright automatically dismisses it before retrying your action.

3

Fast-path SSR URLs through CleanSnap API first

Try CleanSnap's ~300ms HTTP extractor first and only fall back to Playwright when the extracted character count indicates a client-only SPA shell.

Working Code Example

hybrid-cleansnap-playwright.tsts
// Hybrid Fast-Path: Try CleanSnap first, fall back to Playwright only for client-only SPAs
export async function smartExtract(url: string): Promise<string> {
  const fastRes = await fetch("https://usecleansnap.com/api/v1/extract", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ url, stripCookieBanners: true, stripNav: true }),
  });

  if (fastRes.ok) {
    const data = await fastRes.json();
    // If SSR HTML had real content, skip Playwright completely!
    if (data.stats.wordCount >= 80) {
      return data.markdown;
    }
  }

  // Fallback path for pure client-rendered SPAs:
  // Launch Playwright with route blocking on cookielaw.org / cookiebot.com
  throw new Error("Fallback to headless Playwright for client-only SPA: " + url);
}
TRY IT LIVE — ZERO LOGIN

Test this workflow in the CleanSnap Playground

Pre-load the Sample Docs Page or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.

Run Sample Docs Page

Common Edge Cases & Pitfalls

  • Hard-coded `page.waitForTimeout(3000)` adds 3 seconds of dead latency to every URL — prefer route blocking + locator handlers.
  • Blocking first-party scripts too aggressively can prevent the main SPA bundle from executing.
  • Always verify word count on fast-path HTTP extractions so you detect 'Please enable JavaScript' stubs.

Frequently Asked Questions

What percentage of public webpages need Playwright vs direct HTTP extraction?

Because SEO requires indexable HTML, roughly 75%–85% of public docs, blogs, pricing pages, and news articles work immediately with fast HTTP extraction like CleanSnap.

Why is hybrid extraction popular for AI agents?

Running CleanSnap first keeps median latency under 500ms and slashes headless browser infrastructure bills by up to 80%.

Which CMP providers does CleanSnap strip automatically?

CleanSnap matches OneTrust, Cookiebot, TrustArc, Didomi, Quantcast, and generic `cookie`/`consent`/`gdpr`/`ccpa` containers.

Related Developer Guides