Handling Dynamic Consent Modals in Playwright Without Flaky Waits
Eliminate flaky timeout errors caused by late-loading CMP banners in Playwright automation and hybrid AI scrapers.
Why this breaks in production
Unlike server-rendered banners, modern consent managers often load asynchronously 1,500ms after page hydration. A standard Playwright script that checks for a modal immediately after `page.goto` misses it, only for the modal to pop up mid-scrape and block clicks.
Playwright's `page.addLocatorHandler` combined with network route blocking solves dynamic overlays cleanly — while a fast-path HTTP check via CleanSnap avoids booting Playwright entirely on 80% of URLs.
Recommended Engineering Approach
Route-block top CMP script domains before navigation
Use `page.route()` to abort known consent CDN scripts so the modal never renders in the first place.
Register a `page.addLocatorHandler` safety net
If a first-party consent dialog still appears dynamically, Playwright automatically dismisses it before retrying your action.
Fast-path SSR URLs through CleanSnap API first
Try CleanSnap's ~300ms HTTP extractor first and only fall back to Playwright when the extracted character count indicates a client-only SPA shell.
Working Code Example
// Hybrid Fast-Path: Try CleanSnap first, fall back to Playwright only for client-only SPAs
export async function smartExtract(url: string): Promise<string> {
const fastRes = await fetch("https://usecleansnap.com/api/v1/extract", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ url, stripCookieBanners: true, stripNav: true }),
});
if (fastRes.ok) {
const data = await fastRes.json();
// If SSR HTML had real content, skip Playwright completely!
if (data.stats.wordCount >= 80) {
return data.markdown;
}
}
// Fallback path for pure client-rendered SPAs:
// Launch Playwright with route blocking on cookielaw.org / cookiebot.com
throw new Error("Fallback to headless Playwright for client-only SPA: " + url);
}Test this workflow in the CleanSnap Playground
Pre-load the Sample Docs Page or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.
Common Edge Cases & Pitfalls
- Hard-coded `page.waitForTimeout(3000)` adds 3 seconds of dead latency to every URL — prefer route blocking + locator handlers.
- Blocking first-party scripts too aggressively can prevent the main SPA bundle from executing.
- Always verify word count on fast-path HTTP extractions so you detect 'Please enable JavaScript' stubs.
Frequently Asked Questions
What percentage of public webpages need Playwright vs direct HTTP extraction?
Because SEO requires indexable HTML, roughly 75%–85% of public docs, blogs, pricing pages, and news articles work immediately with fast HTTP extraction like CleanSnap.
Why is hybrid extraction popular for AI agents?
Running CleanSnap first keeps median latency under 500ms and slashes headless browser infrastructure bills by up to 80%.
Which CMP providers does CleanSnap strip automatically?
CleanSnap matches OneTrust, Cookiebot, TrustArc, Didomi, Quantcast, and generic `cookie`/`consent`/`gdpr`/`ccpa` containers.