How to Estimate LLM Token Count for Any Webpage Before Calling OpenAI or Claude
Predict prompt costs accurately using fast character-to-token heuristics and live raw-vs-clean compression metrics.
Why this breaks in production
AI agents that browse multiple URLs in parallel can accidentally trigger 128k context explosions or surprise API invoices when a single webpage ships 250 KB of inline SVG and JSON state.
Checking estimated token counts before appending extracted text to your messages array lets your orchestrator decide whether to include the full page, truncate gracefully, or summarize per section.
Recommended Engineering Approach
Use `Math.ceil(chars / 4)` for fast English/code BPE estimation
For OpenAI `cl100k_base` / `o200k_base` and Claude tokenizers, 1 token averages ~3.8–4.2 characters in English Markdown.
Track raw vs clean token savings
Compare `stats.rawEstimatedTokens` against `stats.estimatedTokens` to verify that boilerplate stripping is working as expected.
Enforce per-URL token budgets via `maxChars`
Multiply your per-URL token budget by 4 (e.g., 2,500 tokens → `maxChars: 10000`) and pass it directly to the extraction request.
Working Code Example
const MAX_TOKENS_PER_URL = 3000;
const res = await fetch("https://usecleansnap.com/api/v1/extract", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
url: "https://example.com/pricing",
stripCookieBanners: true,
stripNav: true,
maxChars: MAX_TOKENS_PER_URL * 4,
}),
});
const { stats, markdown } = await res.json();
console.log({
rawTokens: stats.rawEstimatedTokens,
cleanTokens: stats.estimatedTokens,
savedPercent: stats.tokenSavingsPercent,
truncated: stats.truncated,
});Test this workflow in the CleanSnap Playground
Pre-load the Sample Pricing Page or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.
Common Edge Cases & Pitfalls
- CJK text (Chinese, Japanese, Korean) tokenizes closer to 1–2 characters per token — apply a 2x safety buffer for non-Latin pages.
- Dense minified JSON or base64 data URIs tokenize poorly (~1.5 chars/token); always strip `<script>` and inline SVGs first.
- Remember to reserve output completion tokens and system prompt tokens when computing remaining context headroom.
Frequently Asked Questions
How accurate is `Math.ceil(chars / 4)` compared to tiktoken?
On clean English Markdown and technical documentation, `chars / 4` is typically within ±8% of exact BPE tokenization with zero WASM binary overhead.
Does CleanSnap return both raw and cleaned token counts?
Yes — every response includes `stats.rawEstimatedTokens`, `stats.estimatedTokens`, and `stats.tokenSavingsPercent`.
How can I shrink webpage tokens even further?
Enable both `stripCookieBanners: true` and `stripNav: true`, and pass `maxChars` tuned to your agent's budget.