CleanSnapv1 API
Beginner·4 min read·Topic: estimate llm tokens webpage

How to Estimate LLM Token Count for Any Webpage Before Calling OpenAI or Claude

Predict prompt costs accurately using fast character-to-token heuristics and live raw-vs-clean compression metrics.

Why this breaks in production

AI agents that browse multiple URLs in parallel can accidentally trigger 128k context explosions or surprise API invoices when a single webpage ships 250 KB of inline SVG and JSON state.

Checking estimated token counts before appending extracted text to your messages array lets your orchestrator decide whether to include the full page, truncate gracefully, or summarize per section.

Recommended Engineering Approach

1

Use `Math.ceil(chars / 4)` for fast English/code BPE estimation

For OpenAI `cl100k_base` / `o200k_base` and Claude tokenizers, 1 token averages ~3.8–4.2 characters in English Markdown.

2

Track raw vs clean token savings

Compare `stats.rawEstimatedTokens` against `stats.estimatedTokens` to verify that boilerplate stripping is working as expected.

3

Enforce per-URL token budgets via `maxChars`

Multiply your per-URL token budget by 4 (e.g., 2,500 tokens → `maxChars: 10000`) and pass it directly to the extraction request.

Working Code Example

budget-agent-tokens.tsts
const MAX_TOKENS_PER_URL = 3000;

const res = await fetch("https://usecleansnap.com/api/v1/extract", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    url: "https://example.com/pricing",
    stripCookieBanners: true,
    stripNav: true,
    maxChars: MAX_TOKENS_PER_URL * 4,
  }),
});

const { stats, markdown } = await res.json();
console.log({
  rawTokens: stats.rawEstimatedTokens,
  cleanTokens: stats.estimatedTokens,
  savedPercent: stats.tokenSavingsPercent,
  truncated: stats.truncated,
});
TRY IT LIVE — ZERO LOGIN

Test this workflow in the CleanSnap Playground

Pre-load the Sample Pricing Page or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.

Run Sample Pricing Page

Common Edge Cases & Pitfalls

  • CJK text (Chinese, Japanese, Korean) tokenizes closer to 1–2 characters per token — apply a 2x safety buffer for non-Latin pages.
  • Dense minified JSON or base64 data URIs tokenize poorly (~1.5 chars/token); always strip `<script>` and inline SVGs first.
  • Remember to reserve output completion tokens and system prompt tokens when computing remaining context headroom.

Frequently Asked Questions

How accurate is `Math.ceil(chars / 4)` compared to tiktoken?

On clean English Markdown and technical documentation, `chars / 4` is typically within ±8% of exact BPE tokenization with zero WASM binary overhead.

Does CleanSnap return both raw and cleaned token counts?

Yes — every response includes `stats.rawEstimatedTokens`, `stats.estimatedTokens`, and `stats.tokenSavingsPercent`.

How can I shrink webpage tokens even further?

Enable both `stripCookieBanners: true` and `stripNav: true`, and pass `maxChars` tuned to your agent's budget.

Related Developer Guides