Build a Production-Grade Web Extractor Tool for AI Agents
Wire a safe, token-efficient `extract_webpage` tool into your OpenAI, Anthropic, or LangGraph agent in under 25 lines of TypeScript.
Why this breaks in production
Giving an LLM a `read_url` tool looks trivial until it hits production: agents visit slow servers that hang forever, follow redirects to internal metadata IPs, ingest 50,000 tokens of raw HTML, or get tricked by cookie modals.
A production agent extractor needs four guarantees: strict SSRF blocking, hard 10-second timeouts, boilerplate stripping to Markdown, and deterministic token caps.
Recommended Engineering Approach
Define a strict JSON schema for your agent tool
Expose `url` and an optional `focus` or `maxChars` parameter so the model can request concise or deep reads.
Delegate fetching and sanitization to CleanSnap `/api/v1/extract`
Offload SSRF validation, DNS checks, redirect verification, and HTML-to-Markdown parsing to one hardened endpoint.
Return structured errors back to the agent gracefully
Instead of crashing the agent loop on HTTP 403/404/504, return `{ error: data.error.message }` so the LLM can try an alternative source.
Working Code Example
export async function agentReadUrlTool(args: { url: string; maxChars?: number }) {
const res = await fetch("https://usecleansnap.com/api/v1/extract", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
url: args.url,
stripCookieBanners: true,
stripNav: true,
includeMetadata: true,
includeLinks: true,
maxChars: args.maxChars ?? 12000,
}),
});
const payload = await res.json();
if (!res.ok || !payload.ok) {
return {
ok: false,
error: payload?.error?.message ?? "Unable to read webpage.",
};
}
return {
ok: true,
title: payload.metadata?.title,
url: payload.finalUrl,
estimatedTokens: payload.stats.estimatedTokens,
markdown: payload.markdown,
topLinks: (payload.links ?? []).slice(0, 8),
};
}Test this workflow in the CleanSnap Playground
Pre-load the Sample Docs Page or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.
Common Edge Cases & Pitfalls
- Never expose raw `fetch(userUrl)` from a cloud container without blocking `169.254.169.254` and RFC1918 private IPs.
- Always pass `maxChars` so a single huge Wikipedia or RFC page cannot crash your agent's context window.
- Log `X-Extract-Time-Ms` and `X-RateLimit-Remaining` headers for agent observability.
Frequently Asked Questions
What response headers does CleanSnap return for observability?
Every extraction response includes `X-Extract-Time-Ms`, `X-RateLimit-Limit`, `X-RateLimit-Remaining`, and `X-RateLimit-Reset`.
Can my agent follow links found on the extracted page?
Yes — `links` returns deduplicated internal and outbound links with anchor text so agents can decide which follow-up URLs to inspect.
Does CleanSnap support both GET and POST calls?
Yes, both `POST /api/v1/extract` (JSON body) and `GET /api/v1/extract?url=...` are supported.