CleanSnapv1 API
Intermediate·6 min read·Topic: build ai agent web extractor

Build a Production-Grade Web Extractor Tool for AI Agents

Wire a safe, token-efficient `extract_webpage` tool into your OpenAI, Anthropic, or LangGraph agent in under 25 lines of TypeScript.

Why this breaks in production

Giving an LLM a `read_url` tool looks trivial until it hits production: agents visit slow servers that hang forever, follow redirects to internal metadata IPs, ingest 50,000 tokens of raw HTML, or get tricked by cookie modals.

A production agent extractor needs four guarantees: strict SSRF blocking, hard 10-second timeouts, boilerplate stripping to Markdown, and deterministic token caps.

Recommended Engineering Approach

1

Define a strict JSON schema for your agent tool

Expose `url` and an optional `focus` or `maxChars` parameter so the model can request concise or deep reads.

2

Delegate fetching and sanitization to CleanSnap `/api/v1/extract`

Offload SSRF validation, DNS checks, redirect verification, and HTML-to-Markdown parsing to one hardened endpoint.

3

Return structured errors back to the agent gracefully

Instead of crashing the agent loop on HTTP 403/404/504, return `{ error: data.error.message }` so the LLM can try an alternative source.

Working Code Example

agent-read-url-tool.tsts
export async function agentReadUrlTool(args: { url: string; maxChars?: number }) {
  const res = await fetch("https://usecleansnap.com/api/v1/extract", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      url: args.url,
      stripCookieBanners: true,
      stripNav: true,
      includeMetadata: true,
      includeLinks: true,
      maxChars: args.maxChars ?? 12000,
    }),
  });

  const payload = await res.json();
  if (!res.ok || !payload.ok) {
    return {
      ok: false,
      error: payload?.error?.message ?? "Unable to read webpage.",
    };
  }

  return {
    ok: true,
    title: payload.metadata?.title,
    url: payload.finalUrl,
    estimatedTokens: payload.stats.estimatedTokens,
    markdown: payload.markdown,
    topLinks: (payload.links ?? []).slice(0, 8),
  };
}
TRY IT LIVE — ZERO LOGIN

Test this workflow in the CleanSnap Playground

Pre-load the Sample Docs Page or paste your own URL to inspect clean Markdown, token savings, and auto-generated cURL / Node / Python / MCP code.

Run Sample Docs Page

Common Edge Cases & Pitfalls

  • Never expose raw `fetch(userUrl)` from a cloud container without blocking `169.254.169.254` and RFC1918 private IPs.
  • Always pass `maxChars` so a single huge Wikipedia or RFC page cannot crash your agent's context window.
  • Log `X-Extract-Time-Ms` and `X-RateLimit-Remaining` headers for agent observability.

Frequently Asked Questions

What response headers does CleanSnap return for observability?

Every extraction response includes `X-Extract-Time-Ms`, `X-RateLimit-Limit`, `X-RateLimit-Remaining`, and `X-RateLimit-Reset`.

Can my agent follow links found on the extracted page?

Yes — `links` returns deduplicated internal and outbound links with anchor text so agents can decide which follow-up URLs to inspect.

Does CleanSnap support both GET and POST calls?

Yes, both `POST /api/v1/extract` (JSON body) and `GET /api/v1/extract?url=...` are supported.

Related Developer Guides