WebExtract for the Vercel AI SDK

npm @webextract/ai-sdk: two tools for AI SDK agents, built on the @webextract/sdk client. They are thin wrappers over the REST API; see the API docs for the full response schema.

Install

npm i @webextract/ai-sdk ai zod

ESM with TypeScript types. Node ≥ 20.3. Works with ai v5–v7 and zod 3.25+ or 4.

Tools

Tool	Endpoint	Returns	x402	Prepaid
`extractUrl`	`GET /v1/extract`	Markdown + `metadata`, `product` (with `price_detail`), `tables`	$0.005	1 credit
`pageMarkdown`	`GET /v1/markdown`	Markdown, title, word count	$0.002	0.4 credit

Both take { url } (an absolute http(s) URL, validated with zod before any request) and return the API JSON as-is. webextractTools(options) returns both tools sharing one client.

Example

import { generateText, stepCountIs } from "ai";
import { extractUrl, pageMarkdown } from "@webextract/ai-sdk";

const { text } = await generateText({
  model: "openai/gpt-5-mini",
  prompt: "What does https://web-scraping.dev/product/1 cost?",
  tools: { extractUrl: extractUrl(), pageMarkdown: pageMarkdown() },   // free tier, or WEBEXTRACT_API_KEY if set
  stopWhen: stepCountIs(3),
});

Billing

Mode	Options	Notes
Free tier (default)	none, or `billing: "free"`	Sends `X-Billing: free`. 50 calls/day per IP, shared with MCP.
Prepaid key	`apiKey: "lx_..."` or env `WEBEXTRACT_API_KEY`	`Authorization: Bearer`. Buy credits at webextract.dev/#pricing.
x402 pay per call	`paymentFetch`	USDC on Base and 6 other chains. No account. The package never sees your private key: signing happens inside the fetch you pass.
x402 sandbox	`paymentFetch` + `billing: "x402-testnet"`	Base Sepolia test USDC only. Nothing real is charged.

import { extractUrl, pageMarkdown, webextractTools } from "@webextract/ai-sdk";

const free = webextractTools();                              // X-Billing: free
const paid = webextractTools({ apiKey: "lx_..." });           // prepaid credits

WEBEXTRACT_API_KEY: when you don't pass apiKey, the key is read from this environment variable. An explicit apiKey always wins. The variable is ignored when you pass paymentFetch or set billing to anything other than "key".

x402 with `@x402/fetch`

import { wrapFetchWithPaymentFromConfig } from "@x402/fetch";
import { ExactEvmScheme } from "@x402/evm";
import { privateKeyToAccount } from "viem/accounts";
import { webextractTools } from "@webextract/ai-sdk";

const account = privateKeyToAccount(process.env.X402_PRIVATE_KEY as `0x${string}`);
const paymentFetch = wrapFetchWithPaymentFromConfig(fetch, {
  schemes: [{ network: "eip155:8453", client: new ExactEvmScheme(account) }], // only the networks you allow
});
const tools = webextractTools({ paymentFetch });

// Sandbox: register "eip155:84532" (Base Sepolia) instead and add billing: "x402-testnet"

billing: "free" together with paymentFetch is free-first: it uses the free tier, and the wrapper pays once the daily limit is reached. A 402 settlement_failed (verified, not settled, not charged) is retried once with a freshly signed payment (settlementRetries, default 1, 0 turns it off).

Sandbox rate limit: 20 requests per minute per IP (HTTP 429 with Retry-After). Every x402 payment uses 2 requests (the unpaid 402, then the signed retry), so that is about 10 paid calls per minute.

Errors

By default the tools return API errors to the model instead of throwing, so the agent can react to them:

{ "ok": false,
  "error": { "error": "payment_required", "status": 402, "message": "Free daily limit reached ...",
             "charged": false, "retryable": false, "reason": "free_limit_reached", "hint": "...",
             "networks": ["eip155:8453", "..."], "buy_url": "https://webextract.dev/#pricing", "free_remaining": 0 } }

Pass onError: "throw" to throw the error classes from the @webextract/sdk package instead (all extend WebExtractError with status, code, message, charged, retryable):

Class	When
`PaymentRequiredError`	HTTP 402. `reason`: `x402_required`, `x402_requested`, `free_limit_reached`, `invalid_payment`, `unsupported_network`, `settlement_failed`, `no_credits`. Also `networks`, `hint`, buy URL, free calls left.
`TargetFetchError`	The page could not be fetched: `upstream_error`, `blocked_by_site` (422), `robots_disallowed` (451), `dns_failed`, `fetch_failed`, `too_large`, `unsupported_type`. Never charged.
`InvalidRequestError`	400 (missing or invalid `url`, private host, unknown billing mode), or the client-side URL check before any request (`status` 0). Never charged.
`AuthenticationError`	401 invalid API key.
`RateLimitError`	429 (sandbox limits), with retry-after.
`ConnectionError`	No response: network error, timeout, or the payment wrapper threw (for example no matching network).

Response fields

product.price_detail is the recommended price field (prices[] with type list/sale/current, amount, currency, matched variant). It never contains 0; when no price can be trusted, status is unresolved with unresolved.reason. content_source is raw_html (no JavaScript run). When product is null, product_unresolved.reason says why (for example listing_page). See prices and provenance.

Using the client directly

@webextract/sdk is installed with the tools; add it to your own dependencies (npm i @webextract/sdk) if you import it directly.

import { WebExtract, PaymentRequiredError } from "@webextract/sdk";

const we = new WebExtract();                       // same options: apiKey, paymentFetch, billing, baseUrl, timeoutMs
const { data, meta } = await we.extract("https://web-scraping.dev/product/1");
console.log(data.product?.price_detail, meta.freeRemaining);

Also on this site: LangChain (Python and JS) · MCP.