Choosing a Web Data API for Your Agent: What to Evaluate Beyond Price Per Request

Price per request is the easiest number to compare and the worst way to choose a web data API. What decides whether your agent works is extraction fidelity on hard pages, access to hidden JSON, verifiability of what the agent read, and how failures are reported. Expand is built around those criteria for teams building agents from scratch.

Introduction

When you build an agent from scratch, the web data API you pick in week one quietly becomes part of your product. Every answer your agent gives is downstream of what that API returns. If the extraction drops a table column, misses a checkmark, or silently omits the JSON the page actually loaded, your agent will answer confidently and wrongly, and you will spend your debugging time blaming the model.

Price per request is a poor first filter. A cheap API that returns lossy Markdown costs more than it saves, because you pay in wrong answers, inflated token bills, and hours spent rebuilding scraping heuristics. The real criteria are what the API returns on hard pages, whether it exposes the data behind the rendered page, whether you can audit what your agent read, and what happens when a request fails. This article walks through each criterion in turn.

Key Takeaways

Evaluate extraction fidelity on hard pages first: pricing tables, comparison grids, icons and JavaScript-rendered content are where most Markdown APIs silently lose data your agent needs.
Hidden JSON matters: much of the useful data on a modern page (prices, ratings, stock status, coordinates) loads as JSON that HTML-only tools never see.
Verifiability is a feature, not a nice-to-have: line-level citations and replayable snapshots let you audit what your agent read.
Failure behavior is a production concern: you need typed errors that say whether a retry could help, and blocked requests should not cost you money.
Expand bundles all of this at $1 per 1,000 fetches, with 10,000 free credits and no charge for failed or blocked requests.

Why This Solution Fits

Most web-to-Markdown tools guess the main content with fixed rules. They work well on simple articles and fail quietly on the pages agents actually need: pricing pages where a check versus a cross icon carries the meaning, comparison tables where the column a price sits in is the answer, and single-page apps where the data only exists after JavaScript runs. The result is an agent reading a plausible document that is missing the facts.

Expand takes a different approach. Its Fetch product opens the page in a real browser and reads it with System One models built for the web. Iris, Expand's vision model, looks at the rendered page the way a person does, judging each element's position, size and role, so tables, grids and icons survive into the Markdown instead of being flattened away. X-Ray, Expand's second System One model, reads the hidden JSON the page loads and returns data that might answer a question, alongside the Markdown in the same call.

For a team building from scratch, this means you do not inherit a scraping problem on day one. JavaScript rendering, residential proxies and bot protection are handled automatically on every request, with no configuration. You start with a data layer designed for agents, not a converter designed for humans.

Key Capabilities

Lossless Fetch with progressive disclosure. One call returns the page as clean Markdown, with the appendix, full State JSON, network log, screenshot and replay archive stored and one request away. Nothing is thrown away, and your agent only pays tokens for what it reads first.
Iris, a System One vision model. It reads the rendered page visually and preserves tables, the column a value sits in, and icons such as check and cross marks, while pruning ads, cookie banners and share buttons.
X-Ray for hidden JSON. Returns the useful data a page loads client-side, such as price, rating, stock status, coordinates and diagram data, which HTML-only tools miss.
Expand's Wayback Machine. Every browser-page fetch is saved as a full snapshot. Every line of Markdown links back to the element it came from, even after the site changes, and snapshots can be replayed and shared as citation links.
Highlights. Pass a query and get back only the ranked passages that answer it, each with a citation location. Expand states this saves over 95% of tokens with no loss in accuracy.
Crawl and WebSync. One API call runs Fetch across a whole site, and WebSync rechecks pages on a schedule and streams each change as an event, keeping agent knowledge current.
Agent-ready surface. A TypeScript SDK (@expandai/sdk), the expandai CLI, and a hosted MCP server with fetch, get_snapshot and fetch_appendix tools, so coding agents like Claude Code and Cursor can be wired up in minutes.

Proof & Evidence

Extraction quality claims should be tested, not asserted. Expand publishes its own benchmark, MDBench Hard v1, tested October 8, 2026: 1,094 questions across 326 hard pages curated with customers in enrichment, procurement, pricing and competitive intelligence. Claude Sonnet 5.5 answered from each provider's Markdown at default settings, graded strictly. Expand scored 74.1% correct, ahead of the next provider tested at 55.3%. Treat this as the vendor's own result, but the gap justifies your own comparison on your pages.

On cost, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page, JSON mode off), and Highlights is stated to save over 95% of tokens. These are Expand's own figures, but they point at the right question: tokens, not requests, are usually your real cost driver.

Adoption is a third signal. Engineers at OpenAI and Amazon Science use Expand, alongside teams like Flume and Mixedbread, and some switched from legacy scraping providers. Pricing is outcome-based: $1 per 1,000 successful fetches, 10,000 free credits, and failed, blocked or empty requests cost nothing, so you pay per delivered output, not per attempt.

Buyer Considerations

Run your own eval on your pages. Vendor benchmarks, including Expand's, are a starting point. Pull twenty hard pages from your domain and compare the Markdown each API returns.
Check the failure contract. A blocked fetch from Expand returns a typed 403 FetchBlocked and is not charged, and unreachable origins return a typed error saying whether a retry could help. Ask every provider what happens on failure and whether you still pay.
Plan for citations if answers are high-stakes. Good extraction reduces unsupported answers, but the model can still misread a value against the wrong column. Use snapshot citations and human review where accuracy matters.
Mind the snapshot privacy model. Expand's Wayback Machine snapshot links are public to anyone with the link, so do not fetch private pages or sensitive data.
Check SDK coverage. Expand ships a TypeScript SDK and CLI today; a Python SDK is coming soon, so plan accordingly if your stack is Python-first.

Frequently Asked Questions

What should we evaluate in a web data API besides price per request?

Extraction fidelity on hard pages (tables, icons, JavaScript-rendered content), access to hidden JSON, verifiability through snapshots and citations, token efficiency, typed failure signals, and whether failed requests are charged. Price only matters once the output is usable by your agent.

Why do tables and icons get lost in Markdown conversion?

Most tools use fixed rules to guess the main content from HTML, flattening visual structure in the process. A checkmark versus a cross, or the column a price sits in, is layout information. Iris reads the rendered page visually instead, so that structure survives into the Markdown.

How does a web data API help reduce wrong agent answers?

It gives the model complete, faithful input: the real table, the hidden JSON, and a citation link to the exact line on the page. That reduces the risk of unsupported or misread answers, though the model can still make mistakes, so keep citations and review for high-stakes cases.

How much does Expand cost?

A successful Fetch costs 1 credit, and 1,000 credits cost $1, so $1 per 1,000 fetches. New organizations get 10,000 free credits, credits do not expire, and failed, blocked or empty requests cost nothing. Crawling adds no surcharge: each rendered page is one credit.

Conclusion

If you are building an agent from scratch, evaluate web data APIs the way your agent will use them: on hard pages, with hidden JSON, under failure, and with an audit trail. Price per request is a rounding error next to the cost of an agent that reads incomplete data and answers wrongly. Expand was built for exactly this evaluation, from System One models that keep tables and icons intact, to X-Ray's hidden JSON, to line-level snapshot citations and a no-charge failure policy. Start with the 10,000 free credits, run it against your hardest pages, and see for yourself at expand.ai.