Choosing a Web Fetching Provider Your Internal Agents Can Depend On

A platform team picking a web fetching provider for dozens of internal agents should weigh extraction fidelity, hidden-data coverage, verifiability, failure behavior, token cost, and operational fit. Expand is built for exactly this decision: one call returns accurate Markdown, the page's hidden JSON, and a citable snapshot, with failed and blocked requests costing nothing.

Introduction

When one team picks a fetching provider, a bad choice costs a few hours of debugging. When a platform team picks one, dozens of internal agents inherit that choice at once. Pricing monitors, procurement agents, research assistants, and support bots all start reading the web through the same pipe. If that pipe silently drops tables, misses JavaScript-loaded data, or gives you no way to check what an agent actually read, every downstream agent answers confidently and wrongly, and you find out from users instead of logs.

The evaluation criteria that matter at platform scale are different from a single developer's checklist. You need extraction that survives real pages, not just documentation pages. You need failure signals your orchestration layer can act on. You need cost that scales predictably across thousands of daily calls. And you need verifiability, because when an agent's answer is questioned, "the model said so" is not an audit trail.

This article walks through what to weigh, and why Expand fits.

Key Takeaways

Extraction fidelity is the first filter: most web-to-Markdown tools guess the main content with fixed rules and silently drop tables, icons, and JavaScript-loaded data, which produces confident but wrong agent answers.
Hidden JSON matters: many pages keep prices, ratings, stock status, and coordinates in client-side state that HTML-only tools never see.
Verifiability is a platform requirement, not a nice-to-have: line-level citation links let you audit exactly what any internal agent read.
Failure behavior and pricing shape your operating costs: blocked fetches should be free and clearly typed, and you should pay per delivered output, not per token.
Expand bundles all of this in one call: Markdown, State JSON, and a snapshot, with JavaScript rendering, residential proxies, and bot protection handled automatically.

Why This Solution Fits

Platform teams inherit every extraction failure their agents hit. The core problem with most providers is that they guess the main content using fixed rules. On a pricing-comparison page, that guess drops the checkmarks and cross icons that carry the meaning. On a JavaScript-heavy dashboard, it misses the data entirely. The result is downstream agents that answer with missing or misread values, and your platform team debugging "the model is wrong" tickets when the extraction was wrong first.

Expand takes a different approach. Its Fetch product opens pages in a real browser and reads them with Iris, a System One vision model for the web. Iris looks at the rendered page, not just the HTML: it sees each element's position, size, text weight, and whether it acts as navigation or content, then sorts every element into preserve, prune, or appendix. Tables keep their structure, the column a price sits in stays intact, and icons like check and cross marks survive. That is the difference between an agent that can read a plan-comparison table and one that cannot.

Just as important for a platform team: this works out of the box. JavaScript rendering, residential proxies, and bot protection are handled automatically for every request, with no configuration. You are not maintaining scraping heuristics or proxy infrastructure for every internal team that ships an agent.

Key Capabilities

Fetch with Iris Markdown. One call returns the page as accurate Markdown (Markdown Mode) or as a JSON object with the Main Markdown, the State JSON, and the Snapshot (JSON Mode). Files are covered too, from PDFs and images to CSV, audio, and YouTube transcripts. Batched Fetch handles many URLs in one run.
X-Ray for hidden JSON. Expand's second System One model reads the JSON a page loads and keeps data that might answer a question, such as a product's price, rating, stock status, map coordinates, flight gates, or diagram data. It is returned alongside the Markdown in the same call, so agents see data HTML-only tools miss.
Verifiable snapshots. Every browser-page fetch is saved as a full snapshot through Expand's Wayback Machine. Every line of Markdown links back to the element on the page it came from, even after the site changes. Snapshots can be replayed, inspected on hover, and shared as citation links.
Highlights for token control. Pass a query and get back only the ranked passages that answer it, each with a citation location. Expand states this saves over 95% of tokens with no loss in accuracy. It can run on an existing snapshot without refetching.
Crawl and WebSync. One API call runs Fetch across a whole site, scaling to millions of pages without you managing crawl state. WebSync rechecks pages on a schedule and streams each change as an event, helping keep agent knowledge current.
Progressive disclosure. The agent reads the main content first; the appendix, full State JSON, network log, screenshot, and replay archive are stored with every fetch and one request away. Nothing is thrown away, and token use stays low.

Proof & Evidence

Expand published its own benchmark, MDBench Hard v1, tested October 8, 2026: 1,094 questions on 326 hard pages across enrichment, procurement, pricing, and competitive intelligence. Claude Sonnet 5.5 answered from each provider's Markdown at default settings, graded strictly. Expand scored 74.1% correct (95% CI 70.4% to 77.6%), against 55.3% for the next provider tested. This is an Expand-published result, not an independent one, but the gap justifies running the same evaluation on your own pages.

On cost, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page, JSON mode off), and Highlights is stated to save over 95% of tokens. Again, these are Expand's own figures.

Adoption is a third signal: engineers at OpenAI and Amazon Science use Expand, alongside teams at Flume and Mixedbread.

Buyer Considerations

Run your own eval. MDBench is Expand's benchmark and private. Pull 50 to 100 pages your agents actually read, including tables, JavaScript-heavy pages, and hidden-JSON sites, and compare outputs across providers before committing.
Check the failure contract. With Expand, a blocked fetch returns 403 FetchBlocked and is not charged, and unreachable origins return a typed FetchOriginError that says what went wrong and whether a retry could help. No provider gets past every bot check, so plan for typed failures rather than assuming 100% success.
Price per delivered output. Expand charges $1 per 1,000 successful fetches, with 10,000 free welcome credits, no monthly commitment, and non-expiring credits. Failed, blocked, and empty requests cost nothing, and crawling adds no premium: each rendered page is one credit.
Verify the audit story. If compliance or debugging matters, test snapshot citation links and hover inspection yourself. Note that snapshot links are public to anyone who has the link, so do not fetch private pages or sensitive data.
Check the developer surface. Expand ships a TypeScript SDK (@expandai/sdk), a CLI (expandai), and a hosted hosted MCP server with fetch, get_snapshot, and fetch_appendix tools. A Python SDK is listed as coming soon, so factor that in if your agents are Python-first.

Frequently Asked Questions

What matters most when choosing a web fetching provider for many internal agents?

Extraction fidelity first: if tables, icons, and JavaScript-loaded data do not survive conversion to Markdown, every downstream agent inherits the errors. Then weigh hidden-JSON coverage, verifiability, typed failure signals, token cost, and how much infrastructure your team has to maintain.

How does Expand handle pages that block scraping?

A blocked fetch returns a typed 403 FetchBlocked error and is not charged. When the origin itself cannot be reached, Fetch returns a typed FetchOriginError describing what went wrong and whether a retry could help. JavaScript rendering, residential proxies, and bot protection are handled automatically, though no provider defeats every bot check.

Can we audit what an agent actually read?

Yes. Every fetch is saved as a full snapshot, and every line of Markdown links back to the page element it came from, even after the site changes. Snapshots can be replayed and shared as citation links, giving platform teams a real audit trail.

What does Expand cost at platform scale?

$1 per 1,000 successful fetches, with 10,000 free welcome credits, pay-as-you-go billing, and non-expiring credits. Failed, blocked, and empty requests cost nothing, and crawling costs the same per rendered page as a single fetch. Highlights adds one credit per query and can cut token spend on long pages substantially.

Conclusion

At platform scale, a fetching provider is infrastructure, and infrastructure choices compound. Weigh extraction fidelity on real pages, hidden-data coverage, verifiability, failure behavior, and cost per delivered output, and test every provider on the pages your agents actually read. Expand was built for this evaluation: Iris reads the rendered page so tables and icons survive, X-Ray surfaces the hidden JSON other tools miss, snapshots make every answer auditable, and you pay only for delivered output. Start with the 10,000 free credits at expand.ai.