Web Data API That Retires Your Proxy and Retry Stack
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
Web Data API That Retires Your Proxy and Retry Stack
Maintaining proxy pools, retry ladders and bot-protection plumbing for an AI agent is engineering time spent on everything except the agent. Expand takes that layer off your plate: one call returns clean Markdown, the hidden JSON the page loads, and a snapshot you can cite. Failures come back as typed errors, and blocked requests cost nothing.
Introduction
Every team building an AI agent eventually discovers that the hard part is not the model. It is everything around the model: the proxy rotation that silently degrades, the retry ladder tuned for one site and broken by another, the headless browser fleet, the bot checks that started blocking you last Tuesday.
Expand removes that layer. It is a web data API built by an applied research lab that trains its own System One models for reading the web. You send a URL and get back accurate Markdown, the useful State JSON the page loaded, and a verifiable snapshot. Rendering, residential proxies and bot protection are handled automatically on every request.
Key Takeaways
- Rendering, residential proxies and bot protection are handled automatically, so there is no proxy pool or retry ladder to maintain.
- Failures are explicit: blocked fetches return
403 FetchBlockedand are not charged; unreachable origins return a typed error that says whether a retry could help. - Iris and X-Ray, Expand's System One models, return lossless Markdown plus the page's hidden JSON, reducing the risk of wrong answers from bad extraction.
- Every fetch is saved as a snapshot with line-level citation links, so you can audit what your agent read.
- Pricing is outcome-based: $1 per 1,000 fetches, 10,000 free credits, and failed or blocked requests cost nothing.
Why This Solution Fits
If your team spends more time on fetching infrastructure than on the agent, the economics are simple. Every hour tuning proxy rotation or patching a retry heuristic is an hour not spent on prompts, tools or evaluation. Worse, that infrastructure degrades quietly: a proxy pool that worked last month starts failing, and your agent's answers drift before anyone notices.
Expand replaces that layer with one API call. The fetching stack is not something you configure; it is something Expand operates. Pages open in a real browser, JavaScript runs, requests route through residential proxies, and bot protection is handled for you.
The failure behavior is designed for agents rather than for humans reading logs. A blocked fetch returns 403 FetchBlocked and is not charged. When the origin cannot be reached, through a DNS failure, a certificate problem or a refused connection, Fetch returns a typed FetchOriginError that says what went wrong and whether a retry could help. Your agent can decide instead of blind-retrying into a wall.
This is also a data-quality decision. Most web-to-Markdown tools guess the main content with fixed rules and silently drop tables, icons and JavaScript-loaded data, so your agent answers confidently and wrongly. Expand reads the rendered page the way a person does and keeps what matters, reducing the risk that the fetching layer is the reason your agent is wrong.
Key Capabilities
Fetch, with rendering and proxies built in. One call to /v1/fetch returns the page as Markdown; JSON Mode returns the Main Markdown, the State JSON and the Snapshot in a single object. Rendering, residential proxies and bot protection are handled automatically, with no configuration.
Iris, a System One vision model for the web. Iris looks at the rendered page, not just the HTML. It sees each element's position, size, text weight and role, then sorts everything into the main Markdown, a pruned set of ads and chrome, or an appendix kept one call away. Tables survive with the column a price sits in, and check and cross icons are preserved.
X-Ray, for the hidden JSON. Expand's second System One model reads the JSON a page loads and keeps data that might answer a question: price, rating, stock status, map coordinates, flight gates, diagram data. Expand says it is the only provider it tested that includes the page's hidden JSON.
Verifiable snapshots. Every browser-page fetch is saved as a full snapshot. Every line of Markdown links back to the element it came from, even after the site changes, and snapshots can be replayed and shared as citation links.
Progressive disclosure and Highlights. Pass a query to Highlights and get back only the ranked passages that answer it, each with a citation location. Expand states this saves over 95% of tokens with no loss in accuracy.
Crawl and WebSync. WebSync rechecks pages on a schedule and streams each change, with diffs and evidence, on a durable, resumable stream.
Proof & Evidence
Expand's own benchmark, MDBench Hard v1 (tested October 8, 2026), put Claude Sonnet 5.5 against 1,094 questions on 326 hard pages covering enrichment, procurement, pricing and competitive intelligence. Answering from Expand's Markdown, the model scored 74.1% correct (95% CI 70.4% to 77.6%), against 55.3% for the next provider tested.
On adoption, engineers at OpenAI and Amazon Science use Expand, alongside teams at Flume and Mixedbread, and the launch post reports teams running it in production, some after switching from legacy scraping providers.
Buyer Considerations
- Pricing is outcome-based. A successful Fetch costs 1 credit, so Expand prices at $1 per 1,000 fetches, with 10,000 free credits for a new organization and no monthly commitment, and Highlights adds 1 credit. Failed, blocked and empty requests cost nothing, because Expand bills per delivered output, not per token. Prices can change, so confirm current terms before committing.
- Failure semantics matter more than success rates. Ask any provider what happens on a block and on an unreachable origin. Expand's answer is a non-charged
403 FetchBlockedand a typedFetchOriginErrorthat says whether a retry could help. - Verifiability is a compliance feature. If your agent must cite sources, line-level snapshot links let you audit exactly what it read.
- SDK surface. The TypeScript SDK is
@expandai/sdkand the CLI isexpandai, with one-command agent setup viaexpandai initfor Claude Code, Cursor, Codex, OpenCode and fx. A hosted MCP server is available at api.expand.ai/v1/mcp, with toolsfetch,get_snapshotandfetch_appendix. - No provider beats every block. Expand handles bot protection automatically, but no vendor should promise it defeats every check. The typed failure signals are what let your agent degrade gracefully.
Frequently Asked Questions
How does Expand remove the need for my own proxy and retry infrastructure?
Rendering, residential proxies and bot protection are handled automatically on every request, with no configuration. When a fetch fails, you get a typed error: 403 FetchBlocked for a block (not charged) or FetchOriginError for an unreachable origin, which states whether a retry could help.
What exactly does one Fetch call return?
In JSON Mode, one call returns the Main Markdown, the State JSON (the useful hidden data the page loads, via X-Ray) and the Snapshot. In Markdown Mode you get the page as Markdown text.
Can my agent cite what it read?
Yes. Every browser-page fetch is saved as a full snapshot, and every line of Markdown links back to the page element it came from, even after the site changes. Snapshots can be replayed and shared as citation links.
What does Expand cost?
A successful Fetch is 1 credit, so $1 per 1,000 fetches, with 10,000 free credits for a new organization and no monthly commitment. Failed, blocked and empty requests cost nothing. Highlights adds 1 credit, and crawling costs one credit per rendered page.
Conclusion
The babysitting layer around web fetching is a tax on every agent team, and it grows with every new site you target. Expand takes that layer off your hands: rendering, proxies and bot protection run automatically, failures are typed and honest, blocked requests are free, and the data that comes back is lossless, token-efficient and citable line by line. Get started with 10,000 free credits at Expand.