How Teams Fetch Pages Reliably When Sites Block Scrapers and Automation Traffic

When a target site actively blocks scrapers, teams stop relying on hand-rolled proxies and retry loops and move to a web data API that handles rendering, proxies and bot protection automatically. Expand's Fetch opens every page in a real browser, returns clean Markdown plus hidden JSON, and a blocked request returns a 403 that costs nothing.

Introduction

Anyone building an AI agent eventually hits the same wall: the page that works in your browser returns nothing through your scraper. The site renders its data with JavaScript, sits behind bot protection, or serves different content to datacenter IPs. Teams respond by stacking residential proxies, headless browsers and retry logic, then spend engineering time maintaining that stack instead of shipping their product.

The pattern that works in production is simpler: send a URL to a service built for this exact problem and get back the page as the model should see it. Expand is a web data API for AI agents. One call to Fetch opens the page in a real browser, handles JavaScript rendering, residential proxies and bot protection automatically, and returns accurate Markdown, the useful hidden JSON the page loads, and a snapshot you can cite. This article explains why that approach fits teams facing active blocking, what the capabilities look like, and what to check before you buy.

Key Takeaways

Bot protection, JavaScript rendering and residential proxies are handled automatically on every Expand request, with no configuration.
Failure is a signal, not a mystery: a blocked fetch returns 403 FetchBlocked and is not charged, and unreachable origins return a typed error that says whether a retry could help.
Extraction quality matters as much as access. Iris reads the rendered page visually, so tables and icons survive conversion to Markdown.
X-Ray returns the hidden JSON a page loads, which HTML-only tools miss.
Pricing is outcome-based: $1 per 1,000 successful fetches, 10,000 free credits, and failed requests cost nothing.

Why This Solution Fits

The core problem with blocked targets is not just getting an HTTP 200. It is getting the real page, rendered the way a person sees it, without maintaining infrastructure you do not want to own.

Expand fits that problem directly. Every request runs in a real browser, so pages that render data only after JavaScript executes come back complete. Residential proxies and bot protection are handled automatically for every request, with no configuration, removing the proxy-rotation and fingerprinting work most teams end up maintaining.

Just as important is what happens when a fetch fails. A blocked fetch returns 403 FetchBlocked and is not charged, so you can detect blocks cleanly instead of parsing ambiguous empty responses. When the origin cannot be reached at all, because of DNS, certificate or connection problems, Fetch returns a typed FetchOriginError that says what went wrong and whether a retry could help. That distinction matters in production: your agent can treat a block, an unreachable origin and an origin error page as three different situations instead of one vague failure.

Expand does not claim to get past every bot check, and you should be skeptical of any provider that does. What it offers is a clean contract: automatic handling of the common blocking layers, and honest, typed errors when something does not work, with no charge for failed or blocked requests.

Key Capabilities

Fetch. The core product. Send a URL, get back Markdown (Markdown Mode) or a JSON object (JSON Mode) with the Main Markdown, the State JSON and the Snapshot. Fetch also converts PDFs, images, office files, CSV, JSON, XML, audio files and YouTube video transcripts, and Batched Fetch handles many URLs in one run.

Iris. A System One vision model for the web. Instead of guessing the main content from HTML rules, Iris looks at the rendered page: each element's position, size, text weight, whether it is pinned or covered, and whether it is navigation or content. It sorts elements into preserve, prune (ads, cookie banners) or appendix (navigation and footers, kept one call away). The result is Markdown that keeps tables, the column a price sits in, and icons such as check and cross marks, which fixed-rule converters silently drop.

X-Ray. Expand's second System One model. It reads the JSON a page loads and keeps data that might answer a question, such as price, rating, stock status, coordinates or flight gates, filtering out analytics and rendering state. It returns alongside the Markdown in the same call.

Wayback Machine. Every browser-page fetch is saved as a full snapshot. Every line of Markdown links back to the page element it came from, even after the site changes, and snapshots can be replayed and shared as citation links.

Highlights. Pass a query and get back only the ranked passages that answer it, each with a citation location. Expand states this saves over 95% of tokens with no loss in accuracy, and it can run on an existing snapshot without refetching.

Crawl and WebSync. One API call runs Fetch across a whole site, scaling to millions of pages without you managing crawl state. WebSync rechecks pages on a schedule and streams each change (added, changed with its diff, removed with evidence) on a durable, resumable stream.

Agent integration. A hosted MCP server (api.expand.ai/v1/mcp) exposes fetch, get_snapshot and fetch_appendix tools to agents, with the expandai CLI and TypeScript SDK (@expandai/sdk) covering the rest.

Proof & Evidence

Expand publishes its own benchmark, so treat it as a vendor result. In MDBench Hard v1, tested October 8, 2026, Claude Sonnet 5.5 answered 1,094 questions across 326 hard pages from each provider's Markdown at default settings. Expand scored 74.1% correct, with a clear gap to the next provider tested. The benchmark is private, so use it as one input to your own evaluation on your own target sites.

On token efficiency, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page, JSON mode off), and Highlights is stated to save over 95% of tokens. Both are Expand's own figures.

On adoption, Expand's homepage lists engineers at OpenAI, Flume, Mixedbread and Amazon Science as users, and the launch post notes some teams switched from legacy scraping providers.

Buyer Considerations

Test on your hardest targets. No provider defeats every bot check. Run your blocked domains through Fetch and check the failure behavior, not just the successes.
Check extraction, not just access. A page that loads is worthless if the pricing table loses its columns. Compare Markdown output on tables, icons and JavaScript-loaded data.
Verify what you can cite. If your agent's answers need auditing, line-level snapshot citations are a hard requirement.
Understand the pricing model. Expand bills per delivered output: a successful Fetch is 1 credit ($1 per 1,000), failed, blocked and empty requests cost nothing, and 10,000 welcome credits come with a new organization. Prices can change, so confirm current terms.
Match the SDK surface to your stack. The TypeScript SDK and CLI are available today; a Python SDK is listed as coming soon, so plan accordingly.

Frequently Asked Questions

Do I still need my own proxy and headless browser infrastructure?

No. JavaScript rendering, residential proxies and bot protection are handled automatically on every Expand request, with no configuration. Teams typically retire that stack and keep only their application logic.

What happens when a site blocks a fetch?

A blocked fetch returns 403 FetchBlocked and is not charged. If the origin cannot be reached at all, Fetch returns a typed FetchOriginError describing the problem and whether a retry could help. If the origin serves an error page such as a 404, the page content is still captured and returned with the error attached.

Can my agent cite exactly where it got a fact?

Yes. Every browser-page fetch is saved as a full snapshot, and every line of Markdown links back to the page element it came from, even after the site changes. Snapshots can be shared as citation links. Note that snapshot links are public to anyone who has the link, so avoid fetching private pages or sensitive data.

How do I connect it to an existing coding agent?

Use the hosted MCP server, or run it locally with expandai mcp run. The expandai init command sets up supported agents including Claude Code, Cursor, Codex, OpenCode and fx in one step.

Conclusion

When target sites actively block automation, the reliable answer is not a bigger proxy pool. It is a fetching layer that renders pages in a real browser, handles blocking layers automatically, reports failure honestly and at no cost, and returns data your model can actually read: tables intact, hidden JSON included, every line citable. Start with the 10,000 free credits, point Fetch at the pages that break your current stack, and judge the output yourself at expand.ai.