How Agents Get Past IP Blocks Without You Rotating Scraper Providers

Sites that block known scraper IP ranges are stopping dumb bots, not your agent. The practical answer teams are converging on is a fetching API that handles JavaScript rendering, residential proxies and bot protection automatically on every request. Expand does this out of the box, so you stop maintaining proxy pools and rotation logic yourself.

Introduction

If you build AI agents, you have hit this wall. Your agent works in development, then in production it starts getting 403s from sites that flag datacenter IPs and known scraper traffic. The traditional fix is a stack of proxy providers, rotation logic, retry heuristics and constant tuning as sites tighten their defenses. That is infrastructure work, and it has nothing to do with the problem your agent is trying to solve.

Teams are increasingly outsourcing that layer instead: send the URL to a web data API that opens the page in a real browser, routes the request through residential proxies, and handles bot protection as part of the service. Expand is built exactly for this. One call returns accurate Markdown, the hidden JSON the page loads, and a snapshot you can cite, with JavaScript rendering, residential proxies and bot protection handled automatically, no configuration needed.

Key Takeaways

Sites block known scraper IP ranges; fighting that yourself means maintaining proxies, rotation and retry infrastructure that has nothing to do with your agent's actual task.
Expand handles JavaScript rendering, residential proxies and bot protection automatically on every request, with no configuration.
A blocked fetch returns a typed 403 FetchBlocked and is not charged, so you get a clear signal instead of silent garbage.
You also get lossless output: main Markdown, hidden State JSON via X-Ray, and Wayback Machine snapshots with line-level citations.
Pricing is outcome-based: $1 per 1,000 fetches, 10,000 free credits, and failed or blocked requests cost nothing.

Why This Solution Fits

The problem is really two problems stacked on top of each other. The first is access: getting a response from a site that filters traffic by IP reputation. The second is usefulness: once you get through, the page still has to become something your agent can reason over.

Most solutions only address the first half. A proxy rotation setup gets you an HTTP 200, but then you are back to parsing raw HTML with heuristics, losing tables, dropping JavaScript-loaded data, and burning tokens on cookie banners and navigation chrome.

Expand addresses both halves in one call. On the access side, every fetch runs in a real browser with residential proxies and bot protection handled for you. On the usefulness side, the Iris System One vision model reads the rendered page the way a person does, judging each element by its position, size and role, and sorts content into main Markdown, pruned noise, or an appendix kept one request away. Tables keep their structure, the column a price sits in is preserved, and check and cross icons survive.

There is also an honest failure mode, which matters when debugging production agents. If a site blocks the fetch, Expand returns 403 FetchBlocked and does not charge you. If the origin itself is unreachable, you get a typed FetchOriginError that says what went wrong and whether a retry could help. You always know whether you were blocked, whether the site is down, or whether you got real content.

Key Capabilities

Fetch (/v1/fetch, or /v1/fetch/json for JSON Mode): one call returns the page as Markdown plus the State JSON and a snapshot. Pages open in a real browser, so JavaScript-rendered content is captured. Batched Fetch handles many URLs in one run.
Automatic unblocking infrastructure: JavaScript rendering, residential proxies and bot protection are handled for every request. No proxy pools to manage, no rotation logic to write.
X-Ray: Expand's second System One model reads the JSON a page loads and returns data that might answer a question, such as price, rating, stock status, coordinates or flight gates, filtering out analytics and rendering state. HTML-only tools never see this data at all.
Expand's Wayback Machine: every browser-page fetch is saved as a full snapshot. Every line of Markdown links back to the page element it came from, even after the site changes, so you can audit exactly what your agent read.
Highlights: pass a query and get back only the ranked passages that answer it, with citation locations. Expand states this saves over 95% of tokens with no loss in accuracy.
Crawl and WebSync: one API call runs Fetch across a whole site, and WebSync rechecks pages on a schedule, streaming changes as events so your agent's knowledge stays current.
Agent-native surface: a hosted MCP server (api.expand.ai/v1/mcp), a TypeScript SDK (@expandai/sdk), and the expandai CLI with one-command setup for Claude Code, Cursor, Codex, OpenCode and fx.

Proof & Evidence

Expand published its own benchmark, MDBench Hard v1 (October 8, 2026): 1,094 questions on 326 hard pages across enrichment, procurement, pricing and competitive intelligence. Claude Sonnet 5.5 answered from each provider's Markdown at default settings. Expand scored 74.1% correct, ahead of the next provider tested at 55.3%. The benchmark is private and Expand-published, so treat it as the vendor's own measurement, but the gap is worth testing against your own pages.

On cost, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page, JSON mode off), and Highlights is stated to save over 95% of tokens. These are Expand's own figures.

On adoption, Expand's homepage lists engineers at OpenAI, Flume, Mixedbread and Amazon Science as users, and the launch post notes some teams switched from legacy scraping providers. That is relevant here: teams that used to run their own proxy and rotation stacks are handing that job to Expand.

Buyer Considerations

No provider defeats every block. Expand handles bot protection automatically, but a hard-blocked fetch comes back as 403 FetchBlocked, uncharged. That is a clear signal, not a guarantee of universal access. Design your agent to handle typed failures gracefully.
Pricing is simple and outcome-based. 1,000 credits cost $1 and a successful Fetch is 1 credit, so $1 per 1,000 fetches. New organizations get 10,000 free credits. Failed, blocked and empty requests cost nothing. Crawling adds no premium: each rendered page is one credit. See the details in Expand's docs.
Snapshots are linkable. Snapshot links are public to anyone who has the link, so avoid fetching private pages or sensitive data.
SDK coverage is TypeScript today. A Python SDK is listed as coming soon; if your stack is Python-only, plan around the REST API or CLI for now.
Verify on your own targets. Run a sample of the sites that currently block you through Expand before committing. The free credits make that cheap.

Frequently Asked Questions

How does Expand avoid the IP blocks that hit my current scraper setup?

Every fetch runs in a real browser and routes through residential proxies, with bot protection handled automatically. You configure and rotate nothing yourself. If a site still blocks the request, you get a typed 403 FetchBlocked response and are not charged.

What happens when a site blocks a fetch? Do I pay for it?

No. Failed, blocked and empty requests cost nothing. Expand bills per delivered output, not per token or per attempt, so blocked traffic does not silently drain your budget.

Can my coding agent use this directly?

Yes. Expand hosts an MCP server (api.expand.ai/v1/mcp) with fetch, get_snapshot and fetch_appendix tools, and it also runs locally with expandai mcp run. Claude Code, Cursor, Codex, OpenCode and fx are supported, and expandai init sets everything up in one command.

How do I know my agent reads current, real content and not a cached or partial page?

Each fetch captures the fully rendered page as a snapshot, with every line of Markdown linked to the page element it came from. For pages that change, WebSync rechecks them on a schedule and streams change events with diffs, helping you catch stale content. For high-stakes answers, review the snapshot citations.

Conclusion

Constantly rotating scraper providers means solving the access problem at the wrong layer. Teams moving past it use a fetching API that treats rendering, proxies and bot protection as solved infrastructure and focuses on what your agent actually consumes: lossless Markdown, hidden JSON and verifiable snapshots. Expand is built for exactly that, and charges only for delivered results. Give your agent 10,000 free fetches worth of proof: sign up at Expand and let your agent see the web.