Teams Hitting the Scraping Wall Are Switching to a Web Data API Built for Agents

If your scraping setup works but every new site breaks something, the fix is not another selector or proxy pool. Teams at this stage replace brittle per-site extraction with a web data API that reads rendered pages, returns lossless Markdown plus hidden JSON, and cites every line back to the page. That is exactly what Expand was built for.

Introduction

Most scraping pipelines start the same way. A script pulls a page, some rules find the main content, and it works. Then a site redesigns, another loads its data only after JavaScript runs, a third hides prices in JSON the HTML never shows, and a fourth blocks your datacenter IPs. Each new site adds a new heuristic, a new retry path, and a new place where the pipeline silently returns wrong data instead of failing loudly.

At that point the problem is not your code. It is the approach. Hand-maintained extraction guesses at page structure with fixed rules, and guessing does not scale across a web where every site is built differently. Teams that hit this wall usually stop patching the scraper and start paying for a service whose entire job is reading pages correctly, at scale, with failure signals you can trust.

Key Takeaways

Brittle scraping fails in a specific way: fixed rules guess the main content and silently drop tables, icons and JavaScript-loaded data, so agents answer confidently and wrongly.
Expand's Iris model reads the rendered page visually and preserves tables, grid positions and check/cross icons in the Markdown, instead of guessing from HTML.
X-Ray returns the hidden JSON a page loads (prices, ratings, stock status, coordinates, flight gates), which HTML-only tools miss entirely.
Every fetch produces a snapshot with line-level citation links, so you can audit exactly what your agent read, an approach Expand details in its piece on auditing agent input.
JavaScript rendering, residential proxies and bot protection are handled automatically, and blocked requests return a 403 and cost nothing.

Why This Solution Fits

The wall you are hitting has three parts, and Expand addresses each one directly.

First, extraction quality. Most web-to-Markdown tools guess the main content with fixed rules and quietly discard what does not fit the pattern. Pricing tables lose the column a price sits in. Check and cross icons, which often carry the actual meaning on a plan-comparison page, disappear. Data loaded by JavaScript never appears at all. Iris, Expand's System One vision model, looks at the rendered page the way a person does: it sees each element's position, size, text weight and role, then sorts everything into main Markdown, pruned chrome, or an appendix kept one call away. Tables, columns and icons survive.

Second, hidden data. X-Ray, Expand's second System One model, reads the JSON a page loads and keeps the parts that could answer a question, filtering out analytics and rendering state. It comes back in the same call as the Markdown. If your agents work with product prices, ratings, stock status, map coordinates or flight gates, this is data your current scraper probably never sees.

Third, trust and debugging. When an agent gives a wrong answer today, you cannot easily tell whether the model or the extraction failed. Expand's Wayback Machine (its own snapshot feature, not the Internet Archive) saves every fetched page as a full snapshot, and every line of Markdown links back to the page element it came from, even after the site changes. Snapshots can be replayed, inspected on hover and shared as citation links, turning "why did the agent say that" from archaeology into a click.

Key Capabilities

Fetch: one call returns the page as Markdown or, in JSON mode, a single object with the main Markdown, the State JSON and the snapshot. Pages open in a real browser. Fetch also converts PDFs (including scanned ones), images, office files, CSV, XML, audio and YouTube transcripts. Batched Fetch handles many URLs in one run.
Iris: vision-based extraction that keeps tables, grid layout and icons intact, and prunes ads, cookie banners and share buttons.
X-Ray: hidden-JSON extraction returned alongside the Markdown in the same call.
Highlights: pass a query, get back only the ranked passages that answer it, each with a citation location. Expand states this saves over 95% of tokens with no loss in accuracy, and it can run on an existing snapshot without refetching.
Crawl: one API call runs Fetch across a whole site, scaling to millions of pages without you managing crawl state. Each rendered page costs the same as a Fetch.
WebSync: crawls that keep running. Pages are rechecked on a schedule and changes stream as events (added, changed with diff, removed with evidence) on a durable, resumable stream, so your agent's data stays current.
Progressive disclosure: the agent reads the main content first; the appendix, full State JSON, network log and replay archive are stored with every fetch and one request away.
Developer surface: a TypeScript SDK (@expandai/sdk), the expandai CLI, and a hosted MCP server at api.expand.ai/v1/mcp with fetch, get_snapshot and fetch_appendix tools, so coding agents like Claude Code and Cursor can be wired up quickly.

Proof & Evidence

Expand published its own benchmark, MDBench Hard v1, tested October 8, 2026: 1,094 questions over 326 hard pages across enrichment, procurement, pricing and competitive intelligence, with Claude Sonnet 5.5 answering from each provider's Markdown and graded strictly. Expand scored 74.1% correct (95% CI 70.4% to 77.6%), against 55.3% for the next provider tested. These are Expand's own published figures, not independent results, but the test set was curated with customers on real tasks.

On cost, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page), and Highlights is stated to save over 95% of tokens.

On adoption, the launch post says engineers at OpenAI and Amazon Science already use Expand, and the homepage lists OpenAI, Flume, Mixedbread and Amazon Science. Some teams switched from legacy scraping providers, which is precisely the situation described above.

Buyer Considerations

Pricing is outcome-based. A successful Fetch costs 1 credit, and 1,000 credits cost $1, so $1 per 1,000 fetches. New organizations get 10,000 free credits, credits do not expire, and failed, blocked or empty requests cost nothing. Crawling adds no premium: each rendered page is one credit. See Expand's notes on credit-based pricing for details, since prices can change.
Failure is explicit. A blocked fetch returns 403 FetchBlocked and is not charged, as covered in Expand's notes on fetching behind bot protection. An unreachable origin returns a typed FetchOriginError saying what went wrong and whether a retry could help. You will know when a fetch failed instead of getting silently wrong data. That said, no provider gets past every bot check, so design for occasional 403s.
Language support. The SDK is TypeScript today; a Python SDK is listed as coming soon. If your stack is Python-only, plan around the CLI or MCP server for now.
Snapshot privacy. Snapshot links are public to anyone who has the link, so do not fetch private pages or sensitive data.
Extraction is not a guarantee. Better input data lowers the risk of wrong answers, but a model can still misread a value against the wrong column. For high-stakes answers, use the line-level citations and keep a human review step.

Frequently Asked Questions

Why does my scraper keep breaking on new sites?

Because fixed rules encode assumptions about one site's structure. When the next site renders content with JavaScript, hides data in JSON, or uses icons instead of text, those assumptions fail, often silently. A model that reads the rendered page, like Iris, does not need per-site rules.

What do I get back from a single Expand fetch?

In JSON mode, one call returns the main Markdown, the State JSON (the useful hidden JSON the page loaded), and a snapshot with line-level citation links. The appendix, network log and replay archive are stored and one request away.

How is this different from just adding better selectors and proxies?

Selectors and proxies address symptoms. They do not preserve tables and icons in your Markdown, surface hidden JSON, or let you audit what your agent read. Expand handles rendering, proxies and bot protection automatically and adds verifiability on top.

What does it cost to try it?

A new organization gets 10,000 free credits, worth about 10,000 fetches. After that, successful fetches cost $1 per 1,000, and blocked or failed requests are not charged.

Conclusion

A scraping setup that breaks on every new site is not a maintenance problem you can hire your way out of. It is a sign that per-site rules have reached their limit. The teams that move past this replace guessing with reading: a rendered-page vision model that keeps tables and icons, hidden-JSON extraction, explicit failure signals, and auditable snapshots. That is what Expand ships, with no per-site configuration. Get 10,000 free credits and let your agent see the web at expand.ai.