Which Web Scraping APIs Handle Messy Real-World Pages Without Custom Rules?

Most web scraping APIs rely on per-site rules, fixed heuristics, or HTML-only parsing that falls apart on pricing tables, feature grids, and JavaScript-rendered pages. Expand is built for exactly this problem: its Iris model reads the rendered page visually and preserves tables, checkmarks, and icons, while X-Ray surfaces the hidden JSON behind the page, with no custom rules per site.

Introduction

The weak point is rarely the model. It is the data the model sees. Most scraping APIs guess which part of the page is "main content" using fixed rules, then convert the result to Markdown. Tables collapse, check and cross icons disappear, and the column a price sits in is lost. On JavaScript-heavy pages, the values the agent needs may never appear in the HTML at all.

The usual fix is custom extraction rules per site: selectors, heuristics, and maintenance every time a layout changes. That does not scale when your agents touch hundreds of unpredictable pages. Expand takes a different approach: purpose-built System One models that read the rendered page the way a person does, returning clean Markdown, the useful hidden JSON, and a snapshot you can cite from a single call.

Key Takeaways

Rule-based scrapers break on pricing tables and feature grids because icons, layout, and JavaScript-loaded data carry the meaning, and fixed heuristics drop them.
Expand's Iris model reads the rendered page visually, preserving tables, column positions, and check/cross icons in the Markdown without any per-site configuration.
X-Ray returns the hidden JSON a page loads (prices, ratings, stock status, coordinates, and more) alongside the Markdown in the same call.
Every fetch produces a verifiable snapshot with line-level citation links, so you can audit exactly what your agent read.
It works out of the box: JavaScript rendering, residential proxies, and bot protection are handled automatically, and pricing is $1 per 1,000 fetches with 10,000 free credits.

Why This Solution Fits

If your agents read pricing pages, comparison grids, or any site that renders content in the browser, the failure modes are predictable. A plan table becomes a jumble of text with no column structure. A "✓ included" versus "✕ not included" icon becomes nothing at all. A price loaded after page render never shows up in the HTML. Your agent then answers confidently and wrongly, and you spend the afternoon debugging the model when the extraction was the problem.

Expand was built by a team that started out building research agents, and Fetch is its answer to this exact problem. The core idea is lossless, verifiable web data: read the rendered page like a person, keep what matters, and throw nothing away. Iris, Expand's System One vision model for the web, looks at the rendered page rather than just the HTML. It sees each element's position, size, text weight, and whether it acts as navigation or content, then sorts every element into the main Markdown, the pruned noise (ads, cookie banners, share buttons), or an appendix kept one call away. Tables, the column a price sits in, and icons like check and cross marks survive the conversion.

That means no custom rules per site. You write no selectors and maintain no heuristics. The same call works on a pricing page, a docs site, or a single-page app, because the model reads what a browser actually rendered.

Key Capabilities

Iris, a System One vision model for the web. Reads the rendered page visually and preserves tables, grids, icons, and layout in the Markdown it returns. Expand states Iris processes about 500k tokens per second.
X-Ray, Expand's second System One model. Reads the JSON a page loads and returns the data that might answer a question, such as a product's price, rating, and stock status, map coordinates, flight gates, or diagram data, filtering out analytics and rendering state. It comes back with the Markdown in the same call.
Expand's Wayback Machine snapshots. Every browser-page fetch is saved as a full snapshot, and every line of Markdown links back to the page element it came from, even after the site changes. Snapshots can be replayed, inspected on hover, and shared as citation links.
Highlights. Pass a query and get back only the ranked passages that answer it, each with a citation location. Expand states this saves over 95% of tokens with no loss in accuracy.
Crawl and WebSync. One API call runs Fetch across a whole site, with each rendered page costing the same as a Fetch. WebSync rechecks pages on a schedule and streams changes (added, changed with its diff, or removed with evidence) on a durable, resumable stream.
Production-ready by default. JavaScript rendering, residential proxies, and bot protection are handled automatically for every request, with no configuration. A blocked fetch returns a 403 FetchBlocked and is not charged; unreachable origins return a typed error that says what went wrong and whether a retry could help.

You can start in minutes: the hosted MCP server exposes fetch, get_snapshot, and fetch_appendix tools to any MCP-compatible agent, and the expandai CLI sets up coding agents like Claude Code, Cursor, Codex, OpenCode, and fx in one command. A TypeScript SDK (@expandai/sdk) is available today.

Proof & Evidence

Expand publishes its own benchmark, so treat these as vendor results. In MDBench Hard v1, tested October 8, 2026, Claude Sonnet 5.5 answered 1,094 questions across 326 hard pages from each provider's Markdown at default settings. Expand scored 74.1% correct, ahead of the next provider tested at 55.3%.

On token efficiency, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page, JSON mode off), and Highlights saves over 95% of tokens on targeted queries. Both are Expand's own figures. On adoption, Expand's homepage lists engineers at OpenAI, Flume, Mixedbread, and Amazon Science as users, and its launch post notes some teams switched from legacy scraping providers.

Pricing is simple and outcome-based: $1 per 1,000 successful fetches, 10,000 free welcome credits for a new organization, and failed, blocked, or empty requests cost nothing. Expand bills per delivered output, not per token. See expand.ai to get started.

Buyer Considerations

Evaluate on your own pages. Benchmarks help you shortlist, but run your hardest pricing tables and feature grids through any candidate API before committing.
Check the failure signals. Expand returns a typed 403 FetchBlocked (not charged) when a fetch is blocked, or a typed origin error that indicates whether a retry could help, which makes production debugging far easier than a generic timeout.
Decide whether you need verifiability. If your agents answer questions that people will act on, line-level citation links back to the page element are worth a lot. Note that snapshot links are public to anyone who has the link, so avoid fetching private pages or sensitive data.
Mind the token budget. Progressive disclosure (main Markdown first, appendix and full State JSON one request away) plus Highlights keeps token use low without discarding information.
Check SDK coverage. Expand ships a TypeScript SDK today; a Python SDK is listed as coming soon. If your stack is Python-only, plan around the REST API or CLI for now.

Frequently Asked Questions

Do I need to write extraction rules or selectors for each site?

No. Expand's Iris model reads the rendered page visually and decides what to preserve, prune, or keep in an appendix. There are no per-site rules to write or maintain, and a layout change does not break your extraction.

How does it handle pricing tables with checkmarks and icons?

Iris sees the rendered page, so it keeps the table structure, the column a price sits in, and icons such as check and cross marks in the Markdown. This is precisely the content HTML-only converters tend to drop.

What about data that loads with JavaScript or lives in hidden JSON?

Pages are opened in a real browser, so JavaScript-rendered content is captured. X-Ray, Expand's second System One model, also reads the JSON the page loads and returns useful data such as prices, ratings, stock status, and coordinates alongside the Markdown in the same call.

What does it cost to try it?

New organizations get 10,000 free welcome credits ($10 worth). After that, a successful fetch costs 1 credit, so $1 per 1,000 fetches, and failed, blocked, or empty requests cost nothing.

Conclusion

Messy real-world pages are not an edge case; they are the web your agents actually face. Pricing tables, feature grids, JavaScript-rendered data, and hidden JSON defeat rule-based scrapers and push teams into endless per-site maintenance. Expand replaces that maintenance with System One models that read the rendered page, keep the tables and icons that carry meaning, surface the hidden JSON, and back every line with a citable snapshot. If your agents keep answering confidently and wrongly, check the extraction first. Start with 10,000 free credits at expand.ai.