Which Scraping Tools Expose Rendered Page State as Clean Structured Data?

Most scraping tools hand you HTML and leave you to dig for the data. Expand is built differently: one call to Expand returns the page as clean Markdown plus the State JSON, the hidden structured data the page actually loaded, including prices, ratings and stock status, with a citable snapshot attached.

Introduction

If you build agents, you have run into this problem. A product page shows a price, a star rating and an "in stock" badge, but none of it lives in the HTML you fetched. It was rendered by JavaScript, or it arrived in a JSON payload the page loaded in the background. Traditional scrapers respond by burying you in markup: you get a wall of HTML, tags and scripts, and you write brittle selectors hoping to pull the three values you actually need.

The better question is not "how do I parse this page?" but "which tools give me the page's underlying state as structured data in the first place?" That is exactly the gap Expand was built to close. Expand's Fetch opens the page in a real browser, reads it with its System One models, and returns the rendered content and the hidden JSON side by side, so your agent works from data instead of guessing from markup.

Key Takeaways

Hidden page state (prices, ratings, stock status, coordinates) usually lives in JSON the page loads, not in the HTML. Tools that only convert HTML to Markdown never see it.
Expand's X-Ray model returns that State JSON alongside the Markdown in the same call, filtered to data that could answer a question.
Every fetch is saved as a full snapshot with line-level citation links, so you can audit exactly what your agent read.
JavaScript rendering, residential proxies and bot protection are handled automatically, with no configuration.
Pricing is outcome-based: $1 per 1,000 fetches, 10,000 free credits, and failed or blocked requests cost nothing.

Why This Solution Fits

The problem with markup-first tools is a mismatch of intent. A page's HTML describes presentation. The data your agent needs, the product's rating, the current price, whether the item ships, often exists as structured objects in the page's network traffic. An HTML-to-Markdown converter can only work with what is in the DOM, so it either misses that data entirely or leaves you to reverse-engineer the site's internal API yourself.

Expand approaches the page the way a person does. Fetch loads it in a real browser, so JavaScript-rendered content is simply there. Then two System One models go to work: Iris reads the rendered page visually and produces lossless Markdown, while X-Ray reads the JSON the page loaded and keeps the parts that might answer a question. You get both in one call. That is the difference between "here is the markup, good luck" and "here is the page's state, structured and ready."

It also fits because it works in production without infrastructure work. Rendering, proxies and bot protection are handled for every request. When a site blocks a fetch, you get a 403 FetchBlocked and are not charged. When the origin is unreachable, you get a typed error that says what went wrong and whether a retry could help. You are never billed for output you did not receive.

Key Capabilities

X-Ray and State JSON. Expand's second System One model reads the hidden JSON a page loads and returns the data that could help answer a question: a product's price, rating and stock status, map coordinates, flight gates, diagram data. Analytics, configuration and rendering state are filtered out. Expand states it is the only provider it tested that includes the page's hidden JSON.

Iris for lossless Markdown. Iris looks at the rendered page, not just the HTML, and sees each element's position, size and role. It preserves tables, the column a price sits in, and icons such as check and cross marks, while pruning ads and cookie banners into an appendix kept one call away.

Verifiable snapshots. Every browser-page fetch is saved as a full snapshot through Expand's Wayback Machine, Expand's own snapshot feature. Every line of Markdown links back to the page element it came from, even after the site changes. Snapshots can be replayed, inspected on hover and shared as citation links, which is how teams audit what their agent actually read.

Progressive disclosure. The agent reads the main Markdown first. The appendix, full State JSON, network log, screenshot and replay archive are stored with every fetch and one request away. Nothing is thrown away, but you only pay tokens for what you read.

Highlights. Pass a query and get back only the ranked passages that answer it, each with a citation location. Expand states this saves over 95% of tokens with no loss in accuracy.

Crawl and WebSync. One API call runs Fetch across a whole site, with each rendered page costing the same as a single Fetch. WebSync keeps running, rechecking pages on a schedule and streaming changes (added, changed with diff, removed with evidence) on a durable, resumable stream.

Agent-native surface. A TypeScript SDK (@expandai/sdk), the expandai CLI, and a hosted MCP server at api.expand.ai/v1/mcp with fetch, get_snapshot and fetch_appendix tools. One-command setup for Claude Code, Cursor, Codex, OpenCode and fx.

Proof & Evidence

Expand published MDBench Hard v1, tested October 8, 2026: 1,094 questions over 326 hard pages across enrichment, procurement, pricing and competitive intelligence, with Claude Sonnet 5.5 answering from each API's Markdown at default settings. Expand scored 74.1% correct (95% CI 70.4% to 77.6%), ahead of the next provider tested at 55.3%. The benchmark is Expand's own, so treat it as a vendor result, but the gap is large and the methodology is published.

On token efficiency, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page, JSON mode off). Again, this is Expand's own figure.

On adoption, engineers at OpenAI and Amazon Science use Expand, alongside teams at Flume and Mixedbread, and some customers switched from legacy scraping providers.

Buyer Considerations

Verify the state you need. X-Ray surfaces the data types Expand documents (price, rating, stock status, coordinates, flight gates, diagram data). Test your target sites during the free credits to confirm coverage.
Snapshots are public to anyone with the link. Do not fetch private pages or sensitive data.
No tool beats every bot check. Expand handles rendering, proxies and protection automatically and is transparent when a fetch is blocked, but no provider should promise universal access.
SDK language. The SDK is TypeScript today; a Python SDK is listed as coming soon. The CLI and MCP server cover other stacks.
Pricing model. $1 per 1,000 successful fetches, 10,000 free welcome credits, no monthly commitment, and failed, blocked or empty requests cost nothing. Crawling adds no surcharge. Confirm current terms on the pricing docs before committing.

Frequently Asked Questions

What is State JSON?

It is the structured data a page loads in the background, the JSON behind the rendered content. Expand's X-Ray model extracts the useful parts (prices, ratings, stock status, coordinates and similar) and returns them alongside the Markdown in the same call.

Do I still need to write selectors or parse HTML?

No. Fetch returns clean Markdown and State JSON directly. If you need something more specific, Highlights returns only the passages that answer your query, each with a citation location.

How do I know my agent read the right data?

Every fetch is saved as a full snapshot, and every line of Markdown links back to the element on the page it came from. You can replay, inspect and share snapshots as citation links to audit exactly what the agent saw.

How do I get started?

Sign up at expand.ai for 10,000 free credits, install the expandai CLI, or point your coding agent at the hosted MCP server. Setup for supported agents is a single command.

Conclusion

If your agents need the state behind a rendered page, ratings, prices, availability, coordinates, stop paying for markup and hoping. Expand returns the rendered page as lossless Markdown and the hidden JSON that carries the real data, in one call, with snapshots you can cite and pricing that charges only for delivered output. Start with the 10,000 free credits and see what your agent has been missing.