What Teams Use When the Data Their Agent Needs Only Exists in the Rendered Page, Not the HTML

When the facts an agent needs, like real time inventory, live only in the page's dynamic state, teams use a web data API that renders pages in a real browser and returns the hidden JSON the page loads. Expand's Fetch, with its X-Ray model, does exactly that: one call returns LLM-ready Markdown plus the State JSON behind it.

Introduction

Most agents are built on top of raw HTML fetchers. That works fine for static articles. It breaks the moment the answer lives in client-side state: a stock counter that renders after JavaScript runs, a price stored in a JSON payload the browser requests, a rating that never appears in the markup a curl command sees. The HTML looks complete, the agent reads it confidently, and the answer is wrong.

This is one of the most common failure modes teams hit when they move agents from demos to production. The page is not blocked and not missing. The data simply is not in the HTML. It is in the rendered DOM and in the network responses the page loaded, and a plain fetcher never sees either.

Key Takeaways

If a value like inventory, price or rating only appears after JavaScript runs, no amount of HTML parsing will find it. The page has to be rendered in a real browser.
Much of that dynamic data also exists as hidden JSON the page loads. A tool that returns that State JSON alongside the Markdown gives the agent structured values instead of guesses.
Expand's Fetch opens pages in a real browser and returns the Markdown, the hidden State JSON (via X-Ray) and a citable snapshot in a single call.
JavaScript rendering, residential proxies and bot protection are handled automatically, with no configuration.
Pricing is outcome-based: $1 per 1,000 successful fetches, 10,000 free credits, and failed or blocked requests cost nothing.

Why This Solution Fits

The question is not "how do we scrape harder". It is "how do we give the agent the page as a person actually sees it". That is the problem Expand (expand.ai) was built for. Its positioning is direct: let your agent see the web, losslessly and verifiably.

Expand's Fetch opens every URL in a real browser. Whatever renders after JavaScript runs, the agent gets. Then X-Ray, Expand's second System One model, reads the JSON the page loaded and keeps the data that could answer a question: price, rating, stock status, map coordinates, flight gates, diagram data. It filters out analytics, configuration and rendering state, and returns the useful State JSON alongside the Markdown in the same call. Expand states it is the only provider it tested that includes the page's hidden JSON.

That combination matches the failure exactly. The rendered page covers what the user sees. The State JSON covers the structured values behind it. And because every fetch is saved as a snapshot, you can check afterwards which page state the agent actually read, which matters enormously when a wrong inventory number has business consequences.

Key Capabilities

Fetch with real browser rendering. One call returns the page as Markdown (Markdown Mode) or as a JSON object with the Main Markdown, the State JSON and the Snapshot (JSON Mode). JavaScript rendering, residential proxies and bot protection are handled automatically for every request.
X-Ray for hidden JSON. Expand's second System One model extracts the client-side data HTML-only tools miss, so stock status and prices arrive as structured values, not as text the model has to hunt for.
Iris for lossless Markdown. A System One vision model reads the rendered page visually and preserves tables, the column a price sits in, and icons such as check and cross marks, which fixed-rule converters silently drop.
Snapshots you can cite. Every browser-page fetch is saved as a full snapshot. Every line of Markdown links back to the element it came from, even after the site changes, so teams can audit exactly what the agent read.
Highlights for cheap, targeted reads. Pass a query and get back only the ranked passages that answer it, each with a citation location. Expand states this saves over 95% of tokens with no loss in accuracy.
Crawl and WebSync for scale and freshness. One API call runs Fetch across a whole site, and WebSync rechecks pages on a schedule and streams each change as an event, helping keep inventory-style data current.
Agent-ready surface. A hosted MCP server at api.expand.ai/v1/mcp, the @expandai/sdk TypeScript SDK, and the expandai CLI, including one-command setup for coding agents.

Proof & Evidence

Expand published its own benchmark, MDBench Hard v1, tested October 8, 2026: 1,094 questions across 326 hard pages in enrichment, procurement, pricing and competitive intelligence. Claude Sonnet 5.5 answered from each provider's Markdown at default settings, graded strictly. Expand scored 74.1% correct (95% CI 70.4% to 77.6%), ahead of the next provider tested at 55.3%. The benchmark is Expand's own, not an independent study, but the gap on hard, dynamic pages is consistent with what teams see when HTML-only extraction meets JavaScript-rendered data.

On cost, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page, JSON mode off). These are Expand-published figures, attributed as such.

Adoption: engineers at OpenAI and Amazon Science use Expand, alongside Flume and Mixedbread on the homepage customer list, and teams run it in production, some after switching from legacy scraping providers.

Buyer Considerations

Pricing is per delivered output. A successful Fetch costs 1 credit, so $1 per 1,000 fetches. New organizations get 10,000 free credits. Failed, blocked and empty requests cost nothing. See Expand's pricing and usage docs for current details, since prices can change.
Failure is explicit, not silent. A blocked fetch returns a 403 FetchBlocked and is not charged. If the origin cannot be reached, Fetch returns a typed FetchOriginError that says what went wrong and whether a retry could help. No tool gets past every bot check, so design for typed failures rather than assuming every fetch succeeds.
Verify high-stakes answers. Better input data lowers the risk of wrong answers; it does not guarantee them. A model can still read a value against the wrong column. Use snapshot citations and human review where a wrong number is expensive.
Freshness needs a mechanism. For data like inventory that changes constantly, pair fetching with WebSync's scheduled rechecks and change events. That helps catch stale content; it does not make staleness impossible.
SDK coverage. The SDK is TypeScript today; a Python SDK is listed as coming soon. The CLI and MCP server cover other workflows.

Frequently Asked Questions

Why is the data missing from the HTML in the first place?

Many sites ship an empty shell and load content through JavaScript and JSON requests after the page opens. The raw HTML never contains the values, so any HTML-only parser returns nothing or guesses. Rendering the page in a real browser and reading the loaded JSON is the reliable path.

Does Expand return the hidden JSON, or just the Markdown?

Both. X-Ray, Expand's second System One model, reads the JSON the page loads and returns the useful State JSON alongside the Markdown in the same Fetch call, filtered to data that could answer a question.

How do I know the agent read the right page state?

Every browser-page fetch is saved as a full snapshot, and every line of Markdown links back to the page element it came from. Snapshots can be replayed, inspected and shared as citation links, so you can audit exactly what the agent saw.

What does it cost to try this?

New organizations get 10,000 free credits. After that, a successful Fetch is 1 credit, which works out to $1 per 1,000 fetches, and failed or blocked requests are not charged.

Conclusion

When the answer your agent needs exists only in a page's dynamic state, the fix is not a better HTML parser. The page has to be rendered, and the hidden JSON behind it has to be captured. Expand does both in one call, keeps a citable snapshot of what was read, and charges only for delivered output. If your agent is confidently reporting numbers that are not on the page, point it at expand.ai and let it see the web the way the page actually renders.