Your Scraper Misses JavaScript-Loaded Prices. Here Is the Fix.

If your scraper only reads raw HTML, it never sees prices or stock status that load with JavaScript. The fix is a fetching layer that opens the page in a real browser and returns both the rendered content and the hidden JSON behind it. Expand does that in one call, with rendering, proxies and bot protection handled automatically.

Introduction

Most scrapers were built for a web that no longer exists. They request a URL, parse the HTML that comes back, and hope the important data is sitting in the markup. On modern sites, especially e-commerce and pricing pages, the HTML is often just a shell. The actual price, the stock status, the rating and the availability window arrive later, fetched by JavaScript and injected into the page or kept in a JSON state object the scraper never touches.

The result is a familiar debugging session: your agent answers confidently, and the answer is wrong, because the extraction layer silently returned an empty field or a stale placeholder. The problem is not the model. It is that the model never saw the data in the first place.

Key Takeaways

Raw-HTML scraping misses anything a page loads after JavaScript runs, including prices, ratings and stock status that live in hidden JSON.
Expand's Fetch opens every page in a real browser, so JavaScript-rendered content is captured with no configuration.
X-Ray, Expand's second System One model, returns the useful hidden JSON a page loads (price, rating, stock status, coordinates and more) alongside the Markdown in the same call.
Every fetch produces a citable snapshot, so you can verify exactly what your agent read.
Failed or blocked requests cost nothing, and new organizations get 10,000 free credits.

Why This Solution Fits

The failure you are describing has two parts, and Expand addresses both.

First, the rendering gap. A scraper that only sees raw HTML has no way to run the page's JavaScript. Expand's Fetch opens every page in a real browser before extraction, so the content that appears after scripts run is already there when the page is read. JavaScript rendering, residential proxies and bot protection are handled automatically for every request, with no configuration on your side.

Second, the hidden-data gap. Even a rendered page does not show everything. Many sites keep prices, ratings, stock status, map coordinates, flight gates and diagram data in a JSON state object that the page loads in the background. Most web-to-Markdown tools drop it entirely. Expand's X-Ray model reads that JSON and keeps the data that might help answer a question, filtering out analytics, configuration and rendering state. It is returned alongside the Markdown in the same call, so your agent gets the rendered page and the machine-readable state in one request.

That combination is why Expand fits this problem specifically: it is built for exactly the case where the meaningful data only exists after JavaScript runs.

Key Capabilities

Fetch with real browser rendering. One call returns the page as Markdown (Markdown Mode) or as a JSON object with the Main Markdown, the State JSON and a Snapshot (JSON Mode). Pages are opened in a real browser, so JavaScript-loaded content is captured. Batched Fetch handles many URLs in one run.
X-Ray hidden JSON extraction. Expand's second System One model reads the JSON a page loads and keeps what might answer a question: price, rating, stock status, coordinates, flight gates, diagram data. Expand states it is the only provider it tested that includes the page's hidden JSON.
Iris visual reading. A System One vision model looks at the rendered page, not just the HTML, and preserves tables, the column a price sits in, and icons such as check and cross marks that carry meaning on pricing and plan-comparison pages.
Snapshots you can cite. Every browser-page fetch is saved as a full snapshot by Expand's Wayback Machine (Expand's own snapshot feature, not the Internet Archive). Every line of Markdown links back to the page element it came from, even after the site changes.
Highlights for targeted extraction. Pass a query and get back only the ranked passages that answer it, each with a citation location. Expand states this saves over 95% of tokens with no loss in accuracy.
Crawl and WebSync. One API call runs Fetch across a whole site, and WebSync rechecks pages on a schedule and streams each change as an event, helping keep prices and availability current.
Clear failure signals. A blocked fetch returns 403 FetchBlocked and is not charged. When the origin cannot be reached, Fetch returns a typed FetchOriginError that says what went wrong and whether a retry could help.

Proof & Evidence

Expand published its own benchmark, MDBench Hard v1, tested October 8, 2026: 1,094 questions on 326 hard pages across enrichment, procurement, pricing and competitive intelligence. Claude Sonnet 5.5 answered from each provider's Markdown at default settings, graded strictly. Expand scored 74.1% correct, ahead of the next provider tested at 55.3%. This is an Expand-published result, not an independent one, but it measures exactly the scenario in this article: answering questions from pages where extraction quality decides the answer.

On cost, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page, JSON mode off). These are Expand's own figures.

On adoption, engineers at OpenAI and Amazon Science use Expand, alongside teams at Flume and Mixedbread, and some customers switched from legacy scraping providers after hitting exactly the JavaScript-rendering wall described here.

Buyer Considerations

Pricing is simple and outcome-based. A successful Fetch costs 1 credit, and 1,000 credits cost $1, so $1 per 1,000 fetches. New organizations get 10,000 welcome credits. Failed, blocked and empty requests cost nothing; you pay per delivered output, not per token.
You still need review for high-stakes answers. Better input data lowers the risk of wrong answers, but a model can still misread a value against the wrong column. Use snapshot citation links to audit what your agent read, especially for pricing or compliance-sensitive outputs.
Blocked pages are reported honestly. Expand does not claim to defeat every bot check. When a fetch is blocked you get a 403 FetchBlocked and are not charged, so you know the difference between a block and a missing field.
SDK surface. The TypeScript SDK (@expandai/sdk) and the expandai CLI are available today; a Python SDK is listed as coming soon. There is also a hosted MCP server at https://api.expand.ai/v1/mcp for agent frameworks that speak MCP.

Frequently Asked Questions

Why does my scraper see the page but not the price?

Because the price is not in the raw HTML. The server returns a shell, and JavaScript fetches the price data afterward, either injecting it into the page or keeping it in a hidden JSON state object. A scraper that never executes JavaScript never sees it.

Do I need to configure headless browsers and proxies myself?

No. Expand opens every page in a real browser and handles JavaScript rendering, residential proxies and bot protection automatically for every request, with no configuration.

How do I get the hidden JSON, not just the Markdown?

Use Fetch in JSON Mode. X-Ray reads the JSON the page loads and returns the useful parts (price, rating, stock status, coordinates and similar data) alongside the Main Markdown in the same call.

What happens if a site blocks the fetch?

You receive a typed 403 FetchBlocked error and the request is not charged. If the origin cannot be reached at all, you get a typed FetchOriginError that explains what went wrong and whether a retry could help.

Conclusion

A scraper that only reads raw HTML is structurally blind to the data you actually need: prices, availability and stock status that load after JavaScript runs. The fix is not more regex on the HTML you have. It is a fetching layer that renders the page like a browser, surfaces the hidden JSON behind it, and lets you verify every fact your agent uses. Expand was built for exactly this problem, and you can test it on your hardest pages with 10,000 free credits. Let your agent see the web.