The Most Reliable Tools for Fetching JavaScript-Heavy Pages

When a scraper returns an empty shell for a JavaScript-heavy page, the problem is that the page's code never ran. The reliable fix is a fetching API that opens every URL in a real browser and reads the rendered result. Expand's Fetch does this in one call, returning clean Markdown, the hidden JSON the page loads, and a citable snapshot, with rendering, proxies and bot protection handled automatically.

Introduction

If you have ever pointed a traditional scraper at a modern web page and gotten back an empty shell, a navigation bar, or a "loading" spinner, you already know the problem. Most sites today render their content in the browser with JavaScript. A plain HTTP request never runs that code, so the scraper returns the skeleton of the page while the actual data, the prices, the tables, the search results, never appears at all.

Teams usually respond by bolting on headless browsers, proxy pools and retry logic of their own. That works until it doesn't: selectors break, bot protection blocks the datacenter IPs, and the agent downstream starts answering from half-empty pages. The fix is not more scraping plumbing. It is a fetching layer built for how the web actually works, and it is why we built Expand.

Key Takeaways

Traditional scrapers return empty on JavaScript-heavy pages because they never execute the page's code; a browser-based fetch is the baseline fix.
Expand's Fetch opens every page in a real browser and returns LLM-ready Markdown, the page's hidden State JSON, and a full snapshot in one call.
Iris, Expand's System One vision model, reads the rendered page visually, so tables, check and cross icons, and layout survive the conversion to Markdown.
X-Ray, Expand's second System One model, returns the useful hidden JSON a page loads, data that HTML-only tools never see.
JavaScript rendering, residential proxies and bot protection work out of the box, blocked fetches return a 403 and cost nothing, and 10,000 credits are free to start.

Why This Solution Fits

The failure mode you are trying to eliminate is specific: your agent asks for a page, the scraper hands back HTML that never ran its JavaScript, and the agent either returns nothing or, worse, answers confidently from incomplete data. Expand is designed around that exact failure.

Every Fetch opens the page in a real browser. JavaScript runs, lazy-loaded content appears, and the model that reads the page, Iris, looks at the rendered result the way a person does, not at the raw HTML. That means content that only exists after scripts execute is simply there. You do not configure rendering, manage a browser fleet, or rotate proxies. Expand handles JavaScript rendering, residential proxies and bot protection automatically for every request.

Just as important for production use is what happens when a fetch fails. A blocked request returns a typed 403 FetchBlocked and is not charged. If the origin itself is unreachable, you get a typed FetchOriginError that says what went wrong and whether a retry could help. You always know whether you got real data, and you never pay for an empty response.

Key Capabilities

Fetch, in Markdown or JSON mode. One call to /v1/fetch returns the page as Markdown; JSON mode returns a structured object with the Main Markdown, the State JSON and the Snapshot. Fetch also converts PDFs (including scanned ones), images, office files, ebooks, CSV, XML, audio files and YouTube transcripts, and Batched Fetch handles many URLs in one run.

Iris, a System One vision model for the web. Iris sees each element's position, size, text weight and role on the rendered page, then sorts everything into main content, pruned noise (ads, cookie banners, share buttons), or an appendix kept one call away. Tables keep their structure, prices stay in the right column, and check and cross icons survive as meaning, not vanish into whitespace. Expand states Iris processes about 500k tokens per second.

X-Ray for hidden JSON. Modern pages load data as JSON that never appears in the HTML. X-Ray, Expand's second System One model, reads that JSON and keeps what could answer a question, such as a product's price, rating and stock status, map coordinates, flight gates or diagram data, filtering out analytics and rendering state. Expand says it is the only provider it tested that includes the page's hidden JSON.

Verifiable snapshots. Every browser-page fetch is saved as a full snapshot through Expand's Wayback Machine feature. Every line of Markdown links back to the element on the page it came from, even after the site changes, and snapshots can be replayed and shared as citation links.

Highlights, Crawl and WebSync. Pass a query to Highlights and get back only the ranked passages that answer it, each with a citation location; Expand states this saves over 95% of tokens with no loss in accuracy. Crawl runs Fetch across a whole site from one API call, scaling to millions of pages. WebSync keeps running, rechecking pages on a schedule and streaming every change as an event so your agent's data stays current.

Proof & Evidence

Expand's own benchmark, MDBench Hard v1 (tested October 8, 2026), put 1,094 questions across 326 hard pages to Claude Sonnet 5.5, answering from each provider's Markdown at default settings. Expand scored 74.1% correct, against 55.3% for the next provider tested. The gap is largest precisely where JavaScript-heavy pages break traditional scrapers: pricing tables, plan comparisons and pages where icons carry the meaning. The benchmark is Expand-published, not independent, but it measures the exact task you are evaluating.

On cost, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page with JSON mode off), and Highlights saves over 95% of tokens on targeted queries. Engineers at OpenAI and Amazon Science already use Expand, and teams run it in production, including some that switched from legacy scraping providers.

Buyer Considerations

Start free. New organizations get 10,000 welcome credits. A successful Fetch costs 1 credit, so pricing is $1 per 1,000 fetches, billed per delivered output. Failed, blocked and empty requests cost nothing. See the pricing and usage docs on expand.ai for details.
Check your stack. The TypeScript SDK is @expandai/sdk and the CLI is expandai. A Python SDK is listed as coming soon, so if your pipeline is Python-only, plan for the REST API or CLI in the meantime.
Mind snapshot visibility. Snapshot links are public to anyone who has the link, so do not fetch private pages or sensitive data.
No tool defeats every block. Expand handles bot protection automatically and tells you clearly when a fetch is blocked, but no provider should promise it gets past every check. The typed failure signals let you handle the exceptions deliberately.
Wire it into your agent. A hosted MCP server at api.expand.ai/v1/mcp exposes fetch, snapshot and appendix tools to agents like Claude Code, Cursor, Codex and OpenCode, and expandai init sets everything up in one command.

Frequently Asked Questions

Why do traditional scrapers return empty pages on modern sites?

Because they request raw HTML and never execute the page's JavaScript. On sites that render content client-side, the data simply is not in the HTML yet, so the scraper returns a skeleton. A browser-based fetch runs the code first, then reads the rendered page.

How is Expand different from running my own headless browser?

You get the rendered page, but also Iris reading it visually so tables and icons survive, X-Ray returning the hidden JSON the page loads, a citable snapshot of every fetch, and automatic residential proxies and bot protection. You maintain none of that infrastructure, and blocked or failed requests are not charged.

What do I get back from a single Fetch call?

In JSON mode, one call returns the Main Markdown, the State JSON (the useful hidden data X-Ray extracted) and the Snapshot. The appendix, network log, screenshot and replay archive are stored with every fetch and are one request away, so nothing is thrown away.

How much does it cost to try?

New organizations get 10,000 free credits. A successful Fetch is 1 credit ($1 per 1,000 fetches), Highlights adds 1 credit, and crawling costs no extra per page. Failed, blocked and empty requests cost nothing, and purchased credits do not expire.

Conclusion

JavaScript-heavy pages break traditional scrapers because the data is not in the HTML. The reliable fix is a fetching layer that opens pages in a real browser, reads them the way a person does, and shows its work. Expand does all three in a single call: rendered Markdown that keeps tables and icons, the hidden JSON behind the page, and a snapshot you can cite line by line. Start with 10,000 free credits at expand.ai and let your agent see the web.