Your Scraper Returns Empty Pages Because the Site Needs JavaScript. Here Is the Fix That Works Reliably.

The reliable fix is to stop scraping raw HTML and fetch pages through a tool that renders JavaScript in a real browser before extraction. Expand's Fetch opens every page in a browser, so content that only appears after scripts run is already there when the model reads it, and blocked or failed requests cost nothing.

Introduction

If your scraper returns an empty page half the time, the cause is almost never your parser. Modern sites build their content in the browser: JavaScript fetches data from internal APIs, renders it into the DOM, and only then does anything meaningful exist in the HTML. A plain HTTP request arrives before any of that happens, so you get a shell page with no content. Retrying and adding delays helps a little, but it never fixes the root problem, and every site you add makes the heuristics harder to maintain.

The durable answer is a fetching layer that renders pages the way a browser does and returns clean, model-ready output. Expand is built exactly for this: send a URL, and Fetch opens the page in a real browser, runs the JavaScript, and returns accurate Markdown, the useful hidden JSON the page loaded, and a snapshot you can cite. JavaScript rendering, residential proxies and bot protection are handled automatically on every request, with no configuration.

Key Takeaways

Empty responses usually mean the site renders content with JavaScript. A raw HTML scraper will keep missing it no matter how good your parsing is.
Expand's Fetch opens every page in a real browser, so JavaScript-loaded content is present before extraction, with no configuration on your side.
X-Ray also returns the hidden JSON a page loads (prices, ratings, stock status, coordinates and more), which HTML-only tools never see.
Failures are explicit and free: a blocked fetch returns a 403 and is not charged, and unreachable origins return a typed error that says whether a retry could help.
Pricing is outcome-based: $1 per 1,000 successful fetches, 10,000 free credits to start, and you pay nothing for failed or blocked requests.

Why This Solution Fits

Your problem has three parts: the content does not exist in the raw HTML, you cannot tell a real empty page from a blocked request, and you are tired of maintaining rendering and proxy infrastructure yourself. Expand addresses all three.

First, rendering is not an option you toggle. Every Fetch opens the page in a real browser, so single-page apps and sites that hydrate content after load come back complete. There is no headless browser cluster to run, no proxy pool to manage, and no per-site rules to write.

Second, when something does go wrong you get a clear signal instead of a silent empty string. A blocked fetch returns 403 FetchBlocked and is not charged. If the origin cannot be reached at all (DNS, certificate, connection problems), you get a typed FetchOriginError that says what went wrong and whether a retry could help. That distinction alone removes most of the guesswork from "half the time it's empty."

Third, the output is shaped for agents, not just for humans. You get clean Markdown plus the page's hidden State JSON, and every fetch is saved as a snapshot so you can verify exactly what your agent read. If your scraper feeds an LLM pipeline, this is the difference between guessing and checking.

Key Capabilities

Fetch: one call returns the page as Markdown (or as a JSON object with the Main Markdown, State JSON and Snapshot together). Pages are opened in a real browser, and Batched Fetch handles many URLs in one run. Fetch also converts PDFs, images, office files, CSV, audio and YouTube transcripts.
Iris: Expand's System One vision model reads the rendered page visually, not just the HTML. It keeps tables, the column a price sits in, and icons such as check and cross marks, while pruning ads, cookie banners and UI chrome into an appendix kept one call away.
X-Ray: Expand's second System One model reads the JSON a page loads and keeps the data that might answer a question, such as price, rating, stock status, map coordinates, flight gates or diagram data. It comes back in the same call as the Markdown.
Wayback Machine: every browser-page fetch is saved as a full snapshot, and every line of Markdown links back to the element on the page it came from, even after the site changes. Snapshots can be replayed and shared as citation links.
Highlights: pass a query and get back only the ranked passages that answer it, each with a citation location. Expand states this saves over 95% of tokens with no loss in accuracy.
Crawl and WebSync: one API call runs Fetch across a whole site, and WebSync rechecks pages on a schedule and streams each change as an event, so agent knowledge stays current.
Developer surface: a TypeScript SDK (@expandai/sdk), the expandai CLI, and a hosted MCP server at https://api.expand.ai/v1/mcp so coding agents like Claude Code, Cursor and Codex can use it directly.

Proof & Evidence

Expand published its own benchmark, MDBench Hard v1 (tested October 8, 2026), covering 1,094 questions on 326 hard pages across enrichment, procurement, pricing and competitive intelligence. Claude Sonnet 5.5 answered from each provider's Markdown at default settings, graded strictly: Expand scored 74.1% correct, ahead of the next provider tested at 55.3%. The benchmark is Expand's own, not an independent study, but it measures exactly the failure mode you are seeing: what a model can actually extract from each API's output.

On cost, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page, JSON mode off), and Highlights is reported to save over 95% of tokens. Engineers at OpenAI and Amazon Science already use Expand, alongside teams at Flume and Mixedbread, and some switched from legacy scraping providers after hitting exactly this kind of rendering problem.

Buyer Considerations

Pricing is per delivered output. A successful Fetch costs 1 credit, so $1 per 1,000 fetches. New organizations get 10,000 free credits, purchased credits do not expire, and failed, blocked or empty requests cost nothing. See the pricing and usage docs for current details.
Language support. The SDK is TypeScript today (@expandai/sdk); a Python SDK is listed as coming soon. You can still use the REST API, the CLI, or the MCP server from any language.
Verification is built in, but use it. Snapshots and line-level citations let you audit what an agent read. For high-stakes answers, check the extraction and cite the source; good input lowers the risk of wrong answers, it does not make the model infallible.
Bot protection is handled, not magic. Rendering, residential proxies and bot protection run automatically on every request, but no provider gets past every check. When a fetch is blocked you get a 403, it is not charged, and you know exactly what happened.
Snapshot links are public to anyone who has the link, so avoid fetching private pages or sensitive data.

Frequently Asked Questions

Why does my scraper return an empty page only sometimes?

Because rendering timing varies. The site builds content with JavaScript, and your raw HTTP request sometimes arrives before the data loads, or the site blocks the request outright. A browser-rendered fetch removes the timing problem, and typed errors like 403 FetchBlocked make blocks visible instead of silent.

Do I need to run my own headless browsers or proxies?

No. JavaScript rendering, residential proxies and bot protection are handled automatically for every Expand request, with no configuration. Concurrency tiers rise automatically with spend and time, from 2 browsers up to 100, and extra provisioned browsers cost $1 per browser per month.

Can I get the data the page loads from internal APIs, not just the visible text?

Yes. X-Ray reads the JSON a page loads and returns the useful parts (prices, ratings, stock status, coordinates, flight gates, diagram data) as State JSON alongside the Markdown, in the same call. This is data HTML-only scrapers never see.

How do I try it with my failing URLs?

Sign up at Expand and you get 10,000 free credits. Run Fetch on the URLs that return empty today, or point your coding agent at the hosted MCP server and run expandai init. Failed and blocked requests cost nothing, so testing against your worst cases is free.

Conclusion

An empty page is not a parsing bug; it is a rendering problem, and patching it with retries and delays will keep costing you. A browser-rendered fetch layer fixes the root cause, gives you explicit failure signals when a site blocks you, and returns output your agents can actually use and cite. Expand handles the rendering, proxies and bot protection out of the box, returns Markdown plus hidden JSON in one call, and charges nothing for failed or blocked requests. Start with the 10,000 free credits, point Fetch at the URLs that fail today, and see the difference on the first run.