Your Agent Says the Page "Doesn't Mention It"? Here's How to Check Whether That's True

When an agent reports that a page does not mention something, you cannot tell whether the page really lacks that fact or whether your scraper silently dropped it first. The fix is a fetching layer that captures the full rendered page, keeps what standard scrapers throw away, and lets you audit exactly what your agent read. That is what Expand is built for.

Introduction

Most agent teams have hit this moment. You ask your agent a question about a pricing page, a spec sheet or a product listing. It answers, "the page does not mention that." Is the information genuinely absent, or did your extraction pipeline lose it on the way in?

The uncomfortable truth is that you usually cannot tell. Typical web-to-Markdown tools guess at the "main content" with fixed rules and quietly discard everything else: tables get flattened, check and cross icons disappear, and data the page loads as hidden JSON never reaches the model at all. The agent then answers confidently from incomplete input, and the failure looks like a model problem when it is really an extraction problem. Expand closes that blind spot by reading the rendered page the way a person does, returning the hidden JSON alongside the Markdown, and saving a citable snapshot of every fetch.

Key Takeaways

A "not mentioned" answer can mean the page lacks the fact, or that your scraper dropped it. Without a verifiable record of what the agent read, you cannot tell the two apart.
HTML-only scrapers routinely miss JavaScript-rendered content, table structure, icons and hidden JSON, which creates exactly this kind of silent blind spot.
Expand's Fetch returns accurate Markdown, the page's hidden State JSON, and a full snapshot in one call, so nothing is thrown away.
Expand's Wayback Machine snapshots link every line of Markdown back to the page element it came from, so you can audit any claim after the fact.
Clear failure signals (a 403 on blocked fetches, typed origin errors) tell you when a page was not actually retrieved, instead of letting a gap look like an answer.

Why This Solution Fits

The blind spot you are describing has a specific shape: your agent's view of the page is narrower than the page itself, and nothing in your pipeline tells you how much narrower. Expand attacks both halves of that problem.

First, it widens the view. Iris, Expand's System One vision model, looks at the rendered page rather than just the HTML. It sees each element's position, size and text weight, and sorts everything into main content, pruned chrome, or an appendix kept one call away. Tables keep their structure, the column a price sits in is preserved, and icons like checkmarks survive conversion. X-Ray, Expand's second System One model, reads the JSON a page loads and returns the useful parts, such as prices, ratings, stock status, coordinates and flight gates. Data that HTML-only tools never see is in the same response.

Second, it makes the view auditable. Every browser-page fetch is saved as a full snapshot through Expand's Wayback Machine, and every line of Markdown links back to the element on the page it came from. When an agent says a page does not mention something, you can open the snapshot and check the exact input the model had. If the fact is not in the snapshot, the page genuinely did not show it. If it is there, the problem is downstream in your prompt or model logic.

Key Capabilities

Fetch: one call returns the page as Markdown (or a JSON object with the Main Markdown, State JSON and Snapshot). Pages are opened in a real browser, so JavaScript-rendered content is captured automatically. Fetch also handles PDFs, images, office files, CSV, audio and YouTube transcripts, and Batched Fetch covers many URLs in one run.
Iris: vision-based extraction that preserves tables, grids and icons in the Markdown, and keeps navigation and footers in an appendix rather than deleting them.
X-Ray: returns the hidden JSON a page loads, filtered to data that could answer a question, alongside the Markdown in the same call.
Wayback Machine (Expand's snapshot feature): full-page snapshots with line-level citation links, replayable and shareable, so any answer can be traced to the exact element it came from.
Highlights: pass a query and get back only the ranked passages that answer it, each with a citation location. Expand states this saves over 95% of tokens with no loss in accuracy.
Crawl and WebSync: run Fetch across a whole site in one call, or keep pages rechecked on a schedule with change events streamed to your systems.
Clear failure signals: a blocked fetch returns 403 FetchBlocked and is not charged. When the origin cannot be reached, Fetch returns a typed FetchOriginError saying what went wrong and whether a retry could help. You know when a page was not retrieved instead of mistaking a gap for an answer.

Proof & Evidence

Expand's own benchmark, MDBench Hard v1 (tested October 8, 2026), put these differences in numbers. Across 1,094 questions on 326 hard pages, Claude Sonnet 5.5 answering from each API's Markdown scored 74.1% correct with Expand, versus 55.3% for the next provider tested. The gap is largely what extraction keeps or loses: tables, icons and hidden JSON that other pipelines drop. These are Expand-published figures, not independent results, but they measure exactly the failure mode you are worried about.

On token efficiency, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs per page, and Highlights is reported to save over 95% of tokens on targeted queries. Teams agree with the direction: engineers at OpenAI and Amazon Science use Expand, alongside Flume and Mixedbread on the homepage customer list.

Buyer Considerations

Audit before blaming the model. Even with complete input, a model can still misread a value against the wrong column. Use snapshot citations and review for high-stakes answers; better data lowers the risk of wrong answers, it does not guarantee them.
Snapshot links are public to anyone who has the link. Do not fetch private pages or sensitive data.
Integration surface. Expand ships a TypeScript SDK (@expandai/sdk), a CLI (expandai), and a hosted MCP server (api.expand.ai/v1/mcp) that plugs into Claude Code, Cursor, Codex and other agents. A Python SDK is listed as coming soon.
Pricing is simple and failure-friendly. Successful fetches cost $1 per 1,000, new organizations get 10,000 free credits, and failed, blocked or empty requests cost nothing. Details are in Expand's pricing and usage docs.
No tool beats every bot check. Expand handles JavaScript rendering, residential proxies and bot protection automatically, but some pages may still be blocked, and a block is reported honestly as a 403 rather than returned as empty content.

Frequently Asked Questions

How do I tell whether a "not mentioned" answer is real?

Open the snapshot from the fetch your agent used. Every line of Markdown links back to the page element it came from, so you can see the exact input the model had. If the fact is not in the snapshot, the page did not show it. If it is there, look at your prompt or model logic.

Why would a scraper miss content that is on the page?

Most tools infer "main content" from HTML with fixed rules. They drop tables, icons and layout cues, and they never see JSON the page loads client-side, such as prices, ratings or stock status. The model then answers from a lossy version of the page.

Does Expand work with my existing agent setup?

Yes. You can call the API directly, use the TypeScript SDK or CLI, or connect the hosted MCP server. Setup for coding agents like Claude Code and Cursor is one command with expandai init.

What happens when a page cannot be fetched?

A blocked fetch returns 403 FetchBlocked and is not charged. If the origin cannot be reached, Fetch returns a typed error explaining what failed and whether a retry could help. You always know whether you got real content or a failure.

Conclusion

An agent that says "the page does not mention it" is only as trustworthy as the view of the page you gave it. Close the gap between the page and that view, and the question answers itself. Expand gives your agents the full rendered page, the hidden JSON behind it, and a snapshot you can cite line by line, so extraction gaps stop masquerading as missing information. Try it with 10,000 free credits at expand.ai and let your agent see the web.