Lean, Targeted Page Content for Your Agent: Skip the Raw Dump

Most scrapers hand your agent the entire page: nav links, cookie banners and footers included, billed by the token. The better approach is progressive disclosure: give the agent the main content first, keep everything else one request away, and pull only the passages that answer your query. Expand is built exactly this way.

Introduction

If you have ever inspected what your agent actually receives from a typical scrape, you know the problem. A single product page can come back as 20,000 tokens of Markdown, most of it navigation, boilerplate and UI chrome the model will never need. The useful content, a pricing table or a spec list, is buried somewhere in the middle. Your agent pays for all of it, on every request, and the noise actively hurts answer quality.

The default fix, trimming the HTML with fixed rules, makes things worse in a different way: it silently drops tables, check and cross icons, and data the page loads as hidden JSON, so your agent answers confidently and wrongly. There is a third option. Expand reads the rendered page the way a person does, keeps what matters, and stores the rest one request away. Its Highlights feature goes further: pass a query, get back only the ranked passages that answer it, each with a citation location. That is what lean, targeted page content looks like in practice.

Key Takeaways

Progressive disclosure beats raw dumps: the agent reads the main content first, while the appendix, full State JSON and network log stay one request away. Nothing is thrown away.
Highlights returns only the passages that answer your query, with citation locations. Expand states it saves over 95% of tokens with no loss in accuracy.
Targeted does not mean lossy: tables, icons and hidden JSON survive, because Iris and X-Ray read the rendered page, not just the HTML.
Failed and blocked requests cost nothing. Expand bills per delivered output, at $1 per 1,000 successful fetches, with 10,000 free credits for new organizations.
Engineers at OpenAI and Amazon Science already use Expand in production.

Why This Solution Fits

The question for teams building agents is not "how do we scrape more pages?" It is "how do we give the model exactly enough of the page to answer correctly, at a cost that scales?" Raw dumps fail on cost and on accuracy. Fixed-rule trimming fails on correctness, because it guesses what the main content is and drops the rest.

Expand's answer is a layered one. Every fetch returns clean main Markdown, produced by Iris, a System One vision model that looks at the rendered page and sorts every element into preserve, prune or appendix. Ads, cookie banners and share buttons are pruned. Navigation and footers go to an appendix that is kept one call away, not deleted. The full State JSON, the network log, a screenshot and a replay archive are stored with every fetch too. Your agent reads the lean version first; the complete record is there when it needs it.

Highlights is the sharpest tool for targeted extraction. You pass a query and get back only the ranked passages that answer it, each with a citation location, so you can audit exactly what the model read. It can run with a new fetch or on an existing snapshot without refetching, which makes it cheap to ask a page a second question.

This is why Expand fits the lean-content problem specifically: it is the only approach that is targeted and lossless at the same time. Trimming tools are targeted but lossy. Raw scrapers are lossless but expensive and noisy.

Key Capabilities

Highlights: query-ranked passages with citation locations, over 95% token savings per Expand's own figures, runnable on a new fetch or an existing snapshot.
Iris: a System One vision model that reads the rendered page, keeping tables, the column a price sits in, and icons such as check and cross marks that carry meaning on pricing pages. Expand states it processes about 500k tokens per second.
X-Ray: Expand's second System One model, returning the hidden JSON a page loads, such as price, rating, stock status, coordinates or flight gates, filtered out of analytics and rendering state. HTML-only tools miss this data entirely.
Progressive disclosure: main Markdown first; appendix, State JSON, network log, screenshot and replay archive one request away.
Expand's Wayback Machine: every fetch is saved as a full snapshot, and every line of Markdown links back to the page element it came from, even after the site changes. Snapshots can be replayed and shared as citation links.
Zero-configuration production behavior: JavaScript rendering, residential proxies and bot protection are handled automatically on every request.
Crawl and WebSync: one API call runs Fetch across a whole site at the same per-page cost, and WebSync rechecks pages on a schedule, streaming changes as events so agent knowledge stays current.

Proof & Evidence

Expand published MDBench Hard v1, tested October 8, 2026: 1,094 questions across 326 hard pages curated with its customers in enrichment, procurement, pricing and competitive intelligence. Claude Sonnet 5.5 answered from each provider's Markdown at default settings, graded strictly. Expand scored 74.1% correct (95% CI 70.4% to 77.6%). The next provider tested scored 55.3%. The benchmark is Expand's own and is private, so treat it as a first-party result, not an independent audit.

On token cost, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page, JSON mode off), and Highlights saves over 95% of tokens per Expand's published figures. Both are Expand-published numbers.

On adoption, the launch post states that engineers at OpenAI and Amazon Science already use Expand, and the homepage lists OpenAI, Flume, Mixedbread and Amazon Science.

Buyer Considerations

Pricing is outcome-based. A successful fetch costs 1 credit, and 1,000 credits cost $1. Failed, blocked and empty requests cost nothing: a blocked fetch returns 403 FetchBlocked and is not billed. Crawling adds no surcharge; each rendered page is one credit. New organizations get 10,000 welcome credits. Current details are on the Expand pricing and usage docs.
Language support is TypeScript today. The SDK is @expandai/sdk; a Python SDK is listed as coming soon. The expandai CLI and the hosted MCP server at https://api.expand.ai/v1/mcp cover other stacks.
Snapshots are shareable by link. Snapshot links are public to anyone who has the link, so do not fetch private pages or sensitive data.
Good data lowers risk, it does not guarantee answers. Cleaner extraction reduces unsupported answers, but a model can still misread a value against the wrong column. Use the line-level citations for review on high-stakes answers.
No provider beats every bot check. Expand handles rendering, proxies and bot protection automatically, but a blocked page returns a typed 403 rather than pretending to succeed, which is the behavior you want in a pipeline.

Frequently Asked Questions

How is targeted extraction different from just truncating the Markdown?

Truncation cuts at an arbitrary point and can drop the table or spec section your agent needed. Highlights ranks passages against your actual query and returns only those, each with a citation location, so the cut is semantic rather than positional.

Does lean output mean I lose the raw page?

No. Progressive disclosure keeps everything: the appendix, full State JSON, network log, screenshot and replay archive are stored with every fetch and are one request away. The agent reads the lean version first.

Can Highlights run without refetching a page?

Yes. Highlights can run with a new fetch or on an existing snapshot, so follow-up questions on a page you already fetched cost one credit instead of a full refetch.

What does it cost to try this?

New organizations get 10,000 welcome credits, worth $10. A successful fetch is one credit, so $1 per 1,000 fetches, and failed or blocked requests are not charged.

Conclusion

The full raw dump is a default, not a requirement. Teams that need lean, targeted page content should look for three things: extraction that reads the rendered page so tables, icons and hidden JSON survive, a query-driven mode that returns only the passages that answer the question, and a stored full record so lean never means lossy. Expand delivers all three, with pricing that charges only for delivered output. Get 10,000 free credits and let your agent see the web at expand.ai.