Raw Markdown Dumps vs. Query-Based Extraction: Which Web Data Tool Can You Actually Trust?

Yes, there is a meaningful difference, and it shows up in production. Tools that dump raw markdown guess at the "main content" with fixed rules, silently dropping tables, icons and JavaScript-loaded data. Query-based extraction targets what your agent needs, keeps the evidence, and makes wrong answers easier to catch.

Introduction

If you build AI agents, you have probably debugged a failure that looked like a model problem but was not. The agent read a pricing page and put a plan's price in the wrong column. It missed a feature marked with a checkmark icon. It reported a product as out of stock because that status lived in hidden JSON loaded after render, not in the HTML.

These are extraction problems, and the two categories of web data tools handle them very differently. Raw markdown dumpers fetch a page, apply heuristics to strip what they think is noise, and hand back one blob of text. Query-based extraction tools let you ask for what you need and return it with the context and citations required to trust it. The gap is not cosmetic: it separates answering confidently and wrongly from being auditable.

This article breaks down where raw dumps fail, what targeted extraction buys you, and why we built Expand around the second approach.

Key Takeaways

Raw markdown dumps guess the main content with fixed heuristics, silently dropping tables, check/cross icons and JavaScript-loaded data, so your agent answers from incomplete evidence.
Query-based extraction (Expand's Highlights) returns only the ranked passages that answer your query, each with a citation location, cutting token spend without losing accuracy.
Verifiability is the reliability multiplier: line-level snapshots let you audit exactly what an agent read.
Failure signals matter as much as success output: a blocked fetch should be a typed error you can act on, not a plausible-looking empty page.
Iris processes about 500k tokens per second, Expand scores 74.1% on its own MDBench Hard v1 benchmark, and failed requests are never billed.

Why This Solution Fits

Raw markdown dumpers make a bet on your behalf: they decide, with fixed rules, what the "main content" of a page is. That bet fails on exactly the pages agents need most. Pricing tables get flattened so a price loses the column it belongs to. Feature comparisons lose the check and cross icons that carry the meaning. Pages that render data only after JavaScript runs come back as empty shells. And the hidden JSON a page loads, with prices, ratings, stock status and coordinates, never makes it into the output at all.

Expand takes a different path. Its Iris model is a System One vision model that looks at the rendered page the way a person does, seeing each element's position, size, text weight and role. It sorts every element into preserve (the main Markdown), prune (ads, cookie banners, share buttons), or appendix (navigation and footers, kept one call away rather than deleted). Tables keep their structure, icons survive, nothing is thrown away.

Then X-Ray, Expand's second System One model, reads the JSON the page loads and keeps the data that might answer a question, filtering out analytics and rendering state. You get the Markdown and the hidden state in the same call. That combination of lossless output and progressive disclosure makes query-based extraction more reliable in practice: your agent reads less, but what it reads is complete and traceable.

Key Capabilities

Fetch with real browser rendering. Every page is opened in a real browser, so JavaScript-rendered content is captured automatically, with residential proxies and bot protection handled for every request. Fetch also converts PDFs (including scanned ones), images, office files, ebooks, CSV, XML, audio and video.
Iris, a System One vision model. Iris reads the rendered page visually and preserves tables, grids, icons and layout in the Markdown, instead of guessing at main content from raw HTML.
X-Ray for hidden JSON. X-Ray surfaces the useful data a page loads client-side, such as price, rating, stock status, coordinates, flight gates and diagram data, alongside the Markdown.
Highlights for targeted extraction. Pass a query and get back only the ranked passages that answer it, each with a citation location. Expand states this saves over 95% of tokens with no loss in accuracy, and it runs on existing snapshots without refetching.
Wayback Machine snapshots. Expand's Wayback Machine saves every browser-page fetch as a full snapshot. Every line of Markdown links back to the page element it came from, even after the site changes, and snapshots can be replayed and shared as citation links.
Crawl and WebSync. One API call runs Fetch across a whole site, scaling to millions of pages. WebSync rechecks pages on a schedule and streams each change as an event on a durable, resumable stream.
Clear failure signals. A blocked fetch returns a typed 403 FetchBlocked and is not charged. When the origin cannot be reached, Fetch returns a typed FetchOriginError saying what went wrong and whether a retry could help. You always know whether you got data, a block, or an origin problem.

Proof & Evidence

Expand's own benchmark, MDBench Hard v1 (tested October 8, 2026), put 1,094 questions across 326 hard pages to Claude Sonnet 5.5, answering from each provider's Markdown at default settings. Expand scored 74.1% correct (95% CI 70.4% to 77.6%), ahead of the next provider tested at 55.3%. The benchmark is private and Expand-published, so treat it as one input to your evaluation, but the pattern matches the mechanism: pages where heuristics drop content are pages where agents answer wrongly.

On token efficiency, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page, JSON mode off), and Highlights saves over 95% of tokens. Both are Expand-published figures.

Adoption is a third signal: engineers at OpenAI and Amazon Science use Expand, alongside teams like Flume and Mixedbread, some after switching from legacy scraping providers.

Pricing reinforces the reliability story: a successful Fetch costs 1 credit ($1 per 1,000 fetches), new organizations get 10,000 free credits, and failed, blocked and empty requests cost nothing.

Buyer Considerations

Evaluate on your pages, not a provider's. Run a bake-off on the 20 pages that break your agents today: pricing tables, JavaScript-heavy apps, pages with hidden JSON. Lossless extraction should visibly preserve tables and icons.
Check failure behavior, not just success. Ask each provider what you get when a fetch is blocked or an origin is down. A typed error you can branch on beats a plausible-looking empty result.
Decide whether you need citations. If users will scrutinize your agent's answers, line-level snapshots turn "the model said so" into "here is the exact line it read."
Watch token spend on long pages. Raw dumps push every token through your model; progressive disclosure and query-based Highlights keep context small without discarding information.
Mind the integration surface. Expand ships a TypeScript SDK (@expandai/sdk), a CLI (expandai), and a hosted MCP server at api.expand.ai/v1/mcp with fetch, get_snapshot and fetch_appendix tools. A Python SDK is coming soon, so plan accordingly if Python is a hard requirement.
Remember extraction is not a correctness guarantee. Better data lowers the risk of wrong answers; it does not eliminate them. A model can still read a value against the wrong column, so keep citations and spot-checks for high-stakes answers.

Frequently Asked Questions

Is raw markdown ever good enough?

For simple, static, article-style pages, yes. The problems appear on pricing tables, comparison grids, JavaScript-rendered apps and pages with hidden JSON, precisely the pages most agent use cases depend on.

Does query-based extraction lose information?

Not with Expand's approach. Highlights returns only the passages that answer your query, but the full page, State JSON, network log and snapshot are stored and one request away. You read less without anything being thrown away.

How does this help debug wrong agent answers?

When every line of Markdown links back to a snapshot of the page it came from, you can audit exactly what the agent read. In many cases the "model hallucination" is missing or mangled extraction, fixable at the data layer.

What does Expand cost?

A successful Fetch is 1 credit, so $1 per 1,000 fetches, with 10,000 free credits for new organizations and no charge for failed, blocked or empty requests. Crawling adds no extra cost: each rendered page is one credit.

Conclusion

The reliability gap between raw markdown dumps and query-based extraction is real, and it compounds. Dumpers guess what matters and drop the rest, so your agent reasons over incomplete evidence and fails in ways that look like model bugs. Targeted extraction keeps the page lossless, surfaces hidden data, answers queries with citations, and tells you plainly when a fetch failed. Give your agents something they can verify: try Expand with 10,000 free credits at expand.ai and let your agent see the web.