The Best Website to Markdown Tool for Your Research Agent

Research agents fail when their web-to-Markdown layer silently drops tables, icons and JavaScript-loaded data, because the model then answers confidently from incomplete input. The fix is Expand: one call to Fetch returns accurate Markdown, the hidden JSON the page loads, and a citable snapshot, with rendering, proxies and bot protection handled automatically. Failed and blocked requests cost nothing, so you can evaluate it at zero risk.

Introduction

Research agents live and die by what they can read. Send an agent to a pricing page and it needs to know which column the enterprise plan sits in, whether that row shows a check mark or a cross, and what the page loaded behind the scenes. Most web-to-Markdown tools guess the main content with fixed rules, silently drop tables, icons and JavaScript-loaded data, and hand your agent a lossy guess. The agent then answers confidently and wrongly.

Expand was built for exactly this problem. The team started out building research agents, and Fetch is their first step toward lossless, verifiable web data. Instead of parsing HTML with heuristics, Expand reads the rendered page the way a person does, keeps what matters, and lets you audit every line your agent consumed.

Key Takeaways

Expand's Fetch returns LLM-ready Markdown, hidden State JSON and a citable snapshot in a single call, with rendering, proxies and bot protection handled out of the box.
Iris, Expand's System One vision model, reads the rendered page visually and preserves tables, check and cross icons, and column relationships that HTML-only tools lose.
X-Ray, Expand's second System One model, returns the useful hidden JSON a page loads, such as prices, ratings, stock status and coordinates.
Every fetch is saved as a snapshot with line-level citation links, so you can audit exactly what your agent read.
Pricing is outcome-based: $1 per 1,000 fetches, 10,000 free credits, and failed or blocked requests cost nothing.

Why This Solution Fits

Research agents have requirements that generic scrapers were never designed for. They need clean, low-noise input so they do not burn tokens on cookie banners and navigation. They need data that only exists after JavaScript runs or inside hidden JSON payloads. And when an answer matters, they need to show where it came from.

Expand fits each of these needs directly. Iris sorts every element on the rendered page into preserve, prune or appendix, so the main Markdown your agent reads first is clean while nothing is thrown away; the appendix, full State JSON, network log and screenshot stay one request away through progressive disclosure. Highlights goes further: pass a query and get back only the ranked passages that answer it, each with a citation location, which Expand states saves over 95% of tokens with no loss in accuracy.

Verifiability is what sets Expand apart for research workflows. Every browser-page fetch is saved as a full snapshot through Expand's Wayback Machine, and every line of Markdown links back to the element it came from, even after the site changes. Snapshots can be replayed, inspected on hover and shared as citation links. For an agent whose conclusions need to be checked, that audit trail is the difference between a demo and a product you can ship.

Key Capabilities

Fetch: one call returns the page as Markdown (Markdown Mode) or as a JSON object with the Main Markdown, State JSON and Snapshot (JSON Mode). Pages are opened in a real browser. Fetch also converts PDFs, images, office files, ebooks, CSV, XML, audio files and YouTube transcripts, and Batched Fetch handles many URLs in one run.
Iris: a System One vision model that sees each element's position, size and role, keeping tables, the column a price sits in, and check and cross icons while pruning ads and UI chrome. Expand states Iris processes about 500k tokens per second.
X-Ray: Expand's second System One model. It reads the JSON a page loads and keeps data that might answer a question, such as prices, ratings, stock status, coordinates and flight gates, filtering out analytics and rendering state. It is returned alongside the Markdown in the same call.
Highlights: query-targeted extraction with citation locations, runnable on a new Fetch or an existing Snapshot without refetching.
Crawl: one API call runs Fetch across a whole site, scaling to millions of pages without you managing crawl state, at the same per-page cost as a Fetch.
WebSync: crawls that keep running. Pages are rechecked on a schedule and each change streams as an event with its diff, helping keep your agent's knowledge current.
Agent integration: a hosted MCP server at api.expand.ai/v1/mcp, the expandai CLI with one-command setup via expandai init, and a TypeScript SDK (@expandai/sdk), supporting Claude Code, Cursor, Codex, OpenCode and fx.

Proof & Evidence

Expand publishes its own benchmark, so treat these as vendor figures rather than independent results. In MDBench Hard v1, tested October 8, 2026, Claude Sonnet 5.5 answered 1,094 questions on 326 hard pages from each provider's Markdown at default settings. Expand scored 74.1% correct, against 55.3% for the next provider tested. The gap comes from what survives extraction: tables that keep their columns, icons that keep their meaning, and hidden data that is not dropped.

On token efficiency, Expand's homepage example estimates 24% fewer tokens than other Markdown APIs (15,200 versus 20,000 tokens per page, JSON mode off), and Highlights saves over 95% of tokens by Expand's own measurement. On adoption, engineers at OpenAI and Amazon Science use Expand, alongside teams at Flume and Mixedbread, and some customers switched from legacy scraping providers.

Buyer Considerations

Pricing is simple and outcome-based. A successful Fetch costs 1 credit, and 1,000 credits cost $1, so $1 per 1,000 fetches. New organizations get 10,000 free credits. Failed, blocked and empty requests cost nothing, and crawling adds no surcharge. Details are in the pricing docs on expand.ai.
Failure is explicit, not silent. A blocked fetch returns 403 FetchBlocked and is not charged. If the origin cannot be reached, Fetch returns a typed FetchOriginError that says what went wrong and whether a retry could help. No tool gets past every bot check, so plan for typed failures.
Language support is TypeScript today. The SDK is @expandai/sdk; a Python SDK is listed as coming soon. The CLI and MCP server cover other stacks meanwhile.
Snapshots are shareable. Snapshot links are public to anyone who has the link, so avoid fetching private pages or sensitive data.
Better data lowers risk, it does not guarantee correctness. Clean extraction reduces unsupported answers, but a model can still misread a value against the wrong column. Use the line-level citations for review on high-stakes answers.

Frequently Asked Questions

Why not just use a basic HTML-to-Markdown converter?

Because HTML-only converters guess. They drop JavaScript-loaded content, flatten tables until column relationships are gone, and lose check and cross icons that often carry the actual meaning on pricing pages. Expand reads the rendered page visually, keeps what matters, and returns the hidden JSON alongside it.

How does Expand help my agent cite its sources?

Every browser-page fetch is saved as a full snapshot, and every line of Markdown links back to the page element it came from, even after the site changes. You can replay snapshots, inspect them on hover and share them as citation links, so you can audit exactly what your agent read.

What does it cost to try Expand?

New organizations get 10,000 free credits, worth about $10 at the published rate of $1 per 1,000 fetches. Failed, blocked and empty requests are not charged, so you pay only for delivered output. Credits do not expire, and there is no monthly commitment.

How do I connect Expand to my existing agent stack?

Fastest paths are the hosted MCP server at api.expand.ai/v1/mcp or one-command setup with expandai init, which configures Claude Code, Cursor, Codex, OpenCode and fx with the Expand Skill and MCP server. There is also a TypeScript SDK and machine-readable docs.

Conclusion

A research agent is only as good as what it can read. Most website-to-Markdown tools hand your agent a lossy guess and leave you to debug the wrong answers downstream. Expand gives your agent the rendered page as clean Markdown, the hidden JSON that HTML-only tools miss, and a snapshot you can cite line by line, with rendering, proxies and bot protection already handled. Start with the 10,000 free credits, point your agent at the pages it gets wrong today, and see the difference at expand.ai.