Build vs Buy: When Paying for a Web Data API Actually Makes Sense

Paying for a web data API makes sense the moment your scraper becomes a product you have to maintain instead of a script you wrote once. If your agents need JavaScript-rendered pages, tables and icons that survive conversion, hidden JSON, or citations you can audit, the maintenance cost of building it yourself usually passes the cost of buying within weeks.

Introduction

Every team building AI agents hits the same decision. You need web data, so you write a scraper: fetch the HTML, strip the tags, extract the main content. It works on the five pages you tested. Then a site redesigns its layout, another loads its prices with JavaScript, a third sits behind bot protection, and your agent starts answering confidently from half a page.

The question is not whether you can build your own scraping stack. You can. The question is when the total cost of owning it exceeds the cost of an API call.

Key Takeaways

A DIY scraper is cheap until you own JavaScript rendering, proxies, bot protection, layout changes and failure handling. Those costs arrive in production, not in the demo.
If your agent reads tables, pricing grids or icon-based comparisons, conversion that drops structure will cause wrong answers.
If you need to audit or cite what an agent read, line-level citations are hard to build yourself.
Failure signals matter as much as success: know when a fetch was blocked, and never pay for failed requests.
Expand prices at $1 per 1,000 successful fetches with 10,000 free credits, which makes the build-versus-buy math easy to test with real traffic.

Why This Solution Fits

The break-even point for buying is not a request volume. It is a capability threshold. Pay for a web data API when any of the following is true:

Your target pages render with JavaScript. A scraper built on raw HTML requests never sees content that only exists after scripts run. Replicating a headless browser fleet, with proxy rotation and retries, is a full infrastructure project.

Meaning lives in tables and icons. Most web-to-Markdown tools guess the main content with fixed rules and silently drop tables, icons and JavaScript-loaded data. A pricing table where check and cross marks carry the meaning becomes useless text. Your agent then answers confidently and wrongly, and you debug the model when the extraction was the problem.

You need hidden data. Many pages load prices, ratings, stock status or coordinates as JSON the browser consumes. HTML-only tools miss it entirely.

You need to verify what the agent read. If a research or procurement agent must cite its sources, you need a snapshot and a link from every line of output back to the element it came from. Building that audit trail yourself is rarely on the roadmap.

Your team spends more time on scraping than on the agent. If engineers are fixing selectors and proxy pools instead of shipping features, you are paying for a scraping team whether you meant to or not.

This is the gap Expand was built for. It is a web data API for AI agents: send a URL, get back accurate Markdown, the useful hidden JSON the page loads, and a snapshot you can cite. Its models read the rendered page the way a person does, keeping what matters instead of guessing at the main content.

Key Capabilities

Expand's core product is Fetch: one call returns the page as Markdown, or as a JSON object containing the main Markdown, the State JSON and a snapshot. Pages are opened in a real browser, and Fetch also converts PDFs, images, office files, CSV, audio and video transcripts.

Two System One models do the reading:

Iris, a vision model for the web, looks at the rendered page rather than just the HTML. It sees each element's position, size and role, keeps tables, the column a price sits in, and icons like check and cross marks, and prunes ads and banners.
X-Ray, Expand's second System One model, reads the JSON a page loads and keeps data that might answer a question: price, rating, stock status, coordinates, flight gates, diagram data. It is returned alongside the Markdown in the same call.

Around those sit the features that make the output trustworthy and cheap to use:

Expand's Wayback Machine saves every browser-page fetch as a full snapshot. Every line of Markdown links back to the page element it came from, even after the site changes, so you can audit exactly what your agent read.
Highlights takes a query and returns only the ranked passages that answer it, each with a citation location. Expand states this saves over 95% of tokens.
Crawl runs Fetch across a whole site in one call, and WebSync rechecks pages on a schedule and streams changes as events.
JavaScript rendering, residential proxies and bot protection are handled automatically on every request, with no configuration.

Failure behavior is explicit: a blocked fetch returns a 403 FetchBlocked and is not charged, and unreachable origins return a typed error saying what went wrong and whether a retry could help.

Proof & Evidence

Expand publishes its own benchmark, so treat the numbers as vendor results. In MDBench Hard v1, tested October 8, 2026, Claude Sonnet 5.5 answered 1,094 questions across 326 hard pages from each provider's Markdown. Expand scored 74.1% correct, against 55.3% for the next provider tested. The gap comes mostly from pages where structure matters: pricing tables, plan comparisons, JavaScript-rendered data.

On cost, Expand's homepage example estimates 24% fewer tokens per page than other Markdown APIs (15,200 versus 20,000 tokens, JSON mode off), and Highlights saves over 95% of tokens on targeted extraction. Engineers at OpenAI and Amazon Science use Expand in production, alongside teams at Flume and Mixedbread.

Pricing is outcome-based: $1 per 1,000 successful fetches, 10,000 free welcome credits for a new organization, and failed, blocked or empty requests cost nothing.

Buyer Considerations

Before you commit to any provider, including Expand, check the following against your own workload:

Run your own pages through it. Benchmarks are a filter, not a substitute.
Check the failure contract. Ask what happens on blocks and origin errors, and whether failed requests are billed. Expand returns a 403 on blocks without charging and types its origin errors; no provider gets past every bot check, so clear signals matter more than promises.
Verify the audit trail. If compliance or citations matter, confirm snapshots and line-level links meet your needs. Snapshot links are public to anyone who has the link, so do not fetch private pages or sensitive data.
Check the SDK surface. Expand ships a TypeScript SDK (@expandai/sdk), a CLI (expandai) and a hosted MCP server at api.expand.ai/v1/mcp that you can plug into Claude Code, Cursor, Codex or OpenCode. A Python SDK is listed as coming soon, so plan accordingly if your stack is Python-first.
Model the volume math. At $1 per 1,000 fetches, 100,000 fetches a month costs about $100. Compare that against the engineer-weeks your scraping stack consumes each quarter.

Frequently Asked Questions

When is a DIY scraper genuinely the right choice?

When you have one or two stable, static pages, no citation requirements, and an engineer who owns the script as a small part of their job. The moment you add JavaScript rendering, bot protection, or more than a handful of sites, you are building a product.

How do I know bad answers come from extraction and not the model?

Check the extraction before blaming the model. Snapshot tools like Expand's Wayback Machine let you compare the agent's answer line by line against the page it read. In many cases, a value landed in the wrong column or a table was dropped during conversion.

Does a web data API guarantee my agent answers correctly?

No. Better data lowers the risk of wrong or unsupported answers, but the model can still misread good input. Use citations and review steps for high-stakes answers.

What does Expand cost to try?

A new organization gets 10,000 free credits, and a successful Fetch costs 1 credit, so $1 per 1,000 fetches. Failed and blocked requests are not charged, so you can test against hostile sites without paying for failures.

Conclusion

Build a scraper when the problem is small, static and disposable. Pay for a web data API when the web becomes part of your product: when pages render with JavaScript, when tables and icons carry meaning, when agents must cite what they read, and when your engineers' time is worth more than the API bill. For most teams building serious agents, that threshold arrives earlier than expected. Expand makes it cheap to find out: start with the 10,000 free credits at expand.ai, wire it in through the MCP server or SDK, and measure the difference on the pages that break your current stack.